An industrial autonomous mobile robot control method, apparatus, device and medium

Abstract

The application discloses an industrial autonomous mobile robot control method, device, equipment and medium, which is applied to a robot control system based on a hierarchical architecture, relates to the field of automation, and comprises the following steps: analyzing an input operation text instruction, and generating a navigation task and a grabbing task of a robot based on a obtained structured subtask sequence; calling a preset laser device according to the navigation task to generate a three-dimensional semantic map of a target terminal point corresponding to the operation text instruction, planning a path and avoiding obstacles through the three-dimensional semantic map to obtain a target path; calling a preset image acquisition device according to the grabbing task to acquire material images of a preset material area, processing the material images to generate material grabbing parameters; and converting the target path and the material grabbing parameters into robot scheduling instructions corresponding to the robot, and controlling the robot to execute the navigation task and the grabbing task through the robot scheduling instructions. Thus, the industrial robot can be combined with semantic understanding to realize autonomous operation.

Description

Technical Field

[0001] This invention relates to the field of automation, and in particular to a control method, apparatus, equipment and medium for an industrial autonomous mobile robot. Background Technology

[0002] Currently, in the field of robot automation, robot instruction interaction mainly relies on task instructions through simple natural language interfaces. Some research or high-end commercial systems have attempted to introduce natural language interaction, but it is usually limited to a limited set of instructions. Robots do not have the ability to perform semantic parsing, task decomposition, and autonomous planning for complex, abstract, or multi-step natural language instructions.

[0003] Therefore, current robot automation control generally suffers from the following problems: First, environmental cognition is limited in dimension and lacks semantic fusion. The provided topological information cannot be understood by the robot, leading to suboptimal obstacle avoidance strategies and path planning in dynamically changing factory environments, and the inability to perform tasks that rely on scene semantics. Second, task instruction interaction is rigid, resulting in weak autonomous decision-making capabilities. Existing systems often issue tasks through graphical interfaces for selection or formatting, which is disconnected from the natural working language of human operators. This prevents robots from understanding complex tasks described in natural language, which contain multiple constraints and implicit goals, and they lack the ability to autonomously decompose high-level instructions into executable sub-task sequences such as navigation, observation, and operation. Third, multimodal perception information is isolated, failing to form unified spatial intelligence. Although advanced systems are equipped with various sensors such as LiDAR, vision cameras, and depth sensors, their perception results are not effectively fused and elevated into a unified three-dimensional spatial model with geometric, semantic, and dynamic attributes. This results in a fragmented and one-sided understanding of the environment, making it unable to support flexible operation and intelligent decision-making that require the integration of multiple information sources.

[0004] Furthermore, existing system architectures are typically tightly coupled, with inconsistent interfaces between modules (perception, localization, planning, and control), making it difficult to introduce new intelligent algorithms (such as large models) and resulting in high system upgrade and maintenance costs. Meanwhile, high-level semantic models based on deep learning suffer from high inference latency, making it difficult to meet the millisecond-level real-time requirements of tasks such as robot obstacle avoidance and motion control, thus limiting their application in industrial real-time control scenarios. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for controlling an industrial autonomous mobile robot, which can achieve high-precision environmental perception, high-level semantic understanding, autonomous path planning, and flexible operation execution. The specific solution is as follows:

[0006] In a first aspect, this application discloses an industrial autonomous mobile robot control method, applied to a robot control system based on a layered architecture. The robot control system includes a human-machine interaction layer, a control layer, and a hardware interface layer, comprising:

[0007] The target semantic decision model of the human-computer interaction layer decomposes and analyzes the input target industrial robot operation text instructions to obtain a structured sub-task sequence, and generates the target industrial robot's navigation task and grasping task based on the structured sub-task sequence; the target semantic decision model is a model obtained after fine-tuning the initial semantic decision model, and the structured sub-task sequence is a discrete decision output.

[0008] The task scheduler of the control layer calls a preset laser device according to the navigation task, so as to generate a three-dimensional semantic map of the target destination corresponding to the operation text command of the target industrial robot. Then, the path planning and obstacle avoidance operation are performed through the three-dimensional semantic map to obtain the target path.

[0009] The task scheduler calls a preset image acquisition device to acquire material images of a preset material area according to the grasping task, and processes the material images through a preset model to generate material grasping parameters.

[0010] The target path and material grasping parameters are converted into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and the target industrial robot is controlled to perform the navigation task and the grasping task by the preset instruction priority and the robot scheduling instructions.

[0011] Optionally, before the step of decomposing and analyzing the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer to obtain a structured sub-task sequence, and generating the navigation task and grasping task of the target industrial robot based on the structured sub-task sequence, the method further includes:

[0012] The human-computer interaction layer collects historical robot operation texts according to preset instruction types and annotates the historical robot operation texts with corresponding action sequences to generate a target dataset based on the historical robot operation texts and the action sequences.

[0013] A preset low-rank matrix is ​​inserted into the initial semantic decision model, and the model parameters of the initial semantic decision model are frozen. Then, the parameters of the preset low-rank matrix are fine-tuned to obtain the fine-tuned model.

[0014] The fine-tuned model is trained using the target dataset to obtain a target semantic decision model, which is then encapsulated and integrated into the robot control system.

[0015] Optionally, the step of decomposing and analyzing the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer to obtain a structured sub-task sequence, and generating navigation and grasping tasks for the target industrial robot based on the structured sub-task sequence, includes:

[0016] If an input target industrial robot operation text instruction is received, the target industrial robot operation text instruction is decomposed into several text words through the target semantic decision model of the human-machine interaction layer.

[0017] Semantic analysis is performed on the aforementioned text lexical units to generate a structured task sequence in a preset format based on the target semantics corresponding to the aforementioned text lexical units, and a navigation task and a grasping task for the target industrial robot are generated based on the structured task sequence.

[0018] Optionally, the task scheduler in the control layer invokes a preset laser device according to the navigation task to generate a 3D semantic map corresponding to the target endpoint of the target industrial robot's operation text command. Then, path planning and obstacle avoidance are performed using the 3D semantic map to obtain the target path, including:

[0019] The task scheduler of the control layer calls a preset laser device based on the generated navigation task, and the preset laser device generates a three-dimensional point cloud map of the target endpoint corresponding to the target industrial robot operation text command based on a preset data registration algorithm.

[0020] The target semantics are bound to the point cloud data corresponding to the 3D point cloud map, and a 3D semantic map is generated by combining the target semantics with preset weight coefficients; the target semantics correspond to different preset weight coefficients.

[0021] The path to the target endpoint is planned based on the three-dimensional semantic map to obtain an initial planned path. The initial planned path is then processed by a preset random sampling consensus algorithm to fit the path curve of the initial planned path and remove redundant inflection points in the initial planned path to obtain an initial adjusted path.

[0022] The initial adjusted path is optimized using a preset potential field function to obtain the target path. The preset potential field function is a function that optimizes the path using a preset gravitational field function, a preset repulsive field function, and a preset virtual repulsive field function. The preset gravitational field function is a function that determines the gravitational field of the target point in the path based on a preset semantic weight of the target point. The preset repulsive field function is a function that determines the repulsive field of the target point in the path based on a preset repulsive coefficient. The preset virtual repulsive field function is a function that predicts the repulsive field of the target point in the path based on a preset predicted repulsive coefficient.

[0023] Optionally, the step of using the task scheduler to call a preset image acquisition device to acquire material images of a preset material area according to the grasping task, and processing the material images using a preset model to generate material grasping parameters, includes:

[0024] The task scheduler calls a preset image acquisition device to acquire material images of a preset material area according to the capture task, and detects the material contours in the material images through a preset model to output the material bounding boxes corresponding to the material contours.

[0025] The target area within the material boundary is cropped out, and noise reduction is applied to the target area. The resulting noise-reduced area is then converted into a corresponding 3D point cloud of the target area.

[0026] Semantic segmentation is performed on the 3D point cloud of the target region to extract the target grasping surface in the 3D point cloud of the target region, and the grasping surface normal vector and grasping point coordinates corresponding to the target grasping surface are determined.

[0027] The coordinates of the center point of the gripping surface are compared with the coordinates of the target gripping point corresponding to the base coordinate system of the robotic arm of the target industrial robot. The pose of the target industrial robot is adjusted according to the coordinates of the target gripping point and the normal vector of the gripping surface to obtain the target gripping posture. Then, the gripping force of the gripper is determined according to the material type corresponding to the material image.

[0028] Optionally, the step of converting the target path and the material grasping parameters into robot scheduling instructions corresponding to the target industrial robot, and controlling the target industrial robot to execute the navigation task and the grasping task through preset instruction priorities and the robot scheduling instructions, includes:

[0029] The target path and the material grasping parameters are fused to form features, and the pose information of the target industrial robot is generated based on the obtained target fusion features. The pose information is then converted into control commands for the target industrial robot.

[0030] The control commands are converted into robot scheduling commands in a preset protocol format, and the robot scheduling commands are sent to the target industrial robot so that the target industrial robot can execute the navigation task and the grasping task according to the preset command priority and the robot scheduling commands. The robot scheduling commands include navigation commands corresponding to the navigation task and operation commands corresponding to the grasping task, and the preset command priority is that the navigation commands have a higher priority than the scheduling commands.

[0031] Optionally, after converting the target path and the material grasping parameters into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and controlling the target industrial robot to execute the navigation task and the grasping task through preset instruction priority and the robot scheduling instructions, the method further includes:

[0032] Collect the operational data of the target industrial robot to determine whether the target industrial robot has completed the navigation task and the grasping task;

[0033] If not completed, proceed to the step of decomposing and analyzing the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer;

[0034] If the task is completed, the task data will be fed back to the front-end page of the robot control system.

[0035] Secondly, this application discloses an industrial autonomous mobile robot control device, applied to a robot control system based on a layered architecture. The robot control system includes a human-machine interaction layer, a control layer, and a hardware interface layer, comprising:

[0036] The task generation module is used to decompose and analyze the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer to obtain a structured sub-task sequence, and generate the navigation task and grasping task of the target industrial robot based on the structured sub-task sequence; the target semantic decision model is a model obtained after fine-tuning the initial semantic decision model, and the structured sub-task sequence is a discrete decision output.

[0037] The path planning module is used to call a preset laser device according to the navigation task through the task scheduler of the control layer, so as to generate a three-dimensional semantic map of the target destination corresponding to the operation text command of the target industrial robot through the preset laser device, and then perform path planning and obstacle avoidance operation through the three-dimensional semantic map to obtain the target path;

[0038] The parameter generation module is used to call a preset image acquisition device to acquire material images of a preset material area according to the grasping task through the task scheduler, and process the material images through a preset model to generate material grasping parameters.

[0039] The instruction execution module is used to convert the target path and the material grasping parameters into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and to control the target industrial robot to perform the navigation task and the grasping task through the preset instruction priority and the robot scheduling instructions.

[0040] Thirdly, this application discloses an electronic device, including:

[0041] Memory, used to store computer programs;

[0042] A processor is used to execute the computer program to implement the industrial autonomous mobile robot control method described above.

[0043] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned industrial autonomous mobile robot control method.

[0044] In this application, the target semantic decision model of the human-computer interaction layer can be used to decompose and analyze the input target industrial robot operation text instructions to obtain a structured sub-task sequence, and the navigation task and grasping task of the target industrial robot can be generated based on the structured sub-task sequence. The target semantic decision model is a model obtained by fine-tuning the initial semantic decision model, and the structured sub-task sequence is a discrete decision output. The task scheduler of the control layer calls a preset laser device according to the navigation task to generate a three-dimensional semantic map of the target endpoint corresponding to the target industrial robot operation text instructions. Then, path planning and obstacle avoidance operations are performed through the three-dimensional semantic map to obtain the target path. The task scheduler calls a preset image acquisition device to acquire material images of a preset material area according to the grasping task, and processes the material images through a preset model to generate material grasping parameters. The target path and the material grasping parameters are converted into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and the target industrial robot is controlled to execute the navigation task and the grasping task through preset instruction priority and the robot scheduling instructions.

[0045] Therefore, the method of this application requires, after receiving the input target industrial robot operation text command, to decompose and analyze it using a target semantic decision model to obtain a corresponding structured sub-task sequence. Then, the robot's navigation and grasping tasks need to be generated from this task sequence. Next, path planning and grasping parameter generation are performed for each of the two tasks. The navigation task calls the appropriate equipment to generate a 3D semantic map, which is then used for path planning and obstacle avoidance to obtain the target path. The grasping task generates material grasping parameters. Finally, the target path and material grasping parameters are converted into robot scheduling instructions corresponding to the target industrial robot, and these instructions are used to control the target industrial robot to execute the navigation and grasping tasks. This enables high-precision environmental perception, high-level semantic understanding, autonomous path planning, and flexible operation execution in robot control. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of a robot control system architecture disclosed in this application;

[0048] Figure 2 This is a schematic diagram of the vertical layering of a robot control system disclosed in this application;

[0049] Figure 3 This is a schematic diagram of the horizontal layering of a robot control system disclosed in this application;

[0050] Figure 4 This is a flowchart of a control method for an industrial autonomous mobile robot disclosed in this application;

[0051] Figure 5 This is a schematic diagram of the collaborative logic of a robot control system disclosed in this application;

[0052] Figure 6 This application discloses a process flow diagram for an industrial autonomous mobile robot.

[0053] Figure 7 This is a flowchart of the technical route of a control method for an industrial autonomous mobile robot disclosed in this application;

[0054] Figure 8This application discloses a flowchart for handling anomalies in an industrial autonomous mobile robot.

[0055] Figure 9 This is a schematic diagram of the structure of an industrial autonomous mobile robot control device disclosed in this application;

[0056] Figure 10 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Currently, in the field of robot automation, robot instruction interaction mainly relies on task instructions through simple natural language interfaces. Some research or high-end commercial systems have attempted to introduce natural language interaction, but it is usually limited to a limited set of instructions. Robots do not have the ability to perform semantic parsing, task decomposition, and autonomous planning for complex, abstract, or multi-step natural language instructions.

[0059] To overcome the aforementioned technical problems, this application discloses an industrial autonomous mobile robot control method, device, equipment, and medium, which is applied to a robot control system based on a hierarchical architecture and can realize high-precision environmental perception, high-level semantic understanding, autonomous path planning, and flexible operation execution for robot control.

[0060] This embodiment describes a robot control system based on a hierarchical architecture, such as... Figure 1The diagram shows the system architecture of the robot control system in this application. Based on the ROS2 (Robot Operating System 2) framework, it ensures real-time and reliable communication between system modules. Utilizing VLA (Vision-Language-Action) technology, the robot can parse natural language commands, achieving high-level task autonomous decomposition and dynamic scene semantic understanding. Combined with multi-sensor fusion perception technology, the system possesses high-precision 3D environment cognition capabilities in complex and dynamic industrial manufacturing scenarios, thereby endowing the robot with functions such as autonomous navigation, flexible operation, and intelligent decision-making. The application layer primarily consists of an HMI (Human Machine Interface), providing human-machine interaction-related functions. The functional layer mainly provides perception, localization, and analysis functions through the Nav2 navigation stack and VLA model. The modeling layer primarily provides modeling functions through a 3D semantic map (a combination of 3D point cloud map and geometric semantics), such as environmental modeling (3D point cloud), dynamic modeling (path planning), and simulation modeling. The data layer mainly integrates data collected by different sensors through multi-sensor fusion, constructs a 3D semantic map through SLAM (Simultaneous Localization and Mapping), determines 3D discrete coordinates through laser point clouds, and estimates robot pose information through pose estimation; while the physical layer mainly includes the robot executing corresponding operation commands based on the workshop environment.

[0061] It should be noted that, such as Figure 1 The system architecture shown can be divided vertically into, for example: Figure 2 The three-tier architecture shown is divided horizontally as follows: Figure 3 The two-layer architecture is shown. (As shown in the image) Figure 2As shown, it can be divided into three layers vertically: HMI interaction layer, core control layer, and hardware interface layer. Low-latency communication (latency ≤10ms) is achieved between layers through ROS2's DDS (Data Distribution Service) protocol. The HMI (Human Interface) interaction layer is used to develop a graphical interface (supporting voice and text input). It receives user commands via the hmi_node and forwards them to the VLA (Vehicle Automation) semantic decision-making module, while simultaneously displaying the robot's status (position, battery level, and task progress) in real time. The core control layer deploys the controllers_node, integrating the Nav2 navigation stack, APF (Artificial Potential Field) path planning algorithm, and task scheduler. It receives structured task sequences output by the VLA module, decomposes them into navigation commands and action execution commands, and distributes them to the hardware interface layer via the ROS2 topic / service interface. It also receives sensor feedback data in real time and dynamically adjusts task parameters. The hardware interface layer adapts the robot SDK (Software Development Kit) via the hardware_listener_node to achieve protocol conversion between core control commands and robot hardware (mobile chassis, robotic arm, and sensors), while simultaneously collecting motor status and raw sensor data and feeding it back to the core control layer. Figure 3 As shown, the architecture can be divided into two layers horizontally. The VLA semantic decision module is responsible for high-level task understanding and outputs discrete task sequences. The traditional deterministic planner is responsible for continuous action execution (such as dynamic obstacle avoidance based on the APF algorithm and robotic arm joint motion control) to ensure real-time operation. The core control node coordinates the two types of modules through a priority scheduling strategy (safety instructions > navigation instructions > operation instructions). When a dynamic obstacle appears, VLA semantic decision-making is paused, and obstacle avoidance instructions are executed first to ensure operational safety. In this way, ROS2 distributed communication can achieve a low-coupling design between modules, which facilitates the individual upgrading of the VLA model or navigation algorithm and reduces maintenance costs. The vertical layered architecture can clearly define the functional boundaries of each layer. The HMI layer improves the user-friendliness of human-machine interaction, the core control layer ensures decision accuracy, and the hardware interface layer improves device compatibility. The horizontal hybrid decision-making can solve the deficiency of VLA model's insufficient real-time performance. The traditional planner ensures the response speed (≤20ms) of high-frequency decisions such as navigation and obstacle avoidance, while the VLA model improves the system's semantic understanding capability.

[0062] See Figure 4 As shown in the figure, this invention discloses an industrial autonomous mobile robot control method, applied to a robot control system based on a layered architecture. The robot control system includes a human-machine interaction layer, a control layer, and a hardware interface layer. The method includes:

[0063] Step S11: The target semantic decision model of the human-computer interaction layer is used to decompose and analyze the input target industrial robot operation text instructions to obtain a structured sub-task sequence, and the navigation task and grasping task of the target industrial robot are generated based on the structured sub-task sequence; the target semantic decision model is a model obtained after fine-tuning the initial semantic decision model.

[0064] In this embodiment, the received target industrial robot operation text instructions need to be decomposed and analyzed through the target semantic decision model of the human-computer interaction layer to obtain the corresponding structured sub-task sequence. However, before the analysis, the initial semantic decision model needs to be fine-tuned to obtain the target semantic decision model. The model fine-tuning is based on a small sample fine-tuning scheme of low-rank adaptation (LoRA). The initial semantic decision model is the open-source VLA (Vision-Language-Action) model. Specifically, the human-computer interaction layer first collects historical robot operation texts based on preset instruction types and labels the historical robot operation texts with corresponding action sequences to generate a target dataset based on the historical robot operation texts and action sequences. The preset instruction types are workshop logistics instructions, covering three core tasks: "material handling", "loading and unloading scheduling", and "path adjustment". Each instruction needs to be labeled with a corresponding action sequence, such as "move the motor component of station 3 to entrance 2 of the assembly line → navigate to station 3 → robotic arm grabs the component → obstacle avoidance navigation to entrance 2 → place the component", forming a (instruction text, action sequence) paired dataset. The dataset is then fine-tuned using a very small number (50-80) of high-quality samples.

[0065] Furthermore, a pre-set low-rank matrix needs to be inserted into the initial semantic decision model, and the model parameters of the initial semantic decision model are frozen. Then, the parameters of the pre-set low-rank matrix are fine-tuned to obtain the fine-tuned model. It should be noted that in the key Transformer layers of the VLA model, such as the cross-attention layer between the visual encoder and the language encoder, two low-rank matrices A (dimension d×r) and B (dimension r×k) are inserted. The optimal value of rank r is dynamically optimized according to the number of model parameters, as shown in Table 1. The original model parameters are frozen, and only the low-rank matrix parameters are updated. Moreover, the optimal value of r is determined experimentally using the controlled variable method. With 50-80 samples, when the value of r exceeds the optimal range, the accuracy decreases by ≥3% or the parameter update ratio is ≥8%.

[0066] Table 1 Dynamic Optimization Table of Model Parameters

[0067] ;

[0068] Finally, the fine-tuned model needs to be trained on the target dataset to obtain the target semantic decision model. This model is then encapsulated and integrated into the robot control system. Specifically, the AdamW optimizer is used, with a learning rate of 5e-5 to 1e-4, 20 to 30 training epochs, and a batch size of 4 to 8. An early stopping strategy (stopping training if the validation set accuracy shows no improvement for three consecutive epochs) is employed to avoid overfitting. Noise data from industrial scenarios, such as background noise in a workshop and ambiguous command descriptions, is introduced during training to improve model robustness. The fine-tuned VLA model is then encapsulated into a ROS2 package. It receives natural language commands from the HMI (Human-Machine Interface) layer via topic communication and outputs structured task sequences (including action type, target coordinates, and operation parameters) to provide decision input for the core control layer. By optimizing the rank r of the low-rank matrix, the accuracy of the model's semantic parsing of industrial commands is significantly improved while keeping the parameter update rate below 5% of the original model. This greatly reduces the computational requirements for training and the risk of overfitting, enabling the VLA model to be deployed on edge computing devices.

[0069] It should be noted that the human-computer interaction layer and the core control layer are concepts abstracted from system technologies based on robot control systems, specifically as follows: Figure 5 As shown, it can be divided into the human-computer interaction layer, the core control layer, the implementation and execution layer, the hardware operation layer, and the intelligent decision-making layer.

[0070] Furthermore, such as Figure 6 As shown, if an input text command for operating the target industrial robot is received, the target semantic decision model of the human-machine interaction layer decomposes the text command into several text units. Then, semantic analysis is performed on these text units to generate a pre-formatted structured task sequence based on the target semantics corresponding to each text unit. Finally, navigation and grasping tasks for the target industrial robot are generated based on this structured task sequence. Specifically, the operator inputs natural language commands via the HMI graphical interface, either voice or text. Example command: "Move the motor assembly at workstation 3 to assembly line entrance 2." (HMI interaction layer...) pass Receive instruction. Message format: The type includes the original instruction text. The instruction is then forwarded to the VLA semantic decision module, which parses the instruction using a fine-tuned target semantic decision model: first, it identifies the intent: material handling; then it extracts key information: starting position (station 3), target position (assembly line entrance 2), and material type; task decomposition: generating a structured action sequence. Finally, based on the structured task sequence, it generates the navigation and grasping tasks for the target industrial robot. The VLA module outputs the structured task sequence in the following format:

[0071] ;

[0072] It should be noted that the output structured task sequence is sent through the ROS2 service / task_sequence_service. The specific definition of the service type task_msgs / TaskSequence is as follows:

[0073] ;

[0074] The core control layer provides real-time feedback on task execution status via the topic / vla_feedback, including message types. ,Include , , (e.g., "t") "、" ") (0 indicates normal, non-zero indicates abnormal). And the semantic constraints of the output (e.g., " " The core control layer resolves the data into the corresponding control parameters: ": Personnel parameters that trigger the APF semantic repulsion field ( ); "The weight coefficient of the priority path is adjusted to 0.4 during global path planning." ": Start dynamic obstacle trajectory prediction (Kalman filter prediction window 3 seconds).

[0075] In this way, VLA, fine-tuned by LoRA, supports natural language instruction understanding, can autonomously decompose complex tasks into subsequences, and achieve task-level autonomous decision-making without the need for pre-programming or formatted instructions.

[0076] Step S12: The task scheduler of the control layer calls a preset laser device according to the navigation task to generate a three-dimensional semantic map of the target endpoint corresponding to the operation text command of the target industrial robot. Then, the path planning and obstacle avoidance operation are performed through the three-dimensional semantic map to obtain the target path.

[0077] In this embodiment, corresponding processing is required based on the navigation task to perform path planning and obtain the target path. Specifically, such as... Figure 6 As shown, the task scheduler in the control layer needs to invoke a preset laser device based on the generated navigation task. The preset laser device then generates a 3D point cloud map corresponding to the target endpoint of the target industrial robot's operation text command using a preset data registration algorithm. The task scheduler receives the task sequence and prioritizes it according to a priority strategy: safety commands > navigation commands > operation commands. Sub-tasks are executed one by one, and the execution status is monitored. Furthermore, laser SLAM optimizes point cloud matching using an Iterative Closest Point (ICP) algorithm, generating a 3D geometric map with a resolution of 0.05m and outputting the coordinate set of the map point cloud. .

[0078] Then, it is necessary to bind the target semantics obtained from the target semantic decision model analysis with the geometric semantics and point cloud data corresponding to the 3D point cloud map, and generate a 3D semantic map by combining preset weight coefficients. Specifically, when the VLA model parses instructions, it simultaneously outputs scene semantic labels (such as "no entry area", "priority passage", "temporary obstacle", "target workstation"). Through the PointNet 3D point cloud semantic segmentation algorithm, the labels are bound to the geometric point cloud to form a semantic-geometric mapping table as shown in Table 2, which is the 3D semantic map.

[0079] Table 2 Semantic-Geometric Mapping Table

[0080] ;

[0081] Furthermore, path planning to the target endpoint needs to be performed based on the 3D semantic map to obtain an initial planned path. This initial planned path is then processed using a pre-defined random sampling consensus algorithm to fit its curve and remove redundant inflection points, resulting in an initially adjusted path. Specifically, this requires using A... The algorithm performs initial path planning, and its cost function is optimized as follows: Where: g(n) is the geometric distance cost from the starting point to the current node; h(n) is the Manhattan distance heuristic function from the current node to the target node; w(s) is the semantic weight coefficient, dynamically assigned by the semantic label (prohibited area w(s)=∞, priority channel w(s)=0.6, normal area w(s)=1.0, temporary obstacle w(s)=10.0). After the path is generated, the path curve is fitted by the RANSAC algorithm to remove redundant inflection points and ensure the smoothness of the mobile chassis movement. ).

[0082] Finally, the initially adjusted path needs to be optimized using a preset potential field function to obtain the target path. The preset potential field function is a function that optimizes the path using a preset gravitational field function, a preset repulsive field function, and a preset virtual repulsive field function. The preset gravitational field function determines the gravitational field of the target point in the path based on preset semantic weights of the target point; the preset repulsive field function determines the repulsive field of the target point in the path based on preset repulsive coefficients; and the preset virtual repulsive field function predicts the repulsive field of the target point in the path based on preset predicted repulsive coefficients. The specific principle is to address the shortcomings of traditional APF (Artificial Potential Field) methods, which are prone to getting trapped in local optima and have poor adaptability to dynamic obstacles, by incorporating VLA semantic information to optimize the potential field function, achieving local path adjustment with "dynamic obstacle avoidance + semantic compliance". Specifically, the preset potential field function needs to be defined first. Among them: gravitational field The gravitational field of the target point (such as the grabbing station) is dynamically adjusted according to semantic priority, as shown in the formula: , Target semantic weight (core task objective) =2.0, Secondary Task Objective =1.0); Semantic repulsion field Design differentiated repulsive forces based on the semantic type of obstacles, using the following formula: ,in: Repulsion coefficient (personnel) =50, mobile cargo box =30, Fixed equipment =20); Range of repulsive force (personnel) =2.5m, mobile cargo box =1.5m, fixed equipment =1.0m). Obstacle semantic weights (consistent with global planning weights). Predicted potential field. This method predicts the trajectory of dynamic obstacles (people, moving cargo boxes) 3 seconds in advance using Kalman filtering, by pre-setting a "virtual repulsion field" on the predicted path. The formula is as follows: , Let be the predicted coordinates of the obstacle at time t (t∈[0,3]s). Then, force field calculation is required, where The direction of the resultant force is determined using the gradient descent method and used as the direction vector for local path adjustment. The resultant force. Subsequently, local optimal solutions need to be found to avoid this, when the resultant force... When the threshold is determined experimentally, the system is considered trapped in a local optimum. At this point, a "semantic jump mechanism" is triggered—based on the semantic map, the nearest "priority path" or "temporary obstacle avoidance zone" is selected as the intermediate target point, the potential field is recalculated, and the local optimum is escaped. Ultimately, the path update frequency needs to be determined, synchronized with sensor data, updating the local path every 50ms to ensure a dynamic obstacle avoidance response speed ≤20ms. The target path is obtained by optimizing the initially adjusted path using a preset potential field function. In this way, by deeply embedding the semantic information parsed by VLA, such as 'personnel' and 'priority path', into the environmental map (weight coefficients) and control algorithm (potential field function parameters), a transition from geometric obstacle avoidance to semantic obstacle avoidance can be achieved. For example, assigning a larger repulsive force range and coefficient to 'personnel' not only avoids collisions but also induces more human-like deceleration and avoidance behavior in advance, improving operational safety and collaborative friendliness.

[0083] Step S13: The task scheduler calls a preset image acquisition device to acquire material images of a preset material area according to the grasping task, and processes the material images through a preset model to generate material grasping parameters.

[0084] In this embodiment, as Figure 6 As shown, processing is required through a capture task. Specifically, the task scheduler first needs to use a preset image acquisition device to capture material images of a preset material area based on the capture task, and then use a preset model to detect the material contours in the material images to output the corresponding material bounding boxes. Specifically, the preset image acquisition device, i.e., a depth camera, needs to capture RGB images and depth maps of the target material area, with an image resolution of 1280×720 and a frame rate of 30Hz. Then, the preset model, i.e., the YOLOv8 industrial version model, is used to detect the material contours and output the 2D bounding boxes of the materials. That is, the material boundary box.

[0085] Next, the target region needs to be cropped out of the material bounding box, and noise reduction processing needs to be performed on the target region. The resulting noise-reduced region is then converted into a corresponding 3D point cloud of the target region. Specifically, the ROI (Region of Interest) region needs to be cropped out, and then the depth map of the ROI region needs to be noise-reduced (Gaussian filtering). =1.5), converted to 3D point cloud .

[0086] Furthermore, semantic segmentation of the target region's 3D point cloud is required to extract the target grasping surface from the 3D point cloud and determine the grasping surface normal vector and grasping point coordinates corresponding to the target grasping surface. Specifically, the PointNet model needs to be called to perform semantic segmentation on the point cloud, extract the key grasping surfaces of the material, and output the normal vector of the grasping surface. and center point coordinates .

[0087] Finally, the coordinates of the center point of the gripping surface need to be compared with the coordinates of the target gripping point in the base coordinate system of the robot arm. The robot's pose is then adjusted based on the target gripping point coordinates and the gripping surface normal vector to obtain the target gripping posture. Finally, the gripping force of the gripper is determined based on the material type corresponding to the material image. Specifically, the coordinates of the center point of the gripping surface need to be converted to the target gripping point coordinates in the robot arm's base coordinate system using a hand-eye calibration matrix. Then, if the angle between the gripping surface normal vector and the initial posture of the robot arm's end effector is greater than 1, the gripping force is determined. Then, by solving the inverse kinematics problem, the joint angles of the robotic arm are adjusted (the adjustment step for each joint angle is ≤). To ensure the gripping surface is parallel to the jaws, the clamping force needs to be adjusted according to the material type (obtained through VLA semantic parsing). The clamping force is 25-30N for motor components (rigid) and 15-20N for battery modules (soft pack). At the same time, the contact force is fed back in real time by a force sensor (sampling rate 100Hz). If the force exceeds the threshold, the force is automatically released by 0.5N to ensure that the material is not damaged.

[0088] Step S14: Convert the target path and the material grasping parameters into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and control the target industrial robot to execute the navigation task and the grasping task through the preset instruction priority and the robot scheduling instructions.

[0089] In this embodiment, as Figure 6 As shown, robot control commands need to be generated based on the target path corresponding to the navigation task and the material grasping parameters corresponding to the grasping task. Specifically, firstly, feature fusion needs to be performed on the target path and the material grasping parameters, and the pose information of the target industrial robot needs to be generated based on the obtained target fusion features. Then, the pose information is converted into control commands for the target industrial robot. It should be noted that the feature fusion in this embodiment uses the Extended Kalman Filter (EKF) algorithm to fuse data from LiDAR, visual odometry, and IMU to estimate the robot's real-time pose and velocity, providing accurate feedback information for path tracking and motion control. Specifically, in the feature fusion part, the state feature equation needs to be defined first: , where the state vector (Position, attitude, linear velocity, angular velocity), A is the state transition matrix (constructed based on the robot's kinematics model), B is the control input matrix (input u(k) is the chassis linear velocity and angular velocity command), and W(k) is the process noise (Gaussian distribution, variance matrix). Then we need to define the observation equation: The observation vector Z(k) comes from three types of sensors: LiDAR, depth camera, and IMU (Inertial Measurement Unit). The position is output by LiDAR through ICP matching, the position is output by visual SLAM, and the angular velocity is output by IMU. and linear acceleration (The velocity is obtained after integration) Then, Kalman filtering updates and data synchronization are required. Using ROS2's timestamp alignment tool (message_filters), the timestamp error of the three types of sensor data is controlled to ≤5ms to ensure fusion accuracy. Finally, the converted robot scheduling instructions are sent to the target industrial robot so that it can move along the planned path. Upon reaching the target position, the robotic arm performs grasping / placement actions and adjusts its posture in real time to adapt to the actual environment.

[0090] It should be noted that, such as Figure 6As shown, when the robot is performing a task, it is also necessary to monitor whether the robot has completed the task and provide data feedback. Specifically, it is necessary to collect the target robot's operation data to determine whether the target industrial robot has completed the navigation and grasping tasks. If not, the process jumps to the step of decomposing and analyzing the input target industrial robot operation text instructions through the target semantic decision model of the human-machine interaction layer. If completed, the operation data is fed back to the front-end page of the robot control system. Each task needs to be updated with its status updated after completion, and progress information is published via the ROS2 topic / task_progress. Furthermore, the semantic map needs to be updated based on new perception data; visited areas, obstacle changes, and material position changes are marked. Then, it is necessary to check whether all sub-tasks are completed and verify whether the final state meets the expected goals. The results need to be fed back to the HMI, sending information such as task completion status, execution time, and energy consumption to the HMI so that the HMI can update the interface, display the task completion status, update the robot's current position and status, and record operation logs for subsequent analysis and optimization. Finally, the target industrial robot returns to its original standby position or charging station; all modules enter a low-power standby state; the VLA model remains loaded, ready to receive new instructions. In this way, by monitoring the completion rate of the robot's tasks, the accuracy of the industrial autonomous mobile robot's control can be ensured, and by updating the semantic map, the semantic understanding capability can be improved for the next task execution.

[0091] In this embodiment, after receiving the input target industrial robot operation text command, it is necessary to decompose and analyze it through a target semantic decision model to obtain a corresponding structured sub-task sequence. Then, the robot's navigation task and grasping task need to be generated through this task sequence. Next, corresponding path planning and grasping parameter generation are performed through the two tasks respectively. The navigation task calls the corresponding equipment to generate a 3D semantic map, and then path planning and obstacle avoidance operations are performed using the 3D semantic map to obtain the target path. The grasping task generates material grasping parameters. Finally, the target path and material grasping parameters are converted into robot scheduling instructions corresponding to the target industrial robot, and the robot scheduling instructions are used to control the target industrial robot to execute the navigation and grasping tasks. In this way, on the one hand, by using small-sample VLA adaptation and 3D point cloud semantic segmentation, the geometric map is upgraded into a 3D intelligent map that combines geometry, semantics, and dynamics, enabling accurate identification of device type, area attributes, and obstacle dynamic attributes, resulting in better path planning and more flexible obstacle avoidance strategies. On the other hand, the VLA model based on LoRA fine-tuning supports natural language command understanding and can autonomously decompose complex tasks into sub-sequences, achieving task-level autonomous decision-making without the need for pre-programming or formatted commands. Furthermore, by fusing laser, vision, and IMU data into a unified spatial intelligent model, complementary and enhanced perceptual information can be achieved, supporting flexible operation decision-making. Therefore, it is possible to achieve high-precision environmental perception, high-level semantic understanding, autonomous path planning, and flexible operation execution for robot control.

[0092] As a preferred embodiment, such as Figure 7 The diagram shows the technical path of the industrial autonomous mobile robot control method of this application.

[0093] The preparation phase involves environment setup and data preparation, including hardware and software configuration, simulation environment construction, and VLA data preparation. Software configuration requires installing ROS2Humble on Ubuntu 22.04, while hardware configuration requires debugging the LiDAR, IMU, depth camera, and robot arm drive. Simulation environment construction involves building a simplified industrial scene in Gazebo, including elements such as a workbench, materials, and obstacles, and importing the target industrial robot model. VLA data preparation requires collecting and defining approximately 50 core industrial commands to construct a (command, action sequence) paired dataset for model fine-tuning.

[0094] During the perception system development phase, multi-sensor fusion localization and semantic map construction need to be explored. For multi-sensor fusion localization, the `robot_localization` package is used to fuse LiDAR, odometry, and IMU data to provide smooth robot pose estimation. 3D point cloud mapping is performed using `rtab_map_ros` or a similar approach, and loop closure detection incorporating visual features is attempted to improve long-term localization accuracy. For semantic map construction, an offline processing approach is required. First, a high-precision point cloud map is constructed. Then, a pre-trained visual model is used to identify keyframes on the map, and the identified object labels (such as "Machine Tool 1" and "Bag") are injected into the map to generate an initial 3D semantic map.

[0095] During the development phase of the navigation system, lightweight fine-tuning of the VLA model and navigation stack integration and security configuration are required. For lightweight fine-tuning of the VLA model, a smaller, open-source VLA model should be selected. Using the previously prepared industrial instruction dataset, efficient parameter fine-tuning methods such as LoRA are employed to teach the model to parse "go to point A to retrieve object B" into a structured task sequence such as [navigate to (x1, y1), execute the retrieval]. For navigation stack integration and security configuration, Nav2 needs to be configured, using A... Global planning is performed, while APF performs local planning. Preset interfaces are called during simulation to set up virtual security walls and collaborative areas, verifying the security logic.

[0096] During the control system development phase, a full-process integration and debugging of the HMI and control layer development and simulation environment is required. This necessitates developing a simple desktop interface using PyQt to display the map and robot position, and integrating speech recognition and synthesis modules. A core control node is developed to subscribe to the ` / vla_decision` topic and convert it into navigation target requests to Nav2 or MovJ / MovL commands to the robot. In Gazebo, the complete chain of voice command issuance, VLA parsing, navigation, and robotic arm motion simulation is tested to ensure smooth interface communication between modules.

[0097] In the final physical integration and verification phase, hardware system integration is required: deploying the software system onto a mobile robot platform equipped with an NVIDIA Jetson or high-performance industrial PC. Connecting and debugging all physical sensors and the robot arm. Localization and mapping testing: reconstructing a semantic map in a real-world environment and testing localization accuracy and stability. Navigation and obstacle avoidance testing: testing the robot's obstacle avoidance capabilities in dynamic environments. End-to-end task testing: designing 3-5 typical scenario tasks to comprehensively test the system's success rate, response time, and robustness, and optimizing parameters based on the results.

[0098] It should be noted that in practical applications, other solutions can be used to implement the industrial autonomous mobile robot control method of this application. For example, during the development of the control and navigation system, an adapter fine-tuning technique can be used to insert a small adapter module (dimension d×128) into the Transformer layer of the VLA model. This is suitable for scenarios where model structure modifications are more flexible, with the advantage of stronger adaptability. However, the disadvantage is that the number of training parameters is slightly more than LoRA (approximately 8% of the original model), and the computational consumption is slightly increased. Prefix Tuning can also be used to fine-tune the prefix, adding a trainable prefix vector before the input text. The advantage is that the model structure does not need to be modified, resulting in stronger compatibility. The disadvantage is that the structured decomposition accuracy of industrial instructions is slightly lower than LoRA, making it suitable for scenarios with relatively fixed instruction formats.

[0099] In the three development stages mentioned above, FastDDS can be used to build a communication framework independently. Its advantages include lower communication latency (≤5ms), making it suitable for scenarios with extremely high real-time requirements. However, its ecosystem integrity is lower than ROS2, and module integration becomes more difficult. Alternatively, a centralized control architecture can be adopted, integrating semantic decision-making and action planning through a single control node. This approach offers advantages such as simple architecture and short development cycles, but its disadvantages include high coupling, difficulty in subsequent upgrades and maintenance, and inability to resolve computational power constraints.

[0100] As a preferred embodiment, when an abnormality occurs during robot operation, appropriate abnormality handling is required, such as... Figure 8 The diagram illustrates the anomaly handling process of the industrial autonomous mobile robot in this application. When an anomaly is detected, it is first classified into three types: minor anomalies, severe anomalies, and fatal anomalies. For minor anomalies, the robot can adjust itself and record the corresponding anomaly log. For severe anomalies, an alarm notification is issued, and relevant maintenance personnel are notified for manual intervention to repair the system. It should be noted that minor and severe anomalies can be repaired and the robot can continue to operate. When a fatal anomaly occurs, an emergency shutdown is required, followed by a safety check to eliminate the corresponding fault, and then a system restart to determine if the system has returned to normal.

[0101] See Figure 9 As shown, this embodiment of the invention discloses an industrial autonomous mobile robot control device, applied to a robot control system based on a layered architecture. The robot control system includes a human-machine interaction layer, a control layer, and a hardware interface layer, comprising:

[0102] The task generation module 11 is used to decompose and analyze the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer to obtain a structured sub-task sequence, and generate the navigation task and grasping task of the target industrial robot based on the structured sub-task sequence; the target semantic decision model is a model obtained after fine-tuning the initial semantic decision model, and the structured sub-task sequence is a discrete decision output.

[0103] The path planning module 12 is used to call a preset laser device according to the navigation task through the task scheduler of the control layer, so as to generate a three-dimensional semantic map of the target endpoint corresponding to the operation text command of the target industrial robot through the preset laser device, and then perform path planning and obstacle avoidance operation through the three-dimensional semantic map to obtain the target path.

[0104] The parameter generation module 13 is used to call a preset image acquisition device to acquire material images of a preset material area according to the grasping task through the task scheduler, and process the material images through a preset model to generate material grasping parameters.

[0105] The instruction execution module 14 is used to convert the target path and the material grasping parameters into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and to control the target industrial robot to perform the navigation task and the grasping task through the preset instruction priority and the robot scheduling instructions.

[0106] In this embodiment, after receiving the input text command for the target industrial robot, the command is decomposed and analyzed using a target semantic decision model to obtain a corresponding structured sub-task sequence. Then, the robot's navigation and grasping tasks are generated from this task sequence. Next, path planning and grasping parameter generation are performed for each task. The navigation task calls the appropriate equipment to generate a 3D semantic map, which is then used for path planning and obstacle avoidance to obtain the target path. The grasping task generates material grasping parameters. Finally, the target path and material grasping parameters are converted into robot scheduling instructions corresponding to the target industrial robot, and these instructions are used to control the target industrial robot to execute the navigation and grasping tasks. This enables high-precision environmental perception, high-level semantic understanding, autonomous path planning, and flexible operation execution for robot control.

[0107] In some embodiments, the industrial autonomous mobile robot control device may further include:

[0108] The dataset generation unit is used to collect historical robot operation texts according to preset instruction types through the human-computer interaction layer, and to annotate the historical robot operation texts with corresponding action sequences, so as to generate a target dataset based on the historical robot operation texts and the action sequences.

[0109] The model fine-tuning unit is used to insert a preset low-rank matrix into the initial semantic decision model, freeze the model parameters of the initial semantic decision model, and then fine-tune the parameters of the preset low-rank matrix to obtain the fine-tuned model.

[0110] The model encapsulation unit is used to train the fine-tuned model using the target dataset to obtain a target semantic decision model, and to encapsulate the target semantic decision model so that the encapsulated target semantic decision model is integrated into the robot control system.

[0111] In some embodiments, the task generation module 11 may specifically include:

[0112] The instruction decomposition unit is used to decompose the target industrial robot operation text instruction into several text words if it receives the input target industrial robot operation text instruction through the target semantic decision model of the human-machine interaction layer.

[0113] The task generation unit is used to perform semantic analysis on the plurality of text words, generate a structured task sequence in a preset format based on the plurality of target semantics corresponding to the plurality of text words, and generate navigation tasks and grasping tasks for the target industrial robot based on the structured task sequence.

[0114] In some embodiments, the path planning module 12 may specifically include:

[0115] The mapping table generation unit is used to bind the plurality of target semantics to the point cloud data corresponding to the three-dimensional point cloud map, and generate a three-dimensional semantic map by combining the preset weight coefficients; the plurality of target semantics correspond to different preset weight coefficients respectively;

[0116] The path generation unit is used to plan a path to the target endpoint based on the three-dimensional semantic map to obtain an initial planned path, and to process the initial planned path through a preset random sampling consensus algorithm to fit the path curve of the initial planned path and remove redundant inflection points in the initial planned path to obtain an initial adjusted path.

[0117] A path planning unit is used to optimize the initially adjusted path using a preset potential field function to obtain a target path. The preset potential field function is a function that optimizes the path using a preset gravitational field function, a preset repulsive field function, and a preset virtual repulsive field function. The preset gravitational field function is a function that determines the gravitational field of the target point in the path based on a preset semantic weight of the target point. The preset repulsive field function is a function that determines the repulsive field of the target point in the path based on a preset repulsive coefficient. The preset virtual repulsive field function is a function that predicts the repulsive field of the target point in the path based on a preset predicted repulsive coefficient.

[0118] In some embodiments, the parameter generation module 13 may specifically include:

[0119] The bounding box determination unit is used to call a preset image acquisition device to acquire a material image of a preset material area according to the grasping task through the task scheduler, and to detect the material contour in the material image through a preset model, so as to output the material bounding box corresponding to the material contour;

[0120] The three-dimensional point cloud generation unit is used to cut out the target area within the material boundary frame, perform noise reduction on the target area, and convert the noise-reduced area into a corresponding three-dimensional point cloud of the target area.

[0121] The grasping surface data determination unit is used to perform semantic segmentation on the three-dimensional point cloud of the target region to extract the target grasping surface in the three-dimensional point cloud of the target region, and determine the grasping surface normal vector and grasping point coordinates corresponding to the target grasping surface;

[0122] The clamping force determination unit is used to match the coordinates of the center point of the gripping surface with the coordinates of the target gripping point corresponding to the base coordinate system of the robotic arm of the target industrial robot, and to adjust the pose of the target industrial robot according to the coordinates of the target gripping point and the normal vector of the gripping surface to obtain the target gripping posture. Then, it determines the gripping force of the gripper according to the material type corresponding to the material image.

[0123] In some embodiments, the instruction execution module 14 may specifically include:

[0124] The instruction conversion unit is used to perform feature fusion on the target path and the material grasping parameters, generate the pose information of the target industrial robot based on the obtained target fusion features, and convert the pose information into control instructions for the target industrial robot.

[0125] The instruction scheduling unit is used to convert the control instructions into robot scheduling instructions in a preset protocol format, and to send the robot scheduling instructions to the target industrial robot, so that the target industrial robot can execute the navigation task and the grasping task according to the preset instruction priority and the robot scheduling instructions; the robot scheduling instructions include navigation instructions corresponding to the navigation task and operation instructions corresponding to the grasping task, and the preset instruction priority is that the priority of the navigation instructions is higher than that of the scheduling instructions.

[0126] In some embodiments, the industrial autonomous mobile robot control device may further include:

[0127] The operation data acquisition unit is used to collect the operation data of the target industrial robot to determine whether the target industrial robot has completed the navigation task and the grasping task.

[0128] The step jump unit is used to jump to the step of decomposing and analyzing the input target industrial robot operation text instruction through the target semantic decision model of the human-computer interaction layer if the step is not completed.

[0129] The data feedback unit is used to feed back the operation data to the front-end page of the robot control system if the operation has been completed.

[0130] Furthermore, embodiments of this application also disclose an electronic device, Figure 10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0131] Figure 10 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the industrial autonomous mobile robot control method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0132] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0133] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0134] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the industrial autonomous mobile robot control method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0135] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed industrial autonomous mobile robot control method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0137] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0139] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A control method for an industrial autonomous mobile robot, characterized in that, An application to a layered robot control system, the robot control system comprising a human-machine interaction layer, a control layer, and a hardware interface layer, the method comprising: The target semantic decision model of the human-computer interaction layer decomposes and analyzes the input target industrial robot operation text instructions to obtain a structured sub-task sequence, and generates the target industrial robot's navigation task and grasping task based on the structured sub-task sequence; the target semantic decision model is a model obtained after fine-tuning the initial semantic decision model, and the structured sub-task sequence is a discrete decision output. The task scheduler of the control layer calls a preset laser device according to the navigation task, so as to generate a three-dimensional semantic map of the target destination corresponding to the operation text command of the target industrial robot. Then, the path planning and obstacle avoidance operation are performed through the three-dimensional semantic map to obtain the target path. The task scheduler calls a preset image acquisition device to acquire material images of a preset material area according to the grasping task, and processes the material images through a preset model to generate material grasping parameters. The target path and material grasping parameters are converted into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and the target industrial robot is controlled to perform the navigation task and the grasping task by the preset instruction priority and the robot scheduling instructions. The process involves the task scheduler in the control layer invoking a preset laser device based on the navigation task. This preset laser device generates a 3D semantic map corresponding to the target endpoint and the operational text commands of the target industrial robot. The 3D semantic map is then used for path planning and obstacle avoidance to obtain the target path. This includes: The task scheduler of the control layer calls a preset laser device based on the generated navigation task, and the preset laser device generates a three-dimensional point cloud map of the target endpoint corresponding to the target industrial robot operation text command based on a preset data registration algorithm. Several target semantics are bound to the point cloud data corresponding to the 3D point cloud map, and a 3D semantic map is generated by combining preset weight coefficients; the several target semantics are semantics corresponding to several text words, and the several target semantics correspond to different preset weight coefficients; the several text words are words obtained by decomposing the target industrial robot operation text instructions; The path to the target endpoint is planned based on the three-dimensional semantic map to obtain an initial planned path. The initial planned path is then processed by a preset random sampling consensus algorithm to fit the path curve of the initial planned path and remove redundant inflection points in the initial planned path to obtain an initial adjusted path. The initial adjusted path is optimized using a preset potential field function to obtain the target path. The preset potential field function is a function that optimizes the path using a preset gravitational field function, a preset repulsive field function, and a preset virtual repulsive field function. The preset gravitational field function is a function that determines the gravitational field of the target point in the path based on a preset semantic weight of the target point. The preset repulsive field function is a function that determines the repulsive field of the target point in the path based on a preset repulsive coefficient. The preset virtual repulsive field function is a function that predicts the repulsive field of the target point in the path based on a preset predicted repulsive coefficient.

2. The industrial autonomous mobile robot control method according to claim 1, characterized in that, Before the step of decomposing and analyzing the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer to obtain a structured sub-task sequence, and generating the navigation task and grasping task of the target industrial robot based on the structured sub-task sequence, the method further includes: The human-computer interaction layer collects historical robot operation texts according to preset instruction types and annotates the historical robot operation texts with corresponding action sequences to generate a target dataset based on the historical robot operation texts and the action sequences. A preset low-rank matrix is ​​inserted into the initial semantic decision model, and the model parameters of the initial semantic decision model are frozen. Then, the parameters of the preset low-rank matrix are fine-tuned to obtain the fine-tuned model. The fine-tuned model is trained using the target dataset to obtain a target semantic decision model, which is then encapsulated and integrated into the robot control system.

3. The industrial autonomous mobile robot control method according to claim 1, characterized in that, The step involves decomposing and analyzing the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer to obtain a structured sub-task sequence, and generating navigation and grasping tasks for the target industrial robot based on the structured sub-task sequence, including: If an input target industrial robot operation text instruction is received, the target industrial robot operation text instruction is decomposed into several text words through the target semantic decision model of the human-machine interaction layer. Semantic analysis is performed on the aforementioned text lexical units to generate a structured task sequence in a preset format based on the target semantics corresponding to the aforementioned text lexical units, and a navigation task and a grasping task for the target industrial robot are generated based on the structured task sequence.

4. The industrial autonomous mobile robot control method according to claim 1, characterized in that, The process of using the task scheduler to call a preset image acquisition device to acquire material images of a preset material area according to the grasping task, and processing the material images using a preset model to generate material grasping parameters, includes: The task scheduler calls a preset image acquisition device to acquire material images of a preset material area according to the capture task, and detects the material contours in the material images through a preset model to output the material bounding boxes corresponding to the material contours. The target area within the material boundary is cropped out, and noise reduction is applied to the target area. The resulting noise-reduced area is then converted into a corresponding 3D point cloud of the target area. Semantic segmentation is performed on the 3D point cloud of the target region to extract the target grasping surface in the 3D point cloud of the target region, and the grasping surface normal vector and grasping point coordinates corresponding to the target grasping surface are determined. The coordinates of the center point of the gripping surface are compared with the coordinates of the target gripping point corresponding to the base coordinate system of the robotic arm of the target industrial robot. The pose of the target industrial robot is adjusted according to the coordinates of the target gripping point and the normal vector of the gripping surface to obtain the target gripping posture. Then, the gripping force of the gripper is determined according to the material type corresponding to the material image.

5. The industrial autonomous mobile robot control method according to claim 1, characterized in that, The step of converting the target path and the material grasping parameters into robot scheduling instructions corresponding to the target industrial robot, and controlling the target industrial robot to execute the navigation task and the grasping task through preset instruction priorities and the robot scheduling instructions, includes: The target path and the material grasping parameters are fused to form features, and the pose information of the target industrial robot is generated based on the obtained target fusion features. The pose information is then converted into control commands for the target industrial robot. The control commands are converted into robot scheduling commands in a preset protocol format, and the robot scheduling commands are sent to the target industrial robot so that the target industrial robot can execute the navigation task and the grasping task according to the preset command priority and the robot scheduling commands. The robot scheduling commands include navigation commands corresponding to the navigation task and operation commands corresponding to the grasping task, and the preset command priority is that the navigation commands have a higher priority than the scheduling commands.

6. The industrial autonomous mobile robot control method according to any one of claims 1 to 5, characterized in that, After converting the target path and material grasping parameters into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and controlling the target industrial robot to execute the navigation task and the grasping task through preset instruction priority and the robot scheduling instructions, the method further includes: Collect the operational data of the target industrial robot to determine whether the target industrial robot has completed the navigation task and the grasping task; If not completed, proceed to the step of decomposing and analyzing the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer; If the task is completed, the task data will be fed back to the front-end page of the robot control system.

7. A control device for an industrial autonomous mobile robot, characterized in that, An application to a layered architecture-based robot control system, the robot control system comprising a human-machine interaction layer, a control layer, and a hardware interface layer, the device comprising: The task generation module is used to decompose and analyze the input target industrial robot operation text instructions through the target semantic decision model of the human-computer interaction layer to obtain a structured sub-task sequence, and generate the navigation task and grasping task of the target industrial robot based on the structured sub-task sequence; the target semantic decision model is a model obtained after fine-tuning the initial semantic decision model, and the structured sub-task sequence is a discrete decision output. The path planning module is used to call a preset laser device according to the navigation task through the task scheduler of the control layer, so as to generate a three-dimensional semantic map of the target destination corresponding to the operation text command of the target industrial robot through the preset laser device, and then perform path planning and obstacle avoidance operation through the three-dimensional semantic map to obtain the target path; The parameter generation module is used to call a preset image acquisition device to acquire material images of a preset material area according to the grasping task through the task scheduler, and process the material images through a preset model to generate material grasping parameters. The instruction execution module is used to convert the target path and the material grasping parameters into robot scheduling instructions corresponding to the target industrial robot through the hardware interface layer, and to control the target industrial robot to perform the navigation task and the grasping task through the preset instruction priority and the robot scheduling instructions. The path planning module is used for: The task scheduler of the control layer calls a preset laser device based on the generated navigation task, and the preset laser device generates a three-dimensional point cloud map of the target endpoint corresponding to the target industrial robot operation text command based on a preset data registration algorithm. Several target semantics are bound to the point cloud data corresponding to the 3D point cloud map, and a 3D semantic map is generated by combining preset weight coefficients; the several target semantics are semantics corresponding to several text words, and the several target semantics correspond to different preset weight coefficients; the several text words are words obtained by decomposing the target industrial robot operation text instructions; The path to the target endpoint is planned based on the three-dimensional semantic map to obtain an initial planned path. The initial planned path is then processed by a preset random sampling consensus algorithm to fit the path curve of the initial planned path and remove redundant inflection points in the initial planned path to obtain an initial adjusted path. The initial adjusted path is optimized using a preset potential field function to obtain the target path. The preset potential field function is a function that optimizes the path using a preset gravitational field function, a preset repulsive field function, and a preset virtual repulsive field function. The preset gravitational field function is a function that determines the gravitational field of the target point in the path based on a preset semantic weight of the target point. The preset repulsive field function is a function that determines the repulsive field of the target point in the path based on a preset repulsive coefficient. The preset virtual repulsive field function is a function that predicts the repulsive field of the target point in the path based on a preset predicted repulsive coefficient.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the industrial autonomous mobile robot control method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the industrial autonomous mobile robot control method as described in any one of claims 1 to 6.

Images