Path planning model acquisition method, computer device and computer readable storage medium

Abstract

Embodiments of the present application disclose a path planning model acquisition method, a computer device and a computer readable storage medium. The path planning model acquisition method comprises the following steps: obtaining an initial path planning model, the initial path planning model being obtained by training based on path training data in a target virtual scene; issuing the initial path planning model to a robot and obtaining real path parameters fed back by the robot, the real path parameters being collected by the robot moving and / or working according to a path planned by the initial path planning model; and updating the initial path planning model in the target virtual scene according to the real path parameters to obtain a path planning model. Therefore, the present application can obtain an initial model by pre-training in a virtual scene, and then use a robot deployed with the initial model to collect real data to feed back to the virtual scene. Through the closed-loop bidirectional link from virtual to real and from real to virtual, the accuracy of model path planning is improved.

Description

Technical Field

[0001] The present application belongs to the field of robot control technology, and in particular relates to a path planning model acquisition method, a computer device, and a computer-readable storage medium. Background Art

[0002] Path planning for lineside robots is a core component of intelligent manufacturing and Industry 4.0. As production cycles accelerate, traditional fixed-path AGV (Automated Guided Vehicle) path planning methods are no longer sufficient. Current mainstream solutions can be categorized into three types: 1. Improved traditional algorithms, which incorporate dynamic weighting (Dynamic Weighting A) into the A algorithm and integrate the artificial potential field method with rolling windows (APF-Rolling); 2. Machine learning-driven approaches, which use discrete action space path planning based on DQN and imitate learning to replicate manual operation trajectories; and 3. Simulation-assisted approaches, which use Gazebo and ROS to build a training environment and transfer learning to address the Sim2Real problem. However, existing solutions suffer from limitations in response latency, data efficiency, and simulation reliability, making lineside robot path planning inadequate for real-world production scenarios. Improving the accuracy of lineside robot path planning is a pressing technical challenge for those skilled in the art.

[0003] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0004] Based on this, it is necessary to address the above problems and propose a path planning model acquisition method, computer equipment and computer-readable storage medium, which can use higher training efficiency to obtain the effect speed steel block and a path planning model with higher path planning accuracy.

[0005] The present application solves the technical problem by adopting the following technical solutions:

[0006] The present application provides a path planning model acquisition method, which is applied to a server, and the server is associated with a robot; the method includes the following steps: obtaining an initial path planning model, which is obtained by training in a target virtual scene based on path training data; sending the initial path planning model to the robot, and obtaining real path parameters fed back by the robot, which are collected when the robot moves and / or works according to the path planned by the initial path planning model; and updating the initial path planning model in the target virtual scene according to the real path parameters to obtain a path planning model.

[0007] In an optional embodiment of the present application, obtaining an initial path planning model includes: constructing an initial path planning model, the network structure of the initial path planning model includes a first layer and a second layer; the first layer is the input layer of the model, composed of a soft policy gradient algorithm, for receiving map data and task instructions, and processing the map data and task instructions to output a path key point set; the second layer is connected to the first layer, is the output layer of the model, composed of a shear probability optimization algorithm, for receiving a path key point set and real-time environmental data, and processing the path key point set and real-time environmental data to output control instructions, and the control instructions are used to control the movement of the robot.

[0008] In an optional embodiment of the present application, the initial path planning model is updated in the target virtual scene according to the real path parameters to obtain the path planning model, including: updating the target virtual scene according to the real path parameters, setting at least one virtual robot in the updated target virtual scene, and deploying the initial path planning model in the virtual robot; updating the path training data according to the real path parameters to obtain real fine-tuning training data; using the virtual robot for iterative training; each round of iterative training includes: obtaining virtual task instructions and inputting them into the robot, controlling the virtual robot to respond to and complete the task corresponding to the virtual task instructions according to the control instructions, and the control instructions are obtained by the initial path planning model processing the virtual task instructions; obtaining task information generated in the process of the virtual robot completing the task, and the task information is used to record the completion status of the virtual robot; judging whether the initial path planning model of the current round meets the preset training conditions according to the task information; if so, determining that the initial path planning model of the current round has completed training, and using the initial path planning model of the current round as the path planning model; if not, updating the initial path planning model of the current round according to the task information, and performing the next round of iterative training.

[0009] In an optional embodiment of the present application, path training data is updated according to real path parameters to obtain real fine-tuning training data, and obtaining the real fine-tuning training data includes: marking the path training data as a source domain and marking the real path parameters as a target domain; mapping features in the source domain and the target domain to a unified feature space through a feature alignment algorithm based on a generative adversarial network; performing data preprocessing operations on the data output from the feature space to obtain a preliminary data set, the data preprocessing operations including spatiotemporal alignment and anomaly filtering; and reconfiguring the path training data in a target virtual scene according to the preliminary data set to obtain real fine-tuning training data.

[0010] In an optional embodiment of the present application, the initial path planning model is updated in the target virtual scene, including: determining update information of the initial path planning model; determining the noise intensity based on the update information; updating the target virtual scene using a preset noise with a numerical value corresponding to the noise intensity, and updating the initial path planning model in the updated target virtual scene.

[0011] In an optional embodiment of the present application, the noise intensity is determined according to the update information, including: obtaining the training round in the update information, the training round is used to characterize the number of rounds of the current iterative training of the initial path planning model; when the training round is less than a first threshold, determining the noise intensity of the current round according to the current training round; when the training round is greater than or equal to the first threshold and less than a second threshold, obtaining the training stability index of the previous round; if the training stability index is less than a preset stability threshold, updating the noise intensity of the current round according to the training stability index; if the training stability index is greater than or equal to the preset stability threshold, maintaining the noise intensity of the previous round; when the training round is greater than or equal to the second threshold, determining the action variance of the previous round, the action variance is used to characterize the degree of randomness of the movement during the robot training process; if the action variance of the previous round is greater than the preset threshold, freezing the noise intensity of subsequent rounds and keeping it unchanged; if the action variance of the previous round is less than or equal to the preset threshold, updating the noise intensity of the current round according to the action variance.

[0012] In an optional embodiment of the present application, the initial path planning model is updated in the target virtual scene to obtain the path planning model, including: in the target virtual scene, the initial path planning model is subjected to system stability training to obtain the path planning model, the system stability training includes at least one of fault recovery training, multi-robot conflict resolution training and safety protection training; the fault recovery training includes: during the training process, randomly inputting preset faults into the robots in the virtual scene, controlling the robots in the target virtual scene to respond according to the preset faults and perform recovery processing to resolve the preset faults; the multi-robot conflict resolution training includes: adding multiple robots to the target virtual scene and constructing a non-cooperative game model; the robots in the target virtual scene are used as players in the non-cooperative game model and input, and the non-cooperative game model is iterated until a preset equilibrium condition is met; the safety protection training includes: during the training process, randomly injecting disturbance attacks into the robots in the target virtual scene according to a preset attack method.

[0013] The present application also provides a path planning model acquisition method, which is applied to a robot, and the robot is associated with a server; the method includes the following steps: obtaining an initial path planning model issued by the server; obtaining environmental data in response to task instructions; inputting the task instructions and environmental data into the initial path planning model, and moving and / or working according to the operating path planned and output by the initial path planning model; obtaining real path parameters during the movement and / or work process, and uploading the real path parameters to the server, and the real path parameters are used to assist the server in updating the initial path planning model.

[0014] The present application also provides a computer device, comprising a processor and a memory: the processor is configured to execute a computer program stored in the memory to implement the aforementioned method.

[0015] The present application also provides a computer-readable storage medium storing a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0016] The embodiments of the present application have the following beneficial effects:

[0017] This application can use the digital twin method to train the model in the target virtual scene respectively, and use the characteristic of the target virtual scene to speed up the speed to enhance the training effect of the model; and use the robot deployed with the trained model to collect real data to feed back the virtual scene, thereby ensuring the training effect and authenticity. Through the closed-loop bidirectional link from virtual to real and real to virtual, the accuracy of the model path planning is improved. In the fusion of virtual and real bidirectional data links, the gap between simulation and reality is significantly reduced through feature alignment and progressive noise injection, thereby improving the training effect. A hierarchical decision-making mechanism is adopted for the model. The upper-level algorithm handles global task allocation, and the lower-level algorithm focuses on real-time obstacle avoidance, forming a "strategy-tactics" two-layer optimization to improve the robot's work efficiency and response speed.

[0018] The above description is only an overview of the technical solution of this application. In order to more clearly understand the technical means of this application, which can be implemented in accordance with the contents of the description, and to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the following preferred embodiments are specifically described in detail with reference to the accompanying drawings. It should be understood that the above general description and the detailed description below are only exemplary and explanatory and do not limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A flowchart of a path planning model acquisition method applied to a server is provided in an embodiment.

[0021] Figure 2 A flowchart of a method for obtaining a path planning model for a robot is provided in one embodiment.

[0022] Figure 3 A timing diagram of a path planning model acquisition method provided in one embodiment.

[0023] Figure 4 The present invention is a schematic block diagram of the structure of a computer device provided by an embodiment. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] Despite multiple phases of methodological evolution, current mainstream solutions for robot pathfinding still suffer from the following deficiencies: 1. Response delay: The dynamically weighted A* algorithm experiences a planning delay exceeding one second when faced with more than five moving obstacles (measured data from existing production lines). The DWA algorithm (Dynamic Window Approach) is prone to oscillation in narrow channels (with an oscillation probability >65% for channels <0.5m wide). 2. Data efficiency: Deep Q-Network (DQN) algorithms suffer from low sample efficiency. One study showed that imitation learning requires more than 30 hours of labeled video to learn a right-angle turn after 1,000 real collisions. 3. Simulation credibility deficiencies: Dynamic variations in the ground friction coefficient (±15% error), multipath reflections from LiDAR (simulation miss detection rate of 12.7%), and prediction of human motion (error of up to 40% in human-in-the-loop scenarios). 4. Multi-objective optimization: Existing methods struggle to balance path length vs. energy consumption (12% of the Pareto frontier has an infeasible region) and individual robot efficiency vs. overall system throughput (collision rate >25%). To address these shortcomings, this application proposes a path planning model, as well as a training and application solution for this model, specifically including a path planning model acquisition method. This application provides a method for application in a production line application environment called a line-side warehouse (LSW), a micro-storage area adjacent to the production line, for JIT (Just-In-Time Delivery Point) material distribution.

[0026] In general, the method provided in this application first improves the path planning model and adopts a hierarchical decision-making mechanism: the upper-level SAC algorithm (Soft Actor-Critic, soft policy gradient) handles global task allocation, and the lower-level PPO (Proximal Policy Optimization, shear probability optimization) algorithm focuses on real-time obstacle avoidance, forming a "strategy-tactics" two-layer optimization. At the same time, a training paradigm of virtual and real mutual feeding is adopted for the model to build a two-way data closed loop. The digital twin provides a low-cost simulation training environment, and the physical robot feeds back real data to continuously optimize the virtual model. As well as the virtual and real complementary feature alignment, and the progressive noise injection during the training process, the Sim2Real gap is significantly narrowed. In order to clearly describe a path planning model acquisition method provided in this embodiment, please refer to Figures 1-2 .

[0027] The path planning model acquisition method provided in this application is applied to the server and the robot respectively. The server is associated with the robot and a communication connection is established. The server is used to train the model, and the robot is used to apply the model and collect data for training feedback. For the path planning model acquisition method applied to the server, please refer toFigure 1 , including steps S110~S130.

[0028] Step S110: obtaining an initial path planning model, where the initial path planning model is obtained by training in a target virtual scene based on path training data.

[0029] In one embodiment, obtaining an initial path planning model includes: constructing an initial path planning model, the network structure of the initial path planning model includes a first layer and a second layer; the first layer is the input layer of the model, composed of a soft policy gradient algorithm, for receiving map data and task instructions, and processing the map data and task instructions to output a path key point set; the second layer is connected to the first layer, is the output layer of the model, composed of a shear probability optimization algorithm, for receiving a path key point set and real-time environmental data, and processing the path key point set and real-time environmental data to output control instructions, and the control instructions are used to control the movement of the robot.

[0030] In one embodiment, the difference between the initial path planning model and the path planning model described below is that the former is untrained, while the latter is a trained model. The two differ only in internal parameters; the network structure is the same. Specifically, the network structure is divided into the first layer and the second layer.

[0031] The first layer is the model's input layer, comprised of a soft actor-critic (SAC) algorithm. It receives map data and task instructions, processes them, and outputs a set of path keypoints. This layer functionally handles global task scheduling and path frameworks. Map data, specifically a global grid map with a preset resolution, such as 5cm, can be directly fed into the model. For example, when the model is deployed to the robot, a global grid map of the robot's workspace is also injected. This approach is applicable when map data is already known, or when a local map with geo-fences is set in an open environment to constrain the robot's working range. Alternatively, the robot can collect data autonomously through sensors. Map data is processed by multiple layers (e.g., three layers) of fully connected Rectified Linear Units (Rectified Linear Units) in the first layer. In this embodiment, each ReLU unit can consist of 256 nodes. Task instructions are used to control the robot's work or movement. They include the coordinates of the task target, such as a JIT delivery point on a production line, to control the robot's delivery of the correct materials, in the correct quantity, to the correct location at the correct time. In addition to the current robot's task status, task instructions can also include the status of other robots. By knowing the status of multiple robots, the first layer's output can be compatible with all robots' routes, avoiding route conflicts that could impact production. Task instructions can be processed using LSTM (Long Short-Term Memory) units in the first layer to handle temporal dependencies. In this embodiment, the LSTM unit can consist of 128 nodes. The first layer processes map data and task instructions to output a set of path keypoints. This set of path keypoints consists of several path keypoints. Each path keypoint indicates when and where the robot should arrive. All path keypoints are sequentially connected to form the global path for the robot's movement and / or operation. Each path keypoint can be separated by a preset distance, such as 1 meter. For specific scenarios within the global path, the spacing between path keypoints can be adjusted. For example, if the global path needs to pass through a narrow passage on the production line (e.g., a path with a width less than a preset distance, such as 0.4 meters), the spacing between path keypoints can be reduced, for example, by shortening the distance between keypoints to 0.5 meters. Similarly, if the global path requires the robot to perform long-distance linear motion, the key point spacing can be extended to the corresponding length, such as 5 meters. Since the functional goal of the first layer is to achieve global task allocation and multi-objective optimization, the decision frequency is relatively low, for example, it can be set to 1Hz.

[0032] The second layer, connected to the first layer, serves as the model's output and is comprised of a Proximal Policy Optimization (PPO) algorithm. Functionally, the second layer is responsible for low-level control, such as dynamic obstacle avoidance and local trajectory adjustment. It receives input from the path keypoint set output by the first layer, as well as real-time environmental data. This data can include image data (such as visible light and infrared images) collected by the robot's image acquisition device, point cloud data collected by a radar device (such as LiDAR), and motion data collection devices (such as an IMU). The second layer is used to perceive the current local environment and the location of nearby obstacles. The second layer uses a convolutional neural network (CNN) to extract obstacle features, and then uses a gated recurrent unit (GRU) to predict the dynamic trajectory, thereby generating corresponding control instructions for obstacle avoidance. Among them, the CNN network in the second layer can be composed of two layers, each layer processes the input real-time environmental data through a 3×3 convolution kernel to extract features; the GRU unit can be composed of 64 nodes. The control instructions finally generated are used to control the movement or work of the robot. Specifically, it can be adjusted according to a preset frequency (for example, 10Hz) to adjust the linear speed (for example, within the range of ±0-1.5m / s) and / or angular speed (for example, within the range of ±0.5rad / s). The second layer is used for real-time obstacle avoidance and dynamic conflict resolution. Compared with the first layer, it requires a higher frequency of decision-making, for example, it can be set to 10Hz. In addition, for the application scenario of line-side warehouses, the tilt LiDAR data can be pre-processed and decision-making can be performed to avoid blind spots on shelves and avoid collisions.

[0033] In one embodiment, the path training data is a data set for pre-training the initial path planning model. It can be generated or acquired based on historical data collection, and the acquisition method is not limited. Specifically, it may include environmental layout, dynamic obstacles, ground friction, and task sequence, etc. As for the real fine-tuning training data mentioned later, you can also refer to the examples here. The real fine-tuning training data is obtained by feature fusion based on the path training data and combined with the real path parameters. The two have the same content types. Among them, the environment layout may include the length and width of the channel in the scene, for example, channels less than 0.4m are regarded as narrow channels, channels greater than 0.5m are regarded as main channels, and channels between the two are regarded as ordinary channels, and the length of each channel (usually ≥1m). It also includes shelves for stacking materials, and the shelf height is set to a preset height, such as 1.8m (with a tolerance of ±0.1m). The environment layout can be generated based on the line warehouse CAD drawings.

[0034] Dynamic obstacles, which can be specifically other robots in the environment, such as AGV devices (Automated Guided Vehicle), or other workers in the scene, or the movement of other production devices. For AGV devices, the parameters of the robot can also be set, or can be set separately, for example, set a preset number (3-5), and each robot moves at a predetermined speed (for example, 0.3-0.8 m / s). For workers, a random number can also be set, and the speed can be set to, for example, 1.0±0.2 m / s. By setting dynamic obstacles, the random obstacles that may be encountered in the scene are simulated, thereby improving the dynamic obstacle feature recognition capability of the training model and the avoidance capability of the control.

[0035] The ground friction is to simulate the interference of the size of the ground friction in the environment on the movement ability of the robot. In general, the friction in the virtual scene is fixed, but in reality it often changes, for example, oil stains may randomly occur to reduce the friction. For this, the friction μ can be set to an initial size, for example, μ = 0.63±0.02, and an oil stain area can be randomly generated in the path periodically (for example, once every 5 minutes), and the oil stain area will reduce μ by 15%. Through this interference, the training model adjusts the ability of the robot to move, ensuring that the robot can reach the destination in time.

[0036] The task sequence is the production task in the line-side warehouse. By simulating the production plan, the robot is dispatched with JIT distribution tasks, that is, task instructions are issued, so as to control the response of the robot and train the model to process task instruction processing capability. The task instruction issuing period can be 60 seconds (matching 60 JPH), and an emergency order can be inserted with, for example, a 10% probability, thereby simulating the reality of the production line plan. The task sequence can be generated according to the tasks of the production line where the robot to be deployed is located, improving the specificity; or a random task sequence can be generated to train the model, improving the generality.

[0037] In an embodiment, the training process of the initial path planning model is the same as the process of updating the initial path planning model in step S130 hereinafter. For the training and updating process, it will be described in detail hereinafter, which is not expanded here. By pre-training the initial path planning model, the initial path planning model has preliminary path planning and real-time obstacle avoidance capability, which can be deployed in the robot for actual operation subsequently.

[0038] Step S120: issuing the initial path planning model to the robot, and acquiring the real path parameters fed back by the robot. The real path parameters are collected by the robot moving and / or working according to the path planned by the initial path planning model.

[0039] In one embodiment, the training of the initial path planning model is simulated in a virtual environment and cannot actually demonstrate the model's path planning capabilities. In addition, random interference that may occur in reality will further test the model's planning capabilities. To this end, the trained initial path planning model can be sent to the robot, which is then deployed and operated in a real environment. The robot can also obtain the actual path parameters collected during movement and / or operation according to the path planned by the initial path planning model. Correspondingly, the robot needs to be equipped with an operating device for the corresponding operating model and a collection device for collecting the actual path parameters. The operating device can be an edge computing unit with a certain computing power that can run the initial path planning model. The collection device is also the image acquisition device, radar device (for example, it can be a 16-line LiDAR installed at an inclination of ±15°), motion data acquisition device (for example, a 9-axis IMU), etc. mentioned above.

[0040] The real path parameters can correspond to the types of data contained in the path training data, that is, including environmental layout, dynamic obstacles, ground friction, and task sequence. The real path parameters can not only be fed back for retraining the initial path planning model, but are also required for running the initial path planning model. Among them, the environmental layout in the real path parameters fed back can be the actual path trajectory collected by the robot through LiDAR-SLAM mapping (±2cm accuracy), which can be used to correct the deviation of the simulation path planning algorithm (such as narrow road passability optimization). And for the specificity of the line-side warehouse, the narrow road trajectory data can be fed back first for the training of narrow road path planning.

[0041] This data can also include real-world collisions that occurred during robot operation. This data is determined using torque sensors and visual detection. This data is used to generate negative samples during subsequent updates to enhance obstacle avoidance strategies, effectively recreating collision scenarios in simulation. Furthermore, this data can be annotated with collision types, such as AGV side collisions and shelf scrapes, enabling more targeted obstacle avoidance training.

[0042] Real-world path parameters include environmental data, such as dynamic environmental changes, collected and determined using temperature and humidity sensors and tribometers. This data is then used to update simulated physical parameters (e.g., oil contamination causing a decrease in the μ value). This data collection process can include actual measurements of the floor friction coefficient per two square meters. It can also include equipment performance degradation. Motor current monitoring and vibration analysis after eight hours of continuous operation determine the robot's backlash error, which updates the simulated mechanical loss model (e.g., increased gear clearance) to facilitate robot model calibration in the virtual environment. Similarly, how real-world path parameters influence and implement the training process will be described in detail later and will not be elaborated on here.

[0043] Step S130: updating the initial path planning model in the target virtual scene according to the real path parameters to obtain a path planning model.

[0044] In one embodiment, the initial path planning model is updated in the target virtual scene according to the real path parameters to obtain the path planning model, including: updating the target virtual scene according to the real path parameters, setting at least one virtual robot in the updated target virtual scene, and deploying the initial path planning model in the virtual robot; updating the path training data according to the real path parameters to obtain real fine-tuning training data; using the virtual robot for iterative training; each round of iterative training includes: obtaining virtual task instructions and inputting them into the robot, controlling the virtual robot to respond to and complete the task corresponding to the virtual task instructions according to the control instructions, and the control instructions are obtained by the initial path planning model processing the virtual task instructions; obtaining task information generated during the virtual robot completing the task, and the task information is used to record the completion status of the virtual robot; judging whether the initial path planning model of the current round meets the preset training conditions according to the task information; if so, determining that the initial path planning model of the current round has completed training, and using the initial path planning model of the current round as the path planning model; if not, updating the initial path planning model of the current round according to the task information, and performing the next round of iterative training.

[0045] In one embodiment, the initial path planning model is updated, that is, the initial path planning model training is carried out in a target virtual scene. The use of a virtual environment can run the robot at a faster speed and achieve faster iterative training. To this end, it is first necessary to create a target virtual scene, including a virtual environment and a virtual robot. By obtaining the design data of the line-side warehouse, a virtual environment is constructed, which simulates the 3D environment of the line-side warehouse in reality, wherein more realistic dynamic parameters can be set for ground friction, such as μ=0.63±0.02. The virtual robot is a digital model constructed for the robot deployed in the simulation of the real situation using the method of the present application. The specific construction process can be modeled according to the robot design data. Unique modeling can be performed on the robot virtual model, such as setting the motor tooth gap, and by setting the error within the threshold range, such as less than 0.03°, it is ensured that the operation of the virtual robot is more simulated. Several virtual robots are set in the virtual environment, and the virtual environment containing the virtual robots is called the target virtual scene.

[0046] In this regard, the pre-training process of the initial path planning model can be carried out in the target virtual scene. The same training method is adopted for the pre-training process and the subsequent update process. The details will be described in detail later and will not be expanded here.

[0047] In one embodiment, path training data is updated according to real path parameters to obtain real fine-tuning training data, including: marking the path training data as a source domain and marking the real path parameters as a target domain; mapping features in the source domain and the target domain to a unified feature space through a feature alignment algorithm based on a generative adversarial network; performing data preprocessing operations on the data output in the feature space to obtain a preliminary data set, the data preprocessing operations including spatiotemporal alignment and anomaly filtering; and reconfiguring the path training data in a target virtual scene according to the preliminary data set to obtain real fine-tuning training data.

[0048] In one embodiment, the pre-training process of the initial path planning model requires the use of path training data. As mentioned above, path training data is generated as virtual data and does not necessarily correspond to real-world conditions, often exhibiting certain deviations from the real environment. To this end, the path training data can be updated based on real-world path parameters to obtain a more realistic dataset for simulation. This update process can label the path training data as the source domain and the real-world path parameters as the target domain. A generative adversarial network (GAN)-based feature alignment algorithm is used to map features from the source and target domains into a unified feature space, thereby aligning the simulated and real-world sensor data features, ensuring that the L2 error is less than a preset threshold, for example, less than 0.15. Data preprocessing is then performed on the data output from the feature space to obtain a preliminary dataset. This data preprocessing includes spatiotemporal alignment and anomaly filtering. For spatiotemporal alignment, the ICP algorithm (Iterative Closest Point) can be used to match the real and simulated point clouds (for example, ensuring an error of <3 cm), and the NTP (Network Time Protocol) protocol can be used to synchronize timestamps (for example, ensuring a deviation of <1 ms). For abnormal filtering, it is possible to eliminate the manual intervention data in the real path parameters, such as the triggering of the emergency stop button; or to eliminate the data collected during the period of sensor failure during the collection of the real path parameters, such as the point cloud data collected after the LiDAR lens is contaminated. The data that has undergone data preprocessing operations is marked as a preliminary data set. After the preliminary data set is reconfigured in the target virtual scene, the path training data is reconfigured, specifically including the update of the physical engine parameters of the virtual environment and the synchronization of the digital twin, so as to obtain real fine-tuning training data. For the content of the real fine-tuning training data, you can refer to the previous description of the path training data. The two contain the same type of data and will not be repeated here. The real fine-tuning training data can be used to update the initial path planning model later.

[0049] In one embodiment, in addition to GAN-based feature alignment, domain randomization can be used to achieve simulation optimization by updating path training data based on real path parameters to obtain real fine-tuning training data. This method enhances model generalization by randomizing physical parameters in the target virtual scene (for example, updating the friction coefficient by ±25% and the sensor noise by ±30%). Specifically, 200 sets of environment configurations are randomly generated in each training round, and the physical parameters in the target virtual scene are progressively adapted based on parameter space projection. However, this approach may have the following limitations: 1. Training costs increase dramatically, requiring an additional 42% of GPU computing power. 2. Adaptability to dynamic scenarios is reduced. In the human motion prediction task, the accuracy is only 73%, compared to 88% for the original GAN-based feature alignment approach. 3. Simulation accuracy is limited, with the LiDAR multipath reflection modeling error still reaching 8.5%, compared to 2.3% for the original approach.

[0050] In one embodiment, updating path training data based on real path parameters to obtain real fine-tuning training data is actually using real data to feed back virtual environment training. The triggering conditions for this feedback, namely when to obtain the real path parameters fed back by the robot, can be divided into two scenarios: immediate feedback and delayed feedback. Immediate feedback is applicable when using the real path parameters to determine if an abnormality exists in the real environment that was not encountered during virtual environment training, including but not limited to a sudden change in the ground friction coefficient, such as |Δμ| > 0.05, or an unexpected obstacle type not present in the simulation, such as a new type of AGV in the real environment. For these unencountered abnormalities, the corresponding real path parameters can be immediately collected for feedback, stimulating the initial path planning model to retrain. Delayed feedback is applicable when the model is periodically updated. This typically involves periodically (e.g., weekly) summarizing path planning deviation data and reconstructing the simulation map, thereby updating the virtual environment in the target virtual scene. Alternatively, the mechanical wear model can be periodically (e.g., monthly) updated, thereby updating the virtual robot model in the target virtual scene. The update process can also update the virtual robot based on the motor life curve.

[0051] In one embodiment, the initial path planning model is updated using real fine-tuning training data, that is, the initial path planning model is iteratively trained. Specifically, the initial path planning model is deployed in a virtual robot in the target virtual scene, and the task sequence in the real fine-tuning training data, that is, the virtual task instructions, is issued. When the virtual robot receives the virtual task instruction, it will output the corresponding path key point set according to the processing of the initial path planning model, and move and / or work according to the path point response indicated by the path key point set. During the movement and / or work process, the acquisition device in the virtual robot will continuously collect virtual real-time environment data, inject it into the initial path planning model for processing, obtain control instructions and respond to perform real-time obstacle avoidance to avoid collisions. The task is repeatedly performed in this way to train the global path planning capability and real-time obstacle avoidance capability of the initial path planning model.

[0052] After each iteration, the virtual robot obtains task information generated during the task completion process. This task information records the virtual robot's completion status, such as whether it arrived at the correct location at the correct time and with the correct cargo. It also records whether it encountered obstacles during operation, whether it avoided obstacles in a timely manner, and whether collisions occurred. Task information also records the dynamic obstacle avoidance success rate of the initial path planning model. Based on the task information, it is determined whether the initial path planning model for the current round meets the preset training conditions. Specifically, it can be determined whether the obstacle avoidance success rate is greater than a preset threshold, such as greater than or equal to 90%. If the initial path planning model for the current round meets the conditions, training is completed and the initial path planning model is marked as a path planning model. If not, the initial path planning model for the current round is updated based on the task information, and the next round of iterative training is performed.

[0053] In one embodiment, the initial path planning model for the current round is updated based on the task information, that is, SA-reinforcement learning is performed on the initial path planning model. Simulation-Assisted Reinforcement Learning (SA-RL) is a hybrid training paradigm that pre-trains the policy network in a digital twin environment and then fine-tunes it using real environment data. Pre-training the policy network in the digital twin environment means training the initial path planning model based on path training data in the target virtual scene. Fine-tuning with real environment data means updating the target virtual scene and path training data using real environment parameters. The core idea of ​​training is referred to as follows:

[0054] (1)

[0055] In the above formula, Represents the strategy optimization goal, which in this embodiment is the specific arrangement of path planning; and , which represent the expectations of the robot performing the model task in the target virtual scene and the real scene respectively; It is the weight coefficient of virtual and real data, and its value range is 0.3-0.7, which can be set arbitrarily according to needs.

[0056] It is understandable that the initial path planning model uses a hierarchical decision-making network structure to achieve decoupled optimization of global path planning and real-time obstacle avoidance. The first layer uses the SAC algorithm to achieve global task allocation and multi-objective optimization. The optimization objectives may include but are not limited to path length, energy consumption, system throughput, etc. For task information updates and reinforcement learning processes, the following formula can be used for decision making:

[0057] (2)

[0058] In the above formula, Discount Factor is used to train the robot to pay more attention to future rewards rather than just the present. Reward is the "score" given to the robot by the virtual environment. Every time the robot takes an action, the virtual environment will give it a reward score (positive or negative) based on the quality of the action. Temperature Parameter / Exploration Weight is used to balance whether the robot "strictly follows the learned optimal strategy" or "try more new and untraveled paths." Policy Entropy / Randomness: This represents the "uncertainty" or "diversity" of the robot's behavior. A high entropy policy means that the robot will make more random choices when making decisions. is the number of iterations. for The training samples of the round are also called task sequences. for Strategy stability index of the round, calculation method reference:

[0059] (3)

[0060] In the above formula, N is the action vector, which is the number of samples used to calculate the stability index. Specifically, N = 100 sets of continuous actions. for The i-th sample (or i-th time step) of the training round is also a specific task instruction.

[0061] The significance of formula (2) is that the goal of training the SAC algorithm in the second layer is to find a strategy , which maximizes the expected sum of the future discounted rewards and discounted entropy that the robot can obtain from any state under this strategy. This means that the algorithm tries to maximize the cumulative rewards while also keeping the strategy as diverse as possible (high entropy), thereby promoting more extensive exploration and more robust performance. Therefore, the training and update process, that is, the discount factor ,award and temperature parameters Etc. to update. Among them, for the discount factor The parameter selection can be set to 0.99 instead of 0.95 according to the convergence requirement of the Bellman equation in long path planning. When it is set to 0.95, 10.7% of local optimal traps will occur, resulting in a decrease in path planning accuracy.

[0062] With the help of the first layer, a set of path key points can be output to determine the global path. Among them, strategic optimization of line-edge warehouses can also be performed, such as special treatment of narrow roads: if the path contains a 0.4m channel, that is, a narrow road, the key point spacing is compressed from 1m to 0.5m, and the reward function adds a narrow road passing reward item. It can also include beat synchronization. For example, if the production beat is to match 60JPH, a 5-second buffer can be reserved, so that the path planning timeout threshold is set to 55 seconds, thereby achieving beat synchronization. Using SAC as the first-layer global path planning, the path length optimization is improved by 23% compared with the existing technology DQN.

[0063] For the second layer, the PPO algorithm is used to achieve high-frequency real-time obstacle avoidance and handle dynamic obstacles and narrow channels. The following formula can be used for decision making:

[0064] (4)

[0065] In the above formula, It is a clipping parameter (ClippingParameter), which is used to limit the size of the "step" each time the robot learns to update its strategy (that is, change its behavior). Function, with its clipping function (ClippingFunction), with clipping parameters , which is used to force the "new and old strategy probability ratio" to be limited to a preset range (for example, it cannot exceed 1.2 times of the old strategy, nor be less than 0.8 times, the 1.2 and 0.8 are composed of 1+ and 1- decided). , advantage function, measuring the state Next action Compared with the average level of the strategy, how much better is it? The above formula uses the PPO algorithm with the help of the clipping mechanism to make the policy update more stable, avoiding the problem of catastrophic performance degradation caused by excessive single updates in traditional policy gradient methods. At the same time, it is easier to implement and calculate than other trust region methods such as TRPO (Trust Region Policy Optimization, trust region policy optimization), and the success rate of sudden obstacle avoidance is 17% higher than TRPO. Therefore, the training and update process is also the clipping of parameters. Waiting for update.

[0066] Furthermore, for multi-robot obstacle avoidance in the second layer, a non-cooperative game can be used to solve it. Specifically, corresponding payoffs can be set, such as waiting payoff = -0.6T and detour payoff = -0.4E. The payoffs are used to determine the obstacle avoidance path, and the negotiation results are broadcasted between the robots.

[0067] The training and processing steps described above all involve a single round of iterative training of the initial path planning model within the target virtual scene. Whether training the initial path planning model based on path training data or real-world fine-tuning training data, the training process is identical to the one described above, differing only in the training set.

[0068] Training updates within the target virtual scene based on real-world fine-tuning training data and model execution in the real world occur simultaneously. This bidirectional training, both virtual and real, creates a closed-loop data system, enabling continuous interactive optimization between the physical and virtual worlds, forming a complete iterative chain of "perception-decision-execution-feedback." This unique characteristic is that real-world operational results (such as actual path deviations and environmental changes) feed back into model training in the target virtual scene in real time, driving the dynamic evolution of the simulation environment. Optimization strategies from the virtual world are immediately deployed to the real-world robot for optimization, forming a positive cycle.

[0069] The uplink channel, the real-to-virtual process, transmits real-world path parameters, transforming real-world operational data into optimized input for the virtual environment. This enables dynamic calibration and continuous evolution of the simulation model. The specific data feature alignment and target virtual scene updates have been described in detail previously. Please refer to the previous article for details.

[0070] For the downlink channel, that is, the policy migration from virtual to reality, it not only includes sending the initial path planning model to the robot. After obtaining the path planning model in step S130, the robot can also be sent for deployment to transmit the optimized policy network parameters, and safely deploy the optimized policy verified in the virtual environment to the physical world, while retaining the feedback interface. Improve the path planning ability of the real robot. In addition, after sending it, it is also possible to return to step S120, that is, the path planning model trained in step S130 is regarded as the initial path planning model in step S120, and the real path parameters are obtained again for feedback to realize the positive cycle of training.

[0071] In one embodiment, updating the initial path planning model in the target virtual scene also includes: determining update information of the initial path planning model; determining the noise intensity based on the update information; updating the target virtual scene using a preset noise with a numerical value corresponding to the noise intensity, and updating the initial path planning model in the updated target virtual scene.

[0072] In one embodiment, the noise intensity is determined according to the update information, including: obtaining the training round in the update information, the training round is used to characterize the number of rounds of the current iterative training of the initial path planning model; when the training round is less than a first threshold, determining the noise intensity of the current round according to the current training round; when the training round is greater than or equal to the first threshold and less than a second threshold, obtaining the training stability index of the previous round; if the training stability index is less than a preset stability threshold, updating the noise intensity of the current round according to the training stability index; if the training stability index is greater than or equal to the preset stability threshold, maintaining the noise intensity of the previous round; when the training round is greater than or equal to the second threshold, determining the action variance of the previous round, the action variance is used to characterize the degree of randomness of the movement during the robot training process; if the action variance of the previous round is greater than the preset threshold, freezing the noise intensity of subsequent rounds and keeping it unchanged; if the action variance of the previous round is less than or equal to the preset threshold, updating the noise intensity of the current round according to the action variance.

[0073] In one embodiment, during the iterative training process described in step S130 above, noise can also be injected progressively. Specifically, progressive noise injection involves updating the noise intensity of the injected noise based on update information during the iteration of the initial path planning model. The update information can be the number of iterations, for example, exponentially increasing the noise based on the number of training rounds. The types of preset noise can include, but are not limited to, laser multipath reflection, IMU temperature drift, or communication delay.

[0074] In the initial stage of training, that is, when the number of training rounds is less than the first threshold (for example, the first threshold can be set to 100), the noise intensity can be exponentially enhanced with the number of training rounds. The specific calculation process can be referred to the following formula:

[0075] (5)

[0076] In the above formula, is the number of training rounds, is the noise intensity of the tth round. For the initial noise intensity, it can be set to = 0.1. For laser noise, the injected noise can be set to randomly remove a preset number of point clouds (e.g., 12.7%) to simulate missed detections. For IMU noise, it can be set to reflect temperature drift, for example, adding a zero bias of 0.01°C / s × 0.1 = 0.001°C / s to interfere with IMU data acquisition.

[0077] When the training enters the mid-term stage, that is, the number of training rounds is greater than or equal to the first threshold and less than the second threshold (for example, it can be set to the 500th round), it is determined that the noise intensity remains within the preset range, for example ∈(0.1, 0.73). Since the model capability tends to be stable, the noise injection intensity can be adjusted based on the strategy stability index of the previous round. Update. For the strategy stability index of the tth round The calculation of can refer to formula (3). Less than a preset stability threshold (e.g. <0.3), then update the noise intensity of the current round according to the training stability index, specifically by increasing the noise intensity , for example in Relative to Increase by 5%. If the training stability index If it is greater than or equal to the preset stability threshold, the noise intensity of the previous round remains unchanged.

[0078] When the model enters the late stage of training, that is, when the training round is greater than or equal to the second threshold, the action variance of the previous round is determined. The action variance is used to characterize the degree of randomness of the robot's movement during training. If the action variance of the previous round is greater than the preset threshold (for example, greater than 0.5), the noise intensity of subsequent rounds is frozen and remains unchanged. The security policy can be enabled and the FGSM attack (Fast Gradient Sign Method) can be injected to verify the robustness of the model. The training for the injection attack will be explained in detail later and will not be repeated here. If the action variance of the previous round is less than or equal to the preset threshold, the noise intensity of the current round is updated according to the action variance.

[0079] Through progressive noise injection, the real situation can be better simulated. In addition, during the iteration process, the noise can be updated in real time according to the task information. This can be reflected in the following points. The first is the policy entropy value (Exploration), which is used to ensure that the maximum entropy strategy of the SAC algorithm in the first layer is maintained. To avoid premature convergence, the update is to calculate the maximum entropy of the first layer after each training. If the entropy value is lower than the preset threshold, the noise intensity of the next round is increased, for example, by 10%.

[0080] Sim2Real error correction. As previously mentioned, after the initial path planning model is released, it continues to be trained. After collecting the real path parameters, the L2 error between the simulated and real sensor data can be calculated. If the L2 error is greater than a preset threshold, the noise intensity can be reduced (for example, by 5%) and the initial path planning model can be retrained.

[0081] Obstacle avoidance success rate verification. As mentioned above, one of the evaluation metrics for the completion of training of the initial path planning model is whether its obstacle avoidance success rate exceeds a preset threshold. If it does not meet the threshold, the cause can be analyzed and the noise intensity of the corresponding noise type can be increased accordingly. For example, if it is caused by point cloud data, the noise intensity of the laser noise can be increased. If it meets the threshold, the initial path planning model training is considered complete and marked as a path planning model, completing the acquisition of the path planning model.

[0082] In one embodiment, the aforementioned training method is digital twin bidirectional data closed-loop training. In other alternatives, distributed knowledge sharing among multiple robots can be achieved through federated learning. Specifically, each robot can independently train a lightweight DQN network (with <1M parameters) and synchronize model parameters (with differential privacy noise ε = 0.5) via network communication every predetermined period (e.g., a 30-minute period). However, limitations of this training method include: 1. Insufficient real-time performance: global policy update delays increase the response time to sudden obstacles to 450ms, compared to 200ms in the original solution; 2. Communication dependency risk: When the network packet loss rate exceeds 5%, the system throughput decreases by 41%, while the original solution's three-level degradation mechanism only loses 8%; 3. Multi-objective optimization fails: the infeasible region of the Pareto frontier for path length and energy consumption expands to 9%, compared to 3% in the original solution.

[0083] In one embodiment, updating the initial path planning model in the target virtual scene also includes: in the target virtual scene, performing system stability training on the initial path planning model to obtain the path planning model, the system stability training includes at least one of fault recovery training, multi-robot conflict resolution training and safety protection training; the fault recovery training includes: during the training process, randomly inputting preset faults into the robots in the virtual scene, controlling the robots in the target virtual scene to respond according to the preset faults and perform recovery processing to resolve the preset faults; the multi-robot conflict resolution training includes: adding multiple robots to the target virtual scene and constructing a non-cooperative game model; the robots in the target virtual scene are used as players of the non-cooperative game model and input, and the non-cooperative game model is iterated until a preset equilibrium condition is met; the safety protection training includes: during the training process, randomly injecting disturbance attacks into the robots in the target virtual scene according to a preset attack method.

[0084] In one embodiment, during the training process, in addition to verifying the path planning capability of the initial path planning model, the robustness of the model can also be trained. Specifically, this can be done through system stability training, which includes fault recovery training, multi-robot conflict resolution training, and safety protection training.

[0085] Fault recovery training involves randomly inputting preset faults into the robot in the virtual scene during training. The robot in the target virtual scene is then controlled to respond to the preset faults and perform recovery procedures to resolve the preset faults. Preset faults can be categorized into three levels, from low to high, specifically Levels I to III. Each level includes different fault types. For ease of understanding, each level is illustrated with a specific fault: Level I could be a sensor failure, Level II could be a communication interruption, and Level III could be a policy network output anomaly. Each level corresponds to a different level of criticality and indicates the priority for handling. Furthermore, each fault has its own handling method. For example, for sensor failure, the response strategy could include multi-source data consistency verification or automatic switching to dead reckoning using the IMU and wheel speedometer. This allows the fault to be identified and handled within a preset recovery time (e.g., within 200ms). Through training, the sensor failure detection response time can be reduced to less than 100ms, achieving 100% compliance with the ISO 13849 standard for the model. In the event of communication interruptions or 5G signal loss, a locally cached semantic map of the environment, available within a preset timeframe (e.g., 10 seconds), can be activated. The model then switches to a backup model for path planning. For example, within a preset timeframe (e.g., 1 second), a lightweight replanning algorithm based on D* Lite can be used for offline navigation. Training has shown that the local path planning success rate during communication interruptions remains at 92%, a significant improvement over the 65% achieved by traditional approaches. Furthermore, using D* Lite's backup path planning approach, navigation accuracy remains at ±2cm, exceeding the industry standard of ±5cm. For policy network output anomalies—for example, when the model outputs control commands with an abnormality, such as when an action variance exceeds a preset threshold (e.g., greater than 0.5)—a hardware watchdog timer (WDT) can be activated to force a model reset, ensuring recovery within a preset timeframe (e.g., 500 milliseconds). By randomly injecting pre-set faults, the robot is forced to perform fault recovery procedures to train the model's fault handling capabilities.

[0086] For multi-robot conflict resolution training, multiple robots can be added to the target virtual scene and a non-cooperative game model constructed. The robots in the target virtual scene are input as players in the non-cooperative game model, and the model iterates until the preset equilibrium condition is met. The strategy space of the non-cooperative game model is {wait, detour, negotiation}, corresponding to the strategies that the robots can choose when encountering path conflicts. Decisions are made based on a payoff function. The payoff function can be referred to as follows:

[0087] (6)

[0088] In the above formula, is the current decision rate of return, Causes delays for tasks, is the energy cost of waiting, and is the preset negative penalty coefficient. Through iterative virtual game play between robots, the system converges to a Nash equilibrium, resolving path conflicts between multiple robots. Experimental results show that this method improves system throughput by 18% compared to traditional priority methods.

[0089] Safety protection training involves randomly injecting disturbance attacks into the robot in the target virtual scene according to a pre-set attack method. Specifically, referring to the progressive noise injection mentioned above, the pre-set attack method could be to inject FGSM attacks to train the robot's path completion rate. Furthermore, the Flow* tool can be used to verify the dynamic safety of the policy network control, thereby training the robot's reachability.

[0090] In one embodiment, as described above, a path planning model can be obtained through the above-mentioned training and updating. The path planning model can be deployed in the robot again, that is, the path planning model obtained in step S130 can be regarded as the initial planning model in step S120, and returned to step S120 for a loop. Thus, through the real data feedback in step S120 and the virtual training parameter deployment in step S130, a digital twin virtual-real mutual feeding training framework is constructed. The training in step S130 becomes a digital twin, and the physical robot feeds back the real environment data (such as ground friction coefficient, sensor noise) to the digital twin in real time, dynamically optimizes the simulation model parameters (update frequency ≥ 1Hz), and realizes continuous alignment between the virtual environment and the real scene. 80% of the algorithm training is completed through a high-fidelity simulation environment (modeling error <3%), reducing the number of real collision experiments to 5% of the traditional method, and achieving low-cost pre-training.

[0091] According to the above description, the method of the present application uses a digital twin method to train a model in a target virtual scene to update the model parameters, and obtains the real path parameters after sending the real robot to update the target virtual scene and the initial path planning model, thereby improving the training effect through a real and virtual two-way data link. The above description focuses on the training process in the virtual scene, which is applicable to the server. As for the real scene, which is applicable to the robot, the present application also proposes a path planning model acquisition method. The path planning model acquisition method applied to the robot includes steps S210~S240. In order to clearly describe the path planning model acquisition method applied to the robot provided in this embodiment, please refer to Figure 2 .

[0092] Step S210: Obtain the initial path planning model sent by the server.

[0093] In one embodiment, the initial path planning model is obtained by training the server. For details, please refer to the relevant descriptions in steps S110 to S130 above, which will not be repeated here.

[0094] Step S220: Acquire environmental data in response to the task instruction.

[0095] In one embodiment, task instructions are instructions issued by the robot's host computer to control the robot to complete a task, such as when and where the robot should arrive to obtain a specific material. The robot is equipped with data acquisition devices, including but not limited to image acquisition devices, radar devices, and motion data acquisition devices. The corresponding environmental data includes image data (e.g., visible light images, infrared images, etc.), point cloud data (used to determine the robot's environment), and motion data (e.g., the robot's speed and direction).

[0096] Step S230: Inputting the task instructions and environmental data into the initial path planning model, and moving and / or working according to the operation path planned and output by the initial path planning model.

[0097] In one embodiment, an initial path planning model is used to process task instructions and environmental data. The specific processing process has been described in detail above. Here, it is simply understood that the initial path planning model includes a first layer for global path planning and a second layer for real-time obstacle avoidance. The first layer inputs task instructions and processes them to obtain a set of path key points, which are used to indicate the global path of the robot's movement and / or operation. During movement, the robot can continuously acquire environmental data and input it into the second layer. The second layer is used to identify obstacles and, when obstacles are identified, generates control instructions to adjust the robot's linear and angular velocity to achieve obstacle avoidance.

[0098] Step S240: Acquire real path parameters during the movement and / or working process, and upload the real path parameters to the server.

[0099] In one embodiment, as described above, while the robot is moving and / or working, environmental data is continuously collected. This environmental data is data from the real environment and can be used to correct the virtual environment in the target virtual scene. That is, the environmental data collected by the real robot is included in the real path parameters. In addition, the real path parameters can also include the robot's operating data, such as IMU temperature drift, motor friction coefficient changes, battery attenuation, etc. Therefore, the real path parameters can specifically include environmental layout, dynamic obstacles, ground friction, and task sequence. By feeding back the real path parameters to the server, it can be used to assist the server in updating the initial path planning model.

[0100] Furthermore, feedback of real-world path parameters can be categorized into two scenarios: immediate feedback and delayed feedback. Immediate feedback is applicable when using real-world path parameters to identify anomalies in the real environment that were not encountered during virtual environment training. These include, but are not limited to, sudden changes in the ground friction coefficient, such as |Δμ| > 0.05, or unexpected obstacle types not present in the simulation, such as the appearance of a new type of AGV in the real environment. For unforeseen anomalies, the corresponding real-world path parameters can be immediately collected for feedback, stimulating retraining of the initial path planning model. Delayed feedback is applicable when the model needs to be updated periodically. This typically involves periodically (e.g., weekly) summarizing path planning deviation data and reconstructing the simulation map, essentially updating the virtual environment in the target virtual scene. Alternatively, it involves periodically (e.g., monthly) updating the mechanical loss model, essentially updating the virtual robot model in the target virtual scene. This update process can involve adjusting the virtual robot based on the motor life curve.

[0101] The server's detailed description of how to update the initial path planning model has been previously described and will not be repeated here. By using the physical robot to feed real-world path parameters back to the digital twin in real time, the initial path planning model parameters are dynamically optimized (with an update frequency of ≥ 1Hz), achieving continuous alignment between the virtual environment and the real scene, improving training effectiveness.

[0102] From the above description, we can see that the path planning model acquisition methods executed in the server and the robot are relatively independent. In fact, a forward loop training method can be implemented between the two. For this, please refer to Figure 3 The timing diagram of the path planning model acquisition method shown.

[0103] Step S310: The server creates a target virtual scene; in the target virtual scene, an initial path planning model is trained based on the path training data.

[0104] In one embodiment, a target virtual scene is created on a server, comprising a virtual environment and a virtual robot. The virtual environment simulates the 3D environment of a real-world lineside warehouse. More realistic dynamic parameters for ground friction, such as μ = 0.63 ± 0.02, can be set. This can be achieved by obtaining the design data for the lineside warehouse. Simultaneously, a virtual robot model is constructed based on the robot design data. This model can be customized, such as setting motor backlash and ensuring the virtual robot operates more realistically by setting the error within a threshold range, such as less than 0.03°.

[0105] Path training data can specifically include environment layout, dynamic obstacles, ground friction, and task sequences. The environment layout includes the length and width of aisles within the scene, for example, considering aisles less than 0.4m as narrow, aisles greater than 0.5m as main, and those between these as standard. The length of each aisle (typically ≥ 1m) is also included. The shelf height for stacking materials is set to a preset height, such as 1.8m (with a ±0.1m tolerance). The environment layout can be generated based on CAD drawings of line-edge warehouses.

[0106] Dynamic obstacles can specifically be other robots in the environment, such as AGVs (Automated Guided Vehicles), other workers in the scene, or the movement of other production equipment. The parameters for AGVs can be set in the same way as for the robots, or they can be set separately, such as setting a preset number (3 to 5 units), each moving at a predetermined speed (e.g., 0.3 to 0.8 m / s). For workers, a random number can also be set, and the speed can be set to, for example, 1.0 ± 0.2 m / s. By setting dynamic obstacles to simulate random obstructions that may be encountered in the scene, the training model's ability to recognize dynamic obstacle features and its control avoidance capabilities can be improved.

[0107] Ground friction refers to the interference of ground friction in the simulated environment on the robot's motion capabilities. While friction in a virtual scene is typically fixed, real-world conditions often fluctuate. For example, random oil stains may cause friction to decrease. To address this, we can set the friction force μ to an initial value, such as μ=0.63±0.02, and periodically (for example, every five minutes) randomly introduce oil stains into the path. These areas will reduce μ by 15%. This interference training model updates the robot's motion capabilities, ensuring it reaches its destination on time.

[0108] Task sequences, or production tasks within the lineside warehouse, simulate production plans and assign just-in-time delivery tasks, or task instructions, to robots. This controls the robot's response and trains the model's ability to handle task instructions. Task instructions can be issued in 60-second intervals (matching a 60-day period), with a 10% probability for emergency orders added to simulate real-world production line plans. Task sequences can be generated based on the tasks of the production line where the robot will be deployed, improving specificity. Alternatively, randomly generated task sequences can be used to train the model and enhance versatility.

[0109] The network structure of the initial path planning model is specifically divided into the first and second layers. The first layer is the input layer of the model, composed of a soft policy gradient algorithm. It is used to receive map data and task instructions, and process the map data and task instructions to output a path keypoint set. In other words, it functionally handles global task scheduling and path framework. The path keypoint set is composed of several path keypoints. Each path keypoint is used to indicate when and where the robot will arrive. All path keypoints are connected in sequence to form the global path for the robot to move and / or work. Because the functional goal of the first layer is to achieve global task allocation and multi-objective optimization, the decision frequency is relatively low, for example, it can be set to 1Hz. The second layer is connected to the first layer and is the output layer of the model. It is composed of a shear probability optimization algorithm. The second layer is functionally responsible for low-level control such as dynamic obstacle avoidance and local trajectory updates. It inputs the path keypoint set output by the first layer and the collected real-time environmental data. Real-time environmental data can include image data collected by the robot through an image acquisition device (such as visible light images, infrared images, etc.), point cloud data collected by a radar device (such as LiDAR), and the robot's motion data acquisition device (such as an IMU, Inertial Measurement Unit). The second layer is used to perceive the current local environment and the control instructions ultimately generated by the location of nearby obstacles to control the robot's movement or operation. Specifically, it can be adjusted according to a preset frequency (such as 10Hz) to adjust the linear speed (for example, within the range of ±0-1.5m / s) and / or angular velocity (for example, within the range of ±0.5rad / s). The second layer is used for real-time obstacle avoidance and dynamic conflict resolution. Compared with the first layer, it requires a higher frequency of decision-making, for example, it can be set to 10Hz. In addition, for line-side warehouse applications, the tilt LiDAR data can be pre-processed and decision-making can be performed to avoid blind spots on shelves and prevent collisions.

[0110] The initial path planning model training can be carried out in the target virtual scene. The virtual environment can be used to run the robot at a faster speed and achieve faster iterative training. The initial path planning model is deployed in the virtual robot of the target virtual scene, and the task sequence in the real fine-tuning training data, that is, the virtual task instructions, is issued. When the virtual robot receives the virtual task instruction, it will output the corresponding path key point set according to the processing of the initial path planning model, and move and / or work according to the path point response indicated by the path key point set. During the movement and / or work process, the acquisition device in the virtual robot will continuously collect virtual real-time environment data, inject it into the initial path planning model for processing, obtain control instructions and respond to perform real-time obstacle avoidance to avoid collisions. The task is repeatedly performed in this way to train the global path planning capability and real-time obstacle avoidance capability of the initial path planning model.

[0111] After each iteration, the virtual robot obtains task information generated during the task completion process. This task information records the virtual robot's performance, such as whether it arrived at the correct location at the correct time and with the correct cargo. It also records whether obstacles were encountered during operation, whether obstacles were avoided in a timely manner, and whether collisions occurred. Task information also records the dynamic obstacle avoidance success rate of the initial path planning model. Based on the task information, it is determined whether the initial path planning model for the current iteration meets the preset training conditions. Specifically, it can be determined whether the obstacle avoidance success rate is greater than a preset threshold, such as greater than or equal to 90%. If so, the initial path planning model for the current iteration is considered trained and training is complete. If not, the initial path planning model for the current iteration is updated based on the task information, and the next iteration of training is performed.

[0112] Step S320: Send the initial path planning model to the robot.

[0113] In one embodiment, the server and the robot can be connected via wired and / or wireless communication technologies, wherein the wireless communication technologies may include but are not limited to: Global System for Mobile Communication (GSM), Enhanced Data GSMEnvironment (EDGE), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE 802.11n), Voice over Internet protocol (VoIP), Worldwide Interoperability for Microwave Access (WIFI), and / or IEEE 802.11n. Access, Wi-Max), other protocols for email, instant messaging and short messaging, and any other suitable communication protocols, even those that are not yet developed.

[0114] The server sends the trained initial path planning model from step S310 to the robot for deployment. The robot is equipped with a corresponding operating device for running the model and a collection device for collecting actual path parameters. The operating device can be an edge computing unit with sufficient computing power to run the initial path planning model. The collection device can also be the image acquisition device, radar device (for example, a 16-axis LiDAR mounted at a ±15° inclination), or motion data acquisition device (for example, a 9-axis IMU), as described above.

[0115] Step S330: The robot responds to the task instruction and obtains environmental data; inputs the task instruction and environmental data into the initial path planning model, and moves and / or works according to the operation path planned and output by the initial path planning model.

[0116] Step S340: The robot obtains real path parameters during movement and / or work, and uploads the real path parameters to the server.

[0117] In one embodiment, a task instruction is a command issued by the robot's host computer that controls the robot to complete a task, i.e., it specifies when and where the robot should arrive to obtain a specific material. The robot responds to the task instruction by using a data collection device to acquire environmental data. The task instruction and environmental data are then input into an initial path planning model. The initial path planning model processes the data to output a set of global path key points and control instructions. The former is used to control the robot's operation and / or movement, while the latter is used to control the robot's real-time obstacle avoidance.

[0118] In one embodiment, under normal circumstances, if the model is properly trained and there are no unexpected circumstances in the real environment, the robot will maintain a relatively stable state of operation. The robot can continuously obtain the real path parameters collected during operation. The real path parameters can correspond to the types of data contained in the path training data, that is, including environmental layout, dynamic obstacles, ground friction and task sequence, etc. The real path parameters can not only be fed back for retraining the initial path planning model, but are also needed for running the initial path planning model. Among them, the environmental layout in the real path parameters fed back can be the actual path trajectory collected by the robot through LiDAR-SLAM mapping (±2cm accuracy), which is used to correct the deviation of the simulation path planning algorithm (such as narrow road passability optimization). And for the specificity of the line-side warehouse, the narrow road trajectory data can be fed back first for training narrow road path planning.

[0119] Real collision events that occur during the robot running process can also be included, and the data is determined through torque sensors + visual detection. It is used to generate negative sample reinforcement obstacle avoidance strategies in subsequent updating processes, that is, to reproduce collision scenarios in simulation. Moreover, the data can be labeled with collision types, such as AGV side collision / shelf scratching, etc., so as to more specifically achieve obstacle avoidance training.

[0120] The real path parameters include environmental data, such as environmental dynamic changes, which are determined by collecting through temperature and humidity sensors + friction gauges. Thus, the simulation physical parameters (such as oil stains causing μ value to decrease) are updated, and the collection process can be the real measurement of the friction coefficient of an area of two square meters. Device performance degradation can also be included, and the robot backlash error is determined by collecting motor current monitoring + vibration analysis after continuous running for 8 hours, so as to update the mechanical loss model (such as gear clearance increase) in simulation, facilitating robot model calibration in a virtual environment.

[0121] In an embodiment, the subsequent steps are actually a cyclic process, and the model is virtually trained in steps S310 and S350. The trained model is issued to the real robot for running in step S320; and real data is collected by actual running in steps S330 and S340, and fed back to update the model in step S350. Through the alternative updating of virtuality and reality, the training effect of the model is improved. In order to avoid the model training from entering an infinite loop, causing overfitting and other problems, it can be set when the robot performs real path parameter feedback. Specifically, it can be divided into two cases: immediate feedback and delayed feedback. For immediate feedback, it is suitable for determining that there is an abnormal situation in the real environment that has not been encountered in the virtual environment training process according to the real path parameters, including but not limited to sudden changes in ground friction coefficient, such as |Δμ|>0.05; or sudden obstacle types that do not appear in simulation, such as new AGVs appearing in the real environment. For the abnormal situation that has not been encountered, the corresponding real path parameters can be collected for feedback immediately to stimulate the initial path planning model to be trained again. Delayed feedback is suitable for periodically updating the model, which can usually be periodic (for example, every week) to summarize path planning deviation data and reconstruct the simulation map, that is, to update the virtual environment in the target virtual scene. Or periodically (for example, every month) update the mechanical loss model, that is, update the virtual robot model in the target virtual scene. The updating process can update the virtual robot based on the motor life curve.

[0122] Step S350: The server updates the initial path planning model in the target virtual scene according to the real path parameters, and obtains the path planning model.

[0123] In one embodiment, the pre-training process of the initial path planning model requires the use of path training data. As mentioned above, path training data is generated as virtual data and does not necessarily correspond to real-world conditions, often exhibiting certain deviations from the real environment. To this end, the path training data can be updated based on real-world path parameters to obtain a more realistic dataset for simulation. This update process can label the path training data as the source domain and the real-world path parameters as the target domain. A generative adversarial network (GAN)-based feature alignment algorithm is used to map features from the source and target domains into a unified feature space, thereby aligning the simulated and real-world sensor data features, ensuring that the L2 error is less than a preset threshold, for example, less than 0.15. Data preprocessing is then performed on the data output from the feature space to obtain a preliminary dataset. This data preprocessing includes spatiotemporal alignment and anomaly filtering. For spatiotemporal alignment, the ICP algorithm (Iterative Closest Point) can be used to match the real and simulated point clouds (for example, ensuring an error of <3 cm), and the NTP (Network Time Protocol) protocol can be used to synchronize timestamps (for example, ensuring a deviation of <1 ms). For abnormal filtering, it is possible to eliminate the manual intervention data in the real path parameters, such as the triggering of the emergency stop button; or to eliminate the data collected during the period of sensor failure during the collection of the real path parameters, such as the point cloud data collected after the LiDAR lens is contaminated. The data after the data preprocessing operation is marked as a preliminary data set. After the preliminary data set is reconfigured in the target virtual scene, it specifically includes the update of the physical engine parameters of the virtual environment and the synchronization of the digital twin, and obtains the real fine-tuning training data. The initial path planning model is trained with the real fine-tuning training data. The training process is the same as the training of the initial path planning model with the path training data in the previous article, so it will not be repeated here. The initial path planning model updated with the real fine-tuning training data is marked as the path planning model.

[0124] Step S360: The server sends the path planning model to the robot and returns to step S330.

[0125] In one implementation, the path planning model can be distributed to the robot in the same manner as the initial path planning model. This means that any problems encountered during robot operation are addressed, and the model parameters are updated before being distributed to the robot to maintain optimal performance. 80% of algorithm training is completed in a high-fidelity simulation environment (modeling error <3%), reducing the number of real-world collision experiments to 5% of traditional methods. This enables continuous interactive optimization between the physical and virtual worlds, forming a complete iterative chain of "perception-decision-execution-feedback." Physical world operational results (such as actual path deviations and environmental changes) are fed back to the virtual model in real time, driving the dynamic evolution of the simulation environment. Simultaneously, optimization strategies from the virtual world are instantly deployed to the physical entity, forming a positive cycle.

[0126] Therefore, this application innovates the path planning model architecture, achieving decision optimization through a hierarchical reinforcement learning architecture (SAC + PPO combination). Experimental data shows that the path planning model based on this hierarchical architecture has the following improvements: 1. Global path planning response time is reduced to 200ms, compared to 1.2 seconds for the traditional DWA algorithm; 2. Dynamic obstacle avoidance decision frequency is increased to 10Hz, compared to only 3Hz for the traditional APF method; 3. In a scenario with 15 moving obstacles, it maintains a real-time response time of <300ms, an 83% improvement over the existing dynamic weighted A* algorithm, significantly improving the response speed of global path planning and real-time obstacle avoidance.

[0127] The virtual-reality mutual training framework achieves a breakthrough in data efficiency through a two-way data closed loop. The sample utilization rate in the target virtual scene is increased by 42% compared to the pure imitation learning method. In addition, the path planning model obtained after pre-training and updating reduces the number of collisions required for fine-tuning in the real environment to 50 times, which greatly improves efficiency compared to the traditional DQN method that requires 1000 times. In addition, in the face of robot path conflict problems, the path planning model based on this application can achieve a data reuse rate of 78% during multi-robot collaborative training.

[0128] The virtual-reality mutual training framework requires obtaining the real path parameters collected after the robot deploys the initial path planning model to update the initial path planning model. Before the update, the real path parameters will be used to perform GAN-based feature alignment on the path training data, and progressive noise injection will be adopted during the training process to better simulate the target virtual scene. After adopting the solution provided in this application, the error of lidar multipath reflection modeling was reduced from 12.7% to 2.3%, the simulation accuracy of the dynamic change of the ground friction coefficient reached 97.5% (error <±2%), and the prediction accuracy of the human-in-the-loop scene was improved to 88% (traditional methods are only 60%).

[0129] Path planning and real-time obstacle avoidance are achieved through the design of a hierarchical reward function: the infeasible area of ​​the Pareto frontier of path length and energy consumption is reduced to 3%, compared to the current 12%. The overall throughput of the entire line-side warehouse system is increased by 35%, and the multi-robot conflict rate is reduced to 8% compared to the traditional method's 25%.

[0130] In addition, the training process for this application also includes system stability training, which enhances system robustness. Specifically, the three-level fault recovery mechanism of fault recovery training has achieved: a 92% local path planning success rate during communication interruptions (compared to 65% for traditional solutions); a sensor failure detection response time of <100ms (100% compliance with the ISO 13849 standard); and navigation accuracy in degraded mode of up to ±2cm (better than the industry standard of ±5cm).

[0131] Figure 4 FIG1 shows an internal structure diagram of a computer device in an embodiment. The computer device can be a terminal or a server. Figure 4 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement the path planning model acquisition method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may implement the path planning model acquisition method. It will be understood by those skilled in the art that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] In one embodiment, the present application further proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in any of the aforementioned embodiments.

[0133] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0134] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A path planning model acquisition method, characterized in that: applied to a server, said server being associated with the robot; The method comprises the following steps: Acquire an initial path planning model, where the initial path planning model is obtained by training in a target virtual scene based on path training data; Sending the initial path planning model to the robot and obtaining real path parameters fed back by the robot, where the real path parameters are collected when the robot moves and / or works along the path planned by the initial path planning model; updating the initial path planning model in the target virtual scene according to the real path parameters to obtain a path planning model; The updating of the initial path planning model in the target virtual scene includes: determining update information of the initial path planning model; determining noise intensity according to the update information; updating the target virtual scene using preset noise having a value corresponding to the noise intensity, and updating the initial path planning model in the updated target virtual scene; The determining of the noise intensity according to the update information includes: obtaining a training round in the update information, the training round being used to characterize the number of rounds of the current iterative training of the initial path planning model; when the training round is less than a first threshold, determining the noise intensity of the current round according to the current training round; when the training round is greater than or equal to the first threshold and less than a second threshold, obtaining a training stability index of the previous round; if the training stability index is less than a preset stability threshold, updating the noise intensity of the current round according to the training stability index; if the training stability index is greater than or equal to the preset stability threshold, maintaining the noise intensity of the previous round; when the training round is greater than or equal to the second threshold, determining the motion variance of the previous round, the motion variance being used to characterize the degree of randomness of the movement during the robot training process; if the motion variance of the previous round is greater than a preset threshold, freezing the noise intensity of subsequent rounds and keeping it unchanged; if the motion variance of the previous round is less than or equal to the preset threshold, updating the noise intensity of the current round according to the motion variance.

2. The path planning model acquisition method according to claim 1, wherein: The obtaining of the initial path planning model includes: Constructing the initial path planning model, wherein the network structure of the initial path planning model includes a first layer and a second layer; The first layer is the input layer of the model, which is composed of a soft policy gradient algorithm and is used to receive map data and task instructions, and process the map data and the task instructions to output a set of path key points; The second layer is connected to the first layer, is the output layer of the model, and is composed of a cutting probability optimization algorithm. It is used to receive the path key point set and real-time environmental data, and process the path key point set and the real-time environmental data to output control instructions, which are used to control the movement of the robot.

3. The path planning model acquisition method according to claim 1, wherein: The updating of the initial path planning model in the target virtual scene according to the real path parameters to obtain a path planning model includes: updating the target virtual scene according to the real path parameters, setting at least one virtual robot in the updated target virtual scene, and deploying the initial path planning model in the virtual robot; updating the path training data according to the real path parameters to obtain real fine-tuning training data; Iterative training is performed using the virtual robot; each round of iterative training includes: obtaining a virtual task instruction and inputting it into the robot, controlling the virtual robot to respond to and complete a task corresponding to the virtual task instruction according to the control instruction, wherein the control instruction is obtained by processing the virtual task instruction by the initial path planning model; Acquire task information generated by the virtual robot during the task completion process, wherein the task information is used to record the completion status of the virtual robot; Determine whether the initial path planning model of the current round meets the preset training conditions according to the task information; If the conditions are met, it is determined that the training of the initial path planning model of the current round is completed, and the initial path planning model of the current round is used as the path planning model; If not, the initial path planning model of the current round is updated according to the task information, and the iterative training of the next round is performed.

4. The path planning model acquisition method according to claim 3, wherein: The updating of the path training data according to the real path parameters to obtain real fine-tuning training data includes: Marking the path training data as a source domain and marking the true path parameters as a target domain; Mapping the features in the source domain and the target domain to a unified feature space through a feature alignment algorithm based on a generative adversarial network; Performing data preprocessing operations on the data output from the feature space to obtain a preliminary data set, wherein the data preprocessing operations include spatiotemporal alignment and anomaly filtering; In the target virtual scene, the path training data is reconfigured according to the preliminary data set to obtain the real fine-tuning training data.

5. The path planning model acquisition method according to claim 1, wherein: The updating of the initial path planning model in the target virtual scene to obtain a path planning model includes: In the target virtual scene, performing system stability training on the initial path planning model to obtain the path planning model, wherein the system stability training includes at least one of fault recovery training, multi-robot conflict resolution training, and safety protection training; The fault recovery training includes: during the training process, randomly inputting a preset fault into the robot in the virtual scene, controlling the robot in the target virtual scene to respond to the preset fault and perform recovery processing to resolve the preset fault; The multi-robot conflict resolution training includes: adding multiple robots to the target virtual scene and constructing a non-cooperative game model; using the robots in the target virtual scene as players of the non-cooperative game model and inputting them, and iterating the non-cooperative game model until a preset equilibrium condition is met; The safety protection training includes: during the training process, randomly injecting disturbance attacks into the robot in the target virtual scene according to a preset attack method.

6. A computer device, characterized in that: including processor and memory; The processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images