Model training method and device, scheduling method and device and storage medium

Abstract

The invention relates to a model training method and device and a scheduling method and device, and relates to the technical field of engineering machinery. The model training method comprises the steps that state information is determined based on related information of a loader and first related information of an operation scene, and the related information of the loader comprises the working state type of the loader; based on the state information, action information is determined through a reinforcement learning model, and the action information comprises scheduling information of the loader; based on the action information, utilizing a reinforcement learning model to determine a reward value of at least one reward function corresponding to the action information and updated state information; and based on the reward value and the updated state information, training a reinforcement learning model for dispatching the loader. According to the technical scheme, the operation efficiency of the loader can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of engineering machinery, and in particular to a model training method, a model training device, a scheduling method, a scheduling device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Loaders are widely used in many industries, including mines, steel mills, and infrastructure. During the operation of a loader, it is necessary to carry out systematic planning to maximize the efficiency of the loader.

[0003] In related technologies, A* algorithm, Dijkstra algorithm, particle swarm algorithm, genetic algorithm and other optimization algorithms based on swarm intelligence can be used to schedule loaders in complex environments. Summary of the Invention

[0004] The inventors of the present disclosure have discovered the following problems in the above-mentioned related technologies: during the operation of loaders, conflicts in driving routes and long waiting times often occur between loaders, resulting in low operating efficiency of the loaders.

[0005] In view of this, the present disclosure proposes a model training method that can reasonably plan the path of the loader and effectively improve the operating efficiency of the loader.

[0006] According to a first aspect of the present disclosure, a model training method is provided, including: determining state information based on relevant information of a loader and first relevant information of an operation scene, the relevant information of the loader including the working state type of the loader; determining action information based on the state information using a reinforcement learning model, the action information including scheduling information of the loader; determining a reward value of at least one reward function corresponding to the action information and updated state information using a reinforcement learning model based on the action information; and training the reinforcement learning model based on the reward value and the updated state information for scheduling the loader.

[0007] In some embodiments, based on the relevant information of the loader and the first relevant information of the working scene, the status information is determined, including: determining the working status type according to the characteristic information of the loader, the characteristic information including at least one of the operating parameters and hydraulic system parameters of the loader.

[0008] In some embodiments, based on the state information, action information is determined using a reinforcement learning model, including: adding noise to the state information; determining the action information based on the state information after adding the noise; based on the action information, using the reinforcement learning model, determining the reward value of at least one reward function corresponding to the action information and the updated state information, including: adding noise to the action information; determining the reward value based on the action information after adding the noise.

[0009] In some embodiments, based on the relevant information of the loader and the first relevant information of the operation scene, the status information is determined, including: obtaining the second relevant information of the operation scene; aligning the second relevant information using a timestamp to obtain the first relevant information.

[0010] In some embodiments, based on the state information, the action information is determined using a reinforcement learning model, including: verifying the action information according to whether the action information matches the working state type of the loader; and determining the action information based on the verification result.

[0011] In some embodiments, determining the working state type according to the characteristic information of the loader includes: determining the working state type by using a clustering method according to the characteristic information.

[0012] In some embodiments, the loader includes a first loader and a second loader, the first work task of the first loader and the second work task of the second loader are associated, the first reward value of the first loader and the second reward value of the second loader are determined based on the completion status of the first work task and the second work task, and the completion status includes at least one of the efficiency of the operation completion, the safety of the operation completion, and the cooperation status between the first loader and the second loader.

[0013] In some embodiments, the complexity of the first work task is higher than the complexity of the second work task, and the first loader guides the second loader to perform the second work task.

[0014] In some embodiments, at least one reward function includes an efficiency reward function, which is used to characterize the efficiency of job completion. The value of the efficiency reward function of the first loader is greater than the value of the efficiency reward function of the second loader. In response to the higher the working efficiency of the loader, the larger the value of the efficiency reward function.

[0015] In some embodiments, at least one reward function includes a safety reward function for characterizing the safety of job completion, and the safety reward function is determined based on whether the first loader and the second loader collide with obstacles in the job scene, and / or whether the distance between the first loader and the second loader is less than a distance threshold.

[0016] In some embodiments, at least one reward function includes a collaborative reward function, which is used to characterize the collaborative situation of the first loader and the second loader. The collaborative reward function is determined based on at least one of the following: whether the difference between the start time of the first loader and the start time of the second loader is less than a threshold; whether the position of the second loader matches the first work task and the second work task; in response to the presence of a faulty loader in the first loader and the second loader, whether other loaders in the first loader and the second loader take on the work task of the faulty loader.

[0017] In some embodiments, the value of the collaboration reward function is determined according to whether the first loader and the second loader share information related to the first work task and information related to the second work task.

[0018] In some embodiments, at least one reward function includes multiple reward functions, and training a reinforcement learning model based on the reward value and updated state information includes: determining a value function corresponding to each of the multiple reward functions based on the multiple reward functions; and training the reinforcement learning model based on the value function.

[0019] In some embodiments, the first relevant information of the work scene includes at least one of map information, obstacle information, and terrain information of the work scene.

[0020] According to a second aspect of the present disclosure, a scheduling method is provided, comprising: scheduling the loader based on relevant information of the loader and first relevant information of the operation scene using a reinforcement learning model, wherein the reinforcement learning model is obtained by the model training method in any of the above embodiments.

[0021] According to a third aspect of the present disclosure, a model training device is provided, including: a state information determination module, configured to determine state information based on relevant information of the loader and first relevant information of the working scene, the relevant information of the loader including the working state type of the loader; an action information determination module, configured to determine action information based on the state information using a reinforcement learning model, the action information including scheduling information of the loader; a third determination module, configured to determine a reward value of at least one reward function corresponding to the action information and updated state information based on the action information using a reinforcement learning model; a training module, configured to train the reinforcement learning model based on the reward value and the updated state information for scheduling the loader.

[0022] According to a fourth aspect of the present disclosure, a scheduling device is provided, comprising: a scheduling module, configured to schedule the loader based on relevant information of the loader and first relevant information of the operation scene, using a reinforcement learning model, and the reinforcement learning model is obtained by the model training method in any of the above embodiments.

[0023] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the model training method or scheduling method in any one of the above-mentioned embodiments based on instructions stored in the memory.

[0024] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the model training method or scheduling method in any of the above embodiments is implemented.

[0025] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, cause the processor to execute the model training method or scheduling method in any one of the above embodiments.

[0026] In the above embodiment, the reinforcement learning model is used to plan a reasonable path for the loader by considering various factors in the operation scene, thereby effectively improving the operation efficiency of the loader. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0028] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0029] Figure 1 Flowcharts showing some embodiments of the model training method disclosed herein;

[0030] Figure 2 A flowchart illustrating determining map information of a work scene in some embodiments of the model training method disclosed herein;

[0031] Figure 3 A schematic diagram illustrating a reinforcement learning model in some embodiments of the model training method of the present disclosure;

[0032] Figure 4 A flowchart illustrating some embodiments of the scheduling method of the present disclosure;

[0033] Figure 5 A block diagram illustrating some embodiments of the model training apparatus of the present disclosure;

[0034] Figure 6 A block diagram illustrating some embodiments of the scheduling device of the present disclosure;

[0035] Figure 7 A block diagram showing other embodiments of the electronic device disclosed herein;

[0036] Figure 8 A block diagram illustrating some further embodiments of the electronic device disclosed herein. DETAILED DESCRIPTION

[0037] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0038] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0040] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0041] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0042] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] As mentioned above, while some research on scheduling methods has made some progress, limitations remain. For example, algorithms like the A* algorithm and the Dijkstra algorithm are inefficient when dealing with multi-machine collaboration and path planning in complex environments, and they struggle to guarantee a globally optimal solution. Optimization algorithms based on swarm intelligence, such as particle swarm optimization and genetic algorithms, while offering certain advantages, are prone to falling into local optima and suffer from slow convergence.

[0044] On the other hand, the above scheduling method is difficult to quickly plan a reasonable collaborative operation mode for multiple loaders in complex and changeable operation scenarios, and cannot fully utilize the maximum performance of the loaders, resulting in low operation efficiency of the loaders.

[0045] In view of this, the present disclosure proposes a model training method, a model training device, a scheduling method, a scheduling device, a computer-readable storage medium and a computer program product, which can determine the scheduling method of the loader in the operation scenario based on the reinforcement learning model, thereby effectively improving the operation efficiency of the loader.

[0046] Figure 1 A flowchart illustrating some embodiments of the model training method of the present disclosure.

[0047] like Figure 1 As shown, in step 110, state information is determined based on relevant information of the loader and first relevant information of the operation scene, where the relevant information of the loader includes the working state type of the loader.

[0048] For example, the loader's state information, as a key dimension of the reinforcement learning model, can include information such as the loader's operating state type, location coordinates, and driving speed. For example, the loader's operating state type can be configured as shoveling, transporting, unloading, cleaning, stacking, piling, shaking, and other types.

[0049] For example, in complex operating scenarios such as mines and construction sites, loaders can obtain their own position coordinates through high-precision positioning systems and measure their driving speed through on-board sensors.

[0050] In this way, the real-time physical state of the loader can be intuitively reflected through the position coordinates and driving speed, thereby improving the efficiency and accuracy of path planning for the loader.

[0051] For example, the working state type of the loader can be determined by the following embodiments.

[0052] In some embodiments, the working state type is determined based on characteristic information of the loader, where the characteristic information includes at least one of an operating parameter and a hydraulic system parameter of the loader.

[0053] In some embodiments, the working status type is determined using a clustering method based on feature information.

[0054] For example, characteristic information such as loader operating parameters (such as bucket angle θ, engine speed w), hydraulic system parameters (such as cylinder stroke s, pressure p), and time series parameters (such as position coordinates (x, y, z), velocity v, acceleration a), etc., can be used as input feature vectors of the classification model to determine the loader's operating state type. For example, the classification model can be configured based on a clustering method.

[0055] For example, based on empirical data, the action information corresponding to core work status types such as shoveling and transportation can be split into 12 trajectories.

[0056] In this way, by further refining the description of the working state type, we can better cover the subdivision of the action information corresponding to various working state types, and retain a certain amount of redundancy, providing a richer basis for subsequent path planning and decision-making.

[0057] The status information of the loader in step 110 also includes first relevant information about the operation scene. In some embodiments, the first relevant information about the operation scene includes at least one of map information, obstacle information, and terrain information of the operation scene.

[0058] In some embodiments, second relevant information of the operation scene is obtained; the second relevant information is aligned using the timestamp to obtain the first relevant information.

[0059] For example, map information of the work scene can be obtained by a human operator using a loader. Key information such as the location and shape of obstacles, terrain information, and landform information in the work scene can be acquired using multi-source sensors such as lidar and cameras. After determining the map information, obstacle information, and landform information, they can be combined to obtain secondary relevant information about the work scene.

[0060] For example, during the data collection phase of path planning, while collecting obstacle and terrain information, it is also possible to simultaneously introduce map information of the work scene to obtain secondary relevant information about the work scene. Furthermore, the third perspective of the simulation environment (simulated Eagle Eye system) can be used to achieve a comprehensive and multi-level perception of the work scene.

[0061] For example, when each sensor collects the second relevant information, an accurate timestamp can be added to the second relevant information. Furthermore, during the transmission and processing of the second relevant information, the second relevant information from different sources can be aligned based on the timestamp to obtain the first relevant information.

[0062] In this way, the consistency of the second relevant information in the time dimension can be ensured, the error caused by the inconsistent arrival time of the second relevant information can be reduced, the problem of communication delay in the actual work site can be alleviated, and the accuracy of the loader's path planning and collaborative work decision-making can be improved.

[0063] When the state information is determined by the above embodiment, then, Figure 1 In step 130 , the action information of the loader is determined based on the reinforcement learning model.

[0064] In step 130 , based on the state information, action information is determined using a reinforcement learning model, where the action information includes scheduling information of the loader.

[0065] For example, the scheduling information may be at least one of a driving direction, a driving speed, and a work operation corresponding to the work status type.

[0066] The driving direction can be actions such as forward, backward, left turn, right turn, etc., and the steering angle range can be configured. For example, the angle of each left or right turn can be 10°, 20°, etc.

[0067] In this way, the discretization and executableness of actions can be ensured, thereby improving the accuracy of scheduling.

[0068] With respect to the driving direction of the loader, an action verification mechanism may be established through the following embodiments to adjust the action information of the loader.

[0069] In some embodiments, the action information is verified based on whether the action information matches the working state type of the loader; and the action information is determined based on the verification result.

[0070] For example, before a loader performs a turn, the system simulates and verifies the impending turn based on the loader's current position, speed, surrounding obstacle distribution, and other loader status information. If the loader is in the shoveling or unloading state and the loader is turning, the verification result will be abnormal. If the loader is in the transport state and the loader has not turned for an extended period, the verification result will also be abnormal.

[0071] In this way, by establishing an action verification mechanism, it is possible to avoid operational abnormalities caused by improper driving directions output by the reinforcement learning model.

[0072] For example, dispatch information also includes travel speed, which can be divided into different speed gears based on the loader's performance. The low speed gear can be 1.5m / s, suitable for shoveling operations or short-distance movement, and the high speed gear can be 3m / s, used for long-distance transportation.

[0073] For example, when a loader executes motion information, it also needs to decouple travel direction from travel speed. The loader's motion control can be decomposed into lateral control and longitudinal control. Lateral control is related to travel direction (such as steering angle or heading angle), while longitudinal control is related to travel speed.

[0074] In this way, the loader can turn at the maximum steering angle and minimum steering angle Turn on the spot between the vehicles, thereby improving the accuracy of scheduling.

[0075] For example, the scheduling information also includes the work operation corresponding to the work status type. The work operation can be a core work type of the loader, such as shoveling and unloading. When the loader performs the work operation, the conditions and effects of the work operation execution also need to be clarified.

[0076] For example, when a loader performs a shoveling action, the loader's position condition, bucket angle condition, and equipment status condition can be configured.

[0077] For example, the loader needs to be parked at an appropriate distance from the material pile, generally about 1-2 meters away, to ensure the bucket can reach the material and avoid colliding with the pile or other obstacles due to proximity. Before the loader begins loading, the bucket can be adjusted to the appropriate angle, typically with the bottom of the bucket at a 3-5 degree angle to the ground. This allows the bucket to penetrate the material more easily, reduces resistance, and prevents material from sliding off the front of the bucket. Equipment status conditions refer to the loader's engine being in normal operation to ensure sufficient power for the bucket to lift, lower, and tilt.

[0078] When the action information of the loader is determined through the above embodiment, the reinforcement learning model can update the state information and calculate the reward value of the action information through the following embodiment.

[0079] In step 150 , based on the action information, a reinforcement learning model is used to determine a reward value of at least one reward function corresponding to the action information and updated state information.

[0080] In some embodiments, the loader includes a first loader and a second loader, the first work task of the first loader and the second work task of the second loader are associated, the first reward value of the first loader and the second reward value of the second loader are determined based on the completion status of the first work task and the second work task, and the completion status includes at least one of the efficiency of the operation completion, the safety of the operation completion, and the cooperation status between the first loader and the second loader.

[0081] In some embodiments, the complexity of the first work task is higher than the complexity of the second work task, and the first loader guides the second loader to perform the second work task.

[0082] For example, the first and second loaders can form a master-slave loader fleet. The master loader directs the workflow, while the slave loaders collaborate according to established rules. This master-slave relationship facilitates the proper allocation of tasks based on the performance and characteristics of the different loaders. The master loader can handle critical and complex tasks, while the slave loaders can assist with simpler tasks.

[0083] In this way, not only can chaotic and disorderly operations be avoided to improve the collaboration and efficiency of operations, but also optimal allocation of resources can be achieved to give full play to the advantages of each loader.

[0084] In some embodiments, at least one reward function includes an efficiency reward function, which is used to characterize the efficiency of job completion. The value of the efficiency reward function of the first loader is greater than the value of the efficiency reward function of the second loader. In response to the higher the working efficiency of the loader, the larger the value of the efficiency reward function.

[0085] For example, an efficiency reward function can be used to give positive rewards to loaders that quickly complete shoveling, transportation, unloading and other operation processes, encouraging loaders to optimize their operation paths and reduce operation time.

[0086] Furthermore, the efficiency reward function can be further divided into task process optimization rewards and equipment utilization improvement rewards.

[0087] For example, in the material handling area, if the path planned by the first loader enables the second loader to quickly obtain materials while reducing waiting time, then for every 10% reduction in the time to complete a handling task, the first loader can obtain 5 bonus points and the second loader can obtain 3 bonus points as efficiency rewards.

[0088] As another example, if it originally takes 10 minutes to complete a transport task, then in response to the transport task time being shortened to 9 minutes, the first loader can obtain 5 bonus points and the second loader can obtain 3 bonus points as efficiency rewards.

[0089] In this way, the first loader and the second loader can be encouraged to optimize the operation process and shorten the total operation time.

[0090] In some embodiments, at least one reward function includes a safety reward function for characterizing the safety of job completion, and the safety reward function is determined based on whether the first loader and the second loader collide with obstacles in the job scene, and / or whether the distance between the first loader and the second loader is less than a distance threshold.

[0091] For example, you can configure the operating sequence and operating requirements for the first and second loaders within a specific area. If the second loader, under the guidance of the first loader, completes the second task according to the operating process, then for each successful completion of the standard operating procedure, the first loader and the second loader will receive 3 bonus points and 2 bonus points respectively as safety rewards. If any violation occurs, 5 bonus points will be deducted from each of the first and second loaders.

[0092] In this way, a safety reward function can be used to give safety rewards to loaders that have not collided with obstacles or other loaders, so as to guide the loaders to maintain a safe distance during operation and ensure operation safety.

[0093] In some embodiments, at least one reward function includes a collaborative reward function, which is used to characterize the collaboration between the first loader and the second loader. The collaborative reward function is determined based on at least one of the following: whether the difference between the start time of the first loader and the start time of the second loader is less than a threshold; whether the position of the second loader matches the second work task; in response to the presence of a faulty loader between the first loader and the second loader, whether other loaders other than the faulty loader take on the work task of the faulty loader.

[0094] For example, a cooperation reward function may be used to give a cooperation reward to the first loader and the second loader that cooperate in performing work according to the master-slave queue to avoid conflicts and waiting.

[0095] In this way, the efficiency and smoothness of multi-machine collaborative operations can be improved.

[0096] For example, the collaborative reward function may include rewards for synchronized equipment start and stop, rewards for smooth job transitions, and rewards for collaborative fault response.

[0097] For example, when the first loader starts or stops the operation, if the second loader can make a consistent response within a specified short time interval (such as 3 seconds), then the first loader and the second loader can each obtain 3 reward points as a reward for equipment start and stop synchronization.

[0098] In this way, the increase in waiting time caused by asynchronous start and stop of equipment can be effectively avoided, and the overall operation efficiency can be improved.

[0099] For example, in the material transfer link, when the first loader is unloading, if the position of the second loader matches the material receiving task, and the second loader does not spill materials or wait for a long time during the material handover process, then for each successful handover, the first loader and the second loader can each get 4 bonus points as a reward for smooth operation connection.

[0100] This helps reduce material waste and improve material transfer efficiency.

[0101] For example, when the first loader fails, if the second loader can take on more tasks proactively to ensure the overall progress of the operation, the first loader and the second loader can each receive 8 bonus points as a reward for their collaborative response to the failure.

[0102] This not only reduces the impact of failures on operations, but also enhances the collaborative emergency response capabilities among equipment.

[0103] In some embodiments, the value of the collaboration reward function is determined according to whether the first loader and the second loader share information related to the first work task and information related to the second work task.

[0104] For example, in response to the first loader or the second loader sharing the optimized operation strategy, path planning and other information with the other party, and the experience is successfully applied, so that the operation efficiency is improved by more than 10%, then the sharing party can obtain 6 reward points and the applying party can obtain 4 reward points.

[0105] In this way, system learning can be accelerated and the level of collaborative work can be improved.

[0106] Through the above embodiment, not only can multi-objective collaborative optimization be achieved through the set multiple reward functions to reduce the overall cost, but the loader can also be enabled to operate in an orderly manner, reduce waiting time and route conflicts, avoid unnecessary detours and empty driving, and thus greatly shorten the operation cycle.

[0107] In steps 130 and 150 , noise may be added to the input of the reinforcement learning model to improve the robustness of the system.

[0108] In some embodiments, noise is added to the state information; the action information is determined based on the state information after the noise is added; noise is added to the action information; and the reward value is determined based on the action information after the noise is added.

[0109] For example, dynamic noise can be added to the state information, which can be expressed as follows: S(t)'=S(t)(1+

[0110] α),α∈N(0,σ 2 ), S(t) represents the current state information, S(t)' represents the state information after adding noise, and α represents the noise type. The method of adding noise to action information is the same as the above process type and will not be repeated here.

[0111] For example, noise can be added to state information, such as the location and shape of obstacles, or interference signals to the perception of the model's own position, speed, and other states. Adding noise to state information simulates this uncertainty, allowing the model to learn to make correct decisions under inaccurate state information and improving its robustness to environmental noise.

[0112] For example, noise can also be added to action information (adding noise to the actions selected by the agent). For example, in a work scenario, a vehicle's steering, acceleration, and other actions may be interfered with by external factors. Adding action noise can help the model adapt to the uncertainty of action execution, prevent the agent from over-relying on precise action outputs, and enhance its generalization ability in complex real-world environments.

[0113] In this way, when the reinforcement learning model is trained, by adding noise to the state information and action information, the stability and robustness of the model to different environmental errors can be enhanced.

[0114] Once the components of the reinforcement learning model are determined, the model can be trained.

[0115] In step 170 , a reinforcement learning model is trained based on the reward value and the updated state information to schedule loaders.

[0116] In some embodiments, based on the multiple reward functions, a value function corresponding to each of the multiple reward functions is determined; and based on the value function, a reinforcement learning model is trained.

[0117] For example, as mentioned above, the reward function can be configured based on factors such as task completion, efficiency, and safety to provide feedback on the loader's behavior and guide the learning of the agent in the reinforcement learning model.

[0118] For example, the value function can be used to calculate the expected cumulative reward value that the above-mentioned agent can obtain by performing actions in a long term according to a specific strategy, so as to evaluate the different state information and action information of the first loader and the second loader respectively, and thus reasonably allocate tasks according to the results of the value function.

[0119] In this way, the first loader can be responsible for coordinating the overall situation, planning the main path and performing key operations, and the second loader can complete auxiliary tasks according to the instructions of the first loader or the state of the environment. By continuously updating the value function, the intelligent agent can gradually find the strategy that optimizes the overall task completion effect.

[0120] Next, we will combine Figure 2 , an example of how to obtain Figure 1 The map information of the operation scene in the status information mentioned in step 110 is obtained, and the map information is preprocessed.

[0121] Figure 2 A flowchart illustrating determining map information of a work scene in some embodiments of the model training method disclosed herein.

[0122] In step 210, the operation scene is modeled by manually driving a single loader to generate initial map information of the operation scene. The initial map information may include terrain information, landform information, and location coordinates of materials.

[0123] In step 230, a filtering function is used to perform denoising on the data in the initial map information.

[0124] For example, the modeled map may contain various noises, such as sensor measurement noise, environmental interference noise, etc. Therefore, the data can be smoothed by performing a weighted average of the neighborhood around each data point using Gaussian filtering.

[0125] In this way, the impact of noise can be effectively reduced, thereby improving the accuracy of the model.

[0126] In step 250 , clustering is performed on the denoised data to eliminate data with low density.

[0127] For example, using a density-based clustering algorithm, isolated points with low density are identified as outliers and removed.

[0128] In step 270 , the clustering result is used as map information of the work scene.

[0129] In this way, by denoising the initial map information and removing abnormal isolated points, the accuracy and reliability of the acquired map information can be improved, thereby enhancing the effectiveness of the decision-making of the subsequent reinforcement learning model.

[0130] In some embodiments, semantic information can also be extracted from the map information of the work scene (such as identifying object categories, area functions, etc. in the map information), and the semantic information can be combined with the location information in the map information to further obtain map information with richer content information.

[0131] For example, for lidar data, point cloud segmentation algorithms can be used to separate the point clouds of different objects, and then combined with machine learning or deep learning algorithms for object recognition and classification. For camera image data, target detection, semantic segmentation, and other technologies can be used to extract semantic information from the image.

[0132] In this way, by combining semantic information with location information in map information, the content of the map can be enriched, making the map more semantically understandable, and thus providing more valuable information for the decision-making and application of subsequent reinforcement learning models.

[0133] Figure 3 A schematic diagram illustrating a reinforcement learning model in some embodiments of the model training method of the present disclosure.

[0134] like Figure 3 As shown, the reinforcement learning model 300 includes an agent 301, environment information 302, action information 303, state information 304 and a reward function 305.

[0135] In some embodiments, based on relevant information of the loader and first relevant information of the working scene, state information 304 is determined, and the relevant information of the loader includes the working state type of the loader; based on the state information 304, the reinforcement learning model 300 is used to determine action information 303, and the action information 303 includes scheduling information of the loader; based on the action information 303, the reinforcement learning model 300 is used to determine the reward value of at least one reward function 305 corresponding to the action information 303 and the updated state information 304; based on the reward value and the updated state information 304, the reinforcement learning model 300 is trained for scheduling the loader.

[0136] For example, consider a reinforcement learning training process. First, agent 301 has state information S(t) and a reward value R(t). When agent 301 interacts with environment information 302, it can obtain action information a(t) corresponding to state information S(t). Environment information 302 updates state information S(t) based on action information a(t) to obtain S(t+1). It then scores this action information a(t) using reward function 305 to generate a reward value R(t+1).

[0137] Next, the updated state information R(t+1) and reward value R(t+1) are used as inputs for the next training process to continue the above training process until the termination condition of model training is met. For example, the termination condition of model training can be model convergence or reaching a set number of training times.

[0138] In some embodiments, the working state type is determined based on characteristic information of the loader, where the characteristic information includes at least one of an operating parameter and a hydraulic system parameter of the loader.

[0139] In some embodiments, the working status type is determined using a clustering method based on the feature information.

[0140] In some embodiments, the action information 303 is verified based on whether the action information 303 matches the working state type of the loader; and the action information 303 is determined based on the verification result.

[0141] Figure 4 A flowchart illustrating some embodiments of the scheduling method of the present disclosure.

[0142] like Figure 4 As shown, in step 410, based on the relevant information of the loader and the first relevant information of the operation scene, the loader is scheduled using a reinforcement learning model, and the reinforcement learning model is obtained by the model training method in any of the above embodiments.

[0143] In some embodiments, before executing step 410, step 405 may be further executed to obtain relevant information of the loader and first relevant information of the operation scene.

[0144] Figure 5 A block diagram illustrating some embodiments of the model training apparatus of the present disclosure.

[0145] like Figure 5 As shown, the model training device 500 includes: a state information determination module 501, which is configured to determine the state information based on the relevant information of the loader and the first relevant information of the working scene, and the relevant information of the loader includes the working state type of the loader; an action information determination module 502, which is configured to determine the action information based on the state information using the reinforcement learning model, and the action information includes the scheduling information of the loader; a third determination module 503, which is configured to determine the reward value of at least one reward function corresponding to the action information and the updated state information based on the action information using the reinforcement learning model; a training module 504, which is configured to train the reinforcement learning model based on the reward value and the updated state information for scheduling the loader.

[0146] In some embodiments, the state information determination module 501 includes determining the working state type according to characteristic information of the loader, where the characteristic information includes at least one of an operating parameter and a hydraulic system parameter of the loader.

[0147] In some embodiments, the action information determination module 502 includes adding noise to the state information; determining the action information based on the state information after adding noise; the third determination module 503 includes adding noise to the action information; and determining the reward value based on the action information after adding noise.

[0148] In some embodiments, the state information determination module 501 includes obtaining second relevant information of the operation scene; and aligning the second relevant information using a timestamp to obtain the first relevant information.

[0149] In some embodiments, the action information determination module 502 includes verifying the action information according to whether the action information matches the working state type of the loader; and determining the action information according to the verification result.

[0150] In some embodiments, the status information determination module 501 includes determining the working status type based on the characteristic information of the loader, including: merging relevant features based on at least one of the values ​​of the similarity matrix of the clustering function and the density values ​​between the features; and mapping the merged result to the working status type of the loader.

[0151] In some embodiments, the training module 504 includes determining a value function corresponding to each of the multiple reward functions based on the multiple reward functions; and training the reinforcement learning model based on the value function.

[0152] Figure 6 A block diagram showing some embodiments of the scheduling device of the present disclosure.

[0153] In some embodiments, the scheduling module 601 is configured to schedule the loader based on the relevant information of the loader and the first relevant information of the operation scenario using a reinforcement learning model, where the reinforcement learning model is obtained by the model training method in any of the above embodiments.

[0154] Figure 7 A block diagram showing other embodiments of the electronic device of the present disclosure.

[0155] like Figure 7 As shown, the electronic device 7 of this embodiment includes: a memory 71 and a processor 72 coupled to the memory 71, and the processor 72 is configured to execute the model training method or scheduling method in any embodiment of the present disclosure based on instructions stored in the memory 71.

[0156] The memory 71 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0157] Figure 8 A block diagram illustrating some further embodiments of the electronic device disclosed herein.

[0158] like Figure 8 As shown, the electronic device 8 of this embodiment includes: a memory 810 and a processor 820 coupled to the memory 810, and the processor 820 is configured to execute the model training method or scheduling method in any one of the aforementioned embodiments based on instructions stored in the memory 810.

[0159] The memory 810 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0160] Alternatively, the electronic device 8 may further include an input / output interface 830, a network interface 840, a storage interface 850, and the like. These interfaces 830, 840, 850, as well as the memory 810 and the processor 620, may be connected, for example, via a bus 860. The input / output interface 830 provides a connection interface for input / output devices such as a display, mouse, keyboard, touch screen, microphone, and speakers. The network interface 840 provides a connection interface for various networked devices. The storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0161] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The model training method and apparatus, scheduling method and apparatus, and computer-readable storage medium according to the present disclosure have been described in detail. To avoid obscuring the concepts of the present disclosure, some details known in the art have been omitted. Based on the above description, those skilled in the art will fully understand how to implement the technical solutions disclosed herein.

[0163] The methods and systems of the present disclosure may be implemented in many ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0164] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure.

Claims

1. A model training method, comprising: Determining state information based on relevant information about the loader and first relevant information about the operation scene, wherein the relevant information about the loader includes a working state type of the loader; Determining action information based on the state information using a reinforcement learning model, the action information including scheduling information of the loader; Based on the action information, using the reinforcement learning model, determining a reward value of at least one reward function corresponding to the action information and updated state information; Based on the reward value and the updated state information, the reinforcement learning model is trained for scheduling the loader.

2. The model training method according to claim 1, wherein: The determining of the state information based on the relevant information of the loader and the first relevant information of the operation scene includes: The working state type is determined according to characteristic information of the loader, where the characteristic information includes at least one of an operating parameter and a hydraulic system parameter of the loader.

3. The model training method according to claim 1, wherein: The determining the action information based on the state information and utilizing the reinforcement learning model includes: adding noise to the state information; determining the action information according to the state information after adding noise; The determining, based on the action information and utilizing the reinforcement learning model, a reward value of at least one reward function corresponding to the action information and updated state information includes: adding noise to the motion information; The reward value is determined according to the action information after adding noise.

4. The model training method according to claim 1, wherein: The determining of the state information based on the relevant information of the loader and the first relevant information of the operation scene includes: Acquire second relevant information of the operation scenario; The second relevant information is aligned using a timestamp to obtain the first relevant information.

5. The model training method according to claim 1, wherein: The determining the action information based on the state information and utilizing the reinforcement learning model includes: Verifying the action information according to whether the action information matches the working state type of the loader; The action information is determined according to the verification result.

6. The model training method according to claim 2, wherein: The determining the working state type according to the characteristic information of the loader includes: The working status type is determined using a clustering method according to the feature information.

7. The model training method according to any one of claims 1 to 6, wherein: The loaders include a first loader and a second loader, the first work task of the first loader and the second work task of the second loader are associated, the first reward value of the first loader and the second reward value of the second loader are determined according to the completion status of the first work task and the second work task, and the completion status includes at least one of the efficiency of the operation completion, the safety of the operation completion, and the cooperation status of the first loader and the second loader.

8. The model training method according to claim 7, wherein: The complexity of the first work task is higher than that of the second work task, and the first loader guides the second loader to perform the second work task.

9. The model training method according to claim 8, wherein: The at least one reward function includes an efficiency reward function, which is used to characterize the efficiency of completing the operation. The value of the efficiency reward function of the first loader is greater than the value of the efficiency reward function of the second loader. In response to the higher the working efficiency of the loader, the larger the value of the efficiency reward function.

10. The model training method according to claim 8, wherein: The at least one reward function includes a safety reward function, which is used to characterize the safety situation of the completion of the operation. The safety reward function is determined based on whether the first loader and the second loader collide with obstacles in the operation scene, and / or whether the distance between the first loader and the second loader is less than a distance threshold.

11. The model training method according to claim 8, wherein: The at least one reward function includes a collaboration reward function, which is used to characterize the collaboration between the first loader and the second loader. The collaboration reward function is determined according to at least one of the following: whether a difference between the start time of the first loader and the start time of the second loader is less than a threshold; whether the position of the second loader matches the first work task and the second work task; In response to a faulty loader being present between the first loader and the second loader, whether other loaders between the first loader and the second loader take over the work task of the faulty loader.

12. The model training method according to claim 11, wherein: The value of the collaboration reward function is determined according to whether the first loader and the second loader share relevant information of the first work task and relevant information of the second work task.

13. The model training method according to any one of claims 1 to 12, wherein: The at least one reward function comprises a plurality of reward functions, The step of training the reinforcement learning model based on the reward value and the updated state information includes: Determining, based on the multiple reward functions, a value function corresponding to each of the multiple reward functions; Based on the value function, the reinforcement learning model is trained.

14. The model training method according to claim 1, wherein: The first relevant information of the operation scene includes at least one of map information, obstacle information and terrain information of the operation scene.

15. A scheduling method, comprising: Based on the relevant information of the loader and the first relevant information of the operation scene, the loader is scheduled using a reinforcement learning model, and the reinforcement learning model is obtained by the model training method according to any one of claims 1 to 14.

16. A model training device comprising: a state information determination module configured to determine state information based on relevant information about the loader and first relevant information about the operation scene, wherein the relevant information about the loader includes a working state type of the loader; an action information determination module, configured to determine action information based on the state information using a reinforcement learning model, the action information including scheduling information of the loader; a third determination module, configured to determine, based on the action information and using the reinforcement learning model, a reward value of at least one reward function corresponding to the action information and updated state information; A training module is configured to train the reinforcement learning model based on the reward value and the updated state information for scheduling the loader.

17. A scheduling device comprising: The scheduling module is configured to schedule the loader based on the relevant information of the loader and the first relevant information of the operation scene using a reinforcement learning model, and the reinforcement learning model is obtained by the model training method described in any one of claims 1 to 14.

18. An electronic device comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the model training method of any one of claims 1 to 14 or the scheduling method of claim 15 based on instructions stored in the memory.

19. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training method described in any one of claims 1 to 14 or the scheduling method described in claim 15.

20. A computer program product comprising instructions, which, when executed by a processor, cause the processor to perform the model training method according to any one of claims 1 to 14 or the scheduling method according to claim 15.

Images