Path following controller for small vehicle and its implementation method

Abstract

The present invention discloses a path following controller of a small car and its implementation method. The path following controller includes: a comprehensive data processing module, a reinforcement learning processing unit, a model predictive control processing unit, and a control quantity output module; the comprehensive data processing module is used to obtain path following modal control information; the reinforcement learning processing unit is used to update the reinforcement learning network according to the path following modal control information, and obtain the first information according to the reinforcement learning network; the model predictive control processing unit is used to generate a predicted control quantity according to the first information and the path following modal control information; the control quantity output module is used to generate a target control quantity according to the predicted control quantity and the first information. The embodiment of the present application realizes the precise path following control of the small car in a complex transportation environment through the deep fusion processing of the reinforcement learning unit and the model predictive control processing unit, and improves the accuracy of the following control; it can be widely used in the field of mobile robot technology.

Description

Technical Field

[0001] The invention relates to the technical field of mobile robots, in particular to a path following controller of a small vehicle and a realization method thereof. Background Art

[0002] AGV is a mobile robot that can reduce labor losses and shorten logistics cycles through autonomous transportation. It is widely used in production, warehousing and logistics. With the advent of the industrial age, autonomous, unmanned and intelligent AGVs are the development trend in the future logistics field. Path following is one of the basic tasks of AGV motion control in factory workshops, warehouses and other places, and it is also the key to realizing AGV autonomous intelligent motion.

[0003] One method in the related technology is to control the path following of the car through reinforcement learning. However, since the control method of artificial intelligence is essentially formed by optimizing and iterating based on the weight training of the neural network, in the process of continuous interaction with the constructed environment, it will cause resource shortages, excessive time costs, and low sampling efficiency. Another method is to follow the path through model predictive control. Model predictive control is an optimization-based control method. During the solution process, it can constrain the state quantity and control quantity, and finally predict a more accurate control. However, for model predictive control, the weight coefficient of its cost function is usually formulated according to the specific actual situation, and the use of manual coefficient setting and adjustment methods will cause optimization inaccuracy problems. Summary of the invention

[0004] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0005] To this end, an object of the present invention is to provide an efficient path following controller, method, device and storage medium for a small vehicle.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0007] On the one hand, an embodiment of the present invention provides a path following controller for a vehicle, comprising:

[0008] The path following controller of the car in the embodiment of the present invention comprises: a comprehensive data processing module, a reinforcement learning processing unit, a model predictive control processing unit, and a control quantity output module; wherein the comprehensive data processing module is used to obtain path following modal control information, and send the path following modal control information to the reinforcement learning processing unit and the model predictive control processing unit; the reinforcement learning processing unit is used to update the reinforcement learning network according to the path following modal control information, and obtain the first information according to the reinforcement learning network; and send the first information to the model predictive control processing unit and the control quantity output module; the first information includes the decision fusion coefficient required for the target control quantity; the model predictive control processing unit is used to generate a predicted control quantity according to the first information and the path following modal control information, and send the predicted control quantity to the control quantity output module; the control quantity output module is used to generate a target control quantity according to the predicted control quantity and the first information. The embodiment of the present application realizes the precise path following control of the car in a complex transportation environment through the deep fusion processing of the reinforcement learning unit and the model predictive control processing unit, and improves the accuracy of the following control.

[0009] In addition, the path following controller of the vehicle according to the above embodiment of the present invention may also have the following additional technical features:

[0010] Furthermore, in the path following controller of the car in the embodiment of the present invention, the reinforcement learning processing unit includes an environment unit, an agent unit and a parameter output module; the environment unit is used to configure the state quantity, and generate a reward quantity according to the state quantity at consecutive moments, and send the state quantity and the reward quantity to the agent unit; the agent unit is used to generate a first control quantity, a cost function weight coefficient and a decision fusion coefficient based on the reinforcement learning network according to the state quantity and the reward quantity, and send the first control quantity and the decision fusion coefficient to the control quantity output module through the parameter output module, and send the cost function weight coefficient to the model predictive control processing unit.

[0011] Furthermore, in one embodiment of the present invention, the agent unit is also used to send the decision fusion coefficient to the environment unit, and the environment unit is used to update the state quantity and the reward quantity according to the decision fusion coefficient.

[0012] Furthermore, in one embodiment of the present invention, the model predictive control processing unit includes: an error calculation module and a rolling optimization calculation module; wherein the error calculation module is used to calculate the posture error and send the posture error to the rolling optimization calculation module; the rolling optimization calculation module is used to generate the predicted control quantity according to the posture error.

[0013] Furthermore, in one embodiment of the present invention, the rolling optimization calculation module is further used to generate the predicted control amount according to a cost function, and the cost function is: It is used to characterize the pose error from the i-th moment to the i+N-1-th moment, i represents a certain moment, η i is the weight coefficient of the cost function.

[0014] On the other hand, an embodiment of the present invention provides a method for implementing a path following controller of a small vehicle, which is applied to the path following controller of the small vehicle mentioned above, and the method includes:

[0015] Get path following mode control information;

[0016] Establishing a reinforcement learning network according to the acquired path following modal control information;

[0017] Inputting the path following modal control information into the reinforcement learning network to obtain first information; the first information includes a decision fusion coefficient required for the target control amount;

[0018] generating a predicted control amount according to the first information and the path following modal control information;

[0019] A target control amount is generated based on the predicted control amount and the first information.

[0020] Furthermore, the implementation method of the path following controller of the vehicle of the embodiment of the present invention further includes:

[0021] According to the acquisition path following modal control information, the state quantity is configured, and the reward quantity is generated according to the state quantity at consecutive moments;

[0022] According to the state quantity and the reward quantity, a first control quantity, a cost function weight coefficient and a decision fusion coefficient are generated based on the reinforcement learning network.

[0023] Furthermore, the implementation method of the path following controller of the vehicle of the embodiment of the present invention further includes:

[0024] The state quantity and the reward quantity are updated according to the decision fusion coefficient.

[0025] Furthermore, the implementation method of the path following controller of the vehicle in the embodiment of the present invention further includes:

[0026] Calculating the posture error according to the acquired path following modal control information;

[0027] A predicted control amount is determined according to the posture error.

[0028] Furthermore, in the implementation method of the path following controller of the vehicle of the embodiment of the present invention, the step of determining the predicted control amount according to the posture error includes:

[0029] The predicted control amount is determined according to the cost function; the cost function is: It is used to characterize the pose error from the i-th moment to the i+N-1-th moment, i represents a certain moment, η i is the weight coefficient of the cost function.

[0030] On the other hand, an embodiment of the present invention provides a device for implementing a path following controller of a vehicle, comprising:

[0031] at least one processor;

[0032] at least one memory for storing at least one program;

[0033] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned method for implementing the path following controller of the vehicle.

[0034] On the other hand, an embodiment of the present invention provides a storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the implementation method of the path following controller of the above-mentioned vehicle.

[0035] The embodiment of the present application realizes precise path following control of the vehicle in a complex transportation environment and improves the accuracy of following control through deep fusion processing of the reinforcement learning unit and the model predictive control processing unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A schematic structural diagram of an embodiment of a path following controller for a trolley provided by the present invention;

[0038] Figure 2 A schematic structural diagram of another embodiment of a path following controller for a trolley provided by the present invention;

[0039] Figure 3 A structural schematic diagram of an embodiment of a reinforcement learning processing unit provided by the present invention;

[0040] Figure 4 A schematic diagram of the structure of an embodiment of a model predictive control processing unit provided by the present invention;

[0041] Figure 5 A flow chart of an embodiment of a method for implementing a path following controller of a trolley provided by the present invention;

[0042] Figure 6 A schematic structural diagram of an embodiment of a device for implementing a path following controller for a trolley provided by the present invention. DETAILED DESCRIPTION

[0043] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0044] AGV (Automated Guided Vehicle, AGV) refers to a transport vehicle equipped with electromagnetic or optical automatic navigation devices, which can travel along a specified navigation path and has safety protection and various transfer functions. AGV is a type of mobile robot. Through autonomous transportation, it can reduce labor loss and shorten the logistics cycle. It is widely used in production and warehousing logistics. With the advent of Industry 4.0, autonomous unmanned and intelligent AGV is the future development trend in the logistics field. Path following is one of the basic tasks of AGV motion control in factory workshops, warehouses and other places, and it is also the key to realizing AGV autonomous intelligent movement.

[0045] In recent years, with the development of artificial intelligence technology, control theories based on it have been continuously proposed, especially based on reinforcement learning control research, which has produced many research results in recent years. However, since the control method of artificial intelligence is essentially formed by optimizing and iterating the weight training of neural networks, in the process of continuous interaction with the constructed environment, it will cause resource shortages, excessive time costs, and low sampling efficiency. Studies have shown that traditional control methods can be guided in order to optimize the reinforcement learning training stage, but it will increase the problem of hyperparameter adjustment. Model predictive control is a control method based on optimization. In the process of solving the problem, it can constrain the state quantity and control quantity, and finally predict a more accurate control. However, for model predictive control, the weight coefficient of its cost function is usually formulated according to the specific actual situation. The use of manual coefficient setting adjustment method will cause optimization inaccuracy.

[0046] In view of the above problems, the purpose of the present invention is to provide a path following controller and control method based on AGV car, which integrates the control method of deep reinforcement learning and model predictive control, solves the problems of low sampling efficiency and difficult adjustment of model predictive control cost parameters in the process of deep learning control, and enables the AGV car to realize intelligent and accurate path following control movement in a complex transportation environment. Specifically, the controller design includes a comprehensive data processing module, a reinforcement learning processing unit, a model predictive control processing unit, and a control quantity output module; the controller control method is divided into two types, namely training and practice modes. In the training mode, according to the data of the training environment, the data processing module, the model predictive control processing unit and the control quantity output module inside the controller are used to realize the training of the neural network in the reinforcement learning processing unit, so that the module can realize the training based on reinforcement learning control quantity, cost function weight coefficient and fusion discount coefficient; the application mode is based on the real external sensor data and the set reference path, and the final control quantity can be obtained by using the control quantity output module to realize the path following control of the AGV car; the controller completes the deep integration of the control characteristics of deep reinforcement learning and model predictive control, so that the AGV car can realize intelligent and accurate path following control movement in a complex transportation environment.

[0047] A path following controller and implementation method of a vehicle according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings. First, a path following controller of a vehicle according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0048] Figure 1 1 is a schematic diagram of the structure of a path following controller of a car according to an embodiment of the present invention, wherein the path following controller comprises: a comprehensive data processing module 110, a reinforcement learning processing unit 120, a model predictive control processing unit 130, and a control quantity output module 140;

[0049] The integrated data processing module is used to obtain the path following modal control information and send the path following modal control information to the reinforcement learning processing unit and the model predictive control processing unit;

[0050] The reinforcement learning processing unit is used to update the reinforcement learning network according to the path following modal control information, and obtain the first information according to the reinforcement learning network; and send the first information to the model predictive control processing unit and the control quantity output module; the first information includes the decision fusion coefficient required for the target control quantity;

[0051] The model predictive control processing unit is used to generate a predicted control amount according to the first information and the path following modal control information, and send the predicted control amount to the control amount output module;

[0052] The control quantity output module is used to generate a target control quantity according to the predicted control quantity and the first information.

[0053] Optionally, in the path following controller of the car in the embodiment of the present invention, the reinforcement learning processing unit includes an environment unit, an agent unit and a parameter output module; the environment unit is used to configure the state quantity, and generate a reward quantity according to the state quantity at consecutive moments, and send the state quantity and the reward quantity to the agent unit; the agent unit is used to generate a first control quantity, a cost function weight coefficient and a decision fusion coefficient based on the reinforcement learning network according to the state quantity and the reward quantity, and send the first control quantity and the decision fusion coefficient to the control quantity output module through the parameter output module, and send the cost function weight coefficient to the model predictive control processing unit.

[0054] Optionally, in the path following controller of the car in the embodiment of the present invention, the agent unit is also used to send the decision fusion coefficient to the environment unit, and the environment unit is used to update the state quantity and the reward quantity according to the decision fusion coefficient.

[0055] Optionally, in the path following controller of the car in the embodiment of the present invention, the model predictive control processing unit includes: an error calculation module and a rolling optimization calculation module; wherein the error calculation module is used to calculate the posture error and send the posture error to the rolling optimization calculation module; the rolling optimization calculation module is used to generate a predicted control quantity according to the posture error.

[0056] Optionally, in the path following controller of the vehicle in the embodiment of the present invention, the rolling optimization calculation module is further used to generate a predicted control amount according to a cost function, and the cost function is: It is used to characterize the pose error from the i-th moment to the i+N-1-th moment, i represents a certain moment, η i is the weight coefficient of the cost function.

[0057] In some possible implementations, such as Figure 2As shown, the path following controller of the car provided in the embodiment of the present application includes a comprehensive data processing module, a reinforcement learning processing unit, a model predictive control processing unit, and a control quantity output module; the comprehensive data processing module is responsible for collecting, processing and distributing path following modal control information; Figure 3 As shown, the reinforcement learning processing unit mainly includes the environment unit (i.e. Figure 3 environment), the agent unit (i.e. Figure 3 Agents in ) and parameter output modules. Figure 4 As shown, the model predictive control processing unit in the embodiment of the present application includes an error calculation module and a rolling optimization calculation module; the control quantity output module is responsible for generating and distributing the final control quantity of the path following.

[0058] The path following mode control information collected in the embodiment of the present application includes a total of m reference path initialization information p r_k =[x r_k ,y r_k ,θ r_k ] T , k∈[0,m-1], the position state p of the AGV car at the i-th moment and the i+1-th moment collected by the sensor c_i =[x c_i ,y c_i ,θ c_i ] T and p c_i+1 =[x c_i+1 ,y c_i+1 ,θ c_i+1 ] T , and the final control quantity of the AGV car at the i-th moment and the i+1-th moment and i represents a certain moment, x, y, θ represent the horizontal coordinate, vertical coordinate and guidance angle of the AGV in the two-dimensional path, v∈[v ,in , v max ] represents the AGV linear speed, w∈[w min , w max ] represents the AGV angular velocity.

[0059] The path following mode control information processed by the embodiment of the present application includes the state error amount and its error derivative Distance between AGV and reference path i represents a certain moment; the distributed path following modal control information includes the path following modal control information collected and processed by the integrated data processing module.

[0060] The environment unit in the embodiment of the present application is responsible for receiving environmental information from the integrated data processing module and the agent unit and sending it to the agent unit. When receiving information for the first time, only the information of the integrated data processing module is received; the agent unit is responsible for generating parameters related to the control quantity based on reinforcement learning and sending them to the environment unit and the parameter output module; the parameter output module is responsible for distributing the information generated by the received agent unit; the error calculation module uses the information sent by the integrated data processing module; the rolling optimization calculation module is responsible for generating the control quantity based on the model prediction And send it to the control quantity output module; the control quantity output module is responsible for receiving the fusion discount coefficient β i , and generate the final control volume And send it to the AGV car for real-time control, and the final control quantity Passed to the comprehensive data processing module.

[0061] The environmental information in the embodiment of the present application is state S and reward R, where S is the value of setting a round from state s i The starting set, the reward R is the reward r from a round i (s i ,a i ,s i+1 ) starts with a set where action a i Represents the final control amount s i+1 represents a new state, i represents a certain moment; state This includes the AGV state quantity p at two different times c_i and p c_i+1 , state error and its error derivative Control volume Distance between AGV and reference path Weight cost coefficient Decision fusion coefficient β i ∈(0,1). The control quantity related parameters include the control quantity based on reinforcement learning Cost function weight coefficient η i and the fusion discount factor β i , i represents a certain moment. In the embodiment of the present application, the fusion discount coefficient β is sent to the environment unit. i ; The parameters sent to the parameter output module are the control quantity related parameters. The parameter output module is responsible for distributing the information generated by the received agent unit, which is specifically manifested as the control quantity generated by each reinforcement learning model. Fusion discount factor β i Send it to the control quantity output module and convert the cost function weight coefficient η lThe error calculation module is responsible for calculating the posture error from the i-th moment to the i+N-1th moment. n∈[i,i+N-1], N is the model prediction time domain, and the obtained The rolling optimization calculation module is responsible for generating the control quantity based on the model prediction. And send it to the control quantity output module; posture error The control cost function is predicted by the calculation model It turns out that n∈[i,i+N-1], where η i The optimization involved in each rolling window over the entire forecast horizon N remains constant.

[0062] In some other possible implementations, the integrated data processing module is responsible for collecting and distributing the modal control information required for path following, and the collected modal control information required for path following includes a total of m reference path initialization information p r_k =[x r_k ,y r_k ,θ r_k ] T , k∈[0,m-1] (the path is discretized into three-dimensional coordinate points with virtual postures, and the direction is toward the extension direction of the path), and the posture state quantity p of the AGV car at the i-th moment and the i+1-th moment collected by the sensor c_i =[x c_i ,y c_i ,θ c_i ] T and p c_i+1 =[x c_i+1 ,y c_i+1 ,θ c_i+1 ] ( , and the final control quantity of the AGV car at the i-th moment and the i+1-th moment and i represents a certain moment, and x, y, and θ represent the horizontal coordinate, vertical coordinate, and guidance angle of the AGV in the two-dimensional path, respectively.

[0063] At the initialization start, since there is no AGV car posture state quantity and final control quantity at the first moment at the 0th moment, let the posture state quantity p c_0 =p c_1 , control amount The modal control information required for the AGV path following is distributed to the reinforcement learning processing unit and the model predictive control processing unit. The information distributed to the reinforcement learning processing unit includes the state quantity p of the AGV car at the i-th moment and the i+1-th moment. c_i and p c_i+1 , state error and state error derivative Control volume Distance between AGV and reference path i represents a certain moment. The information distributed to the model predictive control processing unit includes the AGV car's posture state quantity p at the i-th moment and the i+1-th moment c_i and p c_i+1 , and also includes the control quantity u of the AGV car at the i-th moment and the i+1-th moment i and u i+1 , path initialization information p r_k =[x r_k ,y r_k ,θ r_k ] T ,k∈[0,m-1].

[0064] like Figure 3 As shown in the figure, the reinforcement learning processing unit mainly includes an environment unit, an agent unit and a parameter output module. The reinforcement learning processing unit can use training networks such as DDPG, PPO, A3C, etc., and configure its required parameters according to the training network structure. PPO is selected as the network structure used for training, and its learning rate, discount factor, trimming parameter, GAE parameter, advantage function, value function coefficient, entropy regularization coefficient, optimization iteration number, batch size, number of hidden layers of the neural network, activation function type, optimizer, etc. are configured. The environment unit includes state S and reward R, where ξ is the value of setting a round from state s i The starting set, reward R, the reward for one round starts from r i (s i ,a i ,s i+1 ) starts with a set where action a i Represents the fusion control amount s i+1 Represents the new state. This includes the AGV state quantity p at two different times c_i and p c_i+1 , state error and its error derivative Control volume Distance between AGV and reference path Weight cost coefficient Decision fusion coefficient β i ∈(0,1), i represents a certain moment. Weight cost coefficient η i and decision fusion coefficient β i During the training of the reinforcement learning neural network, the agent unit is sent to the environment unit for generating new state quantities s i+1 The agent unit is responsible for generating parameters related to the reinforcement learning control quantity, including the reinforcement learning control quantity Cost function weight coefficient ηi And the decision fusion coefficient β i The parameter output module is responsible for distributing the received information in each iteration, converting the control amount generated by each deep reinforcement learning model into Fusion discount factor β i Send it to the measurement output module and convert the cost function weight coefficient η i Sent to the model predictive control module.

[0065] like Figure 4 As shown in FIG. 1 , the model predictive control processing unit includes an error calculation module and a rolling optimization calculation module. The error calculation module uses the information received from the comprehensive data processing module to calculate the kinematic model of the two-wheel differentially driven AGV vehicle. Calculate the i-th moment Then, differentiating time t yields Using the differential value The position p at the i-th moment c_i And the reference path p from the i-th moment to the i+N-1th moment r_n , n∈[i,i+N-1], and finally the pose error from the i-th moment to the i+N-1th moment can be obtained n∈[i,i+N-1], the obtained Sent to the rolling optimization calculation module, where ξ x_i =λ x_i v r_i cosθ e_i ,ξ y_i =λ y_i v r_i sinθ e_i ,ξ θ_i =λ θ_i θ e_i is the posture error interference, is the posture error interference coefficient, N is the time domain of the model prediction control rolling window.

[0066] The rolling optimization calculation module sends a posture error to the receiving error calculation module n∈[i,i+N-1], and at the same time receive the current weight coefficient η sent by the deep reinforcement learning processing unit i , η i Keep constant participation in the optimization of the entire prediction domain N in the rolling window each time, and then calculate the model prediction control cost function x∈[i,i+N-1], and will generate the model-based predictive control quantity Sent to the control quantity output module.

[0067] The control quantity output module receives the fusion discount coefficient βi , and generate control quantities And send it to the AGV car for real-time control, and at the same time Passed to the comprehensive data processing module.

[0068] In summary, the AGV path following controller of the present invention adopts a deep fusion method based on deep reinforcement learning and model predictive control. When in training mode, the deep reinforcement learning module in the controller not only trains the control quantity itself, but also optimizes the weight coefficient in the model predictive control module. The fusion control quantity is obtained by weighted summation of the control quantity, which ensures the accuracy of tracking the reference path during the AGV path following control process, accelerates the convergence speed of the reinforcement learning training process, improves the sampling efficiency, and realizes the adaptability of AGV control under complex environmental and uncertain interference conditions, thereby ensuring the path following control effect.

[0069] Next, a method for implementing a path following controller for a vehicle according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0070] Reference Figure 5 In an embodiment of the present invention, a method for implementing a path following controller of a small car is provided. The method for implementing the path following controller of the small car in the embodiment of the present invention can be applied to a terminal, can be applied to a server, can also be software running in a terminal or a server, etc. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The method for implementing the path following controller of the small car in the embodiment of the present invention is applied to the path following controller of the small car above, and mainly includes the following steps:

[0071] S100: Acquire path following mode control information;

[0072] S200: Establishing a reinforcement learning network according to the acquired path following modal control information;

[0073] S300: Inputting the path following modal control information into the reinforcement learning network to obtain first information; the first information includes a decision fusion coefficient required for the target control amount;

[0074] S400: generating a predicted control amount according to the first information and the path following mode control information;

[0075] S500: Generate a target control amount according to the predicted control amount and the first information.

[0076] Optionally, the method for implementing the path following controller of the vehicle in the embodiment of the present invention further includes:

[0077] According to the acquired path following modal control information, the state quantity is configured, and the reward quantity is generated according to the state quantity at consecutive moments;

[0078] According to the state quantity and the reward quantity, a first control quantity, a cost function weight coefficient and a decision fusion coefficient are generated based on the reinforcement learning network.

[0079] Optionally, the implementation method of the path following controller of the car in the embodiment of the present invention further includes: updating the state quantity and the reward quantity according to the decision fusion coefficient.

[0080] Optionally, the method for implementing the path following controller of the vehicle in the embodiment of the present invention further includes:

[0081] Calculate the posture error based on the acquired path following modal control information;

[0082] According to the posture error, the predicted control amount is determined.

[0083] Optionally, the implementation method of the path following controller of the vehicle in the embodiment of the present invention determines the predicted control amount according to the posture error, including:

[0084] According to the cost function, the predicted control amount is determined; the cost function is: It is used to characterize the pose error from the i-th moment to the i+N-1-th moment, i represents a certain moment, η i is the weight coefficient of the cost function.

[0085] In some possible implementations, the control method provided in the embodiment of the present application is divided into a training mode and an application mode; specifically, the training mode includes the following steps:

[0086] Step S21: The integrated data processing module collects modal control information generated by sensors from the environment and the set initialization information;

[0087] Step S22: the comprehensive data processing module partially processes the collected data and sends them to the reinforcement learning processing unit and the model predictive control processing unit;

[0088] Step S23: The environment unit of the reinforcement learning processing unit configures the reinforcement learning network parameters and receives the comprehensive data information from the comprehensive data processing module. The fused control quantity sent by the control quantity output module As the current state, and randomly initialize η i ,βi , thus forming the state quantity s i , then according to the state quantity s i Generate a new state s i+1 and reward r i+1 Send to the agent unit;

[0089] Step S24: The agent unit receives the new state and reward generated by the environment unit and continues to generate new actions, including the control amount based on reinforcement learning Cost function weight coefficient η i and the fusion discount factor β i ,Will With β i Send to the parameter output module, and η i Send to the model predictive control processing unit;

[0090] Step S25: The agent unit generates the With β i Return to the environment unit for training the new state s in the environment i+1 Update, and update the neural network weights;

[0091] Step S26: If the enhanced training processing unit completes the set rounds of training or reaches a certain training condition, the network will no longer be updated, the entire training is completed, and the trained network model is saved for application, otherwise continue to execute step S27;

[0092] Step S27: The error calculation module of the model predictive control processing unit receives the position and posture p at the i-th moment c_i And the reference path p from the i-th moment to the i+N-1th moment r_n , and the pose error from the i-th moment to the i+N-1th moment is obtained n∈[i,i+N-1], the obtained Sent to the rolling optimization calculation module,

[0093] Step S28: The rolling optimization calculation module of the model prediction control processing unit receives the posture state error Predict the N posture states in the entire prediction time domain n∈[i,i+N-1], and at the same time receive the current weight coefficient η sent by the deep reinforcement learning processing unit i , participate in the optimization of the entire prediction domain N in the rolling window each time, and then calculate the model-based predictive control quantity And send it to the control quantity output module;

[0094] Step S29: The control quantity output module receives the fusion discount coefficient β i , and generate the final control volume And send it to the AGV car for control, and at the same time send the controlled posture information p c_i+1 and the final control volume Pass it to the integrated data processing module and repeat step S21.

[0095] The control steps in the application mode provided in the embodiment of the present application are as follows:

[0096] Step S31: deploying the reinforcement learning neural network model completed in the training mode on the reinforcement learning processing unit;

[0097] Step S32-Step S34: the same as Step S22-Step S24;

[0098] Step S35: The reinforcement learning processing unit is deployed in the reinforcement learning neural network model completed in the training mode, and the agent unit generates the action With β i Returned to the environment unit for updating the state s i+1 ;

[0099] Step S36-Step S38: Same as Step S27-Step S29.

[0100] It can be seen that the contents of the above system embodiments are all applicable to the present method embodiments, the functions specifically implemented by the present method embodiments are the same as those of the above system embodiments, and the beneficial effects achieved are also the same as those achieved by the above system embodiments.

[0101] Reference Figure 6 The embodiment of the present invention provides a device for implementing a path following controller of a vehicle, comprising:

[0102] at least one processor 610;

[0103] At least one memory 620, used to store at least one program;

[0104] When at least one program is executed by at least one processor 610, the at least one processor 610 implements the path following controller implementation method of the vehicle.

[0105] Similarly, the contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0106] An embodiment of the present invention further provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the implementation method of the path following controller of the above-mentioned vehicle.

[0107] Similarly, the contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0108] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0109] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0110] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several programs to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0111] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution systems, devices or apparatuses.

[0112] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0113] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0114] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0115] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0116] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A path following controller for a car, characterized in that: The path following controller includes: a comprehensive data processing module, a reinforcement learning processing unit, a model predictive control processing unit, and a control quantity output module; Wherein, the integrated data processing module is used to obtain path following modal control information, and send the path following modal control information to the reinforcement learning processing unit and the model predictive control processing unit; The reinforcement learning processing unit is used to update the reinforcement learning network according to the path following modal control information, and obtain first information according to the reinforcement learning network; and send the first information to the model predictive control processing unit and the control amount output module; the first information includes the decision fusion coefficient required for the target control amount; The model predictive control processing unit is used to generate a predicted control amount according to the first information and the path following modal control information, and send the predicted control amount to the control amount output module; The control amount output module is used to generate a target control amount according to the predicted control amount and the first information; The reinforcement learning processing unit includes an environment unit, an agent unit and a parameter output module; the environment unit is used to configure the state quantity, and generate a reward quantity according to the state quantity at continuous moments, and send the state quantity and the reward quantity to the agent unit; the agent unit is used to generate a first control quantity, a cost function weight coefficient and a decision fusion coefficient based on the reinforcement learning network according to the state quantity and the reward quantity, and send the first control quantity and the decision fusion coefficient to the control quantity output module through the parameter output module, and send the cost function weight coefficient to the model predictive control processing unit.

2. The path following controller of the vehicle according to claim 1, characterized in that: The agent unit is further used to send the decision fusion coefficient to the environment unit, and the environment unit is used to update the state quantity and the reward quantity according to the decision fusion coefficient.

3. The path following controller of the vehicle according to claim 1, characterized in that: The model predictive control processing unit includes: an error calculation module and a rolling optimization calculation module; wherein the error calculation module is used to calculate the posture error and send the posture error to the rolling optimization calculation module; the rolling optimization calculation module is used to generate the predicted control amount according to the posture error.

4. The path following controller of the vehicle according to claim 3, characterized in that: The rolling optimization calculation module is also used to generate the predicted control amount according to a cost function, and the cost function is: It is used to characterize the pose error from the i-th moment to the i+N-1-th moment, i represents a certain moment, η i is the weight coefficient of the cost function.

5. A method for implementing a path following controller of a small vehicle, characterized in that: The path following controller applied to the vehicle as claimed in claim 1, the method comprising: Get path following mode control information; Establishing a reinforcement learning network according to the acquired path following modal control information; Inputting the path following modal control information into the reinforcement learning network to obtain first information; the first information includes a decision fusion coefficient required for the target control amount; generating a predicted control amount according to the first information and the path following modal control information; A target control amount is generated based on the predicted control amount and the first information.

6. The method for implementing the path following controller of the vehicle according to claim 5, characterized in that: The method further comprises: According to the acquisition path following modal control information, the state quantity is configured, and the reward quantity is generated according to the state quantity at consecutive moments; According to the state quantity and the reward quantity, a first control quantity, a cost function weight coefficient and a decision fusion coefficient are generated based on the reinforcement learning network.

7. The method for implementing the path following controller of the vehicle according to claim 6, characterized in that: The method further comprises: The state quantity and the reward quantity are updated according to the decision fusion coefficient.

8. The method for implementing the path following controller of the vehicle according to claim 6, characterized in that: The method further comprises: Calculating the posture error according to the acquired path following modal control information; A predicted control amount is determined according to the posture error.

9. The method for implementing the path following controller of the vehicle according to claim 8, characterized in that: Determining the predicted control amount according to the posture error includes: The predicted control amount is determined according to the cost function; the cost function is: It is used to characterize the pose error from the i-th moment to the i+N-1-th moment, i represents a certain moment, η i is the weight coefficient of the cost function.

Images