Method and data processing device for controlling motion of a vehicle

Abstract

A computer-implemented method for controlling motion of a vehicle includes training an agent, using a deep reinforcement learning algorithm 402, to result in a trained agent. The deep reinforcement le

Description

TECHNICAL FIELD

[0001] Various embodiments relate to a method and a data processing device for controlling motion of a vehicle. BACKGROUND

[0002] The use of large dimension vehicles, especially tractor trailer systems, is ubiquitous in the transportation industry. The efficient and safe operation of these vehicles is crucial for ensuring timely and cost-effective transportation of goods. However, operating these large vehicles in real-world scenarios can be challenging due to their size, weight, and complex dynamics. These vehicles are larger and heavier than normal passenger vehicles. This means that they have greater momentum and require more space to maneuver. Further, large-dimension vehicles typically have complicated designs. Tractor-trailer systems, for example, consist of multiple articulated segments that are interconnected through a kingpin. Such a design allows for greater cargo capacity but also makes the vehicle more difficult to control. The driver needs to coordinate the movement of the tractor and trailer segments to ensure that the vehicle is stable and balanced. As such, it is challenging to develop autonomous driving techniques for such large-dimension vehicles. SUMMARY

[0003] According to various embodiments, there is provided a computer-implemented method for controlling motion of a vehicle. The method includes training an agent, using a deep reinforcement learning algorithm, to result in a trained agent. The deep reinforcement learning algorithm includes a hybrid reward function configured to reward the agent for following a reference path. The hybrid reward function includes a plurality of reward components and a plurality of penalty components. The method further includes generating, by the trained agent, a control command for steering the vehicle based on observation data comprising vehicle information and lane information.

[0004] According to various embodiments, there is provided a computer program that includes instructions which, when the program is executed by a computer, cause the computer to perform the abovementioned computer-implemented method for controlling motion of a vehicle.

[0005] According to various embodiments, there is provided a non-transitory computer-readable storage medium that includes instructions which, when executed by a computer, causes the computer to perform the abovementioned computer-implemented method for controlling motion of a vehicle.

[0006] According to various embodiments, there is provided a data processing device. The data processing device includes a memory that stores the abovementioned computer program. The data processing device further includes a processor configured to perform the computer-implemented method for controlling motion of a vehicle.

[0007] According to various embodiments, there is provided a vehicle. The vehicle includes the abovementioned data processing device and a sensor. The sensor is configured to output at least part of the observation data.

[0008] Additional features for advantageous embodiments are provided in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings, in which:

[0010] FIG. 1 shows an example of a sampled B-spline curve used for training a deep reinforcement learning policy according to various embodiments.

[0011] FIG. 2 shows a diagram of a kinematic single-track model use for motion planning computations.

[0012] FIG. 3 is a diagram showing a representation of the tractor-trailer system and its coordinates.

[0013] FIG. 4 shows a flow diagram of a computer-implemented method for controlling motion of a vehicle 202, according to various embodiments.

[0014] FIG. 5 shows a simplified functional block diagram of a data processing device according to various embodiments.

[0015] FIG. 6 shows a simplified block diagram of a vehicle according to various embodiments.

[0016] Embodiments described below in context of the devices are analogously valid for the respective methods, and vice versa. Furthermore, it will be understood that the embodiments described below may be combined, for example, a part of one embodiment may be combined with a part of another embodiment.

[0017] It will be understood that any property described herein for a specific device may also hold for any device described herein. It will be understood that any property described herein for a specific method may also hold for any method described herein. Furthermore, it will be understood that for any device or method described herein, not necessarily all the components or steps described must be enclosed in the device or method, but only some (but not all) components or steps may be enclosed.

[0018] The term “coupled” (or “connected”) herein may be understood as electrically coupled or as mechanically coupled, for example attached or fixed, or just in contact without any fixation, and it will be understood that both direct coupling or indirect coupling (in other words: coupling without direct contact) may be provided.

[0019] In this context, the device as described in this description may include a memory which is for example used in the processing carried out in the device. A memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).

[0020] In order that the invention may be readily understood and put into practical effect, various embodiments will now be described by way of examples and not limitations, and with reference to the figures.

[0021] According to various embodiments, a method 400 for controlling motion of a vehicle may include the use of deep reinforcement learning (DRL) policy to control an autonomous large dimensional vehicle. The large dimensional vehicle may be a semi-truck, or a tractortrailer system. A tractor-trailer system includes a tractor coupled to a trailer. Control schemes designed for passenger vehicles are not suitable for a tractor-trailer system, because the tractor and the trailer do not move as a single rigid body. Instead, the trailer is hitched to the tractor and the hitch angle between the trailer and the tractor is dynamic. Applying a passenger vehicle control scheme on a tractor-trailer system will cause off-tracking of the tractor-trailer system. Off-tracking refers to deviation from a reference path. To address the challenges, the method for controlling motion for the tractor-trailer system may include the use of proximal policy optimization (PPO) to train an agent. The agent may be trained with a reward function defined to penalize deviation and reward for proceeding without off-tracking. The reward functions and observation spaces of the deep reinforcement learning algorithm may be designed to result in an agent that is capable of accurately following a reference path, even if it involves sharp turns. The method may also include an incremental control scheme complemented by a local coordination system, to enhance the generalization capacity of the deep reinforcement learning policy. The reference path may be represented using a series of way points and basis spline (B-spline) curve. This may to improve the implementation of partial observation.

[0022] According to various embodiments, the vehicle may be other types of large dimensional vehicles, such as buses, construction vehicles, or others.

[0023] FIG. 1 shows an example of a sampled B-spline curve 102 used for training a deep reinforcement learning policy according to various embodiments. A spline function of order n is a piecewise polynomial function of degree (n - 1) in a variable x. The values of x where the pieces of polynomial meet are known as knots. The key property of spline functions is that they and their derivatives may be continuous, depending on the multiplicities of the knots. B-splines of order n are basis functions for spline functions of the same order defined over the same knots, meaning that all possible spline functions can be built from a linear combination of B-splines, and there is only one unique combination for each spline function. During the training of the deep reinforcement learning policy, a reference path is generated based on a B-spline curve and discretized into way points. By randomly selecting the control points, different tracks may be generated for each training episode to enhance the model’s abilities of generalization. In the example shown, the sampled curve was generated using a 3-degree B-spline curve with 4 control points to manipulate the shape of the curve.

[0024] FIG. 2 shows a diagram of a kinematic single-track model use for motion planning computations. The diagram is taken from a publication “CommonRoad: Vehicle Models” by Althoff et.al. The kinematic single-track model depicts a tractor-trailer system 202 with only three wheels, where the front and rear wheel pairs of the tractor 302 (shown in FIG. 3) and the wheel pair of the trailer 304 (shown in FIG. 3) are each lumped into one wheel. The tractortrailer system may also be referred herein as vehicle 202. This simplification is justified since the roll dynamics are not considered. This also explains the term single-track model. The kinematic single-track model with one on-axle trailer further does not consider any tire slip, so the velocity vector v at the center of the rear axle is always aligned with the link between the front and rear wheel. To derive the differential equation of the kinematic single-track model with one on-axle trailer, the following parameters are required: velocity 206 denoted as V, velocity of the steering angle denoted as Vs , the steering angle denoted as 8 208, the heading and the parameter / ., / ,210 describing the length of the wheelbase. The hitch angle a 204 is introduced as a new state variable. The differential equations of the kinematic single-track model with one on-axle trailer are: 8 = Vs (la) 0=-^-tan 5 lwb a = — <p = ——tan 8 lwb sx = v cosip (Id) sy = vsinip (le)

[0025] The DRL policy may be trained using partial observation data. The partial observation data may include the information about the vehicle 202, also referred herein as “vehicle information”, and information about the lane, also referred herein as “lane information”. In other words, the DRL policy may be trained with only these two information types, without requiring excessive data.

[0026] The vehicle information may include orientation of the vehicle 202, velocity 206 of the vehicle 202, and hitch angle between the tractor and the trailer. The lane information may include the data of two waypoints ahead of the vehicle coordinates system, which may equip the agent of the DRL policy with the ability to anticipate the future condition of the road. The origin of the vehicle coordinates system is the rear axle center point of the tractor. The heading of the tractor is referred to as the longitude direction axis and the axis perpendicular to the heading is referred to as the lateral direction axis.

[0027] Based on the vehicle information and the lane information, the distance between the two waypoints and the vehicle coordinates system origin, and the coordinates of the two waypoints with respect to the vehicle coordinates system origin, also referred herein as local coordinates of the two waypoints, may be determined. In other words, the local coordinate system is the coordinate system attached to the vehicle. The local coordinates of each way point (Xiocai, Yiocai) may be determined as follows: ^locai = (X - * cosip + (Yw - Yv) * simp (2) Yiocai = -(X - ^v) * simp + (Yw - Yv) * cosip (3)

[0028] The waypoints’ global coordinates are denoted as (X, Yw), the vehicle yaw angle is denoted as i / / , and the vehicle world coordinates are denoted as (X, X). The vehicle world coordinates are the coordinates of the vehicle in a world coordinate system. The observation space is listed in the following table: (cosip, simp) Orientation of the tractor V Velocity of the tractor O) Angular velocity of the tractor Dr Distance between vehicle and the first waypoint d2 Distance between vehicle and the second waypoint XX) Local coordinates of the first waypoint Y^ Local coordinates of the second waypoint a Hitch angle between tractor and trailer

[0029] For action space, to mimic the driving behavior of human drivers and to simplify the control problem, two control commands may be generated to control the vehicle. The control commands may be throttle and steering angle.

[0030] The method 400 may seek to achieve the advantage of controlling the vehicle 202 such that it follows a reference path closely without off-tracking. A typical binary reward structure may not achieve the required performance. As such, the DRL policy may include a hybrid reward system. The hybrid reward system may be defined using the hybrid reward function R, as follows: R=rg+rp+rt + rd+r0 (4)

[0031] The reward structure may include (i) reward for reaching waypoints, denoted as rg, (ii) reward for proceeding, denoted as rp, (iii) penalty for exceeding a maximum training timestep, denoted as re, (iv) penalty for deviating from the reference line, denoted as rd, and (v) penalty for off-tracking, denoted as ro. In other words, the hybrid reward function may include a plurality of reward components and a plurality of penalty components. The plurality of reward components may include, for example, rg and rp. The plurality of penalty components may include, for example, re, rd, and r0.

[0032] The reward for reaching way points, rg, is the reward given when the agent reached waypoints on the reference path. The reward for proceeding, rp, is the reward given the agent moves. The penalty for exceeding a maximum training timestep, re, is the penalty given when the agent exceeded a predefined time for reaching the goal. The penalty for deviating from the reference line, rd, is the penalty given when the agent deviates from the reference path, in other words, when the distance between the agent and the reference path exceeds a predetermined threshold. The penalty for off-tracking, ro, is the penalty given when at least one of the trailer and the tractor deviates beyond a predefined threshold, from the reference path.

[0033] Examples of the hybrid reward function components, and notations used in the components are listed below: (10, if D± <Dr 9 I 0, otherwise. _ t—100, if t >maximum timesteps, e I 0, otherwise. _ ( — 100, if Max(dvd2) >Do 0 I 0, otherwise. rd = (di + w * d2)2 * V rp = (DGt-DGt_tf*K where Dr denotes the criteria for reaching a waypoint, t denotes the current timestep, df denotes lateral deviation of the tractor, d2 denotes lateral deviation of the trailer, Do denotes the safety range for lateral deviation, V denotes the coefficient of deviation reward, K denotes the coefficient of proceeding reward, and DG denotes the distance between tractor and the first ahead waypoint.

[0034] FIG. 3 is a diagram showing a representation of the tractor-trailer system and its coordinates. The vehicle 202 may include a tractor 302 and a trailer 304. The vehicle reference coordinate system is represented by 324. A reference path 350 is represented by a B-spline curve that is discretized into a plurality of way points 334. The centre point of the rear axle of the tractor 302 may be regarded as the tractor centre point 310 for calculation of lateral deviation. The centre point of the rear axle of the trailer 304 may be regarded as the trailer centre point 312 for calculation of lateral deviation. As the vehicle 202 moves along its path, the first way point observed may shift from 330 to 330’, while the second way point observed may shift from 332 to 332’. The first way point observed 330 and the second way point observed 332 are part of the plurality of way points 334. The first way point observed 330 and the second way point observed 332 are way points 334 ahead of the tractor 304, in other words, observed by the vehicle 202 before the vehicle 202 approaches these way points.

[0035] The lateral deviation of the tractor 302 from the reference path 350 is indicated as front deviation 320, denoted as di. The lateral deviation of the trailer 304 from the reference path 350 is indicated as rear deviation 322, denoted as d2. The front deviation 320 may be the shortest distance from the tractor centre point 310 to the nearest point on the reference path 350. Similarly, the rear deviation 322 may be the shortest distance from the trailer centre point 312 to the nearest point on the reference path 350. As it may be difficult to directly calculate the distance between the centre point and the nearest point on the reference path 350, the lateral deviation may be approximated based on a vector 340 pointing from the first way point observed 330 to the second way point observed 332. The vector 340 may serve as a proxy to the motion of the vehicle 202. The lateral deviation may be calculated based on coordinates of the first waypoint 330 ahead of the tractor center point 302, coordinates of the second waypoint 332 ahead of the tractor center point 302, and the coordinates of the tractor’s center point 310.

[0036] The lateral deviation of the vehicle 202 may be denoted as d The lateral deviation may be determined based on equation (5): d _ I fe ~ *i) * (yi - y0) - (Xi - x0) * (y2 - yi) I (5) Vfe -X02 + (y2 -yj2 where (xi, yi) are the coordinates of the first waypoint 330, (x2, yj) are the coordinates of the second waypoint 332, and (xo, yo) are the coordinates of the tractor’s center point 310.

[0037] FIG. 4 shows a flow diagram of a computer-implemented method 400 for controlling motion of a vehicle 202, according to various embodiments. The method 400 may include training an agent using a DRL algorithm to result in a trained agent, in 402. The DRL algorithm may include a hybrid reward function configured to reward the agent for following a reference path, for example the reference path 350 shown in FIG. 3. The hybrid reward function may include a plurality of reward components and a plurality of penalty components. The method 400 may further include, generating, by the trained agent, a control command for steering the vehicle 202 based on observation data comprising vehicle information and lane information. The resulting control command may steer a large vehicle to closely follow the reference path.

[0038] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the method 400 may further include representing a lane that the vehicle 202 is travelling on, using at least one basis spline function 102, to result in a lane representation. The method 400 may further include discretizing the lane representation into a plurality of way points 334, and providing the lane information based on the plurality of way points 334. Using a B-spline function to represent the reference path may allow generation of many different reference paths for training the DRL by randomly selecting the control points of the B-spline function. Training the DRL with many difference reference paths may improve the generalization of the DRL to different types of motion paths.

[0039] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the control command may include throttle and steering angle. These control commands are commonly used in vehicles, and as such, may be implemented to automate existing vehicles.

[0040] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the observation data may be obtained from sensors onboard the vehicle 202.

[0041] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the DRL algorithm may include proximal policy optimization.

[0042] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the plurality of reward components may include a first reward for reaching waypoints on the lane the vehicle is travelling on, and a second reward for proceeding.

[0043] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the plurality of penalty components may include a first penalty for exceeding a maximum training time step, a second penalty for deviation from a reference line, and a third penalty for exceeding a deviation safety range.

[0044] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the vehicle information may include orientation of the vehicle and velocity of the vehicle.

[0045] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the lane information may include data on two waypoints of a lane the vehicle is travelling on, the two way points being ahead of the vehicle 202, wherein the data includes respective distances of the two way points to a rear axle centre point of the vehicle, and may further include the respective angles of the two way points to the rear axle centre point of the vehicle. The two waypoints ahead of the vehicle 202 may be the first way point observed 330 and the second way point observed 332. The lateral deviation between the vehicle 202 and positions on the reference path 350 may not be easily determined directly, and hence the distance and angles between the way points to the centre point of the vehicle may be used to approximate the lateral deviation.

[0046] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the vehicle 202 may include a tractor-trailer system. The method 400 may achieve close tracking of a reference path by a tractor-trailer system, even though in a tractor-trailer system, the trailer may exhibit a hitch angle with respect to the tractor.

[0047] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, at least one penalty component of the hybrid reward function is determined based on deviation from each of rear axle centre point of the vehicle and the rear axle centre point of the trailer.

[0048] FIG. 5 shows a simplified functional block diagram of a data processing device 500 according to various embodiments. The data processing device 500 may include a memory 504 and a processor 502. The memory 504 may store a computer program. The computer program may include instructions which, when the computer program is executed by a computer, cause the computer to perform the method 400 according to any above-described embodiment or with any below described further embodiment.. The processor 502 may be configured to perform the method 400. The processor 502 and the memory 504 may be coupled to one another, for example, mechanically, electrically, or communicatively, by a coupling line 530. Various aspects described with respect to the method 400 may be applicable to the data processing device 500.

[0049] FIG. 6 shows a simplified block diagram of a vehicle 600 according to various embodiments. The vehicle 600 may include the data processing device 500. The vehicle 600 may further include a sensor 604 configured to output at least part of the observation data. The sensor 504 may be, for example, a camera configured to capture images of a road that the vehicle 600 is travelling on. The sensor 504 may include, for example, a localization device such as a global positioning system. The vehicle 600 may include the vehicle 202. The data processing device 500 and the sensor 604 may be coupled to one another, for example, mechanically, electrically, or communicatively, by a coupling line 630. Various aspects described with respect to the method 400 may be applicable to the vehicle 600.

[0050] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced. It will be appreciated that common numerals, used in the relevant drawings, refer to components that serve a similar or the same purpose.

[0051] It will be appreciated to a person skilled in the art that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0052] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0053] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term “some” refers to one or more. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

1. A computer-implemented method (400) for controlling motion of a vehicle (202), the method (400) comprising:training an agent, using a deep reinforcement learning algorithm, to result in a trained agent (402);wherein the deep reinforcement learning algorithm comprises a hybrid reward function configured to reward the agent for following a reference path,the hybrid reward function comprising a plurality of reward components and a plurality of penalty components; andgenerating, by the trained agent, a control command for steering the vehicle based on observation data comprising vehicle information and lane information (404).

2. The method (400) of any preceding claim, further comprising:representing a lane that the vehicle is travelling on, using at least one basis spline function, to result in a lane representation;discretizing the lane representation into a plurality of way points; and providing the lane information based on the plurality of way points.

3. The method (400) of any preceding claim, wherein the control command comprises throttle and steering angle.

4. The method (400) of any preceding claim, wherein the observation data is obtained from sensors onboard the vehicle (202).

5. The method (400) of any preceding claim, wherein the deep reinforcement learning algorithm comprises proximal policy optimization.

6. The method (400) of any preceding claim, wherein the plurality of reward components comprises a first reward for reaching waypoints on the lane the vehicle is travelling on, and a second reward for proceeding.

7. The method (400) of any preceding claim, wherein the plurality of penalty components comprises a first penalty for exceeding a maximum training time step, a second penalty for deviation from a reference line, and a third penalty for exceeding a deviation safety range.

8. The method (400) of any preceding claim, wherein the vehicle information comprises orientation of the vehicle (202) and velocity of the vehicle (202).

9. The method (400) of any preceding claim, wherein the lane information comprises data on two waypoints of a lane the vehicle (202) is travelling on, the two way points being ahead of the vehicle (202), wherein the data comprises respective distances of the twoway points to a rear axle centre point of the vehicle (202), and further comprises the respective angles of the two way points to the rear axle centre point of the vehicle (202).

10. The method (400) of any preceding claim, wherein the vehicle (202) comprises a tractortrailer system.

11. The method (400) of claim 10, wherein at least one penalty component of the hybrid reward function is determined based on deviation from each of rear axle centre point of the tractor (302) and the rear axle centre point of the trailer (304).

12. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the computer-implemented method (400) of any one of claims 1 to 11.

13. A non-transitory computer-readable storage medium comprising instructions which, when executed by a processor, cause the processor to carry out the computer-implemented method (400) of any one of claims 1 to 11.

14. A data processing device (500) comprising:a memory (504) storing the computer program of claim 12; anda processor (502) configured to perform the computer-implemented method (400) for controlling motion of a vehicle (202).

15. A vehicle (202, 600) comprising:the data processing device (500) of claim 14; anda sensor (604) configured to output at least part of the observation data.

Images