Method and computer system for controlling movement of a host vehicle

Abstract

Methods and computer systems of controlling movement of a host vehicle. Lane data related to a road item is determined from a data source, and object data acquired from an object is determined via a sensor system. The road item and the object are located in an external environment of the host vehicle. The lane data and the object data are fed to a machine learning algorithm to generate a set of control functions for controlling movement of the host vehicle by performing the following steps: classifying a predetermined set of maneuvers of the host vehicle based on the lane data and the object data, and determining a set of control functions related to kinematics of the host vehicle. Movement of the host vehicle is controlled based on the classified maneuvers and the determined control functions.

Description

Technical Field

[0001] This disclosure relates to a computer implementation method and computer system for controlling the movement of a master vehicle. Background Technology

[0002] In modern vehicles, longitudinal and lateral movement control is typically supported by advanced driver assistance systems (ADAS), such as adaptive cruise control (ACC) which supports longitudinal control and lane keeping assist (LKA) which supports lateral control. Such systems rely primarily on perception of the vehicle's external environment, provided, for example, by vision devices such as cameras and / or radar and / or lidar sensors.

[0003] The perception of the vehicle environment provided as output from these devices is traditionally used by trajectory planners or controllers. Such trajectory planners are capable of modeling the movement of the master vehicle equipped with the perception devices, as well as the movement of other relevant vehicles and / or objects in the master vehicle's environment. The purpose of trajectory planning is to provide a safe and comfortable future trajectory for the master vehicle. However, trajectory planning can be a challenging task. For example, noisy input data needs to be controlled by specific rules, which must be enforced despite the noisy data to provide a stable trajectory.

[0004] Traditional methods for trajectory planning are based on the dynamic characteristics of all relevant objects, such as their position, velocity, and acceleration, which can be determined by sensors in order to predict their individual movements based on the laws of physics. However, common methods for trajectory planning can be computationally expensive and / or inaccurate.

[0005] Therefore, there is a need for a method and system that directly provides reliable parameters for controlling the movement of the master vehicle. Summary of the Invention

[0006] In one aspect, the present invention aims at a computer-implemented method for controlling the movement of a master vehicle. According to the method, lane data related to road items are determined from a data source, and object data is determined from the objects via a sensor system. The road items and objects are located in the external environment of the master vehicle. The lane data and object data are fed to a machine learning algorithm to generate a set of control functions for controlling the movement of the master vehicle by performing the following steps: classifying a predetermined set of maneuvers of the master vehicle based on the lane data and the object data, and determining a set of control functions related to the kinematics of the master vehicle. The movement of the master vehicle is controlled based on the classified maneuvers and the determined control functions.

[0007] Lane data can refer to the lane on which the vehicle is currently moving and / or, where it exists and is available, the adjacent lanes to the right and left. Therefore, lane data can describe the route of one or more of these lanes. Consequently, road items associated with the lane data can include, for example, lane markings or road boundaries that follow the lane route. Road items can also be detected by a sensor system, i.e., in a manner similar to that of objects whose object data has been acquired. Alternatively, lane data can be obtained from a map that can be used to map the intended route of the vehicle.

[0008] Machine learning algorithms can be implemented using artificial neural networks, which can include encoders and corresponding decoders for classification tasks and regression tasks. That is, steps or tasks (i.e., classifying a predetermined set of maneuvers and determining that set of control functions) can use the same common encoder.

[0009] A predetermined set of maneuvers for the primary vehicle can involve both longitudinal and lateral directions with respect to the primary vehicle, wherein the longitudinal direction is defined by the longitudinal axis of the primary vehicle. Longitudinal maneuvers can refer to accelerating, decelerating, and maintaining the speed of the primary vehicle, while lateral maneuvers can refer to maintaining lane position, changing lanes to the right, and changing lanes to the left.

[0010] Control functions related to the kinematics of the host vehicle may include, for example, control parameters for acceleration and steering angle, parameters for the planned curvature, or parameters for the planned trajectory of the host vehicle. Acceleration and steering angle, as well as the planned curvature or planned trajectory, can be represented by continuous functions, such as polynomials of a predetermined order.

[0011] Because the method applies machine learning algorithms to the entire lane and object data, rather than applying physical laws to individual objects in the master vehicle environment, once the machine learning algorithm is trained, the information needed to control the movement of the master vehicle can be obtained more directly with reduced computational effort.

[0012] Furthermore, the so-called "multimodal problem" can be overcome, referring to the fact that drivers of a vehicle may exhibit different behaviors related to their driving style. For example, under the same driving conditions, if another vehicle appears in front of the primary vehicle, one driver may prefer to slow down the primary vehicle and stay in the lane, while another driver may prefer to change lanes to the left and accelerate the vehicle under the same driving conditions. Therefore, controlling the movement of the primary vehicle and predicting its future trajectory can depend not only on perception data, i.e., lane data and object data, but also on the characteristics of the actual driver of the vehicle. If appropriate training information related to a specific driver is available for machine learning algorithms, this method can also overcome the multimodal problem by adapting the machine learning algorithms to the characteristics of the driver.

[0013] Furthermore, machine learning algorithms not only perform regression tasks to determine the optimal values ​​of control functions used to define kinematic parameters, but also classification tasks regarding a predetermined set of maneuvers, which may include corresponding probability values ​​for each maneuver at a given point in time. For example, a machine learning algorithm can provide not only a sequence of optimal values ​​for the acceleration and steering angle of the main vehicle at a near future point in time, but also recommended maneuvers for the main vehicle in the longitudinal and lateral directions.

[0014] The results of the classification step can be linked to the steps of determining the control function and / or actually controlling the vehicle's movement in different ways. The most likely maneuvers relative to the vehicle's longitudinal axis, for example, with respect to the longitudinal and lateral directions respectively, can be selected and used as inputs for determining the control function. Alternatively, the actions predicted by the classification step (i.e., the most likely actions) can be compared with external inputs, such as those provided by the vehicle's driver, for the actions to be performed, to determine which actions can be used as inputs for determining the control function. The connection between the results of the classification step and the steps of determining the control function and / or the actual control steps can improve the reliability and safety of the master vehicle's future trajectory, which can be achieved by applying the output of the method.

[0015] The control function can be a continuous function, that is, a mathematical function that can be expressed in a closed form, such as a polynomial, and the application of such a continuous function can reduce the amount of computation required. Furthermore, continuous functions can allow, for example, the determination of the values ​​of motion parameters such as acceleration and steering angle at any point in time, regardless of whether actual perception of the environment via sensor data is available for that particular point in time.

[0016] According to the implementation, the step of determining the set of control functions (i.e., the regression task) may include predicting control parameters for the acceleration and steering angle of the master vehicle. In other words, for this implementation, the kinematics of the master vehicle can be represented by acceleration and steering angle, i.e., by only two parameters. The control functions determined by the regression task can be represented by corresponding second-order polynomials. In this case, the coefficients of each polynomial can be determined to define a continuous function in time.

[0017] According to another embodiment, the step of determining the set of control functions (i.e., the regression task) may include parameters for predicting the planned curvature of the master vehicle. The planned curvature refers to the future curvature of the master vehicle's predicted trajectory. In this embodiment, the control function may be represented by a third-order polynomial. Alternatively, the regression task may include parameters for predicting the master vehicle's planned trajectory, i.e., instead of the planned curvature. In this case, the control function may be represented by a fifth-order polynomial.

[0018] For all implementations of a given regression task, only one or two kinematic parameters of that regression task may be considered. If the continuous function is represented by the corresponding polynomial, less computational effort is required, and the parameters to be regressed can be assigned to the corresponding layers of a machine learning algorithm or neural network.

[0019] According to another implementation, the data source for determining lane data may include a sensor system. Therefore, the sensor system may be the sole source required to determine the input data for a machine learning algorithm. Alternatively or additionally, the data source for determining lane data may include a predetermined map. Instead, object data obtained from objects in the external environment of the main vehicle is further determined using the sensor system. However, if a predetermined map is used as the data source, the lane data can be independent of the perception capabilities of the sensor system.

[0020] According to another implementation, common ground truth data can be applied to train a machine learning algorithm to classify a predetermined set of maneuvers and determine a control function, such as a time-series continuous function related to the kinematics of the host vehicle. The ground truth data can be acquired as data recorded for the host vehicle during a predetermined time period. To record this data for the host vehicle, a sensor system that may include, at least in part, the vehicle's vision sensors and / or radar sensors and / or lidar sensors can be used. Furthermore, additional pre-classified data can be generated by pre-classifying the predetermined set of maneuvers using data recorded for the host vehicle. This additional pre-classified data can be used for training to determine the continuous function.

[0021] Although the machine learning algorithm simultaneously performs two distinct tasks—a classification task to categorize predetermined maneuvers and a regression task to determine the control function—these two tasks are linked to provide information about the future kinematics or trajectory of the primary vehicle. Therefore, the common ground truth reflects the link between the two tasks in a feasible manner. Furthermore, if the common ground truth data of the primary vehicle is recorded or documented during a predetermined time period, this ground truth data describes and represents the actual behavior of the vehicle's driver. Thus, the aforementioned multimodal problem—the different behaviors of different drivers in the same driving situation—can be overcome by recording or documenting common ground truth data during a predetermined time period.

[0022] Furthermore, multimodal problems can be further facilitated by deriving additional pre-classified data from the data of the master vehicle recorded or documented during a predetermined time period. For example, probability values ​​for each time point and each predetermined operation can be provided to define the additional pre-classified data. Additionally, each maneuver can be individually "given" to the machine learning algorithm during the training phase to temporally "demand" the corresponding control function for the relevant control parameters, such as the steering angle and acceleration for the master vehicle.

[0023] Additional pre-classified data can be fed to the machine learning algorithm during the training phase, such that each training sample in the pre-classified data contains only one baseline ground truth mode. This helps determine the convergence of the control function for the regression task. Such a training process is called "teacher forcing" because the coefficients of the layers used, for example, in the machine learning algorithm or neural network, can be learned for the predetermined motor classification. However, the additional baseline ground truth data or additional pre-classified data should not be "leaked" into the classification task during training. Therefore, the additional pre-classified data is only provided to the regression task; for example, if a computer system with separate decoders for classification and regression tasks is applied, the data is only provided to the regression decoder.

[0024] Furthermore, classification and regression tasks can be associated with corresponding loss functions during the training of machine learning algorithms. That is, for example, there may be a loss function for the classification task and another loss function for the regression task. To train the machine learning algorithm, these loss functions can be minimized based on, for example, applying public perceptual data and additional pre-classified data to encoder and decoder systems that perform classification and regression tasks respectively, and based on a comparison of the machine learning algorithm's output with the corresponding ground truth data.

[0025] Road items may include lane markings, road boundaries, and / or guardrails. For example, lane markings can be monitored by a sensor system to track the route of the lane over which a vehicle is currently moving. Additionally, the route of the current lane can be further verified by tracking road boundaries and / or guardrails. This facilitates lane route tracking. The tracked road item can be represented by a third-order polynomial in a vehicle coordinate system with the vertical and horizontal axes as coordinate axes. That is, monitoring road items in time allows for the representation of the monitored road item, such as lane markings, as a continuous function in time. Therefore, for this embodiment, monitoring lane markings may require a lower computational workload.

[0026] According to another embodiment, object data may include predetermined physical properties of the object. Such physical properties may be the position, velocity, or type of an external object in the main vehicle environment. That is, the term "predetermined physical properties" refers to the pre-selection of a type of physical property to be monitored for other objects (i.e., external objects in the vehicle environment). If more than one physical property of an object is represented by object data, the reliability of motion parameters and corresponding control functions for controlling the movement of the main vehicle can be improved.

[0027] According to another embodiment, a predetermined set of maneuvers for the master vehicle may include lateral maneuvers such as changing lanes to the left, maintaining lane position, and changing lanes to the right, as well as longitudinal maneuvers such as accelerating, maintaining speed, and decelerating. Therefore, in this embodiment, the set of maneuvers may include three lateral maneuvers and three longitudinal maneuvers. Since this set of six predetermined maneuvers has a well-defined relationship with the kinematic parameters of the master vehicle, it can facilitate the control of the master vehicle by using this set of predetermined maneuvers.

[0028] In another aspect, this disclosure relates to a computer system configured to perform several or all of the steps of the computer-implemented methods described herein.

[0029] The computer system is communicatively linked to a sensor system for providing object data acquired from objects. The computer system is also provided with lane data related to road items and determined from a data source, wherein the road items and objects are located in the external environment of the main vehicle. The computer system also includes a machine learning algorithm fed with the lane and object data to generate a set of control functions for controlling vehicle movement in such a manner that a classification decoder classifies a predetermined set of maneuvers of the main vehicle based on the lane and object data, and a regression decoder determines the set of control functions related to the kinematics of the main vehicle. Furthermore, the computer system includes a vehicle control unit configured to control the movement of the main vehicle based on the classified maneuvers and the determined control functions.

[0030] That is, the computer system according to this disclosure includes a machine learning algorithm and two decoders for performing the steps as described above for the corresponding method. Therefore, the benefits, advantages, and disclosures described above for various embodiments of the method are also effective for the computer system.

[0031] According to an implementation, the machine learning algorithm may include an encoder used when classifying a predetermined set of maneuvers and determining that set of control functions.

[0032] As mentioned above, public benchmark ground truth data can be used for training classification and regression tasks, where the public benchmark ground truth data is associated with the corresponding input data or labels used for the public encoder. On the other hand, the concept of "teacherforcing" as described above can be used for training classification and regression tasks.

[0033] The sensor system may include vision sensors and / or radar sensors and / or lidar sensors. Therefore, the sensor system can rely on sensors already available in the vehicle. This can reduce the cost of the computer system configured to perform the method.

[0034] Sensor systems can include vision sensors used only to determine lane data and vision sensors used to determine object data. This can reduce the cost of sensor systems, for example, compared to sensor systems that include LiDAR sensors.

[0035] The computer system may also include a processing unit, at least one memory unit, and at least one non-transitory data memory. The non-transitory data memory and / or memory unit may include computer programs for instructing the computer to perform some or all of the steps or aspects of the computer-implemented methods described herein.

[0036] As used herein, the term "processing unit" can refer to an application-specific integrated circuit (ASIC), electronic circuitry, combinational logic circuitry, field-programmable gate array (FPGA), processor (shared, dedicated, or grouped) that executes code, other suitable components that provide the described functionality, or some or all of the foregoing, such as in a system-on-a-chip. A processing unit may include memory (shared, dedicated, or grouped) storing code executed by the processor.

[0037] In another respect, the present invention relates to a vehicle comprising the computer system described above.

[0038] On the other hand, this disclosure relates to a non-transitory computer-readable medium comprising instructions for performing several or all of the steps or aspects of the computer-implemented methods described herein. The computer-readable medium may be configured as: an optical medium, such as an optical disc (CD) or digital versatile disc (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM); a flash memory; and so on. Furthermore, the computer-readable medium may be configured as a data storage unit accessible via a data connection such as an Internet connection. The computer-readable medium may, for example, be an online database or cloud storage.

[0039] This disclosure also relates to a computer program for instructing a computer to perform some or all of the steps or aspects of the computer-implemented method described herein. Attached Figure Description

[0040] This document describes exemplary embodiments and functions of the present disclosure in conjunction with the following schematically illustrated figures:

[0041] Figure 1 It is an illustration of a vehicle including a computer system configured to perform the methods according to this disclosure;

[0042] Figure 2 It is a diagram of a computer system based on this disclosure;

[0043] Figure 3This is an example of using the method according to the invention to control the master vehicle;

[0044] Figure 4 This is another example of using the master vehicle to control it at a later point in time;

[0045] Figure 5 This is a flowchart illustrating a method for controlling the movement of a master vehicle according to various embodiments;

[0046] Figure 6 Vehicle control systems according to various implementation methods; and

[0047] Figure 7 It is a computer system that includes computer hardware components configured to perform steps of a method for controlling the movement of a master vehicle according to various embodiments. Detailed Implementation

[0048] Figure 1 The illustration shows a primary vehicle or vehicle 110 and another vehicle 120 traveling in front of the primary vehicle 110. Vehicles 110 and 120 are currently traveling in lane 130, which is adjacent to another left lane 131. Lane markings 132 for lanes 130 and 131 are also shown, along with right road boundaries 135 and left road boundaries 136, which may include, for example, guardrails.

[0049] Arrow 140 indicates the current direction of travel and current speed of the primary vehicle 110. According to existing technology, the future trajectory of the primary vehicle 110 can be predicted based on given physical properties of the primary vehicle 110 (such as position, speed, and / or acceleration) and given physical properties of other objects in the external environment of the primary vehicle 110 (e.g., including other vehicles 120). The future trajectory of the primary vehicle 110 can also be influenced by its static environment, which may include, for example, the routes of lanes 130, 131, etc. For example, vehicle 110 may perform braking maneuvers to maintain a safe distance from another vehicle 120, or perform a lane-changing maneuver to the left (as indicated by arrow 150).

[0050] Lane markings 132 and other similar road structures (e.g., guardrails, road boundaries 135, 136, obstacles) are the most important road items, influencing road users' behavior regarding the future trajectory of the primary vehicle 110. Such road items 132, 135, 136 can at least partially represent traffic rules and can modulate individual types of driving behavior by road users into system patterns. However, even for relatively simple highways, modeling such behavior by road users in their driving environment can be challenging. Predicting the future movement of the primary vehicle 110 is an essential component of many vehicle safety features and autonomous driving.

[0051] Figure 2 A diagram is shown of a computer system 200 configured to perform a method for controlling the movement of a master vehicle 110. The computer system 200 includes an encoder 210, a classification decoder 220, and a regression decoder 230.

[0052] As input, encoder 210 receives lane data 211 related to road items and object data 212 related to the attributes of objects in the environment of the master vehicle 110. Lane data 211 includes lane markings 132 (see...). Figure 1 The position of the vehicle 110 relative to the main vehicle and / or the position of the road boundaries 135, 136 relative to the main vehicle 110 are described, for example, as a time function.

[0053] Lane data 211 associated with road items such as lane markings 132 and / or road boundaries 135, 136 is determined by a sensor system installed in the main vehicle 110. This sensor system includes vision sensors. Additionally or alternatively, the sensor system may include radar sensors and / or lidar sensors. In an alternative implementation, lane data 211 associated with road items can be obtained from a map, wherein the map data may be stored in a database of the vehicle 110.

[0054] The channel data 211 is represented as a time-dependent third-order polynomial. That is, the y-coordinate of the location of each road item can be represented by a third-order polynomial, and the x-coordinate is the only variable, where the x-coordinate and y-coordinate refer to a vehicle coordinate system with an x-axis along the longitudinal axis of vehicle 110 and a y-axis along the transverse axis of vehicle 110.

[0055] Lane data 211 also includes specific types of road items monitored by the sensor system of vehicle 110 and a predetermined range of validity for lane data 211. In addition to or as an alternative to road boundaries 135, 136, the location of guardrails can be observed and tracked by the sensor system of vehicle 110. The location of guardrails may also be included in lane data 211. However, lane data 211 typically describes the route of lane 130 in which vehicle 110 is currently moving, and optionally describes the routes of adjacent lanes, such as lane 131 (see...). Figure 1 If such an adjacent lane exists.

[0056] Object data 212 is also detected or determined via the sensor system of vehicle 110 (i.e., via vision sensors, radar sensors, and / or lidar sensors). Object data 212 includes at least the location of objects in the external environment of the main vehicle 110. Depending on the specific sensors used to acquire the object data, object data 212 may also include, for example, other vehicles 120 traveling in front of the main vehicle 110 (see [link to relevant documentation]). Figure 1 The speed, heading angle, and other physical properties of the object.

[0057] Computer system 200 feeds lane data 211 and object data 212 to a machine learning algorithm to perform a classification task that categorizes a predetermined set of maneuvers of the main vehicle 110, and a regression task that determines a continuous function related to the kinematics of the main vehicle 110. The machine learning algorithm is implemented as a neural network, which includes an encoder 210, a classification decoder 220, and a regression decoder 230.

[0058] Both the classification and regression tasks use a common encoder 210 as input to receive channel data 211 and object data 212. Furthermore, the classification task is performed by a classification decoder 220, while the regression task is performed by a regression decoder 230. The output of the classification decoder 220 includes a corresponding probability 225 for each of a predetermined set of maneuvers to be applied to control the master vehicle 110.

[0059] In this embodiment, the predetermined set of maneuvers for the main vehicle 110 includes lateral maneuvers such as changing lanes to the left, maintaining lane position, and changing lanes to the right, as well as longitudinal maneuvers such as accelerating, maintaining speed, and decelerating. That is, the classification decoder 220 uses a predetermined set of six maneuvers to perform a classification task, which outputs the corresponding probability of each of these maneuvers as a time function of a predetermined time period associated with the future movement of the vehicle 110.

[0060] When performing a regression task, the regression decoder 230 outputs a time-continuous control function 235 for the kinematic parameters of the master vehicle 110. In this embodiment, these kinematic parameters are the acceleration and steering angle of the master vehicle 110. Therefore, the regression decoder 230 outputs corresponding time-continuous control functions 235 for acceleration and steering angle, respectively, to control the future movement of the master vehicle 110.

[0061] The outputs of the classification decoder 220 (i.e., the probability 225 of vehicle maneuvering) and the outputs of the regression decoder 230 (i.e., a continuous function 235 of future acceleration and steering angle) are provided to the vehicle control unit 240. The vehicle control unit 240 applies the decoder outputs 225 and 235 (i.e., probability 225 and continuous function 235) to control the future trajectory of the master vehicle 110. This will be explained below. Figure 3 and Figure 4 As explained in the context, the classification decoder 220 and the regression decoder 230 provide their respective outputs 225 and 235 within a predetermined future time period (e.g., five to six seconds), enabling the vehicle control unit 240 to adjust the control parameters accordingly, namely, the acceleration and steering angle in this embodiment.

[0062] In detail, the output of the classification decoder 220 (i.e., the probability 225 of the vehicle maneuver) can be used by the regression decoder 230 and / or the vehicle control unit 240 in different ways. For example, the most probable longitudinal and lateral maneuvers with the highest corresponding probabilities 225 at a given time point can be selected and used as inputs to the regression decoder 230. Alternatively, the corresponding most probable maneuvers can be compared with external inputs to the maneuver to be performed (e.g., provided by the vehicle driver) to determine which maneuver can be used as input to the regression decoder 230 and / or the vehicle control unit 240.

[0063] Furthermore, the continuous control function 235 for acceleration and steering angle can optionally be used for visualization 250 of the future trajectory of the main vehicle. For this visualization 250, a kinematic bicycle model for vehicles, well-known in the art, is applied. (See below...) Figure 3 and Figure 4 In the text, such future or predicted trajectories of 350 and 450 are depicted as examples.

[0064] Specifically, acceleration for future time t α The continuous function of the steering angle δ, 235, is represented by the corresponding second-order polynomial as follows:

[0065] α(t)=a α t 2 +b α t+α0 (1)

[0066] δ(t)=a δ t 2 +b δ t+δ0 (2)

[0067] coefficient a α b α and a δ b δ This is the output of the regression decoder 230 used to determine the continuous control function 235. The constants α0 and δ0 represent the acceleration and steering angle at the current time step, therefore they are given parameters and not part of the regression task. Alternatively, a higher-order polynomial can be used to improve the smoothness of the continuous control function 235, as an alternative to a second-order polynomial.

[0068] In another implementation, the future trajectory of the main vehicle 110 can be directly modeled as a fifth-order multi-vertex equation. In this case, x and y represent the longitudinal and lateral positions of the vehicle 110, respectively. The trajectory can be modeled as follows:

[0069] y(x)=a y x 5+b y x 4 +c y x 3 +d y x 2 +e y x+f y (3)

[0070] In this case, the required coefficient a y b y c y d y e y and f y The output of the regression decoder 230 is used to define the continuous control function 235 given by equation (3). In fact, f y It can be ignored and set to 0 because the origin of the coordinate system defining x and y is usually located at the center of gravity (or another reference position) of the main vehicle 110.

[0071] In another implementation, the curvature of the future trajectory can be used in the continuous control function 235 output by the regression decoder 230. The curvature of the future trajectory is defined, for example, as the second derivative of the trajectory given by equation (3), and can be expressed as follows:

[0072] y″(x)=20a y x 3 +12b y x 2 +6c y x+2d y (4)

[0073] To train the neural network that implements machine learning algorithms for performing classification and regression tasks, various kinematic parameters of a real vehicle 110 are used as ground truth. Kinematic parameters (e.g., acceleration and steering angle in this embodiment) are obtained for a specific predetermined time period. Simultaneously, corresponding lane data 211 and object data 212 are acquired and used as inputs to the encoder 210. The outputs 225 of the classification decoder 220 and 235 of the regression decoder 230 are compared with the acquired or recorded data to determine corresponding loss functions 226 and 236 for the classification and regression tasks, respectively, where the data represents the probability 225 of the corresponding vehicle maneuver and the data represents a continuous function 235 for acceleration and steering angle. During training, these loss functions 226 and 236 must be minimized.

[0074] For training the neural network, if data from real vehicles 110 associated with real drivers is used, a multimodal problem must also be addressed. Different drivers have different driving styles, meaning one driver might prefer to follow a slow-moving vehicle, while another might change lanes and perform overtaking maneuvers. This will result in a significant discrepancy between the actual acceleration data acquired or recorded and the steering angle as a function of time. To properly account for this multimodality, a set of vehicle maneuvers consists of only three longitudinal maneuvers and three lateral maneuvers, as described above: acceleration, speed maintenance, and deceleration in the longitudinal direction, and lane changing to the left, maintaining the lane, and changing lanes to the right in the lateral direction. To account for multimodality, the computer system 200 outputs the probabilities 225 of these six vehicle maneuvers, such that for any given time point, the sum of the probabilities of the longitudinal maneuvers and the sum of the probabilities of the lateral maneuvers are each incremented by 1.

[0075] Furthermore, a so-called "teacher forcing" is applied to the training of the neural network in computer system 200. "Teacher forcing" uses specific training data with pre-classified longitudinal and lateral maneuvers, and this pre-classified data for both sets of maneuvers is used as additional input 260 to regression decoder 230. The additional pre-classified data 260 is only provided to regression decoder 230 during the training of the neural network. In this way, the training data or benchmark ground truth has only one modality at a certain point in time. This supports the convergence of the regression task, which leads to the term "teacher forcing" (if this technique is applied). The pre-classified data 260 is only fed to regression decoder 230 so as not to interfere with the classification task performed by classification decoder 220.

[0076] The application of the additional pre-classified data 260 can also be considered as requiring the neural network to perform a certain maneuver and predict the corresponding continuous function 235 accordingly. That is, providing separate and independent maneuver control models as inputs for training the neural network. Alternatively, the predicted future maneuver can be used as the output from the classification decoder 220, i.e., the output provided by the probability 225 of the corresponding maneuver, which is based on the actual recorded data of vehicle 110.

[0077] Figure 3 and Figure 4 Test results of a computer system 200 and a corresponding method for controlling the movement of a master vehicle 110 according to this disclosure are shown. The steering angle and acceleration of the master vehicle 110 are selected as kinematic parameters, and respectively... Figure 3 and Figure 4 The upper part is shown.

[0078] In detail, the steering angle in radians (rad) and acceleration in m / s² are predicted by computer system 200, i.e., by regression decoder 230 (see Figure 2 The prediction is a function of time. For the prediction, a 7-second time interval is used, which runs from -1.4s to 5.6s relative to the current time point, where the current time point is... Figure 3 and Figure 4 The x-axis in the lower part of the figure is depicted as 0.0.

[0079] For two different points in time... Figure 3 and Figure 4 The corresponding upper left plot depicts the baseline ground truth data 310, 410 of the steering angle over time (in seconds), while the smoothed curves 320, 420 represent the corresponding predicted outputs of the regression decoder 230 (i.e., continuous functions 235 of the steering angle). Similarly, the various upper right plots depict the baseline ground truth data 330, 430 of acceleration over time (in seconds) and also the data from the regression decoder 230 (see...). Figure 2 The predictions 340 and 440 are output as a continuous function of acceleration 235.

[0080] Simultaneously, the classification decoder 220 outputs probabilities 225 for the three lateral maneuvers and three longitudinal maneuvers as described above. In this example, probability 225 is provided by the classification decoder 220 as a function of time for the next 5.6 seconds, corresponding to the "future time period," for which the steering angle and acceleration are also predicted by the regression decoder 230.

[0081] exist Figure 3 and Figure 4 The corresponding lower figure depicts an actual driving scenario for the main vehicle 110. Figure 3 In the diagram, the primary vehicle 110 is located in the right lane 130 and is following a slower vehicle 120. Specifically, the distance of the primary vehicle 110 relative to the center of lane 130 is plotted in seconds to predict and track the trajectory of the primary vehicle 110. Figure 3 and 4 The lower part shows the predicted trajectories 350 and 450 for the current position of the main vehicle 110, represented by a time point of 0.0s. Additionally, corresponding additional trajectories 360 and 460 are shown, which are derived from... Figure 3 and Figure 4 The reference true data for steering angle and acceleration shown in the corresponding upper part are derived, that is, derived from the steering angle data 310, 410 and the acceleration data 330, 430. Conversely, the predicted trajectories 350, 450 are based on the smoothing functions 320, 420 of steering angle and smoothing functions 340, 440 of acceleration output by the regression decoder 230.

[0082] for Figure 3As shown, the classification decoder 220 predicts a probability of almost 1 for lane keeping as a lateral maneuver for the current time point (0.0s) and the short period of approximately 1.5s thereafter. Therefore, since the sensor system of the main vehicle 110 detects the vehicle 120 ahead at a lower speed, the classification decoder 220 predicts a probability of almost 1 for deceleration as a longitudinal maneuver. For this embodiment, the most probable lateral maneuver and the most probable longitudinal maneuver are fed as additional inputs to the regression decoder 230. This additional input reflects driver behavior already “recorded” during the training phase of the computer system 200. Therefore, the regression decoder 230 predicts a steering angle and negative acceleration of almost 0 for the current time point and the short period thereafter.

[0083] However, as Figure 3 As shown, approximately 1.5–2.0 seconds after the current time point 0.0, the prediction of the classification decoder 220 changes drastically for lateral maneuvers, as the probability of a lane change to the left increases significantly, for example, to approximately 0.9. Simultaneously, the prediction for longitudinal maneuvers changes from deceleration to acceleration. Therefore, the prediction of the regression decoder 230 changes for steering angle and acceleration, as shown in… Figure 3 The upper part is visible.

[0084] for Figure 4 In the scenario shown, the primary vehicle 110 is positioned closer to another slowly moving vehicle 120. Therefore, the primary vehicle 110 initiates a lane change to the left lane 131 and a request for acceleration to overtake. Accordingly, as... Figure 3 As shown, the predicted steering angle of 420° has changed relative to the predicted 320°, thus predicting a strong positive acceleration of 440°.

[0085] Regarding Figure 4 The lower part of the graph, showing the current time point as 0.0, predicts that the classification decoder 220 has a probability of changing lanes to the left 1.5 seconds later, and a probability of acceleration as a longitudinal maneuver that is almost 1. Thereafter, the probability of changing lanes to the left decreases sharply, making the probability of maintaining the lane almost 1 approximately 3 seconds after the current time point. Since vehicle 110 is still ahead of another vehicle 120, the acceleration probability remains close to 1 throughout the entire prediction period of approximately 5.6 seconds. Therefore, the prediction of the regression decoder 230 remains constant for acceleration 440, while the prediction for steering angle 420 changes based on the predicted lane-keeping maneuver.

[0086] like Figure 3 and Figure 4As can be seen, the computer system 200 and method of this disclosure can predict appropriate behavior regarding steering angle, acceleration, and the probability of longitudinal and lateral maneuvers. This results in a reasonable predicted trajectory 350, 450 for the main vehicle 110. It should be noted that the deviation between the predicted trajectories 350, 450 and the trajectories 360, 460 based on the reference ground truth data is partly caused by the noise behavior of the reference ground truth data, such as... Figure 3 and Figure 4 This is seen in curves 310, 410, 330, and 430 in the upper part of the graph. Furthermore, the predictions of the neural network, including encoder 210 and two decoders 220 and 230, are affected by uncertainty, causing the ground truth to not be accurately reproduced. Therefore, the deviation between the predicted trajectories 350 and 450 and the "ground truth trajectories" 360 and 460 is expected to occur after a period of more than approximately 70 seconds.

[0087] Figure 5 A flowchart 500 illustrating a method for controlling the movement of a master vehicle according to various embodiments is shown. At 502, lane data related to road items can be determined from a data source, and object data acquired from objects can be determined via a sensor system. Road items and objects can be located in the external environment of the master vehicle. At 504, a machine learning algorithm can be fed the lane data and object data to generate a set of control functions for controlling the movement of the master vehicle by performing the following steps: classifying a predetermined set of maneuvers of the master vehicle based on the lane data and the object data, and determining a set of control functions related to the kinematics of the master vehicle. At 506, the movement of the master vehicle can be controlled based on the classified maneuvers and the determined control functions.

[0088] According to various implementations, the step of determining the control function may include predicting control parameters for the acceleration and steering angle of the master vehicle.

[0089] According to various implementation methods, the step of determining the control function may include parameters for predicting the planned curvature of the master vehicle.

[0090] According to various implementation methods, the data source for determining lane data may include sensor systems and / or predetermined maps.

[0091] According to various implementations, this set of control functions includes functions that are continuous in time.

[0092] According to various implementation methods, in order to train machine learning algorithms, public benchmark truth data can be applied to classify a predetermined set of maneuvers and determine control functions.

[0093] According to various implementation methods, public benchmark truth data can be obtained as data of the main vehicle recorded during a predetermined time period.

[0094] According to various implementation methods, additional pre-classified data can be generated by pre-classifying a predetermined set of maneuvers using data recorded by the main vehicle, and the additional pre-classified data can be applied to the training of determining the control function.

[0095] Depending on the implementation method, road projects may include lane markings, road boundaries, and / or guardrails.

[0096] According to various implementation methods, object data may include predetermined physical attributes of the object.

[0097] According to various implementations, a predetermined set of maneuvers for the main vehicle may include lateral maneuvers such as changing lanes to the left, maintaining lane position, and changing lanes to the right, as well as longitudinal maneuvers such as accelerating, maintaining speed, and decelerating.

[0098] Each of steps 502, 504, and 506, as well as the other steps described above, can be performed by computer hardware components.

[0099] Figure 6 A vehicle control system 600 according to various embodiments is shown. The vehicle control system 600 may include data determination circuitry 602, circuitry 604 for machine learning algorithms, and motion control circuitry 606.

[0100] The data determination circuit 602 can be configured to determine lane data related to a road item from a data source and to determine object data acquired from the object via a sensor system. The road item and the object can be located in the external environment of the main vehicle.

[0101] Circuit 604 for the machine learning algorithm can be configured to feed lane data and object data to the machine learning algorithm to generate a set of control functions for controlling the movement of a master vehicle by performing the following steps: classifying a predetermined set of maneuvers of the master vehicle based on the lane data and the object data, and determining a set of control functions related to the kinematics of the master vehicle. To perform the above steps of classifying the predetermined maneuvers and determining the control functions, circuit 604 may include: encoder 25 (see also...) Figure 2 ), which can be used when performing these two steps; decoder 27 for classification, which can be used when performing only the classification step; and decoder 29 for regression, which can be used when performing only the step of determining the control function.

[0102] The movement control circuit 606 can be configured to control the movement of the master vehicle based on the classified maneuver and the determined control function.

[0103] The data determination circuit 602, the circuit 604 for the machine learning algorithm, and the motion control circuit 606 can be connected to each other, for example, via an electrical connection 608 such as a cable or a computer bus, or via any other suitable electrical connection, to exchange electrical signals.

[0104] "Circuit" can be understood as any type of logical implementation entity, which can be a dedicated circuit or a processor that executes a program stored in memory, firmware, or any combination thereof.

[0105] Figure 7 A computer system 700 with multiple computer hardware components, according to various embodiments, is shown. These multiple computer hardware components are configured to perform steps of a computer-implemented method for determining the attributes of an object at a predetermined point. The computer system 700 may include a processor 702, a memory 704, and a non-transitory data storage device 706.

[0106] Processor 702 can execute instructions provided in memory 704. Non-transitory data storage device 706 can store computer programs, including instructions that can be transferred to memory 704 and then executed by processor 702.

[0107] The processor 702, memory 704, and non-transitory data storage device 706 may be interconnected, for example, via electrical connection 708 (e.g., cable or computer bus) or via any other suitable electrical connection to exchange electrical signals. Thus, the processor 702, memory 704, and non-transitory data storage device 706 may represent circuitry for the machine learning algorithm 604 as described above (i.e., including a common encoder 25), a decoder 27 for classification, and a decoder 29 for regression, for respectively performing the steps of classifying predetermined movements and determining control functions.

[0108] The terms “connection” or “link” are intended to include direct “connection” (e.g., via a physical link) or direct “link” as well as indirect “connection” or indirect “link” (e.g., via a logical link).

[0109] It should be understood that the content already described for one of the above methods can be similarly applied to the vehicle control system 600 and / or the computer system 700.

[0110] List of reference numerals

[0111] 110 Main vehicle

[0112] 120 Another vehicle moving forward

[0113] 130 Current Lane

[0114] 131 Adjacent left lane

[0115] Lane 132 markings

[0116] Road boundaries of 135 and 136

[0117] 140 Arrow indicating the current speed of the main vehicle

[0118] 150 Arrow indicating a left lane change

[0119] 200 computer systems

[0120] 210 encoder

[0121] 211 Lane Data

[0122] 212 Object Data

[0123] 220 Classification Decoder

[0124] 225. Probability of vehicle movement

[0125] Loss function of 226 classifier decoder

[0126] 230 Regression Decoder

[0127] 235 Control Function

[0128] 236 Loss Function of Regression Decoder

[0129] 240 Vehicle Control Unit

[0130] 250 Trajectory Visualization

[0131] 260 pre-classified data

[0132] 310° steering angle reference true value

[0133] 320 Predicted steering angle

[0134] 330 Acceleration Baseline Truth Value

[0135] 340 Prediction Acceleration

[0136] 350 Predicted Trajectory

[0137] 360-degree trajectory based on baseline truth value

[0138] 410° steering angle reference true value

[0139] 420 Predicted steering angle

[0140] 430 Accelerated Benchmark Truth

[0141] 440 Predicted Acceleration

[0142] 450 Predicted Trajectory

[0143] 460 Trajectory Based on Benchmark Truth

[0144] 500 shows a flowchart of a method for controlling the movement of a master vehicle according to various embodiments.

[0145] 502 The steps of determining lane data relevant to the road project from the data source and determining object data acquired via the sensor system.

[0146] 504 The step of feeding lane data and object data to a machine learning algorithm to generate a set of control functions for controlling the movement of a master vehicle by performing the following steps: classifying a predetermined set of maneuvers of the master vehicle based on the lane data and object data, and determining a set of control functions related to the kinematics of the master vehicle.

[0147] 506 Steps for controlling the movement of the master vehicle based on classification-based maneuvers and deterministic control functions

[0148] 600 Vehicle Control System

[0149] 602 Data Determination Circuit

[0150] 604 Machine Learning Algorithm Circuit

[0151] 606 Motion Control Circuit

[0152] 608 connection

[0153] 700 Computer systems according to various implementation methods

[0154] 702 processor

[0155] 704 memory

[0156] 706 Non-transitory data storage

[0157] 708 connection

Claims

1. A computer-implemented method for controlling the movement of a master vehicle (110), the method comprising the following steps: Lane data (211) related to road items (132, 135, 136) is determined from the data source, and object data (212) is determined from the object (120) via a sensor system, wherein the road items (132, 135, 136) and the object (120) are located in the external environment of the master vehicle (110); The lane data (211) and the object data (212) are fed to a machine learning algorithm to generate a set of control functions (235) for controlling the movement of the master vehicle (110) by performing the following steps: Based on the lane data (211) and the object data (212), a predetermined set of maneuvers of the main vehicle (110) is classified; and Determine a set of control functions (235) related to the kinematics of the main vehicle (110); and The movement of the master vehicle (110) is controlled based on the classified maneuvers and the determined control function (235). In order to train the machine learning algorithm, public benchmark ground truth data is used to classify the predetermined set of maneuvers and determine the control function (235). The public benchmark truth data is obtained as data recorded for the master vehicle (110) during a predetermined time period. In this process, additional pre-classified data (260) is generated by pre-classifying the predetermined set of maneuvers using data recorded for the main vehicle (110), and The additional pre-classified data (260) is used to train the determination of the control function (235).

2. The method according to claim 1, wherein, The steps of determining the set of control functions (235) include: predicting control parameters for the acceleration and steering angle of the master vehicle (110).

3. The method according to claim 1, wherein, The steps of determining the set of control functions (235) include: predicting the parameters of the planned curvature of the master vehicle.

4. The method according to any one of claims 1 to 3, wherein, The data source used to determine the lane data (211) includes the sensor system and / or a predetermined map.

5. The method according to any one of claims 1 to 3, wherein, The set of control functions (235) includes continuous functions in time.

6. The method according to any one of claims 1 to 3, wherein, The road items (132, 135, 136) include lane markings (132), road boundaries (135, 136) and / or guardrails.

7. The method according to any one of claims 1 to 3, wherein, The object data (212) includes the predetermined physical attributes of the object (120).

8. The method according to any one of claims 1 to 3, wherein, The predetermined set of maneuvers for the main vehicle (110) includes: lane changing to the left, lane keeping, and lane changing to the right as lateral maneuvers; and acceleration, speed holding, and deceleration as longitudinal maneuvers.

9. A computer system (200, 700) configured to perform a computer-implemented method according to any one of claims 1 to 8. in, The computer systems (200, 700) are communicatively connected to a sensor system for providing object data (212) acquired from the object (120). The computer systems (200, 700) are provided with lane data (211) related to the road projects (132, 135, 136) and determined from the data source. The road projects (132, 135, 136) and the object (120) are located in the external environment of the main vehicle (110). The computer system (200, 700) includes a machine learning algorithm that is fed the lane data (211) and the object data (212) to generate a set of control functions (235) for controlling the movement of the master vehicle (110) by: The classification decoder (220) classifies the predetermined set of maneuvers of the master vehicle (110) based on the lane data (211) and the object data (212); and The regression decoder (230) determines the set of control functions (235) related to the kinematics of the master vehicle (110). The classification decoder (220) and the regression decoder (230) are separate decoders for the classification task and the regression task, respectively, and only provide additional pre-classified data (260) to the regression task. The computer system (200, 700) includes a vehicle control unit (240) configured to control the movement of the master vehicle (110) based on the classified maneuver and the determined control function (235).

10. The computer system (200, 700) according to claim 9, wherein, The machine learning algorithm includes an encoder (210) used to classify the predetermined set of maneuvers and determine the set of control functions (235).

11. A vehicle comprising a computer system (200, 700) according to claim 9 or 10.

12. A non-transitory computer-readable medium comprising instructions for performing a computer-implemented method according to any one of claims 1 to 8.

Images