Vehicle control system

Abstract

Regarding control parameters related to the vehicle's suspension system, even if there are discrepancies between the simulation and the actual environment, control parameters suitable for the actual environment can be obtained. [Solution] The vehicle control device of the embodiment includes a vehicle control unit that transmits target control amount information to a suspension system that performs vehicle control to mitigate the impact received by the vehicle body from the road surface, and a learning unit that uses simulation processing to perform simulations for each of the multiple patterns of road surface shape features using Q learning, and obtains Q values, which are evaluation values ​​that are larger the smaller the impact received by the vehicle body from the road surface, and creates a Q table, and based on the Q table, creates Q value relationship information that holds the top N Q values ​​with large Q values ​​and the set of control parameters corresponding to those Q values ​​for each of the multiple patterns of road surface shape features.

Description

Technical Field

[0006] , ,

[0005] , , , ,

[0001] The present invention relates to a vehicle control device.

Background Art

Means for Solving the Problem

[0007] The present invention is a vehicle control device mounted on a vehicle, comprising: a vehicle control unit that transmits target control amount information to a suspension system that performs vehicle control for mitigating an impact received by the vehicle body of the vehicle from a road surface; and using simulation processing, by Q-learning which is reinforcement learning, for each of one or more road surface shape feature amounts of a plurality of patterns, when simulations are performed for each of M patterns (M: a predetermined integer of 3 or more) of one or more control parameters for the vehicle control unit, a Q value which is an evaluation value having a larger value as the impact received by the vehicle body from the road surface is smaller is obtained to create a Q table, and based on the Q table, for each of the road surface shape feature amounts of the plurality of patterns, Q value relationship information is created which holds a set of the top N (N: an integer satisfying 2 ≤ N < M) Q values having large Q values and the control parameters corresponding to those Q values, when the vehicle travels on an actual road surface, using the Q value relationship information and the obtained road surface shape feature amount, the control parameter corresponding to any of the Q values corresponding to the road surface shape feature amount in the Q value relationship information is transmitted to the vehicle control unit to cause vehicle control to be performed, and an update process is performed to obtain the Q value in that case and update the Q value in the Q value relationship information, the update process is repeated, and when selecting the control parameter for the obtained road surface shape feature amount, in the Q value relationship information, when all N Q values for the road surface shape feature amount have been updated, the control parameter corresponding to the largest Q value among the held Q values is selected and transmitted to the vehicle control unit, and a learning unit.

Effect of the Invention

[0008] According to the vehicle control device of the present invention, even if there is a difference between the simulation and the actual environment regarding the control parameters related to the control of the suspension system, it is possible to obtain control parameters that are suitable for the actual environment. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a perspective view showing a transparent view of a portion of the passenger compartment of the vehicle according to the embodiment. [Figure 2] Figure 2 is a plan view showing the vehicle body of the embodiment as viewed through the lens. [Figure 3] Figure 3 is a block diagram showing the configuration of the vehicle control system in the embodiment of the vehicle. [Figure 4] Figure 4 is an explanatory diagram of the simulation process by the vehicle control device of the embodiment. [Figure 5] Figure 5 is an explanatory diagram illustrating the processing performed in a real environment by the vehicle control device of the embodiment. [Figure 6] Figure 6 shows an example of a Q-table and other elements in an embodiment. [Figure 7] Figure 7 shows an example of Q-value relationship information in the embodiment. [Figure 8] Figure 8 shows an example of positional relationship information in an embodiment, which includes a set of road surface shape features and location information for each point. [Figure 9] Figure 9 is an explanatory diagram illustrating the updating of positional relationship information in the embodiment. [Figure 10] Figure 10 is a flowchart showing the process of creating Q-value relationship information by the vehicle control device of the embodiment. [Figure 11] Figure 11 is a flowchart showing the processing in a real environment by the vehicle control device of the embodiment. [Modes for carrying out the invention]

[0010] Illustrative embodiments of the present invention are disclosed below. The configurations of the embodiments shown below, as well as the operations, results, and effects brought about by such configurations, are examples only. The present invention can be realized by configurations other than those disclosed in the following embodiments, and it is possible to obtain at least one of the various effects based on the basic configuration or derived effects. Furthermore, in order to simplify the explanation below, the descriptions of "data" and "information" may be omitted, for example, by simply referring to "acceleration" as "acceleration data" or "acceleration information."

[0011] Vehicle 1 in this embodiment may be, for example, an automobile powered by an internal combustion engine (not shown), i.e., an internal combustion engine automobile; an automobile powered by an electric motor (not shown), i.e., an electric vehicle or a fuel cell vehicle; a hybrid automobile powered by both; or an automobile equipped with other power sources. Furthermore, Vehicle 1 can be equipped with various transmissions and various devices, such as systems and components, necessary for driving the internal combustion engine or electric motor. In addition, the type, number, and layout of the devices related to driving the wheels 3 in Vehicle 1 can be set in various ways.

[0012] Figure 1 is a perspective view showing a part of the passenger compartment of the vehicle 1 of the embodiment. Figure 2 is a plan view showing the vehicle body of the embodiment as seen through. Figure 3 is a block diagram showing the configuration of the vehicle control system of the vehicle of the embodiment.

[0013] First, an example of the configuration of the vehicle 1 of this embodiment will be described using Figures 1 to 3. As illustrated in Figure 1, the vehicle body 2 constitutes a passenger compartment 2a where an occupant (not shown) sits. Inside the passenger compartment 2a, facing the driver's seat 2b, are the steering unit 4, acceleration control unit 5, braking control unit 6, gear shift control unit 7, etc.

[0014] The steering unit 4 is, for example, a steering wheel protruding from the dashboard 24. The acceleration operation unit 5 is, for example, an accelerator pedal located under the driver's feet. The braking operation unit 6 is, for example, a brake pedal located under the driver's feet. The shift operation unit 7 is, for example, a shift lever protruding from the center console. Note that the steering unit 4, the acceleration operation unit 5, the braking operation unit 6, the shift operation unit 7, etc. are not limited to these.

[0015] Also, in the passenger compartment 2a, a display device 8 as a display output unit and an audio output device 9 as an audio output unit are provided. The display device 8 is, for example, an LCD (Liquid Crystal Display), an OELD (Organic Electroluminescent Display), etc. The audio output device 9 is, for example, a speaker. Also, the display device 8 is covered with a transparent operation input unit 10 such as a touch panel. The passenger can visually recognize the image displayed on the display screen of the display device 8 through the operation input unit 10. Also, the passenger can execute an operation input by touching, pressing, or moving the operation input unit 10 with a finger or the like at a position corresponding to the image displayed on the display screen of the display device 8.

[0016] These display device 8, audio output device 9, operation input unit 10, etc. are provided, for example, in a monitor device 11 located at the center in the vehicle width direction, that is, the left - right direction, of the dashboard 24. The monitor device 11 can have operation input units not shown, such as switches, dials, joysticks, push buttons, etc. Also, an audio output device not shown can be provided at another position in the passenger compartment 2a different from the monitor device 11, and audio can be output from the audio output device 9 of the monitor device 11 and the other audio output device. Note that the monitor device 11 can be used in combination with, for example, a navigation system or an audio system. Also, in the passenger compartment 2a, a display device 12 different from the display device 8 is provided.

[0017] Also, as illustrated in FIG. 2, the vehicle 1 is, for example, a four-wheel automobile and has two front wheels 3F on the left and right and two rear wheels 3R on the left and right. All of these four wheels 3 can be configured to be steerable. As illustrated in FIG. 3, the vehicle 1 has a steering system 13 that steers at least two wheels 3.

[0018] As illustrated in FIG. 3, the steering system 13 has actuators 101, 104 and a torque sensor 13b. The steering system 13 is electrically controlled by an ECU 14 (Electronic Control Unit) or the like to operate the actuators 101, 104. Here, as shown in FIG. 2, the actuator 101 is connected to the front wheels 3F and is a front steering actuator for steering the front wheels 3F. Also, the actuator 104 is connected to the rear wheels 3R and is a rear steering actuator for steering the rear wheels 3R.

[0019] The steering system 13 is, for example, an electric power steering system, a SBW (Steer By Wire) system, or the like. The steering system 13 adds torque, that is, assist torque, to the steering part 4 by the actuators 101, 104 to supplement the steering force, or steers the wheels 3 by the actuators 101, 104. In this case, the actuators 101, 104 may steer one wheel 3 or may steer a plurality of wheels 3. Also, the torque sensor 13b detects, for example, the torque applied by the driver to the steering part 4.

[0020] Furthermore, as illustrated in Figure 3, the vehicle body 2 is equipped with multiple imaging units 15, for example, eight imaging units 15a to 15f. Each imaging unit 15 is a digital camera incorporating an image sensor such as a CCD (Charge Coupled Device) or CIS (CMOS Image Sensor). Each imaging unit 15 can output video data at a predetermined frame rate. Each imaging unit 15 has either a wide-angle lens or a fisheye lens and can capture a range of, for example, 140° to 190° in the horizontal direction. The optical axis of each imaging unit 15 is set to point diagonally downward. Therefore, the imaging unit 15 sequentially captures the external environment around the vehicle body 2, including the road surface on which the vehicle 1 can move and the area where the vehicle 1 can park, and outputs it as captured images (image data).

[0021] The imaging unit 15a is located, for example, at the rear end 2e of the vehicle body 2 and is provided on the lower wall of the rear trunk door 2h. The imaging unit 15ba is located, for example, at the right side of the vehicle body 2, i.e., at the right end in the vehicle width direction, and is provided on the front side of the door mirror 2g, which is a right-side projection. The imaging unit 15bb is located, for example, at the right side of the vehicle body 2, i.e., at the right end in the vehicle width direction, and is provided on the rear side of the door mirror 2g, which is a right-side projection.

[0022] The imaging unit 15c is located, for example, at the front of the vehicle body 2, that is, at the front end in the vehicle's longitudinal direction, and is provided on the front bumper or the like. The imaging unit 15da is located, for example, at the left side of the vehicle body 2, that is, at the left end in the vehicle's width direction, and is provided on the front side of the door mirror 2g, which is a protruding part on the left side. The imaging unit 15db is located, for example, at the left side of the vehicle body 2, that is, at the left end in the vehicle's width direction, and is provided on the rear side of the door mirror 2g, which is a protruding part on the left side.

[0023] The imaging unit 15e is located, for example, on the right side of the vehicle body 2, that is, at the right end in the vehicle width direction, and is provided near the right-side door. The imaging unit 15f is located, for example, on the left side of the vehicle body 2, that is, at the left end in the vehicle width direction, and is provided near the left-side door.

[0024] The ECU 14 performs calculations and image processing based on image data obtained from multiple imaging units 15, enabling it to generate images with a wider field of view or a virtual overhead view image of the vehicle 1 as seen from above. The overhead view image can also be referred to as a planar image. In this embodiment, each imaging unit 15 captures an image of the area around the vehicle 1.

[0025] Furthermore, as illustrated in Figure 1, the vehicle body 2 is equipped with multiple distance measuring units 16 and 17, such as four distance measuring units 16a to 16d and eight distance measuring units 17a to 17h. The distance measuring units 16 and 17 are, for example, sonars that emit ultrasonic waves and capture the reflected waves. Sonars can also be called sonar sensors or ultrasonic detectors. The ECU 14 can measure the presence or absence of objects such as obstacles located around the vehicle 1 and the distance to such objects based on the detection results of the distance measuring units 16 and 17. In other words, the distance measuring units 16 and 17 are examples of detection units that detect objects. The distance measuring unit 17 can be used, for example, to detect objects at a relatively short distance, while the distance measuring unit 16 can be used, for example, to detect objects at a relatively long distance that are further away than the distance measuring unit 17. In addition, the distance measuring unit 17 can be used, for example, to detect objects in front of and behind the vehicle 1, while the distance measuring unit 16 can be used to detect objects to the sides of the vehicle 1.

[0026] Furthermore, as illustrated in Figure 3, the vehicle control system 100 includes the ECU 14, monitoring device 11, steering system 13, distance measuring units 16 and 17, as well as the brake system 18, suspension system 30, steering angle sensor 19, accelerator sensor 20, shift sensor 21, wheel speed sensor 22, acceleration sensor 25, vehicle height sensor 26, spring sensor 27, actuator 107, etc., all of which are electrically connected via the in-vehicle network 23, which serves as an telecommunications line. The in-vehicle network 23 is configured, for example, as a CAN (controller area network). The ECU 14 is an example of an estimation device and vehicle control device.

[0027] The ECU 14 controls the steering system 13, brake system 18, suspension system 30, actuator 107, etc., by sending control signals via the in-vehicle network 23. The ECU 14 also receives detection results from the torque sensor 13b, brake sensor 18b, steering angle sensor 19, distance measuring unit 16, distance measuring unit 17, accelerator sensor 20, shift sensor 21, wheel speed sensor 22, acceleration sensor 25, vehicle height sensor 26, etc., as well as operation signals from the operation input unit 10, etc., via the in-vehicle network 23.

[0028] The ECU14 includes, for example, a CPU (Central Processing Unit) 14a, a ROM (Read Only Memory) 14b, a RAM (Random Access Memory) 14c, a display control unit 14d, an audio control unit 14e, an SSD (Solid State Drive, flash memory) 14f, and the like.

[0029] The CPU 14a performs various calculations and controls, such as image processing related to images displayed on the display devices 8 and 12, determining the target position of the vehicle 1, calculating the movement path of the vehicle 1, determining whether or not there is interference with an object, automatic control of the vehicle 1, release of automatic control, damping control, spring constant switching control, steering control, stabilizer control, and driving force control. The CPU 14a reads a program installed and stored in a non-volatile storage device such as the ROM 14b and performs calculations according to the program.

[0030] RAM 14c temporarily stores various data used in calculations performed by CPU 14a. The display control unit 14d primarily performs image processing using image data obtained by the imaging unit 15 and image data synthesis displayed on the display device 8, among the calculation processes performed by ECU 14. The audio control unit 14e primarily processes audio data output by the audio output device 9, among the calculation processes performed by ECU 14. SSD 14f is a rewritable non-volatile storage unit that can store data even when the power to ECU 14 is turned off. Note that CPU 14a, ROM 14b, RAM 14c, etc., can be integrated within the same package. Also, ECU 14 may use other logic processors such as DSP (Digital Signal Processor) or logic circuits instead of CPU 14a. Furthermore, HDD (Hard Disk Drive) may be provided instead of SSD 14f, and SSD 14f and HDD may be provided separately from ECU 14.

[0031] The location information sensor 31 is a sensor that acquires the current position of vehicle 1, and is, for example, a GPS (Global Positioning System) receiver. The location information sensor 31 sends the acquired current position as location information to the CPU 14a.

[0032] The brake system 18 includes, for example, an ABS (Anti-lock Brake System) to suppress brake lock, an Electronic Stability Control (ESC) to suppress skidding of the vehicle 1 during cornering, an electric brake system to enhance braking force (perform brake assist), and a BBW (Brake By Wire). The brake system 18 applies braking force to the wheels 3 and thus to the vehicle 1 via an actuator 18a. The brake system 18 can also detect signs of brake lock, wheel spinning, skidding, etc., from the rotational difference between the left and right wheels 3, and perform various controls. The brake sensor 18b is, for example, a sensor that detects the position of the movable part of the braking operation unit 6. The brake sensor 18b can detect the position of the brake pedal as a movable part. The brake sensor 18b includes a displacement sensor.

[0033] Furthermore, actuator 107 is a rear drive force control actuator as shown in Figure 2, and is electrically controlled by the ECU 14, etc., to control the drive force of the rear wheel 3R.

[0034] The steering angle sensor 19 is a sensor that detects the amount of steering of the steering unit 4, such as a steering wheel. The steering angle sensor 19 is constructed using, for example, a Hall element. The ECU 14 obtains the amount of steering of the steering unit 4 by the driver, the amount of steering of each wheel 3 during automatic steering, etc. from the steering angle sensor 19 and performs various controls. The steering angle sensor 19 detects the rotation angle of the rotating part included in the steering unit 4.

[0035] The accelerator sensor 20 is, for example, a sensor that detects the position of a movable part of the acceleration control unit 5. The accelerator sensor 20 can detect the position of the accelerator pedal as a movable part. The accelerator sensor 20 includes a displacement sensor.

[0036] The shift sensor 21 is, for example, a sensor that detects the position of a movable part of the gear shift operation unit 7. The shift sensor 21 can detect the position of a movable part such as a lever, arm, or button. The shift sensor 21 may include a displacement sensor or may be configured as a switch.

[0037] The wheel speed sensor 22 is a sensor that detects the amount of rotation of the wheel 3 and the number of rotations per unit time. The wheel speed sensor 22 outputs the number of wheel speed pulses indicating the detected rotation speed as a sensor value. The wheel speed sensor 22 may be constructed using, for example, a Hall element. The ECU 14 calculates the amount of movement of the vehicle 1 based on the sensor value obtained from the wheel speed sensor 22 and executes various controls. In some cases, the wheel speed sensor 22 may be provided in the brake system 18. In that case, the ECU 14 obtains the detection result of the wheel speed sensor 22 via the brake system 18. The vehicle height sensor 26 is a sensor that detects the stroke displacement of each wheel.

[0038] The suspension system 30 is a vehicle control system that mitigates the impact that the vehicle body 1 receives from the road surface, and is positioned between the vehicle body 2 and the wheels 3 of the vehicle 1. The suspension system 30 includes a spring that absorbs vibrations of the vehicle 1 caused by impacts from the road surface, and a variable damping damper that dampens the vibrations of the spring and can change the damping force of the spring's vibrations. In this embodiment, the suspension system 30 works in cooperation with the ECU 14 to control a damping force adjustment device such as a solenoid actuator to change the damping force of the variable damping damper. As a result, the suspension system 30 realizes a damping force control system that dampens vertical, lateral, and longitudinal vibrations of the vehicle body caused by impacts from the road surface.

[0039] Acceleration sensors 25 are provided near the vehicle body 2 side of the suspension system 30 (hereinafter also referred to as the "sprung mass") and near the vehicle 1 side of the wheel 3 side of the suspension system 30 (hereinafter also referred to as the "unsprung mass"). The acceleration sensors 25 include vertical acceleration sensors (i.e., sprung mass G sensor and unsprung mass G sensor) that detect and output the vertical acceleration of the vehicle body 2, longitudinal acceleration sensors that detect and output the longitudinal acceleration of the vehicle body 2 (vehicle 1), and lateral acceleration sensors that detect and output the lateral acceleration, which is the lateral (widthwise) acceleration of the vehicle body 2 (vehicle 1).

[0040] Here, the vertical acceleration sensors consist of a sprung mass acceleration sensor (hereinafter also referred to as the "sprung mass G sensor") that detects sprung mass acceleration (hereinafter also referred to as "sprung mass G"), and an unsprung mass acceleration sensor (hereinafter also referred to as the "unsprung mass G sensor") that detects unsprung mass acceleration (hereinafter also referred to as "unsprung mass G").

[0041] The suspension system 30 is electrically controlled by the ECU 14, etc., to operate actuators 102, 103, 105, and 106 for damping force control and stabilizer control.

[0042] Here, actuator 102 is a front active stabilizer actuator for controlling the stabilizer on the front wheel 3F side, as shown in Figure 2, and is provided on the suspension on the rear wheel 3R side, as shown in Figure 2, and is a rear active stabilizer actuator for controlling the stabilizer on the rear wheel 3R side.

[0043] Furthermore, as shown in Figure 2, actuator 103 is provided on the suspension on the front wheel 3F side and is a front damping force control actuator for controlling the damping force on the front wheel 3F side. Actuator 106 is provided on the suspension on the rear wheel 3R side and is a rear damping force control actuator for controlling the damping force on the rear wheel 3R side.

[0044] The configurations, arrangements, and electrical connection types of the various sensors and actuators described above are examples only and can be configured (changed) in various ways.

[0045] Next, the vehicle control device implemented by the ECU 14 will be described. Hereinafter, the ECU 14 may be referred to as the vehicle control device 14. In this specification, the term "road surface shape" refers not only to the shape of the road surface itself, but also to all elements of the road surface that are related to the impact on the vehicle body, such as the condition of the road surface (dry, wet, covered with snow, covered with ice, etc.) and the material of the road.

[0046] Figure 4 is an explanatory diagram of the simulation process by the vehicle control device 14 of the embodiment. As shown in Figure 4, the vehicle control device 14 mainly comprises a vehicle behavior data processing unit 141, a road surface shape output unit 142, a reward calculation unit 143, a learning unit 144, and a vehicle control unit 145 as functional units.

[0047] These functional units are realized by the CPU 14a reading and executing a program stored in the memory unit 50 (Figure 3). Here, the memory unit 50 includes ROM 14b, RAM 14c, and SSD 14f. That is, the program may include, for example, modules corresponding to each functional unit of the vehicle control device 14 shown in Figure 4. It is also possible to realize each functional unit module using independent hardware such as circuits including ASICs (Application Specific Integrated Circuits).

[0048] Here, the vehicle behavior sensors are the sprung mass G sensor, wheel speed sensor 22, and vehicle height sensor 26 within the acceleration sensor 25 in the simulation. The vehicle behavior data output by the vehicle behavior sensors is data related to the behavior of vehicle 1, and includes, for example, sprung mass acceleration (sprung mass G), vehicle height, vehicle speed, sprung mass velocity, vehicle height change velocity, and unsprung mass velocity.

[0049] The vehicle behavior data processing unit 141 acquires vehicle behavior data output by the vehicle behavior sensor and calculates the vehicle behavior (including sprung mass G).

[0050] For example, the vehicle behavior data processing unit 141 receives the sprung acceleration (sprung G), which is the detection signal from the sprung G sensor in the acceleration sensor 25, and outputs the sprung acceleration as vehicle behavior data. The vehicle behavior data processing unit 141 also receives the wheel speed, which is the detection signal from the wheel speed sensor 22, calculates the vehicle speed, which is the speed of vehicle 1, from the wheel speed, and outputs the calculated vehicle speed as vehicle behavior data. The vehicle behavior data processing unit 141 also receives the vehicle height, which is the detection signal from the vehicle height sensor 26, and outputs the vehicle height as vehicle behavior data.

[0051] Furthermore, the vehicle behavior data processing unit 141 integrates the sprung mass acceleration detected by the sprung mass G sensor to obtain the sprung mass velocity and outputs the sprung mass velocity as vehicle behavior data. The vehicle behavior data processing unit 141 also differentiates the vehicle height sensor data detected by the vehicle height sensor to obtain the vehicle height change rate and outputs the vehicle height change rate as vehicle behavior data. In addition, the vehicle behavior data processing unit 141 obtains the unsprung mass velocity from the difference between the sprung mass velocity and the vehicle height change rate and outputs the unsprung mass velocity as vehicle behavior data.

[0052] In other words, the vehicle behavior data processing unit 141 outputs vehicle behavior data such as sprung mass acceleration (sprung mass G), vehicle height, vehicle speed, sprung mass velocity, vehicle height change velocity, and unsprung mass velocity to the vehicle control unit 145. The vehicle behavior data processing unit 141 also outputs the sprung mass G to the reward calculation unit 143.

[0053] The road surface shape output unit 142 comprises a feature calculation unit 1421 and an output unit 1422. Here, Figure 6 shows an example of a Q table in the embodiment. The feature calculation unit 1421 calculates road surface shape features A, B, and C as shown in Figure 6(a).

[0054] The road surface shape feature quantity A is the heave (vertical movement) component of the vibration of the center of gravity part of the vehicle 1. The road surface shape feature quantity B is the roll (lateral sway) component of the vibration of the center of gravity part of the vehicle 1. The road surface shape feature quantity C is the pitch (longitudinal sway) component of the vibration of the center of gravity part of the vehicle 1.

[0055] The feature quantity calculation unit 1421 calculates each of the road surface shape feature quantities A, B, and C, for example, using information such as the acceleration under the spring of each of the four wheels 3. Hereinafter, the road surface shape feature quantities A, B, and C may be simply referred to as "road surface shape feature quantities".

[0056] The output unit 1422 outputs the road surface shape feature quantity received from the feature quantity calculation unit 1421 to the state index acquisition unit 1441 of the learning unit 144.

[0057] The reward calculation unit 143 inputs the acceleration above the spring from the vehicle behavior data processing unit 141, calculates a reward value (evaluation value) using a predetermined reward function, and outputs the calculated reward value to the table update unit 1443 of the learning unit 144.

[0058] The learning unit 144 uses simulation processing to obtain, by Q-learning which is reinforcement learning, for each of one or more (in the example of FIG. 6(a), 3) road surface shape feature quantities of a plurality of patterns (in the example of FIG. 6(a), 4096 patterns), a Q value which is an evaluation value (reward value) with a larger value as the impact received by the vehicle body from the road surface is smaller when performing a simulation for each of one or more (in the example of FIG. 6(b), 4) control parameters (control values) of M patterns (M: a predetermined integer of 3 or more) (in the example of FIG. 6(b), 256 patterns) for the vehicle control unit 145, and creates a Q table (FIG. 6(c)). Further, the learning unit 144 creates Q-value relationship information (FIGS. 7(b)(c)) that holds, for each of the road surface shape feature quantities of a plurality of patterns (4096 patterns), the top N (N: an integer satisfying 2 ≦ N < M) (in the example of FIG. 7, the top 3) Q values with large Q values and the set of control parameters corresponding to those Q values. These will be described in detail below.

[0059] The learning unit 144 includes a state index acquisition unit 1441, a control index selection unit 1442, a table update unit 1443, a table 1444, and a control gain conversion unit 1445.

[0060] The state index acquisition unit 1441 receives road surface shape features from the output unit 1422 and transmits the state index (Figure 6(a)) corresponding to those road surface shape features to the control index selection unit 1442 and the table update unit 1443.

[0061] Here, Figure 6(b) is a table showing the control values ​​for each control index. The control values ​​can take on four levels: 1, 2, 3, and 4, for each of R (roll), P (pitch), H (heave), and B (base).

[0062] The control index selection unit 1442, based on the state index input from the state index acquisition unit 1441, outputs control indices sequentially from 1 to 256 to the table update unit 1443 and the control gain conversion unit 1445, according to the number of times that state index has been selected.

[0063] The control gain conversion unit 1445 converts the control index into a control gain. The control gain is a parameter used to calculate the target control amount for the vehicle control unit 145, which provides the target control amount to the suspension system 30. The vehicle control unit 145 transmits the target control amount to the suspension system 30. Specifically, the vehicle control unit 145 calculates the target control amount based on the vehicle behavior data input from the vehicle behavior data processing unit 141 and the control gain, and outputs the calculated target control amount to the suspension system 30. As a result, the suspension system 30 commands the actuators 102, 103, 105, and 106 to control current to reach the target control amount and performs damping force control, etc.

[0064] The table update unit 1443 obtains evaluation values ​​(reward values) (Q values) when performing a simulation (simulation by the suspension system 30) using control parameters (control values) corresponding to the control index obtained from the control index selection unit 1442, based on the state index obtained from the state index acquisition unit 1441, and creates (updates) a Q table (Figure 6(c)).

[0065] By repeating the above process, the Q table (Figure 6(c)) is completed. Then, as shown in Figure 7, the table update unit 1443 creates Q-value relationship information (Figure 7(b)(c)) from the Q table (Figure 7(a)), which holds the top three Q values ​​with the largest Q values ​​and the set of control indexes corresponding to those Q values ​​for each state index.

[0066] Next, the processing in a real environment will be described. Figure 5 is an explanatory diagram of the processing in a real environment by the vehicle control device 14 of the embodiment. When the vehicle 1 is traveling on an actual road surface, the learning unit 144 uses the Q-value relationship information (Figure 7(b)(c)) and the acquired road surface shape features to send a control parameter corresponding to one of the Q values ​​corresponding to the road surface shape features in the Q-value relationship information (Figure 7(b)(c)) to the vehicle control unit 145 to perform vehicle control, acquires the Q value in that case, and performs an update process to update the Q value in the Q-value relationship information (Figure 7(b)(c)).

[0067] Furthermore, when the learning unit 144 repeatedly performs the update process and selects control parameters for the acquired road surface shape features, if all N (3) Q values ​​for the road surface shape features have been updated in the Q-value relationship information (Figure 7(b)(c)), it selects the control parameter corresponding to the largest Q value among the stored Q values ​​and transmits it to the vehicle control unit 145. These details are described below.

[0068] The road surface shape output unit 142 further includes a storage unit 1423. The storage unit 1423 stores road location information and road surface shape features in association. If a road surface shape feature associated with location information exists in the storage unit 1423, the output unit 1422 outputs that road surface shape feature; otherwise, it outputs the road surface shape feature calculated by the feature calculation unit 1421.

[0069] The control index selection unit 1442 obtains a state index from the state index acquisition unit 1441 and selects a control index from the control index table (Figure 7(c)) according to the number of times that state index has been selected. For example, the first selection outputs the control index of rank 1 (Figure 7(b)(c)), the second selection outputs the control index of rank 2 (Figure 7(b)(c)), the third selection outputs the control index of rank 3 (Figure 7(b)(c)), and for the fourth selection and beyond, it outputs the control index corresponding to the value with the largest evaluation value (Q value). Alternatively, the first selection may output a predetermined control index, the second selection outputs the control index of rank 1 (Figure 7(b)(c)), the third selection outputs the control index of rank 2 (Figure 7(b)(c)), the fourth selection outputs the control index of rank 3 (Figure 7(b)(c)), and for the fifth selection and beyond, it outputs the control index corresponding to the value with the largest evaluation value.

[0070] Here, Figure 8 shows an example of positional relationship information in an embodiment, which includes a set of road surface shape features and location information for each point. For example, when vehicle 1 travels on a predetermined road surface multiple times, the learning unit 144 stores the acquired road surface shape features (state index) and the location information of each point in the positional relationship information of the storage unit 50 for the first time. For subsequent trips, it acquires road surface shape features (state index) based on the current location information of vehicle 1 and the positional relationship information, and then performs the following processing.

[0071] Furthermore, for example, when the learning unit 144 selects control parameters for acquired road surface shape features, if all N (3) Q values ​​for the road surface shape features have been updated in the Q-value relationship information, it may select the control parameter corresponding to the largest Q value among the stored Q values ​​and transmit it to the vehicle control unit. If the new Q value acquired during vehicle control by the vehicle control unit 145 has changed by a predetermined change threshold, the Q-value relationship information may be updated with that Q value.

[0072] Furthermore, Figure 9 is an explanatory diagram illustrating the updating of positional relationship information in the embodiment. The learning unit 144 stores the acquired road surface shape feature quantities and the positional information of each location in the positional relationship information of the storage unit 50 for a predetermined road surface. If a predetermined period has elapsed, the learning unit 144 may acquire road surface shape feature quantities again for that location and update the positional relationship information. When driving on subsequent occasions, the road surface shape feature quantities may be acquired based on the vehicle's current position information and the updated positional relationship information, and subsequent processing may be carried out.

[0073] Next, the process of creating Q-value related information by the vehicle control device 14 will be described. Figure 10 is a flowchart of the process of creating Q-value related information by the vehicle control device 14 in the embodiment. Please also refer to Figure 4.

[0074] In step S11, the learning unit 144 uses simulation processing to obtain Q values ​​when performing simulations for each of the multiple patterns of road surface shape features with each of the M patterns of control parameters (control values) for the vehicle control unit 145, and creates a Q table (Figure 6(c)).

[0075] Next, in step S12, the learning unit 144 creates a reduced table (Q-value relationship information) (Figure 7(b)(c)) based on the Q-table (Figure 6(c)), which holds the top N Q-values ​​with the largest Q-values ​​and the set of control parameters corresponding to those Q-values ​​for each of the multiple patterns of road surface shape features.

[0076] Next, we will describe the processing performed by the vehicle control device 14 in a real environment. Figure 11 is a flowchart showing the processing performed by the vehicle control device 14 in a real environment according to the embodiment. Please also refer to Figure 5.

[0077] In step S21, the state index acquisition unit 1441 acquires road surface shape features from the output unit 1422 of the road surface shape output unit 142.

[0078] Next, in step S22, the control index selection unit 1442 identifies a state index corresponding to the road surface shape feature obtained in step S21 (Figure 7(b)).

[0079] Next, in step S23, the control index selection unit 1442 determines whether the state index identified in step S22 was selected for the first time. If Yes, the unit proceeds to step S24; otherwise, the unit proceeds to step S26.

[0080] In step S24, the control index selection unit 1442 identifies the control index of rank 1 (Figure 7(c)).

[0081] Next, the suspension system 30 controls the vehicle using the control index, and in step S25, the table update unit 1443 updates the Q value in Figure 7(b) accordingly, and returns to step S21.

[0082] In step S26, the control index selection unit 1442 determines whether the state index identified in step S22 has been selected for the second time. If Yes, the unit proceeds to step S27; otherwise, the unit proceeds to step S29.

[0083] In step S27, the control index selection unit 1442 identifies the second control index (Figure 7(c)).

[0084] Next, the suspension system 30 controls the vehicle using the control index, and in step S28, the table update unit 1443 updates the Q value in Figure 7(b) accordingly, and returns to step S21.

[0085] In step S29, the control index selection unit 1442 determines whether the state index identified in step S22 has been selected for the third time. If yes, the unit proceeds to step S30; otherwise, it proceeds to step S32.

[0086] In step S30, the control index selection unit 1442 identifies the third control index (Figure 7(c)).

[0087] Next, the suspension system 30 controls the vehicle using the control index, and in step S31, the table update unit 1443 updates the Q value in Figure 7(b) accordingly, and returns to step S21.

[0088] After processing up to step S31, the evaluation value table in Figure 7(b) will have content (Q value) that is appropriate for the actual environment.

[0089] In step S32, the control index selection unit 1442 identifies the control index with the highest Q value (Figure 7(b)). Subsequently, the suspension system 30 performs vehicle control using that control index.

[0090] As described above, the vehicle control device 14 of this embodiment creates a Q table (Figure 6(c)) using simulation processing, then creates a reduced table (Q value relationship information) (Figure 7(b)(c)), and then updates the reduced table when the vehicle 1 is actually driving on the road surface. This makes it possible to obtain control parameters that are suitable for the actual environment even if there are differences between the simulation and the actual environment regarding the control parameters related to the control of the suspension system 30.

[0091] Furthermore, when Vehicle 1 travels on a predetermined road surface multiple times, when it travels for the first time, as shown in Figure 8, a set of acquired road surface shape features and their location information is stored in the positional relationship information of the storage unit 50 for each point. This allows Vehicle 1 to use the positional relationship information to acquire appropriate road surface shape features and perform subsequent processing when it travels on the predetermined road surface for the second time or later. Therefore, for example, when Vehicle 1 travels on the same road every day, a comfortable ride for the user in Vehicle 1 can be achieved.

[0092] Furthermore, as shown in Figure 9, if a predetermined period of time has elapsed since the positional relationship information was stored in the storage unit 50, the road surface shape features are acquired again for that location and the positional relationship information is updated. This allows the vehicle 1 to be controlled using optimal control parameters corresponding to the change, even if the road surface shape changes due to road construction or other reasons.

[0093] Furthermore, if, after updating the Q-value relationship information (Figure 7(b)(c)) in a real-world environment, the new Q-value acquired during vehicle control by the vehicle control device 14 changes significantly beyond a predetermined change threshold, the Q-value relationship information may be updated with that Q-value. This allows the vehicle 1 to be controlled using optimal control parameters corresponding to any changes that affect the Q-value, such as changes in tire pressure or deterioration of parts.

[0094] Furthermore, conventional deep learning methods using DNNs generally require a large amount of memory due to the large number of parameters. Additionally, repeated training is necessary, which is time-consuming; therefore, directly updating the DNN in vehicle 1 is difficult.

[0095] On the other hand, the method of this embodiment does not have those disadvantages. Specifically, for example, by narrowing down the Q table to create a reduced table (Figure 7(b)(c)), memory usage can be kept significantly smaller.

[0096] The vehicle control program executed by the vehicle control device 14 in the above embodiment is provided pre-installed in, for example, a ROM 14b.

[0097] Furthermore, the vehicle control program may be configured to be provided as an installable or executable file stored on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0098] Furthermore, the vehicle control program may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the vehicle control program may be configured to be provided or distributed via a network such as the Internet.

[0099] While embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. Furthermore, these embodiments and their variations are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as described in the claims.

[0100] For example, in the example in Figure 6(a), the number (types) of road surface shape features is set to 3, but it is not limited to this and may be 6 or other numbers. Similarly, the number of state indices is not limited to 4096 and may be other numbers.

[0101] Furthermore, in the example in Figure 6(b), the number (types) of control values ​​is set to four, but this is not limited to this, and other numbers such as three, five, or six may be used. Similarly, the number of control indices is not limited to 256, and other numbers may be used. Also, the number of control value levels is not limited to four, and other levels may be used.

[0102] Also, in the examples of FIGS. 7(b) and 7(c), although the top three data in terms of Q-value were retained, the number is not limited to three, and other numbers of data such as four may be retained.

[0103] 〔Summary of this embodiment〕 This embodiment at least includes the following configuration.

[0104] This embodiment is a vehicle control device mounted on a vehicle, and includes a vehicle control unit that transmits target control amount information to a suspension system that performs vehicle control for alleviating the impact received by the vehicle body of the vehicle from the road surface; and a simulation process is used to obtain Q-values, which are evaluation values with larger values as the impact received by the vehicle body from the road surface is smaller, for each of one or more road surface shape feature amounts in a plurality of patterns when simulations are performed for each of one or more control parameters in M patterns (M: a predetermined integer of 3 or more) for the vehicle control unit by Q-learning, which is reinforcement learning, and a Q-table is created. Based on the Q-table, for each of the road surface shape feature amounts in a plurality of patterns, Q-value relationship information is created that holds the top N (N: an integer satisfying 2 ≤ N < M) Q-values with large Q-values and the sets of the control parameters corresponding to those Q-values. When the vehicle travels on an actual road surface, using the Q-value relationship information and the obtained road surface shape feature amount, the control parameter corresponding to any of the Q-values corresponding to the road surface shape feature amount in the Q-value relationship information is transmitted to the vehicle control unit to cause vehicle control to be performed, and the Q-value in that case is obtained and an update process for updating the Q-value in the Q-value relationship information is performed. The update process is repeated, and when selecting a control parameter for the obtained road surface shape feature amount, in the Q-value relationship information, when all N Q-values for the road surface shape feature amount have been updated, the control parameter corresponding to the largest Q-value among the retained Q-values is selected and transmitted to the vehicle control unit.

[0105] With such a configuration, even when there are differences between the simulation and the actual environment regarding the control parameters related to the control of the suspension system, control parameters suitable for the actual environment can be obtained.

[0106] Furthermore, when the vehicle travels on a predetermined road surface multiple times, the learning unit stores, for each point, a set of acquired road surface shape features and their location information in the positional relationship information of the storage unit when the vehicle travels on the road surface for the first time. When the vehicle travels on the road surface for the second time or later, it acquires the road surface shape features based on the vehicle's current location information and the positional relationship information, and then performs the subsequent processing.

[0107] With this configuration, when a vehicle travels on a predetermined road surface for the second time or later, it can use the positional relationship information to acquire appropriate road surface shape features and perform subsequent processing.

[0108] Furthermore, the learning unit, for each location on the predetermined road surface, stores the acquired road surface shape feature quantity and the location information of that location in the positional relationship information of the storage unit. After a predetermined period has elapsed, the learning unit acquires the road surface shape feature quantity again for that location and updates the positional relationship information. When driving on subsequent occasions, the learning unit acquires the road surface shape feature quantity based on the vehicle's current location information and the updated positional relationship information, and then performs the subsequent processing.

[0109] With this configuration, even if the road surface shape changes due to road construction or other reasons, for example, the vehicle can be controlled using optimal control parameters that correspond to the change.

[0110] Furthermore, when the learning unit selects the control parameters for the acquired road surface shape features, if all N Q values ​​for the road surface shape features have been updated in the Q-value relationship information, the learning unit selects the control parameter corresponding to the largest Q value among the stored Q values ​​and transmits it to the vehicle control unit. If the new Q value acquired during vehicle control by the vehicle control unit has changed by a predetermined change threshold, the learning unit updates the Q-value relationship information with that Q value.

[0111] With this configuration, when any change that affects the Q-value occurs in the vehicle, such as a change in tire pressure or deterioration of parts, the vehicle can be controlled using optimal control parameters corresponding to that change.

[0112] Furthermore, the effects of the dependent claims and embodiments are additional effects separate from the effects of the independent claims. [Explanation of Symbols]

[0113] 1...Vehicle, 14...ECU (Vehicle Control Unit), 15, 15a, 15c...Imaging Unit, 30...Suspension System, 31...Position Information Sensor, 102, 103, 105, 106...Actuators, 141...Vehicle Behavior Data Processing Unit, 142...Road Surface Shape Output Unit, 143...Reward Calculation Unit, 144...Learning Unit, 145...Vehicle Control Unit

Claims

1. A vehicle control device installed in a vehicle, A vehicle control unit that transmits target control amount information to a suspension system that performs vehicle control to mitigate the impact received by the vehicle body from the road surface, Using simulation processing, and employing Q-learning, which is a form of reinforcement learning, a Q-table is created by obtaining Q-values, which are evaluation values ​​that increase as the impact received by the vehicle body from the road surface decreases, when simulations are performed for each of the multiple patterns of road surface shape features using each of the M patterns (M: a predetermined integer of 3 or more) of control parameters for the vehicle control unit. Based on the Q-table, Q-value relationship information is created for each of the multiple patterns of road surface shape features, holding the top N Q-values ​​(N: an integer satisfying 2 ≤ N < M) and the set of control parameters corresponding to those Q-values. When the vehicle is traveling on an actual road surface, the control unit is given the control parameter corresponding to one of the Q values ​​corresponding to the road surface shape feature in the Q value relationship information, using the Q value relationship information and the acquired road surface shape feature, and the vehicle is controlled. The Q value in this case is acquired, and an update process is performed to update the Q value in the Q value relationship information. This update process is repeated. A vehicle control device comprising: a learning unit that, when selecting the control parameter for an acquired road surface shape feature, selects the control parameter corresponding to the largest Q value among the stored Q values ​​and transmits it to the vehicle control unit if, in the Q value relationship information, all N Q values ​​for the road surface shape feature have been updated.

2. The aforementioned learning unit, When the aforementioned vehicle travels on a predetermined road surface multiple times, When driving for the first time, for each point, the acquired road surface shape features and their location information are stored in the memory unit's positional relationship information. The vehicle control device according to claim 1, which, when driving for the second time or later, acquires the road surface shape feature quantity based on the vehicle's current position information and the positional relationship information, and performs subsequent processing.

3. The aforementioned learning unit, Regarding the aforementioned predetermined road surface, For each location, if a predetermined period has elapsed since the acquired road surface shape feature quantity and the location information of that location were stored in the location relationship information of the storage unit, the road surface shape feature quantity is acquired again for that location and the location relationship information is updated. The vehicle control device according to claim 2, which, when driving for the next time or later, acquires the road surface shape feature quantity based on the vehicle's current position information and the updated positional relationship information, and then performs the subsequent processing.

4. The aforementioned learning unit, The vehicle control device according to claim 1, wherein when selecting a control parameter for an acquired road surface shape feature, if all N Q values ​​for the road surface shape feature have been updated in the Q value relationship information, the control parameter corresponding to the largest Q value among the held Q values ​​is selected and transmitted to the vehicle control unit, and if the new Q value acquired during vehicle control by the vehicle control unit has changed by a predetermined change threshold, the Q value relationship information is updated with the new Q value.

Images