Vehicle control device, vehicle control method, and program

Abstract

To provide a vehicle control device capable of controlling a vehicle with a small load based on an image obtained by a camera. [Solution] A vehicle control device generates a traveling potential field based on an input image, generates a target trajectory based on the gradient of the traveling potential, and operates an electric power steering device based on the target trajectory. The vehicle control device generates a traveling potential field by executing the following processes: setting a low potential point within a central region of the input image at a position determined based on reference subject information, setting a first high potential point in an area where an obstacle is captured, setting the value of the traveling potential at the low potential point to a first set value, setting the value of the traveling potential at the first high potential point to a second set value, and interpolating the value of the traveling potential in an area of ​​the input image between the low potential point and the first high potential point.

Description

Technical Field

[0005] ,

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a program. More specifically, the present invention relates to a vehicle control device, a vehicle control method, and a program for controlling a host vehicle based on an image captured by a camera.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. In order to achieve this, research and development have focused on further improving traffic safety and convenience through research and development related to preventive safety technologies.

[0003] For example, Patent Document 1 describes a preventive safety technology for performing driving control of a host vehicle based on the so-called potential method. Here, the potential method refers to a method of defining a potential function (hereinafter also referred to as a "potential field") corresponding to obstacles and the like existing around the host vehicle, and generating a target trajectory of the host vehicle according to the gradient of this potential function.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in the vehicle control device disclosed in Patent Document 1, the shape of the road on which the host vehicle travels and the situation of surrounding objects are recognized by using a plurality of external sensors such as a camera, a radar device, and a lidar device, and map information, etc., and the shape of the potential function is determined based on this recognition result. Therefore, in the in-vehicle computer that generates the potential function and the target trajectory, there is a tendency for a large load to be applied in order to integrate the outputs of a plurality of external sensors and map information, etc.

[0006] The present invention aims to provide a vehicle control device, a vehicle control method, and a program that can control a vehicle with minimal load based on images obtained from a camera, and ultimately contribute to the development of a sustainable transportation system. [Means for solving the problem]

[0007] (1) The vehicle control device according to the present invention (for example, the vehicle control device 1 described later) comprises: an input image acquisition means (for example, the input image acquisition unit 2 described later) that acquires an image as an input image taken by a camera (for example, the onboard camera C described later) directed forward from the vehicle (for example, the vehicle V described later); a driving potential field generation means (for example, the driving potential field generation unit 3 described later) that generates a driving potential field showing the distribution of driving potential for the future driving position of the vehicle based on the input image; a target trajectory generation means (for example, the target trajectory generation unit 4 described later) that generates a target trajectory for the vehicle based on the gradient of the driving potential in the driving potential field; and a driving control means (for example, the driving control unit 5 described later) that operates a steering mechanism (for example, the electric power steering device 9 described later) based on the target trajectory, wherein the driving potential field generation means acquires the position of a reference subject or The method is characterized by generating a driving potential field by performing the following steps: extracting reference subject information regarding the position of the boundary line of the reference subject; setting a low potential point within the central region of the input image (for example, the central region CC described later) and at a position determined based on the reference subject information; identifying the position of an obstacle that hinders the safe driving of the vehicle within the input image; setting a first high potential point in the region of the input image in which the obstacle is visible; setting the value of the driving potential at the low potential point to a first set value; setting the value of the driving potential at the first high potential point to a second set value greater than the first set value; and interpolating the value of the driving potential in the region of the input image between the low potential point and the first high potential point with a value between the first set value and the second set value.

[0008] (2) In this case, the driving potential field generation means acquires the position of the fitting line (e.g., the left empty fitting line Fa1 described later) of the left empty boundary line (e.g., the left empty boundary line La1 described later) and the position of the fitting line (e.g., the right empty fitting line Fa2 described later) of the right empty boundary line (e.g., the right empty fitting line Fa2 described later) in the input image as reference subject information, and if the intersection point (e.g., the intersection point Pa described later) of the fitting line of the left empty boundary line and the fitting line of the right empty boundary line exists within the central region, it is preferable to set the low potential point at that intersection point.

[0009] (3) In this case, the driving potential field generation means preferably obtains the position of the fitting line of the left road boundary line (for example, the left road fitting line Fb1 described later) and the position of the fitting line of the right road boundary line (for example, the right road fitting line Fb2 described later) in the input image as reference subject information, and if the intersection point of the fitting line of the left road boundary line and the fitting line of the right road boundary line (for example, the intersection point Pb described later) is located within the central region, it is preferable to set the low potential point at that intersection point.

[0010] (4) In this case, the driving potential field generation means preferably acquires the position of the road boundary line in the input image as the reference subject information, and if the uppermost endpoint of the road boundary line in the input image (for example, the uppermost endpoint Pc described later) is within the central region, it is preferable to set the low potential point at that uppermost endpoint.

[0011] (5) In this case, the driving potential field generation means preferably acquires the position of the left empty fitting line of the left empty boundary line in the input image, the position of the right empty fitting line of the right empty boundary line in the input image, the position of the left road fitting line of the left road boundary line in the input image, the position of the right road fitting line of the right road boundary line in the input image, and the position of the road boundary line in the input image as reference subject information, and sets the low potential point at a position determined based on two or more points that exist within the central region among the intersection of the left empty fitting line and the right empty fitting line, the intersection of the left road fitting line and the right road fitting line, and the uppermost point of the road boundary line in the input image.

[0012] (6) In this case, if a preceding vehicle recognized as a target to follow exists within the central region, it is preferable for the driving potential field generation means to acquire the position of the preceding vehicle in the input image as the reference subject information.

[0013] (7) In this case, if the vehicle is traveling on a road where the sky above is obstructed by an overhead structure, it is preferable that the driving potential field generating means acquires the position of the road boundary line or the position of the preceding vehicle recognized as the target to follow as the reference subject information.

[0014] (8) In this case, it is preferable that the running potential field generating means sets the value of the running potential such that the gradient becomes steeper as it approaches the low potential point within the low potential range centered on the low potential point, sets the value of the running potential such that the gradient becomes steeper as it approaches the first high potential point within the first high potential range centered on the first high potential point, and sets the value of the running potential such that the gradient is constant and gentler than that inside the low potential range and the first high potential range outside the low potential range and the first high potential range.

[0015] (9) In this case, it is preferable that the driving potential field generation means generates the driving potential field by further performing the following: a process to identify the position of the driving lane lines of the vehicle in the input image; a process to set a second high potential point in the region of the input image in which the driving lane lines are visible; a process to set the value of the driving potential at the second high potential point to a third setting value that is greater than the first setting value; and a process to interpolate the value of the driving potential between the low potential point and the second high potential point in the input image with a value between the first setting value and the third setting value.

[0016] (10) In this case, it is preferable that the driving potential field generating means does not set the second high potential point when the vehicle is in the process of changing lanes or is scheduled to do so. [Effects of the Invention]

[0017] (1) In the present invention, the input image acquisition means acquires an image taken by a camera pointed forward from the perspective of the vehicle as an input image, the driving potential field generation means generates a driving potential field showing the distribution of driving potential for the future driving position of the vehicle based on the input image, the target trajectory generation means generates a target trajectory of the vehicle based on the gradient of the driving potential in the driving potential field, and the driving control means operates the steering mechanism based on the target trajectory. In addition, in the present invention, the driving potential field generation means extracts reference subject information relating to a predetermined reference subject from the input image, sets a low potential point in the central region of the input image and at a position determined based on the reference subject information, identifies the position of an obstacle in the input image, and further sets a first high potential point in the region of the input image in which the obstacle is visible. Furthermore, the driving potential field generation means sets the value of the driving potential at low potential points to a first set value, sets the value of the driving potential at first high potential points to a second set value greater than the first set value, and interpolates the value of the driving potential in the region of the input image between low potential points and first high potential points with a value between these first and second set values ​​to generate a driving potential field for the input image. In this way, according to the present invention, a driving potential field can be generated and a target trajectory can be generated using only the positions of reference subjects and obstacles captured in the input image, without using external sensors other than the camera or map information, so that the vehicle can be controlled with less load, and in turn, it can contribute to the development of a sustainable transportation system.

[0018] (2) In the present invention, the driving potential field generation means acquires the position of the fitting line of the left sky boundary line (i.e., the boundary line between the sky and ground objects on the left side as viewed from the vehicle) (hereinafter also referred to as the "left sky fitting line") and the position of the fitting line of the right sky boundary line (i.e., the boundary line between the sky and ground objects on the right side as viewed from the vehicle) (hereinafter also referred to as the "right sky fitting line") in the input image as reference subject information. Furthermore, if the intersection point of the left sky fitting line and the right sky fitting line exists within the central region, the driving potential field generation means sets a low potential point at this intersection point, which will be the end of the target path. Thus, according to the present invention, a driving potential field can be generated by simple calculations on the input image.

[0019] (3) In the present invention, the driving potential field generation means acquires the position of the fitting line of the left road boundary line (i.e., the left edge of the road on which the vehicle is traveling as viewed from the vehicle) (hereinafter also referred to as the "left road fitting line") and the position of the fitting line of the right road boundary line (i.e., the right edge of the road on which the vehicle is traveling as viewed from the vehicle) (hereinafter also referred to as the "right road fitting line") in the input image as reference subject information. Furthermore, if the intersection of the left road fitting line and the right road fitting line exists within the central region, the driving potential field generation means sets a low potential point, which is the end of the target path, at this intersection. Thus, according to the present invention, a driving potential field can be generated by simple calculations on the input image.

[0020] (4) In the present invention, the driving potential field generation means acquires the position of the road boundary line in the input image (i.e., the line formed by combining the left road boundary line and the right road boundary line) as reference subject information. Furthermore, if the uppermost endpoint of the road boundary line in the input image is located within the central region, the driving potential field generation means sets a low potential point, which is the end of the target path, at this uppermost endpoint. Thus, according to the present invention, a driving potential field can be generated by simple calculations on the input image.

[0021] (5) In the present invention, the driving potential field generation means acquires the positions of the left empty fitting line, the right empty fitting line, the left road fitting line, the right road fitting line, and the road boundary line in the input image as reference subject information. The driving potential field generation means also sets a low potential point which is the end of the target path at a position determined based on two or more points located within the central region among the intersection of the left empty fitting line and the right empty fitting line, the intersection of the left road fitting line and the right road fitting line, and the uppermost point of the road boundary line in the input image. Thus, according to the present invention, a driving potential field can be generated by simple calculations on the input image.

[0022] (6) In the present invention, if a preceding vehicle recognized as a target to be followed exists within the central region, the driving potential field generation means acquires the position of the preceding vehicle in the input image as reference subject information, and sets a low potential point which is the end of the target path at a position determined based on the position of this preceding vehicle. Thus, according to the present invention, a driving potential field that automatically follows a preceding vehicle can be generated by simple calculations on the input image.

[0023] (7) In the present invention, when the vehicle is traveling on a road where the sky is obscured by an overhead structure, such as inside a tunnel, that is, when the sky is hardly visible in the input image, the driving potential field generation means acquires the position of the road boundary line or the position of the preceding vehicle recognized as the target to follow as reference subject information, and sets the position of the low potential point based on these positions of the road boundary line and the position of the preceding vehicle. Therefore, according to the present invention, even when the sky is not sufficiently visible in the input image, the low potential point can be set at an appropriate position.

[0024] (8) In the present invention, the travel potential field setting means sets the value of the travel potential such that the gradient becomes steeper as it approaches the center inside the low potential range centered on the low potential point, and sets the value of the travel potential such that the gradient becomes steeper as it approaches the center inside the first high potential range centered on the first high potential point. Outside the low potential range and the first high potential range, the value of the travel potential is set such that the gradient is constant and gentler than inside the low potential range and the first high potential range. According to the present invention, by generating a travel potential field by the above procedure, a travel potential field that avoids the first high potential point where an obstacle exists and generates a target path that ends at the low potential point can be generated by simple calculation.

[0025] (9) In the present invention, the travel potential field generation means sets a second high potential point in the region of the input image where the travel lane line of the host vehicle is captured. Further, the travel potential field generation means sets the value of the travel potential at the second high potential point to a third set value larger than the first set value, and interpolates the value of the travel potential between the low potential point and the second high potential point in the input image with a value between the first set value and the third set value, thereby generating a travel potential field. Therefore, according to the present invention, by generating a travel potential field by the above procedure, a travel potential field that avoids the first high potential point where an obstacle exists and the second high potential point where the travel lane line exists and generates a target path that ends at the low potential point can be generated by simple calculation.

[0026] When the second high potential point is set at the position where the driving lane line of the host vehicle exists as described above, a target path that avoids the driving lane line is generated. Therefore, in the present invention, when the host vehicle is in the process of executing or planning to execute a lane change, the driving potential field generation means does not set the second high potential point as described above. Therefore, according to the present invention, a driving potential field that generates a target path straddling the driving lane line can be generated by simple calculations.

Brief Description of the Drawings

[0027] [Figure 1] It is a diagram schematically showing the configuration of a vehicle equipped with a vehicle control device according to an embodiment of the present invention. [Figure 2] It is a functional block diagram of the vehicle control device. [Figure 3] It is a diagram schematically showing the driving potential field generated by the driving potential field generation unit. [Figure 4] It is a flowchart showing the specific procedure of the driving potential field generation process. [Figure 5] It is a diagram for explaining the procedure of the low potential point search process. [Figure 6] It is a diagram for explaining the procedure of the high potential point search process. [Figure 7] It is a diagram showing an example of setting the value of the driving potential in the part including the first high potential point and the low potential point. [Figure 8] It is a diagram showing an example of the driving potential field generated by the driving potential field generation process.

Embodiments for Carrying Out the Invention

[0028] Hereinafter, a vehicle control device according to an embodiment of the present invention will be described with reference to the drawings.

[0029] Figure 1 is a schematic diagram showing the configuration of a vehicle V equipped with the vehicle control device 1 according to this embodiment. The upper part of Figure 1 shows a plan view of the vehicle V, and the lower part of Figure 1 shows a side view. In the following description, the vehicle V is described as a so-called right-hand drive four-wheeled vehicle in which the driver's seat is located on the right side in the vehicle width direction when viewed along the direction of travel, but the present invention is not limited to this. The vehicle V may also be a so-called left-hand drive four-wheeled vehicle in which the driver's seat is located on the left side in the vehicle width direction when viewed along the direction of travel.

[0030] Vehicle V comprises an electric power steering system 9 as a steering mechanism for steering the left and right front wheels Wf, a power plant 8 as a driving drive system that generates driving force to rotate the front wheels Wf which are the drive wheels of vehicle V, a braking system 7 that generates braking force to stop the rotation of the front wheels Wf and rear wheels Wr, an onboard camera C that takes images of the area around vehicle V, and a vehicle control device 1 that controls the electric power steering system 9, the power plant 8, and the braking system 7 based on the images taken by the onboard camera C.

[0031] The electric power steering system 9 includes a gearbox 93 that connects a pinion shaft 92 extending from a steering wheel 91 that accepts steering input from the driver to the left and right front wheels Wf, an electric motor 94 provided in the gearbox 93, and a steering sensor 95 that detects the amount of steering input of the steering wheel 91.

[0032] The gearbox 93 includes a rack shaft that extends along the width of the vehicle and meshes with the pinion shaft 92, and tie rods that connect both ends of the rack shaft to the left and right front wheels Wf. By converting the rotational motion of the steering wheel 91 caused by the driver's steering operation into motion along the width of the vehicle, the left and right front wheels Wf are steered in the direction of travel. The electric motor 94 rotates in accordance with the control signal output from the vehicle control device 1 and generates driving force to assist the driver's steering operation or to automatically steer the front wheels Wf without the driver's steering operation. The steering sensor 95 detects the amount of steering of the steering wheel 91 and transmits a signal corresponding to the detected value to the vehicle control device 1.

[0033] The power plant 8 is a power source that generates driving force to rotate the front wheels Wf in order to move the vehicle V forward or backward in the direction of travel, in response to acceleration and deceleration operations of the accelerator pedal (not shown) by the driver or control signals output from the vehicle control device 1. In the following description, the power plant 8 will be described in which a drive motor is used to generate driving force by consuming electricity supplied from a high-voltage battery or fuel cell stack (not shown), but the present invention is not limited to this. The power plant 8 may also be an engine that generates driving force by consuming fuel stored in a fuel tank (not shown), or a transmission that changes the output of this engine and transmits it to the front wheels Wf.

[0034] The braking system 7 includes a disc brake system that generates braking force to decelerate or stop the rotation of each wheel Wf, Wr by tightening discs provided on the axles of each wheel Wf, Wr, mainly during driving, based on braking operations by the driver using the brake pedal (not shown) or control signals output from the vehicle control device 1, and a parking brake that generates braking force to maintain the rotation of each wheel Wr, Wf stopped, mainly during parking.

[0035] The on-board camera C is oriented forward along the direction of travel when viewed from the vehicle V. In this embodiment, the on-board camera C is described as being located in the center of the vehicle body in the width direction, but the present invention is not limited to this.

[0036] The vehicle control device 1 controls the electric power steering system 9, the power plant 8, and the braking system 7 based on an image of the front of the vehicle V taken by the onboard camera C. The vehicle control device 1 is a computer composed of hardware such as a CPU and other arithmetic processing means, an auxiliary storage means such as an HDD or SSD that stores a program that causes the arithmetic processing means to execute the driving potential field generation process described later, and a main memory means such as RAM for storing data that the arithmetic processing means temporarily needs to execute the program.

[0037] Figure 2 is a functional block diagram of the vehicle control device 1. The vehicle control device 1 consists of an input image acquisition unit 2, a driving potential field generation unit 3, a target trajectory generation unit 4, and a driving control unit 5, with the hardware configuration described above.

[0038] The input image acquisition unit 2 acquires an image of the front of the vehicle V, captured by the on-board camera C, as the input image. The input image acquisition unit 2 transmits information about the acquired input image to the driving potential field generation unit 3.

[0039] The driving potential field generation unit 3 generates a driving potential field based on the input image transmitted from the input image acquisition unit 2, which shows the distribution of the driving potential on the input image for the future driving position of the vehicle V (i.e., the distribution of the driving potential on the two-dimensional image coordinate system in which the input image is defined). The driving potential field generation unit 3 transmits information about the generated driving potential field to the target trajectory generation unit 4.

[0040] Figure 3 schematically shows the driving potential field generated by the driving potential field generation unit 3. In Figure 3, the values ​​of the driving potential are shown in different colors against the input image as the background. More specifically, darker colors indicate higher values ​​of the driving potential. In Figure 3, the point with the smallest driving potential value (potential minimum point) is indicated by a white circle. As shown in Figure 3, the driving potential field is a scalar function of the driving potential defined on a two-dimensional image coordinate system. The specific procedure for generating the driving potential field by the driving potential field generation unit 3 will be explained later with reference to Figures 4 to 8, etc.

[0041] Returning to Figure 2, the target trajectory generation unit 4 calculates the gradient of the running potential field generated by the running potential field generation unit 3, and generates a target trajectory on the image coordinates of the vehicle V based on this gradient of the running potential field and the current steering angle of the vehicle V. The target trajectory generation unit 4 transmits information about the generated target trajectory to the running control unit 5. Here, the gradient of the running potential field is a vector function that corresponds to the partial derivative of each image coordinate component of the running potential field defined on the two-dimensional image coordinates. As a result, the target trajectory generation unit 4 generates a target trajectory (see the thick dashed line in Figure 3) that starts from the front of the vehicle V and follows the trough of the running potential to the potential minimum point.

[0042] The driving control unit 5 operates the electric power steering system 9, the power plant 8, and the braking system 7 based on the target trajectory generated by the target trajectory generation unit 4. More specifically, the driving control unit 5 operates the electric power steering system 9, the power plant 8, and the braking system 7 so that the vehicle V travels along the target trajectory defined on the image coordinates.

[0043] Figure 4 is a flowchart showing the specific procedure for generating the driving potential field. The driving potential field generation process shown in Figure 4 is repeatedly executed in the driving potential field generation unit 3 under a predetermined control cycle while the vehicle V is in motion.

[0044] First, in step ST1, the driving potential field generation unit 3 acquires the input image transmitted from the input image acquisition unit 2, and then proceeds to step ST2.

[0045] Next, in step ST2, the driving potential field generation unit 3 performs segmentation processing on the input image acquired in step ST1 to classify the subjects in the input image into multiple classes, and generates an image (hereinafter also called an "edge image") with the boundaries of each class extracted, and then proceeds to step ST3.

[0046] Next, in step ST3, the driving potential field generation unit 3 performs a low-potential point search process to search for the location of low-potential points on the image coordinates based on edge images extracted from the input image and information about each class captured in the input image (hereinafter also referred to as "class information"), and then proceeds to step ST4. Here, a low-potential point is the point on the image coordinates where the value of the driving potential is at its minimum, and as shown in Figure 3, it is the point that marks the end of the target trajectory. The specific procedure for the low-potential point search process will be explained in detail below with reference to Figure 5.

[0047] Figure 5 is a diagram showing an example of an input image and is intended to illustrate the procedure for low-potential point search processing. First, the driving potential field generation unit 3 extracts information as reference subject information from the edge image and class information, etc., regarding the position of a reference subject on the image coordinate system or the position of the boundary line of this reference subject, which is determined to set the position of the low-potential point. Next, the driving potential field generation unit 3 sets the low-potential point at a position determined based on the extracted reference subject information, within the central region CC which is defined to include the center of the image coordinate system. Several embodiments of the low-potential point search processing will be described below.

[0048] <Example 1> In Example 1, the driving potential field generation unit 3 sets the position of the low potential point using a subject classified as "sky" among multiple subjects in the input image as the reference subject. In this case, the driving potential field generation unit 3 obtains the position of the fitting line Fa1 of the left sky boundary line La1 and the position of the fitting line Fa2 of the right sky boundary line La2 as reference subject information from the edge image and class information, etc. Here, the left sky boundary line La1 refers to the boundary line between the sky and ground objects on the left side as viewed from the vehicle V in the input image, and the right sky boundary line La2 refers to the boundary line between the sky and ground objects on the right side as viewed from the vehicle V in the input image. The fitting line Fa1 of the left sky boundary line (hereinafter also referred to as the "left sky fitting line") refers to a line obtained by fitting the left sky boundary line La1 based on a known algorithm, for example, by a linear function, and the fitting line Fa2 of the right sky boundary line (hereinafter also referred to as the "right sky fitting line") refers to a line obtained by fitting the right sky boundary line La2 based on a known algorithm, for example, by a linear function.

[0049] Furthermore, if the intersection point Pa of the left empty fitting line Fa1 and the right empty fitting line Fa2 obtained as described above is located within a predetermined central region CC, the driving potential field generation unit 3 sets a low potential point at this intersection point Pa. If the intersection point Pa is not located within the central region CC, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 2 to 5, etc.

[0050] Furthermore, if vehicle V is traveling on a road where the sky is obstructed by an overhead structure (for example, if vehicle V is traveling inside a tunnel), the left sky fitting line Fa1 and the right sky fitting line Fa2 cannot be obtained. Therefore, even in such cases, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 2 to 5, etc.

[0051] <Example 2> In Example 2, the driving potential field generation unit 3 sets the position of the low potential point using a subject classified as a "road" among multiple subjects in the input image as the reference subject. In this case, the driving potential field generation unit 3 obtains the position of the fitting line Fb1 of the left road boundary and the position of the fitting line Fb2 of the right road boundary as reference subject information from the edge image and class information, etc. Here, the left road boundary refers to the left edge of the road on which the vehicle V is traveling in the input image, as viewed from the vehicle V, and the right road boundary refers to the right edge of the road on which the vehicle V is traveling in the input image, as viewed from the vehicle V. The fitting line of the left road boundary (hereinafter also called the "left road fitting line") Fb1 refers to a line obtained by fitting the left road boundary based on a known algorithm, for example, by a linear function, and the fitting line of the right road boundary (hereinafter also called the "right road fitting line") Fb2 refers to a line obtained by fitting the right road boundary based on a known algorithm, for example, by a linear function. In the example shown in Figure 4, the left and right road boundaries and their fitting lines Fb1 and Fb2 almost completely overlap, so only fitting lines Fb1 and Fb2 are shown.

[0052] Furthermore, if the intersection point Pb of the left road fitting line Fb1 and the right road fitting line Fb2 obtained as described above is located within the central region CC, the driving potential field generation unit 3 sets a low potential point at this intersection point Pb. If the intersection point Pb is not located within the central region CC, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 1, 3 to 5, etc.

[0053] <Example 3> In Example 3, the driving potential field generation unit 3 sets the position of the low potential point by using a subject that is classified as a "road" class from among the multiple subjects captured in the input image, similar to Example 2. In this case, the driving potential field generation unit 3 obtains the position of the road boundary line as reference subject information from the edge image and class information, etc. Here, the road boundary line refers to the line formed by combining the left road boundary line and the right road boundary line as described above.

[0054] Furthermore, if the uppermost endpoint Pc in the input image of the road boundary line acquired as described above is located within the central region CC, the driving potential field generation unit 3 sets a low potential point at this uppermost endpoint Pc. If the uppermost endpoint Pc is not located within the central region CC, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 1-2, 4-5, etc.

[0055] <Example 4> In Example 4, the driving potential field generation unit 3 generates a driving potential field from multiple subjects captured in the input image. Of these, subjects classified as "sky" or "road" are used as reference subjects to set the position of the low potential point. In this case, the driving potential field generation unit 3 obtains the position of the left sky fitting line Fa1, the position of the right sky fitting line Fa2, the position of the left road fitting line Fb1, the position of the right road fitting line Fb2, and the position of the road boundary line as reference subject information from edge images and class information, etc.

[0056] Furthermore, the driving potential field generation unit 3 sets low potential points at positions determined based on two or more points located within the central region CC, which are among the intersection points Pa of the left empty fitting line Fa1 and the right empty fitting line Fa2, the intersection point Pb of the left road fitting line Fb1 and the right road fitting line Fb2, and the uppermost point Pc of the road boundary line. More specifically, the driving potential field generation unit 3 sets low potential points at the geometric centroid of two or more points located within the central region CC, which are among the three points Pa, Pb, and Pc mentioned above.

[0057] <Example 5> In Example 5, the driving potential field generation unit 3 sets the position of the low potential point using a subject classified as a "preceding vehicle" from among multiple subjects captured in the input image as the reference subject. In this case, the driving potential field generation unit 3 obtains the position of the preceding vehicle (more specifically, the position of the center point Pd of the preceding vehicle) as reference subject information from the edge image and class information, etc. Here, a preceding vehicle refers to a vehicle that is in the same lane as vehicle V and is traveling in front of vehicle V in the same direction as vehicle V.

[0058] Furthermore, if the preceding vehicle extracted as described above is recognized as a target for vehicle V to follow through a process not shown, and the center point Pd of the preceding vehicle is within the central region CC, the driving potential field generation unit 3 sets a low potential point at the center point Pd of the preceding vehicle. If there is no preceding vehicle, if the preceding vehicle is not recognized as a target for following, or if the center point Pd of the preceding vehicle is not within the central region CC, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 1 to 4, etc.

[0059] Returning to Figure 4, in step ST4, the driving potential field generation unit 3 performs a high-potential point search process to search for the location of high-potential points on the image coordinates based on edge images and class information extracted from the input image, and then proceeds to step ST5. Here, a high-potential point is a point on the image coordinates where the value of the driving potential is at its maximum. The specific procedure for the high-potential point search process will be explained in detail below with reference to Figure 6.

[0060] Figure 6 is a diagram showing an example of an input image and is intended to explain the procedure for searching for high-potential points. First, the driving potential field generation unit 3 identifies the position in image coordinates of obstacles that hinder the safe driving of vehicle V from edge images and class information, etc. Obstacles that hinder the safe driving of vehicle V include, for example, other vehicles other than the preceding vehicle, curbs, and street trees. Next, as shown by the circles in Figure 6, the driving potential field generation unit 3 sets a first high-potential point in the region where the obstacles identified by the above procedure are visible in the image coordinates.

[0061] Furthermore, the driving potential field generation unit 3 identifies the position of the vehicle V in the image coordinates of the driving lane lines from the edge image and class information, etc. Next, the driving potential field generation unit 3 sets a second high potential point in the region where the driving lane lines identified by the above procedure are visible in the image coordinates, as shown by the square marks in Figure 6.

[0062] Unlike obstacles set at the first high-potential point, these lane markings do not in themselves hinder the safe driving of vehicle V. However, if vehicle V drives across a lane marking, it may endanger the safe driving of other vehicles. Therefore, the driving potential field generation unit 3 sets a second high-potential point at such a lane marking. However, if a second high-potential point is set at such a lane marking, vehicle V will be unable to move across the lane marking, and consequently will be unable to change lanes. Therefore, it is preferable for the driving potential field generation unit 3 not to set a second high-potential point when vehicle V is in the process of changing lanes or is planning to change lanes.

[0063] Returning to Figure 4, in step ST5, the driving potential field generation unit 3 executes a driving potential value setting process, which sets the driving potential value on the image coordinates based on the positions of the low potential points searched in step ST3 and the high potential points searched in step ST4, and then terminates the process in Figure 4. The specific procedure for the driving potential value setting process will be explained below with reference to Figure 7.

[0064] Figure 7 shows an example of setting the value of the driving potential in the region including the first high-potential point and the low-potential point, as shown by line VI-VI in Figure 6.

[0065] First, the running potential field generation unit 3 sets the running potential value at the low potential point to a predetermined first set value, sets the running potential value at the first high potential point to a second set value greater than the first set value, and sets the running potential value at the second high potential point to a third set value greater than the first set value. The following description will focus on the case where the second set value and the third set value are set to equal magnitudes, but the present invention is not limited to this. The second set value and the third set value may be set to different values.

[0066] Next, the driving potential field generation unit 3 interpolates the value of the driving potential in the region between the low potential point and the first high potential point in the image coordinates using a value between the first set value and the second set value, and interpolates the value of the driving potential in the region between the low potential point and the second high potential point using a value between the first set value and the third set value.

[0067] More specifically, the running potential field generation unit 3 sets the running potential value such that the gradient becomes steeper as you approach the low potential point within the low potential range centered on the low potential point, sets the running potential value such that the gradient becomes steeper as you approach the first high potential point within the first high potential range centered on the first high potential point, and sets the running potential value such that the gradient becomes steeper as you approach the second high potential point within the second high potential range centered on the second high potential point.

[0068] Next, the driving potential field generation unit 3 sets the value of the driving potential so that the gradient is constant outside the low potential range, the first high potential range, and the second high potential range, and is gentler than the gradient inside the low potential range, the first high potential range, and the second high potential range. The driving potential field generation unit 3 generates the driving potential field by setting the value of the driving potential over all regions on the image coordinates using the procedure described above.

[0069] Figure 8 shows an example of a running potential field generated by the running potential field generation process described above. In Figure 8, the gradient of the running potential field, which is a vector function, is indicated by an arrow. Also, in the example in Figure 8, to make the illustration clearer, the case where only the second high potential point is set and the first high potential point is not set is shown.

[0070] In the example shown in Figure 8, the ridges of the driving potential peaks are formed on the two driving lanes extending from both the left and right sides of the vehicle toward the center of the input image. Therefore, in the example shown in Figure 8, the valleys of the driving potential are formed extending between the two driving lanes toward the center of the input image. Consequently, under a driving potential field like the one shown in Figure 8, the target trajectory is generated by the target trajectory generation unit 4, extending along the valleys of the driving potential field as shown by the thick dashed lines in Figure 8.

[0071] The vehicle control device 1 according to this embodiment provides the following effects. (1) The input image acquisition unit 2 acquires an image taken by an onboard camera C pointed forward from the vehicle V as an input image, the driving potential field generation unit 3 generates a driving potential field showing the distribution of driving potential for the future driving position of the vehicle V based on the input image, the target trajectory generation unit 4 generates a target trajectory for the vehicle V based on the gradient of the driving potential in the driving potential field, and the driving control unit 5 operates the electric power steering device 9, power plant 8, braking device 7, etc. based on the target trajectory. The driving potential field generation unit 3 also extracts reference subject information relating to a predetermined reference subject from the input image, sets a low potential point within the central region CC of the input image and at a position determined based on the reference subject information, identifies the position of an obstacle in the input image, and further sets a first high potential point in the region of the input image in which the obstacle is visible. Furthermore, the driving potential field generation unit 3 sets the value of the driving potential at the low potential point to a first set value, sets the value of the driving potential at the first high potential point to a second set value which is greater than the first set value, and interpolates the value of the driving potential in the region of the input image between the low potential point and the first high potential point with a value between these first and second set values ​​to generate a driving potential field for the input image. In this way, according to this embodiment, a driving potential field can be generated and a target trajectory can be generated using only the positions of reference subjects and obstacles captured in the input image, without using external sensors other than the on-board camera C or map information, so that the vehicle can be controlled with less load, and in turn, it can contribute to the development of a sustainable transportation system.

[0072] (2) The driving potential field generation unit 3 acquires the positions of the left empty fitting line and the right empty fitting line in the input image as reference subject information. Furthermore, if the intersection point Pa of the left empty fitting line and the right empty fitting line is located within the central region CC, the driving potential field generation unit 3 sets a low potential point at this intersection point Pa, which will be the end of the target path. Thus, according to the present invention, a driving potential field can be generated by simple calculations on the input image.

[0073] (3) The driving potential field generation unit 3 acquires the positions of the left road fitting line and the right road fitting line in the input image as reference subject information. Furthermore, if the intersection point Pb of the left road fitting line and the right road fitting line is located within the central region CC, the driving potential field generation unit 3 sets a low potential point at this intersection point Pb, which will be the end of the target path. Thus, according to the present invention, a driving potential field can be generated by simple calculations on the input image.

[0074] (4) The driving potential field generation unit 3 acquires the position of the road boundary line in the input image as reference subject information. Furthermore, if the uppermost endpoint Pc in the input image of the road boundary line is located within the central region CC, the driving potential field generation unit 3 sets a low potential point at this uppermost endpoint Pc, which will be the end of the target path. Thus, the vehicle control device 1 can generate a driving potential field by simple calculations on the input image.

[0075] (5) The driving potential field generation unit 3 acquires the positions of the left empty fitting line, the right empty fitting line, the left road fitting line, the right road fitting line, and the road boundary line in the input image as reference subject information. The driving potential field generation unit 3 also sets a low potential point, which is the end of the target path, at a position determined based on two or more points that exist within the central region CC, among the intersection point Pa of the left empty fitting line and the right empty fitting line, the intersection point Pb of the left road fitting line and the right road fitting line, and the uppermost point Pc of the road boundary line in the input image. Thus, the vehicle control device 1 can generate a driving potential field by simple calculations on the input image.

[0076] (6) When a preceding vehicle recognized as a target to follow exists within the central region CC, the driving potential field generation unit 3 acquires the position of the preceding vehicle in the input image as reference subject information, and sets a low potential point which will be the end of the target path at a position determined based on the position of this preceding vehicle. Thus, the vehicle control device 1 can generate a driving potential field that automatically follows the preceding vehicle by performing simple calculations on the input image.

[0077] (7) When a vehicle is traveling on a road where the sky is obscured by overhead structures, such as inside a tunnel, that is, when the sky is hardly visible in the input image, the driving potential field generation unit 3 acquires the position of the road boundary line or the position of the preceding vehicle recognized as the target to follow as reference subject information, and sets the position of the low potential point based on these positions of the road boundary line and the preceding vehicle. Thus, the vehicle control device 1 can set the low potential point at an appropriate position even when the sky is not sufficiently visible in the input image.

[0078] (8) The driving potential field generation unit 3 sets the value of the driving potential such that the gradient becomes steeper as you approach the center within the low potential range centered on the low potential point, the value of the driving potential such that the gradient becomes steeper as you approach the center within the first high potential range centered on the first high potential point, and the value of the driving potential such that the gradient is constant and gentler than the gradient inside the low potential range and the first high potential range outside the low potential range and the first high potential range. According to the vehicle control device 1, by generating a driving potential field in the above procedure, a driving potential field that generates a target path that avoids the first high potential point where an obstacle exists and ends at the low potential point can be generated by simple calculations.

[0079] (9) The driving potential field generation unit 3 sets a second high potential point in the region of the input image in which the driving lane markings of the vehicle V are visible. The driving potential field generation unit 3 also sets the value of the driving potential at the second high potential point to a third setting value which is greater than the first setting value, and generates a driving potential field by interpolating the value of the driving potential between the low potential point and the second high potential point in the input image with a value between the first setting value and the third setting value. Thus, the vehicle control device 1 can generate a driving potential field by the above procedure, thereby generating a target path that avoids the first high potential point where obstacles exist and the second high potential point where driving lane markings exist, and ends at the low potential point, through simple calculations.

[0080] (10) As described above, if a second high potential point is set at a location where a lane marking exists for vehicle V, a target path that avoids the lane marking is generated. Therefore, in the vehicle control device 1, the driving potential field generation unit 3 does not set the second high potential point as described above when vehicle V is performing or planning to perform a lane change. Thus, according to the present invention, a driving potential field that generates a target path that crosses a lane marking can be generated by simple calculations.

[0081] Although one embodiment of the present invention has been described above, the present invention is not limited thereto. Within the scope of the spirit of the present invention, the details of the configuration may be modified as appropriate. [Explanation of symbols]

[0082] V... Vehicle (own vehicle) C...In-car camera (camera) 1…Vehicle control system (vehicle control system) 2…Input image acquisition unit (input image acquisition means) 3…Driving potential field generation unit (driving potential field generation means) 4...Target trajectory generation unit (target trajectory generation means) 5…Travel control unit (travel control means) 7...Brake device 8…Power plant (driving system) 9…Electric power steering system (steering mechanism)

Claims

1. An input image acquisition means that acquires an image as an input image taken by a camera pointed forward from the perspective of the vehicle, A driving potential field generation means generates a driving potential field that shows the distribution of driving potential for the future driving position of the vehicle based on the input image, A target trajectory generation means that generates a target trajectory for the vehicle based on the gradient of the running potential in the running potential field, A vehicle control device comprising a driving control means for operating a steering mechanism based on the aforementioned target trajectory, The aforementioned driving potential field generating means is A process for extracting reference subject information from the input image, relating to the positions of multiple reference subjects or the positions of the boundaries of said multiple reference subjects, A process of setting a low-potential point within the central region of the input image and at a position determined based on the reference subject information, A process to identify the position of an obstacle that hinders the safe driving of the vehicle within the input image, The process of setting a first high-potential point in the region of the input image in which the obstacle is visible, A process to set the value of the driving potential at the low potential point to a first set value, A process to set the value of the driving potential at the first high-potential point to a second set value that is greater than the first set value, A vehicle control device characterized by generating a driving potential field by performing a process of interpolating the value of the driving potential in the region between the low potential point and the first high potential point in the input image with a value between the first set value and the second set value.

2. The aforementioned driving potential field generating means is The position of the left empty fitting line of the left empty boundary line and the position of the right empty fitting line of the right empty boundary line in the input image are acquired as reference subject information. The vehicle control device according to claim 1, characterized in that, if the intersection point of the left empty fitting line and the right empty fitting line is located within the central region, the low potential point is set at the intersection point.

3. The aforementioned driving potential field generating means is The positions of the left road fitting line on the left road boundary line and the right road fitting line on the right road boundary line in the input image are acquired as reference subject information. The vehicle control device according to claim 1, characterized in that, if the intersection point of the left road fitting line and the right road fitting line is located within the central region, the low potential point is set at the intersection point.

4. The aforementioned driving potential field generating means is The position of the road boundary in the input image is acquired as the reference subject information. The vehicle control device according to claim 1, characterized in that, if the uppermost endpoint of the road boundary line in the input image is located within the central region, the low potential point is set at the uppermost endpoint.

5. The aforementioned driving potential field generating means is The position of the left empty fitting line of the left empty boundary line in the input image, the position of the right empty fitting line of the right empty boundary line in the input image, the position of the left road fitting line of the left road boundary line in the input image, the position of the right road fitting line of the right road boundary line in the input image, and the position of the road boundary line in the input image are acquired as reference subject information. The vehicle control device according to claim 1, characterized in that the low potential point is set at a position determined based on two or more points located within the central region among the intersection of the left empty fitting line and the right empty fitting line, the intersection of the left road fitting line and the right road fitting line, and the uppermost point of the road boundary line in the input image.

6. An input image acquisition means that acquires an image as an input image taken by a camera pointed forward from the perspective of the vehicle, A driving potential field generation means generates a driving potential field that shows the distribution of driving potential for the future driving position of the vehicle based on the input image, A target trajectory generation means that generates a target trajectory for the vehicle based on the gradient of the running potential in the running potential field, A vehicle control device comprising a driving control means for operating a steering mechanism based on the aforementioned target trajectory, The aforementioned driving potential field generating means is A process for extracting reference subject information from the input image, relating to the position of the reference subject or the position of the boundary line of the reference subject, A process of setting a low-potential point within the central region of the input image and at a position determined based on the reference subject information, A process to identify the position of an obstacle that hinders the safe driving of the vehicle within the input image, The process of setting a first high-potential point in the region of the input image in which the obstacle is visible, A process to set the value of the driving potential at the low potential point to a first set value, A process to set the value of the driving potential at the first high-potential point to a second set value that is greater than the first set value, The process of interpolating the value of the driving potential in the region between the low-potential point and the first high-potential point in the input image with a value between the first set value and the second set value is performed to generate the driving potential field. The vehicle control device is characterized in that, when a preceding vehicle recognized as a target to be followed exists within the central region, the position of the preceding vehicle in the input image is acquired as the reference subject information.

7. An input image acquisition means that acquires an image as an input image taken by a camera pointed forward from the perspective of the vehicle, A driving potential field generation means generates a driving potential field that shows the distribution of driving potential for the future driving position of the vehicle based on the input image, A target trajectory generation means that generates a target trajectory for the vehicle based on the gradient of the running potential in the running potential field, A vehicle control device comprising a driving control means for operating a steering mechanism based on the aforementioned target trajectory, The aforementioned driving potential field generating means is A process for extracting reference subject information from the input image, relating to the position of the reference subject or the position of the boundary line of the reference subject, A process of setting a low-potential point within the central region of the input image and at a position determined based on the reference subject information, A process to identify the position of an obstacle that hinders the safe driving of the vehicle within the input image, The process of setting a first high-potential point in the region of the input image in which the obstacle is visible, A process to set the value of the driving potential at the low potential point to a first set value, A process to set the value of the driving potential at the first high-potential point to a second set value that is greater than the first set value, The process of interpolating the value of the driving potential in the region between the low-potential point and the first high-potential point in the input image with a value between the first set value and the second set value is performed to generate the driving potential field. The aforementioned driving potential field generating means is A vehicle control device characterized in that, when the vehicle is traveling on a road where the sky is obstructed by an overhead structure, it acquires the position of the road boundary line or the position of a preceding vehicle recognized as a target to follow as reference subject information.

8. The aforementioned driving potential field generating means is Within the low-potential range centered on the low-potential point, the value of the driving potential is set such that the gradient becomes steeper as one approaches the low-potential point. Within the first high-potential range centered on the first high-potential point, the value of the running potential is set such that the gradient becomes steeper as it approaches the first high-potential point. The vehicle control device according to claim 1, characterized in that the value of the driving potential is set such that the gradient is constant outside the low potential range and the first high potential range and is gentler than the gradient inside the low potential range and the first high potential range.

9. The aforementioned driving potential field generating means is The process of determining the position of the vehicle's lane markings within the input image, The process of setting a second high-potential point in the region of the input image in which the lane markings are visible, A process to set the value of the driving potential at the second high-potential point to a third set value that is greater than the first set value, The vehicle control device according to claim 1, further comprising the process of interpolating the value of the driving potential between the low-potential point and the second high-potential point in the input image with a value between the first set value and the third set value to generate the driving potential field.

10. The vehicle control device according to claim 9, characterized in that the driving potential field generating means does not set the second high potential point when the vehicle is performing or planning to perform a lane change.

11. A vehicle control method in which a computer controls the vehicle based on an image captured by a camera pointed forward from the vehicle's perspective, The steps include acquiring an image captured by the aforementioned camera as an input image, The steps include generating a driving potential field that shows the distribution of driving potential for the future driving position of the vehicle based on the input image, A step of generating a target trajectory for the vehicle based on the gradient of the running potential in the running potential field, The system includes the step of operating the steering mechanism based on the target trajectory, In the step of generating the aforementioned driving potential field, A process for extracting reference subject information from the input image, relating to the positions of multiple reference subjects or the positions of the boundaries of said multiple reference subjects, A process of setting a low-potential point within the central region of the input image and at a position determined based on the reference subject information, A process to identify the position of an obstacle that hinders the safe driving of the vehicle within the input image, The process of setting a first high-potential point in the region of the input image in which the obstacle is visible, A process to set the value of the driving potential at the low potential point to a first set value, A process to set the value of the driving potential at the first high-potential point to a second set value that is greater than the first set value, A vehicle control method characterized by generating a driving potential field by performing a process of interpolating the value of the driving potential in the region between the low potential point and the first high potential point in the input image with a value between the first set value and the second set value.

12. A program that controls the vehicle by computer based on images captured by a camera pointed forward from the vehicle's perspective, To the aforementioned computer, The steps include acquiring an image captured by the aforementioned camera as an input image, The steps include generating a driving potential field that shows the distribution of driving potential for the future driving position of the vehicle based on the input image, A step of generating a target trajectory for the vehicle based on the gradient of the running potential in the running potential field, The step of operating the steering mechanism based on the aforementioned target trajectory is performed, In the step of generating the aforementioned driving potential field, the computer is instructed to: A process for extracting reference subject information from the input image, relating to the positions of multiple reference subjects or the positions of the boundaries of said multiple reference subjects, A process of setting a low-potential point within the central region of the input image and at a position determined based on the reference subject information, A process to identify the position of an obstacle that hinders the safe driving of the vehicle within the input image, The process of setting a first high-potential point in the region of the input image in which the obstacle is visible, A process to set the value of the driving potential at the low potential point to a first set value, A process to set the value of the driving potential at the first high-potential point to a second set value that is greater than the first set value, A program characterized by generating a driving potential field by performing a process of interpolating the value of the driving potential in the region between the low potential point and the first high potential point in the input image with a value between the first set value and the second set value.

Images