Vehicle control device, vehicle control method, and storage medium

Abstract

The problem to be solved by the present invention is to provide a vehicle control device that can control a vehicle with a small load on the basis of an image captured by a camera. In order to solve the above problem, the vehicle control device of the present invention generates a travel potential field based on an input image, generates a target trajectory based on a gradient of a travel potential, and operates an electric power steering device based on the target trajectory. A vehicle control device generates a travel position potential field by performing: a process for setting a low position potential point at a position determined on the basis of reference subject information within a central region of an input image; setting a first high potential point in the area where the obstacle is reflected; setting the value of the driving potential at the low potential point as a first set value; setting the value of the driving potential at the first high potential point as a second set value; and interpolating the value of the driving potential of a region between the low potential point and the first high potential point in the input image.

Description

Technical Field

[0001] This invention relates to a vehicle control device, a vehicle control method, and a storage medium. More specifically, it relates to a vehicle control device, a vehicle control method, and a storage medium for controlling a vehicle based on images captured by a camera. Background Technology

[0002] In recent years, there has been a strong push for various initiatives to provide sustainable transportation systems, particularly targeting vulnerable groups among transportation users. To achieve this goal, significant efforts are being made in research and development related to preventative safety technologies to further improve the safety and convenience of transportation.

[0003] For example, Patent Document 1 describes a safety prevention technology that uses a so-called potential method to control the vehicle's movement. Here, the potential method refers to defining a potential function (hereinafter also called a "potential field") corresponding to obstacles or other objects existing around the vehicle, and generating a target trajectory for the vehicle according to the gradient of the potential function.

[0004] [Previous Technical Documents]

[0005] (Patent Documents)

[0006] Patent Document 1: International Publication No. 2018 / 131090 Summary of the Invention

[0007] [The problem the invention aims to solve]

[0008] Furthermore, in the vehicle control device shown in Patent Document 1, multiple external sensors, such as cameras, radar devices, and optical detection and ranging devices, along with map information, are used to identify the shape of the driving road and the condition of surrounding objects, and the shape of the potential function is determined based on the identification results. Therefore, in order to integrate the outputs of multiple external sensors and map information, a large load is often applied to the onboard computer that generates the potential function and the target trajectory.

[0009] The purpose of this invention is to provide a vehicle control device, a vehicle control method, and a storage medium, wherein the vehicle control device can control a vehicle with less load based on images captured by a camera, thereby promoting the development of sustainable transportation systems.

[0010] [Technical means to solve the problem]

[0011] (1) The vehicle control device of the present invention (e.g., the vehicle control device 1 described later) is characterized by comprising: an input image acquisition means (e.g., the input image acquisition unit 2 described later), which acquires an image taken by a camera (e.g., the vehicle camera C described later) viewed from the vehicle itself (e.g., the vehicle V described later) facing forward as an input image; a driving potential field generation means (e.g., the driving potential field generation unit 3 described later), which generates a driving potential field based on the aforementioned input image, the driving potential field representing the distribution of driving potential with respect to the future driving position of the aforementioned vehicle; a target trajectory generation means (e.g., the target trajectory generation unit 4 described later), which generates a target trajectory for the aforementioned vehicle based on the gradient of the aforementioned driving potential in the aforementioned driving potential field; and a driving control means (e.g., the driving control unit 5 described later), which operates a steering mechanism (e.g., the electric power steering device 9 described later) based on the aforementioned target trajectory; and the aforementioned driving potential... The field generation method generates the aforementioned driving potential field by performing the following processes: extracting reference subject information related to the position of a reference subject or the position of the boundary line of the reference subject from the aforementioned input image; setting a low potential point at a position determined based on the aforementioned reference subject information within the central region (e.g., the central region CC described later) of the aforementioned input image; determining the position of an obstacle that would hinder the safe driving of the aforementioned vehicle within the aforementioned input image; setting a first high potential point in the region of the aforementioned input image where the obstacle is reflected; setting the value of the driving potential at the aforementioned low potential point as a first set value; setting the value of the driving potential at the aforementioned first high potential point as a second set value that is greater than the aforementioned first set value; and interpolating the value of the driving potential in the region between the aforementioned low potential point and the aforementioned first high potential point in the aforementioned input image using a value between the aforementioned first set value and the aforementioned second set value.

[0012] (2) Preferably, the aforementioned driving potential field generation means obtains the position of the fitting line (e.g., the left sky fitting line Fa1, as described later) of the left sky boundary line (e.g., the left sky boundary line La1, as described later) and the position of the fitting line (e.g., the right sky fitting line Fa2, as described later) of the right sky boundary line (e.g., the right sky fitting line La2, as described later) in the aforementioned input image as the aforementioned reference subject information. When the intersection point (e.g., the intersection point Pa, as described later) of the aforementioned left sky boundary line fitting line and the aforementioned right sky boundary line fitting line exists in the aforementioned central region, the aforementioned low potential point is set at the intersection point.

[0013] (3) Preferably, the aforementioned driving potential field generation means obtains the position of the fitting line of the left road boundary line (e.g., the left road fitting line Fb1 described later) and the position of the fitting line of the right road boundary line (e.g., the right road fitting line Fb2 described later) in the aforementioned input image as the aforementioned reference subject information. When the intersection point (e.g., the intersection point Pb described later) of the aforementioned fitting line of the left road boundary line and the aforementioned fitting line of the right road boundary line exists in the aforementioned central region, the aforementioned low potential point is set at the intersection point.

[0014] (4) Preferably, the aforementioned driving potential field generation means obtains the position of the road boundary line in the aforementioned input image as the aforementioned reference subject information, and when the uppermost point of the aforementioned road boundary line in the aforementioned input image (e.g., the uppermost point Pc described later) exists in the aforementioned central region, the aforementioned low potential point is set at the uppermost point.

[0015] (5) Preferably, the aforementioned driving potential field generation means obtains the position of the left sky fitting line of the left sky boundary line, the position of the right sky fitting line of the right sky boundary line, the position of the left road fitting line of the left road boundary line, the position of the right road fitting line of the right road boundary line, and the position of the road boundary line in the aforementioned input image as the aforementioned reference subject information, and sets the aforementioned low potential point at the position determined by two or more points existing in the aforementioned central region among the intersection of the aforementioned left sky fitting line and the aforementioned right sky fitting line, the intersection of the aforementioned left road fitting line and the aforementioned right road fitting line, and the uppermost point of the aforementioned road boundary line in the aforementioned input image.

[0016] (6) Preferably, when there is a vehicle in the aforementioned central region that is identified as a follower, the aforementioned driving potential field generation means obtains the position of the aforementioned vehicle in the aforementioned input image as the aforementioned reference subject information.

[0017] (7) In this case, it is preferable that the aforementioned driving potential field generation method obtains the position of the road boundary line or the position of the preceding vehicle identified as the object to be followed as the aforementioned reference subject information when the aforementioned vehicle is driving on a road that is blocked by the overhead structure.

[0018] (8) Preferably, the aforementioned driving potential field generation means sets the value of the aforementioned driving potential in the inner side of the low potential range centered on the aforementioned low potential point, in a manner that the gradient becomes steeper as it approaches the aforementioned low potential point; sets the value of the aforementioned driving potential in the inner side of the first high potential range centered on the aforementioned first high potential point, in a manner that the gradient becomes steeper as it approaches the aforementioned first high potential point; and sets the value of the aforementioned driving potential in the outer side of the aforementioned low potential range and the aforementioned first high potential range, in a manner that the gradient is fixed and gentler than that of the inner side of the aforementioned low potential range and the aforementioned first high potential range.

[0019] (9) Preferably, the aforementioned driving potential field generation means generates the aforementioned driving potential field by further performing the following processing: determining the position of the driving dividing line of the aforementioned vehicle in the aforementioned input image; setting a second high potential point in the region in the aforementioned input image where the aforementioned driving dividing line is reflected; setting the value of the aforementioned driving potential at the aforementioned second high potential point to a third setting value that is greater than the aforementioned first setting value; and interpolating the value of the aforementioned driving potential between the aforementioned low potential point and the aforementioned second high potential point in the aforementioned input image using the value between the aforementioned first setting value and the aforementioned third setting value.

[0020] (10) Preferably, the aforementioned driving potential field generation method does not set the aforementioned second high potential point when the vehicle is performing or is scheduled to perform a lane change.

[0021] (The effect of the invention)

[0022] (1) In this invention, the input image acquisition means acquires an image taken by a camera viewed from the vehicle's front side as the input image. The driving potential field generation means generates a driving potential field based on the input image, wherein the driving potential field represents the distribution of driving potential relative to the future driving position of the vehicle. The target trajectory generation means generates a target trajectory for the vehicle based on the gradient of the driving potential in the driving potential field. The driving control means operates the steering mechanism based on the target trajectory. In addition, in this invention, the driving potential field generation means extracts reference subject information related to a specific reference subject from the input image, sets a low potential point in the central region of the input image and at a position determined based on the reference subject information, determines the position of obstacles in the input image, and then sets a first high potential point in the region in the input image where the obstacles are reflected. Furthermore, the driving potential field generation method sets the driving potential value at the low potential point as a first preset value, and sets the driving potential value at the first high potential point as a second preset value that is larger than the first preset value. Using the values ​​between these first and second preset values, the driving potential values ​​in the region between the low potential point and the first high potential point in the input image are interpolated, thereby generating a driving potential field from the input image. Thus, according to the present invention, without using external sensors other than cameras and map information, the driving potential field and target track can be generated solely by the positions of reference subjects and obstacles reflected in the input image. Therefore, the vehicle can be controlled with less load, thereby promoting the development of sustainable transportation systems.

[0023] (2) In this invention, the driving potential field generation means acquires the positions of the fitted lines (hereinafter also referred to as "left sky fitted lines") of the left sky boundary line (i.e., the boundary line between the sky and ground objects on the left side when viewed from the vehicle) and the fitted lines (hereinafter also referred to as "right sky fitted lines") of the right sky boundary line (i.e., the boundary line between the sky and ground objects on the right side when viewed from the vehicle) in the input image as reference subject information. Furthermore, when the intersection of the left sky fitted lines and the right sky fitted lines exists within the central region, the driving potential field generation means sets a low potential point at that intersection point as the end of the target path. Thus, according to this invention, a driving potential field can be generated by simple calculations on the input image.

[0024] (3) In this invention, the driving potential field generation means acquires the positions of the fitted lines (hereinafter also referred to as "left road fitted lines") of the left road boundary line (i.e., the left edge line of the road on which the vehicle is traveling, as viewed from the vehicle) and the fitted lines (hereinafter also referred to as "right road fitted lines") of the right road boundary line (i.e., the right edge line of the road on which the vehicle is traveling, as viewed from the vehicle) in the input image as reference subject information. Furthermore, when the intersection of the left road fitted lines and the right road fitted lines exists within the central region, the driving potential field generation means sets a low potential point at that intersection as the end of the target path. Thus, according to this invention, a driving potential field can be generated by simple computation of the input image.

[0025] (4) In this invention, the driving potential field generation means acquires the position of the road boundary line (i.e., the line that combines the aforementioned left and right road boundary lines) in the input image as reference subject information. Furthermore, if the uppermost point of the road boundary line in the input image exists within the central region, the driving potential field generation means sets a low potential point at that uppermost point as the end of the target path. Therefore, according to this invention, a driving potential field can be generated by simple computation of the input image.

[0026] (5) In this invention, the driving potential field generation method acquires the positions of the left sky-fitting line, the right sky-fitting line, the left road-fitting line, the right road-fitting line, and the road boundary line in the input image as reference subject information. Furthermore, the driving potential field generation method sets a low potential point as the end of the target path at a position determined by at least two points existing within the central region of the intersection of the left and right sky-fitting lines, the intersection of the left and right road-fitting lines, and the uppermost endpoint of the road boundary line in the input image. Therefore, according to this invention, a driving potential field can be generated by simple computation of the input image.

[0027] (6) In this invention, when a vehicle identified as a follower is present in the central region, the driving potential field generation method acquires the position of the vehicle in the input image as reference subject information, and sets a low potential point at the position determined based on the position of the vehicle as the end of the target path. Thus, according to this invention, a driving potential field such as automatically following a vehicle can be generated by simple calculation of the input image.

[0028] (7) In this invention, when the vehicle is traveling on a road where the sky is obscured by overhead structures, such as in a tunnel, i.e., when the sky is barely visible in the input image, the method acquires the position of the road boundary line or the position of a preceding vehicle identified as a follower as reference subject information, and sets the position of the low potential point based on the positions of these road boundary lines and the positions of the preceding vehicles. Thus, according to this invention, even when the sky is not sufficiently visible in the input image, the low potential point can be set at an appropriate location.

[0029] (8) In this invention, the driving potential field setting means sets the driving potential value inside the low potential range centered on the low potential point, in a manner that the gradient becomes steeper as it approaches the center; inside the first high potential range centered on the first high potential point, the driving potential value is set in a manner that the gradient becomes steeper as it approaches the center; and outside the low potential range and the first high potential range, the driving potential value is set in a manner with a fixed gradient, which is gentler than inside the low potential range and the first high potential range. According to this invention, by generating the driving potential field according to the above process, a driving potential field such as generating the following target path can be generated by simple calculation, the target path avoiding the first high potential point where obstacles exist and ending at the low potential point.

[0030] (9) In this invention, the driving potential field generation means sets a second high potential point in the region of the vehicle's driving dividing line reflected in the input image. Furthermore, the driving potential field generation means sets the driving potential value at the second high potential point to a third setting value that is larger than a first setting value, and interpolates the driving potential value between the low potential point and the second high potential point in the input image using the value between the first and third setting values, thereby generating a driving potential field. Therefore, according to this invention, by generating a driving potential field according to the above process, a driving potential field such as generating a target path can be generated by simple calculations, the target path avoiding the first high potential point where an obstacle exists and the second high potential point where a driving dividing line exists, and ending at a low potential point.

[0031] (10) If a second high potential point is set at the location of the vehicle's driving lane divider as described above, a target path such as avoiding the driving lane divider will be generated. Therefore, in this invention, the driving potential field generation means does not set such a second high potential point when the vehicle is performing or is scheduled to perform a lane change. Thus, according to this invention, a driving potential field such as generating a target path across the driving lane divider can be generated by simple calculation. Attached Figure Description

[0032] Figure 1This is a schematic diagram illustrating the structure of a vehicle equipped with a vehicle control device according to an embodiment of the present invention.

[0033] Figure 2 This is a functional block diagram of the vehicle control device.

[0034] Figure 3 It is a schematic diagram illustrating the driving potential field generated by the driving potential field generating unit.

[0035] Figure 4 It is a flowchart illustrating the specific process of generating and processing the driving potential field.

[0036] Figure 5 This is a diagram used to illustrate the process of searching for low potential points.

[0037] Figure 6 This is a diagram used to illustrate the process of searching for high potential points.

[0038] Figure 7 This is a diagram illustrating a setting example of the driving potential values ​​at a portion including the first high potential point and the low potential point.

[0039] Figure 8 This is a diagram illustrating an example of a driving potential field generated through a driving potential field generation process. Detailed Implementation

[0040] Hereinafter, a vehicle control device according to an embodiment of the present invention will be described with reference to the drawings.

[0041] Figure 1 This is a schematic diagram illustrating the structure of a vehicle V equipped with the vehicle control device 1 of this embodiment. Figure 1 The upper section shows a plan view of vehicle V. Figure 1 The lower section shows a side view. Furthermore, the following description pertains to the case where vehicle V is a so-called right-hand drive four-wheeled vehicle, meaning that when viewed along the direction of travel, the driver's seat is located on the right side in the vehicle's width direction; however, the invention is not limited to this. Vehicle V can also be a so-called left-hand drive four-wheeled vehicle, meaning that when viewed along the direction of travel, the driver's seat is located on the left side in the vehicle's width direction.

[0042] The vehicle V includes: an electric power steering system 9, which is a steering mechanism for steering the left and right front wheels Wf; a power unit 8, which is a driving drive device that generates driving force to rotate the drive wheels, i.e., the front wheels Wf, in the vehicle V; a braking device 7, which generates braking force to stop the rotation of the front wheels Wf and the rear wheels Wr; an onboard camera C, which captures images of the area around the vehicle V; and a vehicle control device 1, which controls the electric power steering system 9, the power unit 8, and the braking device 7 based on the images captured by the onboard camera C.

[0043] The electric power steering system 9 includes: a gearbox 93, a pinion shaft 92 extending from the steering wheel 91 that receives the driver's steering operation and connecting it to the left and right front wheels Wf; an electric motor 94, disposed in the gearbox 93; and a steering sensor 95, which detects the steering amount of the steering wheel 91.

[0044] The gearbox 93 includes a rack shaft extending along the vehicle width direction and meshing with a pinion shaft 92, and tie rods connecting the two ends of the rack shaft to the left and right front wheels Wf. By converting the rotational motion of the steering wheel 91 caused by the driver's steering operation into movement along the vehicle width direction, the left and right front wheels Wf are turned in the direction of travel. The electric motor 94 rotates according to the control signal output from the vehicle control unit 1, generating a driving force to assist the driver's steering operation or to automatically turn the front wheels Wf without the driver's steering operation. The steering sensor 95 detects the steering amount of the steering wheel 91 and sends a signal corresponding to the detected value to the vehicle control unit 1.

[0045] The power unit 8 is the source of driving force. Based on the driver's acceleration / deceleration operation on the accelerator pedal (not shown) or control signals output from the vehicle control device 1, it generates a driving force that causes the front wheels Wf to rotate, enabling the vehicle V to move forward or backward in the direction of travel. Hereinafter, the power unit 8 will be described using a drive motor that generates driving force by consuming electricity supplied from a high-voltage battery or fuel cell stack (not shown); however, the invention is not limited to this. Alternatively, the power unit 8 may be an engine that generates driving force by consuming fuel stored in a fuel tank (not shown), or a transmission that transmits the output of such an engine to the front wheels Wf after gear shifting.

[0046] The braking device 7 includes a disc brake and a parking brake, which generate braking force based on the driver's braking operation on the brake pedal (not shown) or control signals output from the vehicle control device 1. The disc brake mainly generates braking force to fasten the discs mounted on the axles of each wheel Wf and Wr when driving, thereby slowing down or stopping the rotation of each wheel Wf and Wr. The parking brake mainly generates braking force to keep the rotation of each wheel Wr and Wf in a stopped state when parked.

[0047] Viewed from vehicle V, the vehicle-mounted camera C faces forward along the direction of travel. Furthermore, this embodiment describes the case where the vehicle-mounted camera C is positioned at the center of the vehicle body in the width direction, but the invention is not limited thereto.

[0048] The vehicle control unit 1 controls the electric power steering system 9, the power unit 8, and the braking system 7 based on an image of the front side of the vehicle V captured by the onboard camera C. The vehicle control unit 1 is a computer composed of the following hardware: a processing unit such as a central processing unit (CPU), an auxiliary storage unit such as a hard disk drive (HDD) or solid state drive (SSD) that stores the program that enables the processing unit to perform the driving potential field generation process described later, and a main storage unit such as random access memory (RAM) that stores data temporarily required by the processing unit when executing the program.

[0049] Figure 2 This is a functional block diagram of the vehicle control device 1. The vehicle control device 1 comprises an input image acquisition unit 2, a driving potential field generation unit 3, a target trajectory generation unit 4, and a driving control unit 5, using the hardware structure described above.

[0050] The input image acquisition unit 2 acquires an image of the front side of the vehicle V captured by the vehicle-mounted camera C as the input image. The input image acquisition unit 2 then sends information related to the acquired input image to the driving potential field generation unit 3.

[0051] The driving potential field generation unit 3 generates a driving potential field based on the input image sent from the input image acquisition unit 2. The driving potential field represents the distribution of the driving potential on the input image with respect to the future driving position of the vehicle V (i.e., the distribution of the driving potential on the two-dimensional image coordinates of the input image). The driving potential field generation unit 3 sends information related to the generated driving potential field to the target track generation unit 4.

[0052] Figure 3 This is a schematic diagram illustrating the driving potential field generated by the driving potential field generation unit 3. Figure 3 In the diagram, using the input image as a background, the driving potential values ​​are illustrated by distinguishing them with different colors. More specifically, the larger the driving potential value, the darker the color used to represent it. Additionally, in... Figure 3 In the diagram, a white circle represents the point where the driving potential value is minimum (the point of minimum potential). For example... Figure 3 As shown, the so-called driving potential field is a scalar function of the driving potential defined on two-dimensional image coordinates. Furthermore, the specific process of generating the driving potential field using the driving potential field generation unit 3 will be discussed later. Figures 4-8 We'll explain later.

[0053] Return to Figure 2The target trajectory generation unit 4 calculates the gradient of the driving potential field generated by the driving potential field generation unit 3, and generates a target trajectory for the vehicle V in image coordinates based on the gradient of the driving potential field and the current steering angle of the vehicle V. The target trajectory generation unit 4 sends information related to the generated target trajectory to the driving control unit 5. Here, the gradient of the driving potential field is equivalent to a vector function of the partial derivative of each image coordinate component of the driving potential field defined in two-dimensional image coordinates. Thus, the target trajectory generation unit 4 generates a target trajectory such as starting from the front of the vehicle V and moving along the valley of the driving potential to the point of minimum potential (see reference). Figure 3 (thick dashed line in the middle).

[0054] The driving control unit 5 operates the electric power steering unit 9, the power unit 8, and the braking device 7 based on the target track generated by the target track generation unit 4. More specifically, the driving control unit 5 operates the electric power steering unit 9, the power unit 8, and the braking device 7 in a manner that causes the vehicle V to travel along the target track defined on the image coordinates.

[0055] Figure 4 It is a flowchart illustrating the specific process of generating and processing the driving potential field. Figure 4 The driving potential field generation process shown is repeatedly executed in the driving potential field generation unit 3 under a specific control cycle during the driving process of vehicle V.

[0056] First, in step ST1, the driving potential field generation unit 3 acquires the input image sent from the input image acquisition unit 2 and moves to step ST2.

[0057] Next, in step ST2, the driving potential field generation unit 3 performs segmentation processing on the input image obtained in step ST1, classifies the subject reflected in the input image into multiple categories, generates an image (hereinafter also referred to as "edge image") by extracting the boundary lines of each category, and transfers it to step ST3.

[0058] Next, in step ST3, the driving potential field generation unit 3 performs a low potential point search process based on the edge image extracted from the input image and information related to each category reflected in the input image (hereinafter also referred to as "category information"), and then proceeds to step ST4. Here, a low potential point refers to a point in the image coordinate system where the driving potential value is at its minimum, such as... Figure 3 The point shown refers to the end point of the target orbit. The following refers to... Figure 5 The specific process of searching for low potential points is explained in detail.

[0059] Figure 5This diagram illustrates an example of an input image and serves to explain the flow of the low-potential point search process. The driving potential field generation unit 3 first extracts information from edge images and category information, such as the position of a reference subject in the image coordinates or information related to the position of the boundary line of that reference subject, which is determined for setting the position of the low-potential point, as reference subject information. Next, the driving potential field generation unit 3 sets the low-potential point at a position determined based on the extracted reference subject information within a central region CC, which is determined in a manner including the center of the image coordinates. Several embodiments of the low-potential point search process will be described below.

[0060] <Example 1>

[0061] In Embodiment 1, the driving potential field generation unit 3 uses subjects classified as "above" from among the multiple subjects reflected in the input image as reference subjects to set the position of the low potential point. In this case, the driving potential field generation unit 3 obtains the positions of the fitted lines Fa1 of the left sky boundary La1 and Fa2 of the right sky boundary La2 from edge images and category information, etc., as reference subject information. Here, the left sky boundary La1 refers to the boundary line between the sky and ground objects on the left side of the input image when viewed from vehicle V, and the right sky boundary La2 refers to the boundary line between the sky and ground objects on the right side of the input image when viewed from vehicle V. Furthermore, the so-called fitting line of the left sky boundary (hereinafter also referred to as the "left sky fitting line") Fa1 refers to the line obtained by fitting the left sky boundary La1 with a known algorithm, such as by fitting a linear function, and the so-called fitting line of the right sky boundary (hereinafter also referred to as the "right sky fitting line") Fa2 refers to the line obtained by fitting the right sky boundary La2 with a known algorithm, such as by fitting a linear function.

[0062] Furthermore, when the intersection point Pa of the left sky fitting line Fa1 and the right sky fitting line F2a obtained above exists within a predetermined central region CC, the driving potential field generation unit 3 sets a low potential point at the intersection point Pa. Alternatively, when the intersection point Pa does not exist within the central region CC, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 2 to 5, etc.

[0063] Furthermore, when the vehicle V is traveling on a road where the overhead structure obstructs its view (for example, when the vehicle V is traveling inside a tunnel), the left sky fitting line Fa1 and the right sky fitting line Fa2 cannot be obtained. Therefore, in this case, the driving potential field generation unit 3 also sets the position of the low potential point based on other embodiments 2 to 5, etc.

[0064] <Example 2>

[0065] In Embodiment 2, the driving potential field generation unit 3 uses subjects classified as "road" among the multiple subjects reflected in the input image as reference subjects to set the position of the low potential point. In this case, the driving potential field generation unit 3 obtains the positions of the fitting line Fb1 of the left road boundary and the fitting line Fb2 of the right road boundary from edge images and category information as reference subject information. Here, the left road boundary refers to the left edge line of the road on which vehicle V is traveling in the input image as viewed from vehicle V, and the right road boundary refers to the right edge line of the road on which vehicle V is traveling in the input image as viewed from vehicle V. In addition, the fitting line of the left road boundary (hereinafter also referred to as "left road fitting line") Fb1 refers to the line obtained by fitting the left road boundary based on a known algorithm, such as by fitting a linear function, and the fitting line of the right road boundary (hereinafter also referred to as "right road fitting line") Fb2 refers to the line obtained by fitting the right road boundary based on a known algorithm, such as by fitting a linear function. Furthermore, in Figure 4 In the example, since the left and right road boundary lines almost coincide with their fitted lines Fb1 and Fb2, only the fitted lines Fb1 and Fb2 are shown.

[0066] Furthermore, when the intersection point Pb of the left road fitting line Fb1 and the right road fitting line Fb2 obtained above exists within the central region CC, the driving potential field generation unit 3 sets a low potential point at the intersection point Pb. Alternatively, when the intersection point Pb does not exist within the central region CC, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 1, 3 to 5, etc.

[0067] <Example 3>

[0068] In Embodiment 3, the driving potential field generation unit 3, similar to Embodiment 2, uses subjects classified as "roads" as reference subjects among the multiple subjects reflected in the input image to set the position of low potential points. In this case, the driving potential field generation unit 3 obtains the position of the road boundary line as reference subject information from edge images and category information, etc. Here, the so-called road boundary line refers to the line that combines the aforementioned left road boundary line and right road boundary line.

[0069] Furthermore, when the uppermost endpoint Pc of the road boundary line obtained above exists within the central region CC, the driving potential field generation unit 3 sets a low potential point at the uppermost endpoint Pc. Alternatively, when the uppermost endpoint Pc does not exist within the central region CC, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 1-2, 4-5, etc.

[0070] <Example 4>

[0071] In Embodiment 4, the driving potential field generation unit 3 uses subjects classified as "above" or "road" among the multiple subjects reflected in the input image as reference subjects to set the position of the low potential point.

[0072] In this case, the driving potential field generation unit 3 obtains the positions of the left sky fitting line Fa1, the right sky fitting line Fa2, the left road fitting line Fb1, the right road fitting line Fb2, and the road boundary line from edge images and category information, etc., as reference subject information.

[0073] Furthermore, the driving potential field generation unit 3 sets low potential points at positions determined based on two or more points existing within the central region CC, among the intersection point Pa of the left sky fitting line Fa1 and the right sky fitting line Fa2, the intersection point Pb of the left road fitting line Fb1 and the right road fitting line Fb2, and the uppermost endpoint Pc of the road boundary line. More specifically, the driving potential field generation unit 3 sets low potential points, for example, at the geometric centroids of two or more points existing within the central region CC among the aforementioned three points Pa, Pb, and Pc.

[0074] <Example 5>

[0075] In Embodiment 5, the driving potential field generation unit 3 uses subjects classified as "moving vehicles" from among the multiple subjects reflected in the input image as reference subjects to set the position of the low potential point. In this case, the driving potential field generation unit 3 obtains the position of the moving vehicle (more specifically, the position of the center point Pd of the moving vehicle) from edge images and category information, etc., as reference subject information. Here, "moving vehicle" refers to a vehicle traveling in the same lane as vehicle V and in front of vehicle V, heading in the same direction as vehicle V.

[0076] Furthermore, when the preceding vehicle, as extracted above, is identified as a follower of vehicle V through a process not shown, and the center point Pd of the preceding vehicle exists within the central region CC, the driving potential field generation unit 3 sets a low potential point at the center point Pd of the preceding vehicle. Additionally, when there is no preceding vehicle, the preceding vehicle is not identified as a follower, or the center point Pd of the preceding vehicle does not exist within the central region CC, the driving potential field generation unit 3 sets the position of the low potential point based on other embodiments 1 to 4, etc.

[0077] Return to Figure 4In step ST4, the driving potential field generation unit 3 performs a high potential point search process based on edge images and category information extracted from the input image, and then proceeds to step ST5. Here, a high potential point refers to a point on the image coordinate system where the driving potential value is at its maximum. (Refer to the following...) Figure 6 The specific process of searching for high potential points is explained in detail.

[0078] Figure 6 This is a diagram illustrating an example of an input image, used to explain the process of searching for high potential points. The driving potential field generation unit 3 first determines the positions of obstacles that would hinder the safe driving of vehicle V on the image coordinates from edge images and category information, etc. Here, obstacles that would hinder the safe driving of vehicle V include, for example, other vehicles besides the vehicle in front, curbs, and roadside trees. Next, the driving potential field generation unit 3... Figure 6 As indicated by the circular notation, a first high potential point is set in the region of the obstacle that is determined in the image coordinates by the above process.

[0079] Furthermore, the driving potential field generation unit 3 determines the position of the driving demarcation line of vehicle V on the image coordinates from edge images and category information, etc. Next, the driving potential field generation unit 3... Figure 6 As indicated by the square symbol, a second high potential point is set in the region of the driving dividing line determined by the above process in the image coordinates.

[0080] These driving lane markings, unlike obstacles set at the first high potential point, are not objects that themselves would obstruct the safe driving of vehicle V. However, if vehicle V crosses the driving lane markings, it may threaten the safe driving of other vehicles. Therefore, the driving potential field generation unit 3 sets a second high potential point on these driving lane markings. However, if a second high potential point is set on these driving lane markings, vehicle V will be unable to cross the driving lane markings and thus will be unable to change lanes. Therefore, it is preferable that the driving potential field generation unit 3 does not set a second high potential point when vehicle V is performing or planning to perform a lane change.

[0081] Return to Figure 4 In step ST5, the driving potential field generation unit 3 performs a driving potential value setting process, which sets the driving potential value on the image coordinates based on the positions of the low potential point searched in step ST3 and the high potential point searched in step ST4, and then ends the process. Figure 4 The following refers to the processing. Figure 7 The specific process for setting and processing the driving potential value is explained.

[0082] Figure 7 It is depicted as follows Figure 6 The diagram shown in line VI-VI illustrates an example of setting the driving potential values ​​at a portion of the first high potential point and the low potential point.

[0083] First, the driving potential field generation unit 3 sets the driving potential value at the low potential point to a predetermined first set value, sets the driving potential value at the first high potential point to a second set value that is larger than the first set value, and sets the driving potential value at the second high potential point to a third set value that is larger than the first set value. The following description pertains to the case where the second and third set values ​​are set to equal values, but the present invention is not limited thereto. The second and third set values ​​may also be set to different values.

[0084] Next, the driving potential field generation unit 3 interpolates the driving potential value of the region between the low potential point and the first high potential point on the image coordinate using the value between the first set value and the second set value, and interpolates the driving potential value of the region between the low potential point and the second high potential point using the value between the first set value and the third set value.

[0085] More specifically, the driving potential field generation unit 3 sets the value of the driving potential in the inner side of the low potential range centered on the low potential point, in the manner that the gradient becomes steeper as it approaches the low potential point; in the inner side of the first high potential range centered on the first high potential point, in the manner that the gradient becomes steeper as it approaches the first high potential point; and in the inner side of the second high potential range centered on the second high potential point, in the manner that the gradient becomes steeper as it approaches the second high potential point.

[0086] Next, the driving potential field generation unit 3 sets the value of the driving potential outside the low potential range, the first high potential range, and the second high potential range, with a fixed gradient that is gentler than the inner areas of the low potential range, the first high potential range, and the second high potential range. Through this process, the driving potential field generation unit 3 sets the value of the driving potential across the entire region on the image coordinates, thereby generating a driving potential field.

[0087] Figure 8 This is a diagram illustrating an example of a driving potential field generated through the driving potential field generation process described above. Figure 8 In the diagram, the gradient of the driving potential field, which is a vector function, is indicated by an arrow. Additionally, in... Figure 8 In the example, to make the illustration clearer, the case where only the second high potential point is set without setting the first high potential point is shown.

[0088] exist Figure 8 In the example shown, the ridgelines of a mountain form the driving position on the two driving lanes extending towards the center of the input image on the left and right sides of the vehicle. Therefore, in Figure 8 In the example shown, a valley line of driving potential is formed, extending between the two driving lanes and reaching the center of the input image. Therefore, in Figure 8 Under the driving potential field shown, the target trajectory is generated by the target trajectory generation unit 4. Figure 8 The target track extends along the valley line of the driving potential field, as shown by the thick dashed line.

[0089] The vehicle control device 1 according to this embodiment has the following effects.

[0090] (1) The input image acquisition unit 2 acquires an image taken by the vehicle-mounted camera C facing forward from the vehicle V as the input image. The driving potential field generation unit 3 generates a driving potential field based on the input image. The driving potential field represents the distribution of driving potential relative to the future driving position of the vehicle V. The target track generation unit 4 generates a target track for the vehicle V based on the gradient of the driving potential in the driving potential field. The driving control unit 5 operates the electric power steering device 9, the power equipment 8, and the braking device 7, etc., based on the target track. In addition, the driving potential field generation unit 3 extracts reference subject information related to a specific reference subject from the input image. In the central region CC of the input image and at a position determined based on the reference subject information, a low potential point is set, the position of the obstacle in the input image is determined, and then a first high potential point is set in the region in the input image where the obstacle is reflected. Furthermore, the driving potential field generation unit 3 sets the driving potential value at the low potential point as a first set value, and sets the driving potential value at the first high potential point as a second set value that is larger than the first set value. Using the values ​​between these first and second set values, it interpolates the driving potential values ​​in the region between the low potential point and the first high potential point in the input image, thereby generating a driving potential field from the input image. Thus, according to this embodiment, without using external sensors other than the vehicle-mounted camera C and map information, the driving potential field and target track can be generated solely by the positions of the reference subject and obstacles reflected in the input image. Therefore, the vehicle can be controlled with less load, thereby promoting the development of a sustainable transportation system.

[0091] (2) The driving potential field generation unit 3 acquires the positions of the left and right sky-fitting lines in the input image as reference subject information. Furthermore, when the intersection point Pa of the left and right sky-fitting lines exists within the central region CC, the driving potential field generation unit 3 sets a low potential point at that intersection point Pa as the end of the target path. Therefore, according to the present invention, a driving potential field can be generated by simple calculation of the input image.

[0092] (3) The driving potential field generation unit 3 acquires the positions of the left and right road fitting lines in the input image as reference subject information. Furthermore, when the intersection point Pb of the left and right road fitting lines exists within the central region CC, the driving potential field generation unit 3 sets a low potential point at this intersection point Pb as the end of the target path. Therefore, according to the present invention, a driving potential field can be generated by simple calculation of the input image.

[0093] (4) The driving potential field generation unit 3 acquires the position of the road boundary line in the input image as reference subject information. Furthermore, when the uppermost endpoint Pc of the road boundary line in the input image exists within the central region CC, the driving potential field generation unit 3 sets a low potential point at the uppermost endpoint Pc as the end of the target path. Thus, according to the vehicle control device 1, the driving potential field can be generated by simple calculations on the input image.

[0094] (5) The driving potential field generation unit 3 acquires the positions of the left sky fitting line, the right sky fitting line, the left road fitting line, the right road fitting line, and the road boundary line in the input image as reference subject information. Furthermore, the driving potential field generation unit 3 sets a low potential point as the end of the target path at a position determined by at least two points within the central region CC, based on the intersection point Pa of the left and right sky fitting lines, the intersection point Pb of the left and right road fitting lines, and the uppermost point Pc of the road boundary line in the input image. Thus, according to the vehicle control device 1, the driving potential field can be generated by simple calculations on the input image.

[0095] (6) When a vehicle identified as a follower is present in the central region CC, the driving potential field generation unit 3 acquires the position of the vehicle in the input image as reference subject information, and sets a low potential point at the end of the target path at the position determined based on the position of the vehicle. Thus, according to the vehicle control device 1, a driving potential field such as automatically following a vehicle can be generated by simple calculation of the input image.

[0096] (7) When the vehicle is traveling on a road where the sky is obscured by overhead structures, such as in a tunnel, i.e., when the sky is barely visible in the input image, the driving potential field generation unit 3 acquires the position of the road boundary line or the position of the preceding vehicle identified as the object being followed as reference subject information, and sets the position of the low potential point based on the position of these road boundary lines and the position of the preceding vehicle. Thus, according to the vehicle control device 1, even when the sky is not fully visible in the input image, the low potential point can be set at an appropriate position.

[0097] (8) The driving potential field generation unit 3 sets the driving potential value inside the low potential range centered on the low potential point, in a manner that the gradient becomes steeper as it approaches the center; inside the first high potential range centered on the first high potential point, the driving potential value is set in a manner that the gradient becomes steeper as it approaches the center; outside the low potential range and the first high potential range, the driving potential value is set in a manner with a fixed gradient and a gentler gradient than inside the low potential range and the first high potential range. According to the vehicle control device 1, by generating the driving potential field according to the above process, a driving potential field such as generating the following target path can be generated by simple calculation, the target path avoiding the first high potential point where there is an obstacle and ending at the low potential point.

[0098] (9) The driving potential field generation unit 3 sets a second high potential point in the region of the driving dividing line of vehicle V reflected in the input image. Furthermore, the driving potential field generation unit 3 sets the driving potential value at the second high potential point to a third setting value that is larger than the first setting value, and interpolates the driving potential value between the low potential point and the second high potential point in the input image using the value between the first setting value and the third setting value, thereby generating a driving potential field. Thus, according to the vehicle control device 1, by generating the driving potential field according to the above process, a driving potential field such as generating a target path can be generated by simple calculation, the target path avoiding the first high potential point where an obstacle exists and the second high potential point where a driving dividing line exists, and ending at a low potential point.

[0099] (10) If a second high potential point is set at the location of the driving lane divider of vehicle V as described above, a target path such as avoiding the driving lane divider will be generated. Therefore, in vehicle control device 1, the driving potential field generation unit 3 does not set such a second high potential point when vehicle V is performing or is scheduled to perform a lane change. Thus, according to the present invention, a driving potential field such as generating a target path across the driving lane divider can be generated by simple calculation.

[0100] The above description illustrates one embodiment of the present invention, but the invention is not limited thereto. Appropriate modifications to the details and structure are also possible within the scope of the spirit of the invention.

[0101] Figure Labels

[0102] V: Vehicle (this vehicle)

[0103] C: Vehicle-mounted camera (camera)

[0104] 1: Vehicle control device (Vehicle control device)

[0105] 2: Input Image Acquisition Unit (Input Image Acquisition Method)

[0106] 3: Driving potential field generation unit (driving potential field generation method)

[0107] 4: Target orbit generation unit (target orbit generation method)

[0108] 5: Driving Control Unit (Driving Control Methods)

[0109] 7: Braking device

[0110] 8: Power equipment (driving drive device)

[0111] 9: Electric power steering system (steering mechanism)

Claims

1. A vehicle control device characterized by comprising: Possess: input image acquisition means, acquire an image captured by a camera observing toward a front side from a host vehicle as an input image; travel position potential field generation means, generate a travel position potential field based on the input image, the travel position potential field indicating a distribution of a travel position potential for a future travel position of the host vehicle; target trajectory generation means, generate a target trajectory of the host vehicle based on a gradient of the travel position potential in the travel position potential field; and travel control means, operate a steering mechanism based on the target trajectory; and The travel position potential field generation means generates the travel position potential field by performing the following processes: extracting reference subject information related to a position of a reference subject or a position of a boundary line of the reference subject from the input image; setting a low potential point at a position within a central region of the input image and determined based on the reference subject information; determining a position of an obstacle within the input image that becomes an obstacle to safe travel of the host vehicle; setting a first high potential point in a region of the input image in which the obstacle is represented; setting a value of the travel position potential at the low potential point to a first set value; setting a value of the travel position potential at the first high potential point to a second set value that is greater than the first set value; and interpolating a value of the travel position potential for a region between the low potential point and the first high potential point in the input image with a value between the first set value and the second set value.

2. The vehicle control device according to claim 1, wherein The travel position potential field generation means: acquires a position of a fitting line of a left sky boundary line and a position of a fitting line of a right sky boundary line in the input image as the reference subject information, in a case where an intersection of the fitting line of the left sky boundary line and the fitting line of the right sky boundary line exists within the central region, sets the low potential point at the intersection.

3. The vehicle control device according to claim 1, wherein The travel position potential field generation means: acquires a position of a fitting line of a left road boundary line and a position of a fitting line of a right road boundary line in the input image as the reference subject information, in a case where an intersection of the fitting line of the left road boundary line and the fitting line of the right road boundary line exists within the central region, sets the low potential point at the intersection.

4. The vehicle control device according to claim 1, wherein The travel position potential field generation means: acquires a position of a road boundary line in the input image as the reference subject information, in a case where an uppermost end point of the road boundary line in the input image exists within the central region, sets the low potential point at the uppermost end point.

5. The vehicle control device according to claim 1, wherein The travel position potential field generation means: acquires a position of a left sky fitting line of a left sky boundary line in the input image, a position of a right sky fitting line of a right sky boundary line in the input image, a position of a left road fitting line of a left road boundary line in the input image, a position of a right road fitting line of a right road boundary line in the input image, and a position of a road boundary line in the input image as the reference subject information, A low potential point is set at a position determined based on 2 or more points existing within the central region among an intersection of the aforementioned left sky fitting line and the aforementioned right sky fitting line, an intersection of the aforementioned left road fitting line and the aforementioned right road fitting line, and an uppermost end point of the aforementioned road boundary line in the aforementioned input image.

6. The vehicle control device according to claim 1, wherein The aforementioned travel potential field generation means acquires a position of a preceding vehicle recognized as a follow-up object in the aforementioned input image as the aforementioned reference subject information in a case where the preceding vehicle exists within the aforementioned central region.

7. The vehicle control device according to claim 1, wherein The aforementioned travel potential field generation means: In a case where the host vehicle is traveling in a road on which the host vehicle is obstructed from above by an over-head structure, a position of a road boundary line or a position of a preceding vehicle recognized as a follow-up object is acquired as the aforementioned reference subject information.

8. The vehicle control device according to claim 1, wherein The aforementioned travel potential field generation means: Within a low potential range centered on the aforementioned low potential point, the value of the aforementioned travel potential is set in a manner in which the gradient becomes steeper as the aforementioned low potential point is approached, Within a first high potential range centered on the aforementioned first high potential point, the value of the aforementioned travel potential is set in a manner in which the gradient becomes steeper as the aforementioned first high potential point is approached, Outside the aforementioned low potential range and the aforementioned first high potential range, the value of the aforementioned travel potential is set in a manner in which the gradient is fixed and gentler than within the aforementioned low potential range and the aforementioned first high potential range.

9. The vehicle control device according to claim 1, wherein The aforementioned travel potential field generation means generates the aforementioned travel potential field by further performing the following processing: determining a position of a travel division line of the host vehicle within the aforementioned input image; setting a second high potential point in a region of the aforementioned input image in which the travel division line is reflected; setting the value of the aforementioned travel potential at the aforementioned second high potential point to a third set value that is greater than the aforementioned first set value; and interpolating the value of the aforementioned travel potential between the aforementioned low potential point and the aforementioned second high potential point in the aforementioned input image with values between the aforementioned first set value and the aforementioned third set value.

10. The vehicle control device according to claim 9, wherein The aforementioned travel potential field generation means does not set the aforementioned second high potential point in a case where the host vehicle is performing or scheduled to perform a lane change.

11. A vehicle control method characterized by, A host vehicle is controlled by a computer based on an image captured by a camera that observes a forward direction from the host vehicle, and the vehicle control method includes the following steps: an image captured by the camera is acquired as an input image; a travel potential field is generated based on the input image, the travel potential field indicating a distribution of travel potential for a future travel position of the host vehicle; a target track of the host vehicle is generated based on a gradient of the travel potential in the travel potential field; and a steering mechanism is operated based on the target track. In the step of generating the travel potential field, the travel potential field is generated by performing the following processing: reference subject information about a position of a reference subject or a position of a boundary line of the reference subject is extracted from the input image; a low potential point is set at a position determined within a central region of the input image and based on the reference subject information; ​ determining a position of an obstacle that is an obstacle to safe travel of the host vehicle within the input image; setting a first high potential point in a region in which the obstacle is represented in the input image; setting a value of the travel potential at the low potential point to a first set value; setting a value of the travel potential at the first high potential point to a second set value that is greater than the first set value; and interpolating a value of the travel potential in a region between the low potential point and the first high potential point in the input image with a value between the first set value and the second set value.

12. A storage medium, characterized by stores a program that controls a host vehicle by a computer based on an image captured by a camera that observes a forward side with respect to the host vehicle, and the program causes the computer to perform the steps of: acquiring the image captured by the camera as an input image; generating a travel potential field that indicates a distribution of a travel potential with respect to a future travel position of the host vehicle based on the input image; generating a target track of the host vehicle based on a gradient of the travel potential in the travel potential field; and operating a steering mechanism based on the target track. In the step of generating the travel potential field, the travel potential field is generated by causing the computer to perform the processes of: extracting reference subject information related to a position of a reference subject or a position of a boundary line of the reference subject from the input image; setting a low potential point at a position within a central region of the input image and determined based on the reference subject information; determining a position of an obstacle that is an obstacle to safe travel of the host vehicle within the input image; setting a first high potential point in a region in which the obstacle is represented in the input image; setting a value of the travel potential at the low potential point to a first set value; setting a value of the travel potential at the first high potential point to a second set value that is greater than the first set value; and interpolating a value of the travel potential in a region between the low potential point and the first high potential point in the input image with a value between the first set value and the second set value. ​

Images