Information processing device, computer program for information processing, and information processing method

Abstract

To provide an information processing device that can detect, at an early stage, a branching road that branches off from a road on the vehicle is traveling, based on a front image that represents a front environment of the vehicle.SOLUTION: An information processing device comprises: a dissipation point detecting unit that detects a dissipation point where two lane compartment lines intersect within a front image which represents a front environment of a vehicle; a direction detection unit that detects a direction of movement of the vehicle from a predetermined reference position in the front image to the dissipation point; a first determination unit that, when two movement directions are detected by the direction detection unit, determines whether or not an orientation of each of the two movement directions differs from a movement direction previously detected by the direction detection unit by more than a predetermined reference range; and a second determination unit that determines that the branching road that branches off from the road on which the vehicle travels has appeared in the front image when the first determination unit determines that the direction of either of the two movement directions is different.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a computer program for information processing, and an information processing method. [Background technology]

[0002] An automatic control system installed in a vehicle generates a navigation route for the vehicle based on the current position of the vehicle, the destination position of the vehicle, and a navigation map. The automatic control system estimates the current position of the vehicle using map information and controls the vehicle to travel along the navigation route.

[0003] The automatic control system uses a camera to obtain a forward image that represents the environment in front of the vehicle, detects lane markings (or lane boundary lines) that appear in the forward image, identifies the lanes that are defined by the lane markings, and controls the vehicle so that the vehicle stays within the lane.

[0004] When there is a branch point where a branch road branches off from the driving road on which the vehicle is traveling, the driving road and the branch road are displayed in the forward image. The automatic control system detects the lane markings of the driving road and the lane markings of the branch road from the forward image.

[0005] When the vehicle is to travel on the driving road, the automatic control system selects the lane markings of the driving road to control the vehicle, whereas when the vehicle is to travel on the branching road, the automatic control system selects the lane markings of the branching road to control the vehicle.

[0006] Therefore, the automatic control system determines whether or not a branch road appears ahead of the vehicle based on the forward image.

[0007] For example, Patent Document 1 proposes a technology that recognizes the lane in which a vehicle is traveling and adjacent lanes adjacent to the traveling lane based on an image, and determines whether or not the situation is a branching scene in which a branching lane exists that branches off from the traveling lane based on the difference in the tendency of lane width changes between the traveling lane and the adjacent lanes. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2017-123009 A Summary of the Invention [Problem to be solved by the invention]

[0009] However, the technology described in Patent Document 1 has a problem in that it is not possible to determine whether or not a branching scene occurs until an adjacent lane appears to the lane in which the vehicle is traveling.

[0010] Therefore, an object of the present disclosure is to provide an information processing device that can quickly detect a branch road that branches off from a road on which a vehicle is traveling, based on a forward image that represents the environment ahead of the vehicle. [Means for solving the problem]

[0011] (1) According to one embodiment, an information processing device is provided, which is characterized by having a vanishing point detection unit that detects a vanishing point where two lane markings intersect in a forward image that represents an environment in front of a vehicle, a direction detection unit that detects a moving direction of the vehicle from a predetermined reference position in the forward image toward the vanishing point, a first determination unit that, when two moving directions are detected by the direction detection unit, determines whether or not the orientation of each of the two moving directions differs from the moving direction previously detected by the direction detection unit by exceeding a predetermined reference range, and a second determination unit that, when the first determination unit determines that the orientation of either of the two moving directions differs, determines that a branch road branching off from the road on which the vehicle is traveling has appeared in the forward image.

[0012] (2) In the information processing device of (1), it is preferable to have a third judgment unit that judges whether or not the vehicle will exit onto the branch road based on a driving plan representing the planned driving trajectory of the vehicle, which is generated based on the judgment result that a branch road has appeared by the second judgment unit, and a first removal unit that removes the moving direction that is judged to be different from the moving direction previously detected by the direction detection unit when the third judgment unit judges that the vehicle will not exit onto the branch road.

[0013] (3) In the information processing device of (2), when the third judgment unit judges that the vehicle is exiting to a branch road, it is preferable that the direction detection unit determines that the direction of movement that is judged to be different is the direction of movement that was previously detected by the direction detection unit.

[0014] (4) In any of the information processing devices of (1) to (3), it is preferable that the information processing device has a fourth judgment unit that judges, based on the vehicle's navigation route, whether or not there is a plan to exit onto a branch road that branches off from the driving road on which the vehicle is traveling within a predetermined range from the vehicle's current position in the direction of travel, and has a second removal unit that, when the second judgment unit judges that a branch road has appeared and the fourth judgment unit judges that the vehicle does not plan to exit onto the branch road from the driving road, removes from the lane markings detected in the forward image a vanishing point that represents a direction of travel that is judged to be facing in a different direction.

[0015] (5) In the information processing device according to any one of (1) to (4), it is preferable that the predetermined reference position in the forward image represents the position of the vehicle.

[0016] (6) According to another embodiment, there is provided a computer program for information processing, which causes a processor to execute a process including: detecting a vanishing point where two lane markings intersect in a forward image that represents the environment in front of the vehicle; detecting a moving direction of the vehicle from a predetermined reference position in the forward image toward the vanishing point; determining, when two moving directions are detected, whether or not the orientation of each of the two moving directions differs from a previously detected moving direction by more than a predetermined reference range; and, when it is determined that the orientation of either of the two moving directions differs, determining that a branch road branching off from the road on which the vehicle is traveling has appeared in the forward image.

[0017] (7) According to another embodiment, there is provided an information processing method, characterized in that an information processing device executes the following: detects a vanishing point where two lane markings intersect in a forward image that represents the environment in front of the vehicle, detects the moving direction of the vehicle from a predetermined reference position in the forward image toward the vanishing point, and, when two moving directions are detected, determines whether or not the orientation of each of the two moving directions differs from a previously detected moving direction by more than a predetermined reference range, and, when it is determined that the orientation of either of the two moving directions differs, determines that a branch road branching off from the road on which the vehicle is traveling has appeared in the forward image. Effect of the Invention

[0018] An information processing device according to the present disclosure can quickly detect a branch road branching off from a road on which a vehicle is traveling, based on a forward image that represents the environment ahead of the vehicle. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram (part 1) for explaining an outline of the operation of the information processing device according to the present embodiment. [Diagram 2] FIG. 2 is a diagram (part 2) for explaining the outline of the operation of the information processing device according to the present embodiment. [Diagram 3]1 is a hardware configuration diagram of a vehicle in which an information processing device according to an embodiment of the present invention is implemented; [Figure 4] 5 is an example of an operational flowchart relating to information processing of the information processing device of the present embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a forward image. [Figure 6] FIG. 1 is a diagram (part 1) for explaining detection of a vanishing point and a moving direction from a front image. [Figure 7] FIG. 13 is a diagram showing another example of a front image. [Figure 8] FIG. 2 is a diagram (part 2) for explaining detection of a vanishing point and a moving direction from a front image. [Figure 9] FIG. 13 is a diagram showing yet another example of a front image. [Figure 10] FIG. 3 is a diagram (part 3) for explaining detection of a vanishing point and a moving direction from a front image. [Figure 11] 13 is an example of an operational flowchart relating to information processing in the first modified example of the information processing device of the present embodiment. [Figure 12] 13 is an example of an operational flowchart relating to information processing in the second modified example of the information processing device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] 1 and 2 are diagrams for explaining an overview of the operation of the information processing device 12 of this embodiment. Hereinafter, an overview of the information processing operation of the information processing device 12 disclosed in this specification will be described with reference to FIGS.

[0021] The vehicle 10 has an automatic control device 11 and an information processing device 12. The automatic control device 11 controls the operation of the vehicle 10 based on a forward image captured by the camera 2, etc. The forward image shows the environment in front of the vehicle 10. The vehicle 10 may be an autonomous vehicle. The information processing device 12 generates information for the automatic control device 11 to control the vehicle 10 based on the forward image.

[0022] 1, a vehicle 10 is traveling on a road 50. The road 50 has a left lane marking 51, a right lane marking 52, and a lane 53 defined by the left lane marking 51 and the right lane marking 52. The vehicle 10 is traveling on the lane 53.

[0023] The information processing device 12 detects two lane markings 101, 102 in a forward image 100 captured by the camera 2. The lane marking 101 corresponds to the left lane marking 51, and the lane marking 102 corresponds to the right lane marking 52. Then, the information processing device 12 detects a vanishing point F1 where the two lane markings 101, 102 intersect in the forward image 100.

[0024] The information processing device 12 detects a moving direction V1 of the vehicle 10 from a predetermined reference position R in the forward image 100 toward the vanishing point F1. The reference position R may be, for example, a position corresponding to the center of the vehicle 10 in the width direction in the forward image 100. The width direction of the vehicle 10 is a direction perpendicular to the front-rear direction of the vehicle 10. The reference position R is located in the center of the lower end of the forward image 100.

[0025] Since only one movement direction V1 was detected, the information processing device 12 determines the lane markings 101 and 102 in the forward image 100, which define the vanishing point F1 corresponding to the movement direction V1, as the lane markings that define the area in which the vehicle 10 is traveling.

[0026] The information processing device 12 notifies the automatic control device 11 of the lane markings 101 and 102 in the forward image 100.

[0027] The automatic control device 11 controls the operation of the vehicle 10 based on the lane markings 101 and 102 in the forward image 100. For example, the automatic control device 11 drives the vehicle 10 so as to travel in an area 103 between the lane markings 101 and 102 based on the forward image 100. As a result, the vehicle 10 is controlled so as to travel on a lane 53 defined by a left lane marking 51 and a right lane marking 52 of the road 50.

[0028] 2, vehicle 10 is also traveling on road 50. Ahead of vehicle 10, road 60 branches off from road 50. Road 60 has a left lane marking 61, a right lane marking 62, and a lane 63 defined by left lane marking 61 and right lane marking 62.

[0029] The information processing device 12 detects two lane markings 101, 102 in the forward image 110. The lane marking 101 corresponds to the left lane marking 51 of the road 50, and the lane marking 102 corresponds to the right lane marking 52 of the road 50. Then, the information processing device 12 detects a vanishing point F1 where the two lane markings 101, 102 intersect in the forward image 110.

[0030] Furthermore, the information processing device 12 detects two lane markings 104, 105 in the forward image 110. The lane marking 104 corresponds to the left lane marking 61 of the road 60, and the lane marking 105 corresponds to the right lane marking 62 of the road 60. Then, the information processing device 12 detects a vanishing point F2 where the two lane markings 104, 105 intersect in the forward image 110.

[0031] The information processing device 12 detects a moving direction V1 of the vehicle 10 moving from a reference position R to a vanishing point F1 in the forward image 110. The information processing device 12 also detects a moving direction V2 of the vehicle 10 moving from a reference position R to a vanishing point F2 in the forward image 110. The moving direction V2 is a newly detected moving direction.

[0032] Since the movement direction V2 differs in orientation from the previously detected movement direction V1 by more than a predetermined reference range, the information processing device 12 determines that a road 60 branching off from the road 50 on which the vehicle 10 is traveling has appeared in the forward image 110.

[0033] When a road 60 branching off from the road 50 is displayed in the forward image 110, the information processing device 12 can determine that the road 60 has appeared based on the moving direction V2 representing the road 60. The timing at which the road 60 appears in the forward image 110 depends on the field of view of the camera 2 or the mounting angle of the camera 2 to the vehicle 10, etc., but the information processing device 12 can determine that the road 60 has appeared at an early stage.

[0034] The information processing device 12 determines, in the forward image 110, the lane markings 101 and 102 that define the vanishing point F1 corresponding to the movement direction V1 as the lane markings of the road 50 on which the vehicle 10 is currently traveling.

[0035] In addition, the information processing device 12 determines, in the forward image 110, the lane markings 104 and 105 that define the vanishing point F2 corresponding to the movement direction V2 as the lane markings of the branch road 60 that branches off from the road 50 on which the vehicle 10 is currently traveling.

[0036] The information processing device 12 notifies the automatic control device 11 of the lane markings 101 and 102 in the forward image 110 as the lane markings of the road 50 on which the vehicle 10 is currently traveling.

[0037] Furthermore, the information processing device 12 notifies the automatic control device 11 of the lane markings 104 and 105 in the forward image 110 as the lane markings of the road 60 branching off.

[0038] When the vehicle 10 travels along the road 50 without exiting from the road 50 to the road 60, the automatic control device 11 controls the operation of the vehicle 10 based on the lane markings 101 and 102 in the forward image 110. For example, the automatic control device 11 drives the vehicle 10 so as to travel in an area 103 between the lane markings 101 and 102 based on the forward image 110. As a result, the vehicle 10 is controlled to travel on a lane 53 defined by the left lane marking 51 and the right lane marking 52 of the road 50.

[0039] On the other hand, when the vehicle 10 exits from the road 50 to the road 60, the automatic control device 11 controls the operation of the vehicle 10 based on the lane markings 104 and 105 in the forward image 110 after the vehicle 10 exits the road 50. For example, the automatic control device 11 drives the vehicle 10 so as to travel in an area 106 between the lane markings 104 and 105 based on the forward image 110. As a result, the vehicle 10 is controlled to travel on a lane 63 defined by a left lane marking 61 and a right lane marking 62 of the road 60.

[0040] As described above, the information processing device 12 can detect a branch road branching off from the road on which the vehicle is traveling at an early stage, based on a forward image that represents the environment ahead of the vehicle 10.

[0041] In addition, the information processing device 12 notifies the automatic control device 11 of the lane markings of the road 50 on which the vehicle 10 is traveling and the road 60 which branches off therefrom, so that the automatic control device 11 can correctly recognize the area in which the vehicle 10 is traveling and generate a driving plan.

[0042] 3 is a hardware configuration diagram of a vehicle 10 in which the information processing device 12 of the present embodiment is implemented. The vehicle 10 includes a camera 2, an automatic control device 11, an information processing device 12, and the like. Furthermore, the vehicle 10 may include other distance measuring sensors (not shown) such as a LiDAR sensor for measuring distances to objects around the vehicle 10, and the like.

[0043] The camera 2, the automatic control device 11, and the information processing device 12 are communicatively connected via an in-vehicle network 13 that complies with a standard such as a controller area network.

[0044] The camera 2 is attached to the vehicle 10 so as to face forward of the vehicle 10. The camera 2 captures a forward image showing the environment of a predetermined area in front of the vehicle 10, for example, at a predetermined cycle. The forward image may show a road included in the predetermined area in front of the vehicle 10 and road features such as lane markings on the road surface. The forward image may also show other vehicles located in front of the vehicle 10. The camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS, and an imaging optical system that forms an image of the area to be captured on the two-dimensional detector.

[0045] Each time the camera 2 captures a forward image, it outputs the forward image and the camera image capture time at which the forward image was captured to the automatic control device 11, information processing device 12, etc. via the in-vehicle network 13. The forward image is used in the automatic control device 11 for processing to estimate the position of the vehicle 10. The forward image is also used in the automatic control device 11 for processing to detect objects around the vehicle 10. The forward image is further used for information processing in the information processing device 12. The vehicle 10 may have another camera attached to the vehicle 10 so as to face the rear of the vehicle 10.

[0046] The automatic control device 11 controls the operation including the traveling of the vehicle 10. The automatic control device 11 generates a navigation route from the current position of the vehicle 10 to the destination position based on navigation map information, the destination position of the vehicle 10, and positioning information indicating the current position of the vehicle 10 input from a GNSS receiver (not shown). The automatic control device 11 outputs the current position of the vehicle 10 and the navigation route to the information processing device 12 via the in-vehicle network 13.

[0047] The automatic control device 11 also estimates the position and orientation of the vehicle 10 at the time the camera image was captured, based on the lane markings shown in the high-precision map information and the lane markings around the vehicle 10 shown in the forward image captured by the camera 2. The automatic control device 11 then estimates the driving lane on the road on which the vehicle 10 is located, based on the lane markings shown in the high-precision map information and the estimated position and estimated azimuth angle of the vehicle 10.

[0048] Furthermore, the automatic control device 11 detects the objects and their types around the vehicle 10 based on the forward image. The objects include other vehicles traveling around the vehicle 10.

[0049] In addition, the automatic control device 11 selects lanes within the road on which the vehicle 10 is traveling based on the high-precision map information, the navigation route, and the current position of the vehicle 10 for the nearest driving section (e.g., 10 km) selected from the navigation route, and generates a driving lane plan that represents the planned driving lanes on which the vehicle 10 is to travel.

[0050] Furthermore, the automatic control device 11 generates a driving plan that indicates a planned driving trajectory of the vehicle 10 up to a predetermined time (e.g., 5 seconds) ahead based on the forward image, the driving lane plan, the high-precision map information, the current position of the vehicle 10, surrounding environment information, vehicle status information, etc. The surrounding environment information includes the positions of other vehicles driving around the vehicle 10, etc. The vehicle status information includes the current position, vehicle speed, acceleration, and traveling direction of the vehicle 10, etc. The automatic control device 11 generates a driving plan that controls the steering, driving, braking, and other operations of the vehicle 10 so that the vehicle 10 drives in the area between the lane markings detected in the forward image.

[0051] Furthermore, the automatic control device 11 controls each part of the vehicle 10 based on the driving plan. For example, the automatic control device 11 outputs a control signal based on the driving plan via the in-vehicle network 13 to an actuator (not shown) that controls the steering wheel, a drive device (not shown), or a brake (not shown).

[0052] The information processing device 12 executes a detection process, a judgment process, a decision process, and a removal process. To this end, the information processing device 12 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the information processing device 12 to the in-vehicle network 13.

[0053] The memory 22 is an example of a storage unit, and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores computer programs and various data of applications used in information processing executed by the processor 23. The memory 22 also stores internal parameters such as the optical axis direction of the camera 2, the focal length and angle of view of the imaging optical system, and external parameters such as the mounting position and attitude of the camera 2.

[0054] All or part of the functions of the information processing device 12 are functional modules realized by a computer program running on the processor 23, for example. The processor 23 has a detection unit 231, a judgment unit 232, a decision unit 233, and a removal unit 234. The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The information processing device 12 is, for example, an electronic control unit (ECU). Details of the operation of the information processing device 12 will be described later.

[0055] In FIG. 3, the automatic control device 11 and the information processing device 12 are illustrated as separate devices, but all or part of these devices may be configured as a single device.

[0056] Fig. 4 is an example of an operation flowchart related to information processing of the information processing device of this embodiment 12. Hereinafter, the information processing of the information processing device 12 will be described with reference to Fig. 4. The information processing device 12 executes information processing according to the operation flowchart shown in Fig. 4 at an information processing time set at a predetermined cycle.

[0057] First, the detection unit 231 detects a vanishing point where two lane markings intersect in a forward image captured by the camera 2 (step S101). The position of the vanishing point is expressed in an image coordinate system. The image coordinate system has an origin at the upper left corner of the forward image, an x-axis extending horizontally to the right from the origin, and a y-axis extending downward from the origin perpendicular to the x-axis. The detection unit 231 is an example of a vanishing point detection unit. The process of the detection unit 231 detecting a vanishing point will be described later with reference to Figs. 5 to 10.

[0058] Next, the detection unit 231 detects the moving direction of the vehicle 10 from a predetermined reference position in the forward image toward the vanishing point (step S102). The detection unit 231 is an example of a direction detection unit. The reference position can be a position in the forward image corresponding to the vehicle 10. For example, the center position in the width direction of the vehicle 10 can be used as the reference position. The reference position is preferably located at the lower end of the forward image. The reference position is expressed in the image coordinate system. The position at which the reference position is located in the forward image can be determined based on the internal parameters and external parameters of the camera 2, etc. For example, when the camera 2 is attached at the center position in the width direction of the vehicle 10 with its optical axis facing the front of the vehicle 10, the reference position is the center position of the lower end of the forward image.

[0059] The moving direction of the vehicle 10 is represented by the direction of a vector in the image coordinate system, starting from the reference position and ending at the vanishing point. When two or more vanishing points are detected in the forward image, the moving direction of the vehicle 10 is detected for each vanishing point.

[0060] Next, the determination unit 232 determines whether or not two movement directions have been detected by the detection unit (step S103).

[0061] When two moving directions are detected (step S103-Yes), the determination unit 232 determines whether or not the orientation of each of the two moving directions differs from a previously detected moving direction by exceeding a predetermined reference range (step S104). As the previously detected moving direction, an average moving direction in the most recent period (for example, 3 seconds) can be used. The determination unit 232 is an example of a first determination unit.

[0062] First, in the image coordinate system, the start points of the vector representing the currently detected moving direction and the vector representing the previously detected moving direction are superimposed on a reference position. Then, the angle between the vector representing the currently detected moving direction and the direction of the previously detected moving direction is calculated. It is determined whether this angle exceeds a predetermined reference angle. For example, the reference angle may be 30 milliradians (mils). 100 m ahead of the vehicle 10, both ends of a road with a width of 3.5 m are approximately 30 milliradians.

[0063] In the image coordinate system, when the direction of the vector representing the currently detected moving direction exceeds a reference angle with respect to the direction of the vector representing the previously detected moving direction, the determination unit 232 determines that this moving direction is different. When the direction of the vector representing the currently detected moving direction represents a branch road branching off from the road on which the vehicle 10 is currently traveling, this moving direction is determined to be different. The direction of the vector representing the currently detected moving direction indicates the appearance of a branch road.

[0064] On the other hand, in the image coordinate system, if the orientation of the vector representing the currently detected moving direction does not exceed the reference angle with respect to the orientation of the vector representing the previously detected moving direction, the determination unit 232 determines that this moving direction is not different. It may be said that the orientation of the vector representing the currently detected moving direction is consistent with the orientation of the vector representing the previously detected moving direction. If the orientation of the vector representing the currently detected moving direction represents the road on which the vehicle 10 is currently traveling, this moving direction is determined to be not different. The orientation of the vector representing the currently detected moving direction is within the reference angle with respect to the orientation of the vector representing the previously detected moving direction.

[0065] When a branch road branching off from the travel road is displayed in the forward image, the information processing device 12 can determine that the branch road has appeared based on the travel direction representing the branch road. The timing at which the branch road appears in the forward image depends on the field of view of the camera 2 or the mounting angle of the camera 2 to the vehicle 10, but the information processing device 12 can determine that the branch road has appeared early on.

[0066] Here, the smaller the reference angle, the earlier the information processing device 12 can determine that a branch road has appeared. On the other hand, if the reference angle is too small, there is a risk of erroneously determining that a branch road has appeared. Therefore, it is preferable to determine the magnitude of the reference angle in consideration of the balance with accuracy.

[0067] If it is determined that the orientation of either of the two moving directions is different (step S104-Yes), the determination unit 232 determines that a branch road branching off from the road on which the vehicle 10 is traveling has appeared in the forward image (step S105). If it is determined that the orientation of either of the two moving directions is different, it is considered that the lane markings of the traveling road on which the vehicle 10 is currently traveling and the lane markings of the branch road are depicted in the forward image. The determination unit is an example of a second determination unit.

[0068] A vector indicating the same direction as the previously detected moving direction corresponds to the road on which the vehicle 10 is currently traveling, while a vector indicating a different direction from the previously detected moving direction corresponds to a branch road.

[0069] Next, the determination unit 233 determines the lane markings of the road on which the vehicle 10 is currently traveling and the lane markings of the branch road in the forward image (step S106). The two lane markings that define the vanishing point that represents the end point of the vector that represents the same direction as the previously detected moving direction are the lane markings of the road. The two lane markings that define the vanishing point that represents the end point of the vector that represents the direction different from the previously detected moving direction are the lane markings of the branch road.

[0070] Next, the determination unit 233 notifies the automatic control device 11 of information indicating the positions of the lane markings in the forward image, and ends the series of processes (step S107). Specifically, the determination unit 233 notifies the automatic control device 11 of information indicating the positions of two lane markings that define the driving lane of the driving road on which the vehicle 10 is currently traveling, and information indicating the positions of two lane markings that define the lanes of the branch road, together with the forward image.

[0071] When the vehicle 10 travels along the travel road without exiting from the travel road to a branch road, the automatic control device 11 generates a driving plan for the vehicle 10 based on the forward image so that the vehicle 10 travels in an area between two lane markings that divide the lanes of the travel road. The automatic control device 11 may generate a driving plan by converting the forward image into a bird's-eye view based on the internal parameters and external parameters of the camera 2, etc.

[0072] On the other hand, when the vehicle 10 exits from road 50 onto road 60, the automatic control device 11 generates a driving plan for the vehicle 10 based on the forward image so as to drive in the area between the two lane markings that separate the lanes of the branch road.

[0073] If the two moving directions are not detected (step S103-No), or if it is determined that the orientation of either of the two moving directions is not different (step S104-No), the determination unit 233 determines the lane markings of the road on which the vehicle 10 is currently traveling in the forward image (step S108). The two lane markings that define the vanishing point that represents the end point of the vector that represents the same orientation as the previously detected moving direction are the lane markings of the road. For example, if the vehicle 10 is traveling on a road having two or more lanes, it may be determined that the orientation of either of the two moving directions is not different.

[0074] Next, the determination unit 233 notifies the automatic control device 11 of information indicating the positions of the lane markings in the forward image and the forward image (step S107), and ends a series of processes. Specifically, the determination unit 233 notifies the automatic control device 11 of information indicating the positions of two lane markings that define the driving lane of the driving road on which the vehicle 10 is currently traveling, together with the forward image. The automatic control device 11 generates a driving plan for the vehicle 10 based on the forward image so that the vehicle 10 travels in an area sandwiched between the two lane markings that define the lanes of the driving road.

[0075] Next, the process in step S101 described above in which the detection unit 231 detects a vanishing point in the front image will be described below with reference to FIGS.

[0076] FIG. 5 is a diagram showing an example of a forward image. A forward image 200 shown in FIG. 5 corresponds to the forward image 100 in FIG. 1. A detection unit 231 detects areas 201 and 202 corresponding to two lane markings shown in the forward image 200. The detection unit 231 has a classifier that has been trained to identify lane markings shown on the road surface from an image. The detection unit 231 inputs the forward image 200 to this classifier, thereby detecting areas 201 and 202 corresponding to the two lane markings shown in the forward image 200. The positions of pixels in areas 201 and 202 are expressed in an image coordinate system.

[0077] 6 is a diagram for explaining detection of a vanishing point and a moving direction from a forward image 200. The detection unit 231 obtains a polynomial representing a lane marking based on the positions of pixels representing the lane marking for each of the regions 201 and 202. For example, the detection unit 231 obtains the polynomial using the least squares method.

[0078] The detection unit 231 determines, as a vanishing point, an intersection F1 of curves P1 and P2 expressed by polynomials for each of the regions 201 and 202. If the road is straight, the lane markings are approximated to straight lines, and if the road is curved, the lane markings are approximated to curved lines.

[0079] Then, the detection unit 231 obtains a vector V1 having a starting point at the reference position R arranged in the center of the lower edge of the forward image 200 and an ending point at the vanishing point F1. The vector V1 represents the moving direction of the vehicle 10.

[0080] In the above example, two lane markings were detected, but if three or more lane markings are detected, the detection unit 231 determines the vanishing point and movement direction for two adjacent lane markings.

[0081] Fig. 7 is a diagram showing another example of a forward image. A forward image 300 shown in Fig. 7 corresponds to the forward image 110 in Fig. 2. The detection unit 231 detects areas 301, 302, 303, and 304 corresponding to the four lane markings shown in the forward image 300. The positions of pixels in the areas 301, 302, 303, and 304 are expressed in the image coordinate system.

[0082] 8 is a diagram for explaining detection of a vanishing point and a moving direction from the forward image 200. The detection unit 231 obtains a polynomial representing the lane marking for each of the regions 301, 302, 303, and 304, based on the positions of the pixels representing the lane marking.

[0083] The detection unit 231 determines, as a vanishing point, an intersection F1 of curves P3 and P4 expressed by polynomials for adjacent regions 301 and 302. The detection unit 231 also determines, as a vanishing point, an intersection F2 of curves P5 and P6 expressed by polynomials for adjacent regions 303 and 304. Here, at a predetermined y-axis position, the forward image is scanned in the x-axis direction, and two regions that are arranged adjacent to each other without any other region intervening are adjacent regions.

[0084] Then, the detection unit 231 obtains a vector V1 having a starting point of a reference position R arranged at the center of the lower end of the forward image 200 and an ending point of a vanishing point F1. The vector V1 represents the moving direction of the vehicle 10. The detection unit 231 also obtains a vector V2 having a starting point of a reference position R arranged at the center of the lower end of the forward image 200 and an ending point of a vanishing point F2. The vector V2 represents the other moving direction of the vehicle 10.

[0085] The method by which the detection unit 231 determines the vanishing point is not limited to the above-mentioned method. For example, the detection unit 231 may detect the vanishing point for the front image shown in FIG.

[0086] Fig. 9 is a diagram showing yet another example of a forward image. A forward image 300 shown in Fig. 9 corresponds to the forward image 110 in Fig. 2. The detection unit 231 detects pixels that become edges by applying an edge detection filter, such as a Sobel filter, to the forward image 300, and detects pixels having a luminance equal to or higher than a predetermined threshold value among the detected pixels as candidate points. The positions of the candidate points are expressed in the image coordinate system.

[0087] Then, the detection unit 231 groups candidate points that are within a predetermined reference number of pixels from the bottom to the top of the forward image 300, for example, to generate regions 401, 402, and 403. The reference number of pixels can be, for example, 5 to 10 pixels.

[0088] 10 is a diagram for explaining detection of a vanishing point and a moving direction from the front image 300. The detection unit 231 obtains a polynomial for each of the regions 401, 402, and 403 based on candidate points.

[0089] The detection unit 231 determines, as a vanishing point, an intersection F1 of curves P8 and P9 expressed by polynomials for the adjacent regions 402 and 403. The detection unit 231 also determines, as a vanishing point, an intersection F2 of curves P7 and P9 expressed by polynomials for the adjacent regions 401 and 403.

[0090] Then, the detection unit 231 obtains a vector V1 having a starting point of a reference position R arranged at the center of the lower end of the forward image 200 and an ending point of a vanishing point F1. The vector V1 represents the moving direction of the vehicle 10. The detection unit 231 also obtains a vector V2 having a starting point of a reference position R arranged at the center of the lower end of the forward image 200 and an ending point of a vanishing point F2. The vector V2 represents the other moving direction of the vehicle 10.

[0091] As described above in detail, the information processing device can quickly detect a branch road that branches off from the road on which the vehicle is traveling, based on a forward image that represents the environment ahead of the vehicle.

[0092] In addition, by the information processing device notifying the automatic control device of the lane markings of the road on which the vehicle is traveling and the branch roads, the automatic control device can correctly recognize the area in which the vehicle is traveling and generate a driving plan.

[0093] Next, a first modified example and a second modified example of the information processing device 12 of the above-described embodiment will be described below with reference to FIGS.

[0094] FIG. 11 is an example of an operational flowchart relating to information processing in the first variation of the information processing device 12 of this embodiment.

[0095] This modified example differs from the operational flowchart shown in Fig. 4 in that steps S209 to S212 have been added. The processes in steps S201 to S208 are the same as steps S101 to S108 described above.

[0096] In this modified example, after the automatic control device 11 is notified of information indicating the positions of the lane markings in the forward image (step S207), the automatic control device 11 generates a driving plan for the vehicle 10 as described above.

[0097] When the vehicle 10 travels along the travel road without exiting from the travel road to a branch road, the automatic control device 11 generates a driving plan for the vehicle 10 based on a forward image so that the vehicle 10 travels in the area between two lane dividing lines that divide the lanes of the travel road.

[0098] In addition, when the vehicle 10 exits from road 50 onto road 60, the automatic control device 11 generates a driving plan for the vehicle 10 based on the forward image so that the vehicle 10 travels in the area between two lane markings that separate the lanes of the branch road.

[0099] Then, the automatic control device 11 notifies the information processing device 12 of the operation plan via the network 13.

[0100] The determination unit 232 acquires a driving plan from the automatic control device 11 (step S209). Next, the determination unit 232 determines whether the vehicle 10 will exit to a branch road based on the driving plan (step S210). The driving plan represents a planned driving trajectory of the vehicle 10 up to a predetermined time ahead. The determination unit 232 is an example of a third determination unit. The information processing time period is preferably longer than the period in which the automatic control device 11 generates the driving plan.

[0101] When the planned travel path of the vehicle 10 is shown as traveling in an area between two lane markings that divide the lanes of the branch road, it is determined that the vehicle 10 will exit onto the branch road.

[0102] On the other hand, if the planned travel path of the vehicle 10 is shown as traveling in an area between two lane markings that divide the lanes of the travel road, it is determined that the vehicle 10 will not exit to the branch road.

[0103] If it is determined that the vehicle 10 does not exit to the branch road (step S210-No), the removal unit 234 removes the moving direction determined to be different from the moving directions detected in the past (step S211), and ends the series of processes. The removal unit 234 is an example of a first removal unit.

[0104] As a result, in the next information processing, if a branch road is displayed on the forward image, this branch road will be determined as a newly appeared branch road.

[0105] On the other hand, if it is determined that the vehicle 10 is exiting to a branch road (step S210-Yes), the detection unit 231 sets the moving direction determined to be different from the previously detected moving direction by exceeding a predetermined reference range as the previously detected moving direction (step S212), and ends the series of processes. In addition, the removal unit 234 removes the moving direction of the travel road from the previously detected moving directions.

[0106] As a result, in the next information processing, the branch road is treated as a travel road on which the vehicle 10 is traveling.

[0107] As described above in detail, in this modified example, the information processing device changes the previously detected moving direction based on the driving plan of the vehicle, so that the information processing device can appropriately determine the appearance of a branch road in the next information processing.

[0108] FIG. 12 is an example of an operational flowchart relating to information processing in the second modified example of the information processing device of this embodiment.

[0109] This modified example differs from the operational flowchart shown in Fig. 4 in that steps S309 to S311 have been added. The processes in steps S301 to S308 are the same as steps S101 to S108 described above.

[0110] After it is determined that a branch road branching off from the road on which the vehicle 10 is traveling appears in the forward image (step S305), the determination unit 232 refers to the navigation route and determines whether the vehicle 10 is scheduled to exit from the traveling road to a branch road within a predetermined range in the traveling direction from the current position of the vehicle 10 (step S309). The predetermined range can be, for example, 1 km to 5 km.

[0111] If there is no plan to exit onto the branch road (step S309-No), the removal unit 234 removes the lane markings of the branch road (step S310). Specifically, the removal unit 234 removes, from among the lane markings detected in the forward image, a lane marking that represents a vanishing point that indicates a moving direction that is determined to have a different orientation. The removal unit 234 is an example of a second removal unit. The removal unit 234 determines that the lane marking that represents a vanishing point that indicates a moving direction that is determined to have a different orientation marks an adjacent lane adjacent to the lane in which the vehicle 10 is currently traveling.

[0112] Next, the determination unit 233 determines, in the forward image, the lane markings of the road on which the vehicle 10 is currently traveling (step S311).

[0113] Next, the determination unit 233 notifies the automatic control device 11 of the information indicating the positions of the lane markings in the forward image and the forward image (step S307), and ends the series of processes. Specifically, the determination unit 233 notifies the automatic control device 11 of information indicating the positions of two lane markings that divide the driving lane of the driving road on which the vehicle 10 is currently traveling, together with the forward image.

[0114] On the other hand, if there is a plan to exit onto a branch road (step S309-Yes), the process proceeds to step S306. The process of step S306 is the same as step S106 described above.

[0115] As described above in detail, in this variant, if the vehicle 10 has no plans to exit from the traveling road to a branching road within a specified range in the direction of travel from the current position of the vehicle 10, a forward image with the lane markings of the branching road removed is notified to the automatic control device 11.

[0116] The automatic control device 11 generates a driving plan for the vehicle 10 based on a forward image showing only the lane markings of the road to be traveled, so that the vehicle 10 travels in an area between two lane markings that divide the lane of the road to be traveled. This enables the automatic control device 11 to generate a safer driving plan.

[0117] In the present disclosure, the information processing device, the computer program for information processing, and the information processing method of the above-mentioned embodiments can be modified as appropriate without departing from the spirit of the present disclosure. Furthermore, the technical scope of the present disclosure is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents. [Explanation of symbols]

[0118] 2 Camera 10 Vehicles 11 Automatic control device 12 Information processing equipment 21 Communication Interface 22 Memory 23 Processors 231 Detection unit 232 Judgment section 233 Decision Section 234 Removal section 13 In-vehicle network

Claims

1. a vanishing point detection unit that detects a vanishing point where two lane markings intersect in a forward image that represents the environment in front of the vehicle; a direction detection unit that detects a moving direction of the vehicle from a predetermined reference position in the forward image toward the vanishing point; a first determination unit that, when two movement directions are detected by the direction detection unit, determines whether or not an orientation of each of the two movement directions differs from a movement direction previously detected by the direction detection unit by exceeding a predetermined reference range; a second determination unit that, when the first determination unit determines that the orientation of either of the two movement directions is different, determines that a branch road branching off from a road on which the vehicle is traveling has appeared in the forward image; and 13. An information processing device comprising:

2. a third determination unit that determines whether or not the vehicle will exit onto the branch road based on a driving plan that represents a planned driving path of the vehicle and that is generated based on a determination result that a branch road has appeared by the second determination unit, 2. The information processing device according to claim 1, further comprising a first removal unit that, when the third determination unit determines that the vehicle will not exit onto a branch road, removes the movement direction determined to be different from the movement direction previously detected by the direction detection unit.

3. 3. The information processing device according to claim 2, wherein when the third determination unit determines that the vehicle is exiting onto a branch road, the direction detection unit sets the moving direction determined to be different to the moving direction previously detected by the direction detection unit.

4. a fourth determination unit that determines whether or not the vehicle has a plan to exit onto a branch road that branches off from a road on which the vehicle is traveling within a predetermined range from a current position of the vehicle in a traveling direction based on a navigation route of the vehicle, 2. The information processing device according to claim 1, further comprising a second removal unit that, when the second determination unit determines that a branch road has appeared and the fourth determination unit determines that the vehicle has no plans to exit from the driving road to the branch road, removes from the lane markings detected in the forward image a lane marking that represents the vanishing point representing the direction of travel that is determined to have a different orientation.

5. The information processing device according to claim 1 , wherein the predetermined reference position in the forward image represents a position of a vehicle.

6. detecting a vanishing point where two lane markings intersect in a forward image representing the environment in front of the vehicle; Detecting a moving direction of the vehicle from a predetermined reference position in the forward image toward the vanishing point; When two moving directions are detected, it is determined whether or not each of the two moving directions differs from a previously detected moving direction by exceeding a predetermined reference range; If it is determined that the orientation of either of the two moving directions is different, it is determined that a branch road branching off from the road on which the vehicle is traveling has appeared in the forward image. A computer program for information processing, comprising: a processor for executing a process including the steps of:

7. An information processing device, detecting a vanishing point where two lane markings intersect in a forward image representing the environment in front of the vehicle; Detecting a moving direction of the vehicle from a predetermined reference position in the forward image toward the vanishing point; When two moving directions are detected, it is determined whether or not each of the two moving directions differs from a previously detected moving direction by exceeding a predetermined reference range; If it is determined that the orientation of either of the two moving directions is different, it is determined that a branch road branching off from the road on which the vehicle is traveling has appeared in the forward image.

2. An information processing method comprising:

Images