Travel path inference device and vehicle

Abstract

A travel path inference device according to an embodiment of the present disclosure is provided with a computing circuit that is capable of generating first processing data by performing convolution processing on the basis of a captured image generated by an imaging device that captures images forward of a vehicle, is capable of storing the first processing data, is capable of performing time-series processing on the basis of the first processing data and one or more pieces of stored second processing data generated by convolution processing on the basis of one or more past captured images, and is capable of inferring a position of a travel path on which the vehicle is traveling by performing estimation processing on the basis of the processing results of the time-series processing.

Description

Travel path inference device and vehicle

[0001] The present disclosure relates to a travel path inference device capable of inferring a travel path on which a vehicle is traveling, and a vehicle equipped with such a travel path inference device.

[0002] In vehicles, an image of the area ahead of the vehicle is often captured, and the travel path is detected based on the captured image. For example, Patent Literature 1 discloses a technology for inferring the future route of the vehicle based on the image of the area ahead of the vehicle.

[0003] Special table 2019-509534 publication

[0004] A travel path inference device according to an embodiment of the present disclosure includes an arithmetic circuit that is capable of generating first processed data by performing convolution processing on captured images generated by an imaging device that captures images of the area ahead of a vehicle, storing the first processed data, performing time-series processing on the first processed data and one or more second processed data that are each generated by convolution processing on one or more past captured images and stored, and performing estimation processing on the results of the time-series processing to infer the position of a travel path on which a vehicle is traveling.

[0005] A vehicle according to an embodiment of the present disclosure includes an imaging device and an arithmetic circuit. The imaging device is capable of generating an imaged image by capturing an image of an area ahead of the vehicle. The arithmetic circuit is capable of generating first processed data by performing convolution processing on the captured image, storing the first processed data, performing time-series processing on the first processed data and one or more second processed data items that have been generated and stored by convolution processing on one or more past captured images, and performing estimation processing on the results of the time-series processing to infer the position of a road on which the vehicle is traveling.

[0006] The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate one embodiment and, together with the description, serve to explain the principles of the disclosure.

[0007] FIG. 1 is an explanatory diagram illustrating an example configuration of a vehicle equipped with a driving assistance device according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example configuration of the driving assistance device illustrated in FIG. 1. FIG. 3 is an explanatory diagram illustrating an example operation of a device that infers the position of a driving path according to a reference example. FIG. 4 is an explanatory diagram illustrating an example operation of a driving path inference processing unit illustrated in FIG. 2. FIG. 5 is an explanatory diagram illustrating an example of detailed operation of the driving path inference processing unit illustrated in FIG. 2. FIG. 6 is an explanatory diagram illustrating an example of parameters handled by the control parameter generation unit illustrated in FIG. 2. FIG. 7 is an explanatory diagram illustrating an example of other parameters handled by the control parameter generation unit illustrated in FIG. 2.

[0008] A lane inference device that infers the position of a lane infers the position of the lane, for example, based on lane markings provided on the lane. However, there may be portions of the lane where lane markings are not provided, such as at intersections. It is desirable for the lane inference device to be able to appropriately infer the position of the lane even in such cases.

[0009] It is desirable to provide a roadway inference device and a vehicle that can appropriately infer the position of a roadway.

[0010] Some exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the following description illustrates one specific example of the present disclosure and should not be construed as limiting the present disclosure. For example, each element, including numerical values, shapes, materials, parts, the position of each part, and the connection method of each part, is merely an example and should not be construed as limiting the present disclosure. Furthermore, in the following exemplary embodiments, components not described in independent claims based on the highest concept of the present disclosure are optional and may be provided as needed. The drawings are schematic and are not intended to be drawn to scale. Throughout this specification and the drawings, components having substantially the same function and configuration are designated by the same reference numerals, and redundant description will be omitted. Furthermore, components not directly related to one embodiment of the present disclosure are not shown in the drawings.

[0011] 1 and 2 show an example of the configuration of a driving assistance device 1 equipped with a driving path inference device according to an embodiment. The driving assistance device 1 is mounted on a vehicle 9 and is configured to assist a driver in driving the vehicle 9. The driving assistance device 1 includes an imaging device 10 and a processing device 20.

[0012] The imaging device 10 is configured to generate captured images PIC by capturing an image of the area ahead of the vehicle 9. In this example, the imaging device 10 is disposed inside the vehicle 9 near the upper portion of the windshield of the vehicle 9, as shown in FIG. 1 . In this example, the imaging device 10 is a so-called monocular camera including one image sensor. However, this is not limited thereto, and instead, the imaging device 10 may be, for example, a stereo camera including two image sensors. The imaging device 10 generates a series of captured images PIC by performing an imaging operation at a predetermined frame rate (e.g., 60 fps). The imaging device 10 then supplies image data of the generated series of captured images PIC to the processing device 20.

[0013] The processing device 20 is configured to perform processing based on the captured image PIC, thereby controlling the operation of the driving assistance device 1. The processing device 20 is configured using, for example, one or more processors, one or more memories, etc., and is configured to perform processing by executing a program. The processing device 20 has a roadway inference processing unit 25, a control parameter generation unit 26, and a driving assistance processing unit 27.

[0014] The travel path inference processing unit 25 is configured to generate travel path data DT by inferring the position of the travel path on which the vehicle 9 is traveling based on the multiple captured images PIC supplied from the imaging device 10. The travel path inference processing unit 25 infers the position of the travel path based on, for example, marking lines on the travel path. However, there may be portions of the travel path where marking lines are not provided, such as intersections. The travel path inference processing unit 25 is also able to infer the position of the travel path based on the multiple captured images PIC even in portions of the travel path where marking lines are not provided.

[0015] As a reference example, the operation of a device that infers the position of a road based on one captured image PIC, rather than a plurality of captured images PIC, will be described below.

[0016] 3 shows an example of the operation of a device that infers the position of a roadway based on a single captured image PIC. In this example, a vehicle 8 equipped with this device is traveling along the roadway from bottom to top in FIG. 3.

[0017] In the example of FIG. 3(A), a vehicle 8 is traveling on a roadway 111 and passing through an intersection CR11 located on the roadway 111. Left and right demarcation lines 111L and 111R are provided on the roadway 111 in a portion before the intersection CR11 and a portion behind the intersection CR11. On the other hand, no demarcation lines 111L or 111R are provided at the intersection CR11. In this example, the demarcation lines 111L and 111R before the intersection CR11 and the demarcation lines 111L and 111R behind the intersection CR11 are located at the same positions in a direction intersecting the extension direction of the roadway 111 (the horizontal direction in FIG. 3). The imaging device of the vehicle 8 generates a captured image PIC by performing an imaging operation when the vehicle 8 is at the position shown in FIG. 3(A). The captured image PIC captures the area indicated by region R11. For convenience of explanation, the shape of the region R11 is shown as a rectangle, but in reality, the shape is more complex. For example, the vehicle 8 infers the positions of the lane markings 111L and 111R at the intersection CR11, as shown by the dashed lines, based on the image of the lane markings 111L and 111R included in the single captured image PIC. In this example, the inferred lane matches the actual lane 111.

[0018] In the example of FIG. 3(B), the vehicle 8 is traveling on the roadway 112 and passing through an intersection CR12 located on the roadway 112. Left and right demarcation lines 112L and 112R are provided on the roadway 112 in a portion before the intersection CR12 and a portion behind the intersection CR12. On the other hand, the intersection CR12 does not have any demarcation lines 112L and 112R. In this example, the demarcation lines 112L and 112R before the intersection CR12 and the demarcation lines 112L and 112R behind the intersection CR12 are located at different positions in a direction intersecting the extension direction of the roadway 112 (the horizontal direction in FIG. 3). In other words, the roadway 112 is offset in a direction intersecting the extension direction of the roadway 112. The imaging device of the vehicle 8 generates a captured image PIC by performing an imaging operation when the vehicle 8 is in the position shown in FIG. 3(B). This captured image PIC captures the range indicated by region R12. For example, the vehicle 8 infers the position of the lane markings, as indicated by the dashed lines, at intersection CR12 based on the images of lane markings 112L and 112R included in this single captured image PIC. In this example, the inferred lane does not match the actual lane 112. That is, because lane 112 is offset in a direction intersecting the extension direction of lane 112, it is difficult for the vehicle 8 to correctly infer the position of the lane based on this captured image PIC.

[0019] On the other hand, the travel path inference processing unit 25 according to this embodiment infers the position of the travel path on which the vehicle 9 is traveling based on the multiple captured images PIC supplied from the imaging device 10. As a result, the travel path inference processing unit 25 can infer the position of the travel path based on the multiple captured images PIC even in a portion of the travel path where no dividing line is provided.

[0020] 4 shows an example of the operation of the travel path inference processing unit 25. In this example, the vehicle 9 travels on the travel path 112 shown in FIG. 3B from bottom to top in FIG.

[0021] In this example, the path inference processing unit 25 infers the position of the path based on five captured images PIC generated each time the vehicle 9 travels a predetermined distance. The imaging device 10 of the vehicle 9 generates a captured image PIC in which the range shown in region R1 is captured when the vehicle 9 is at position POS1, generates a captured image PIC in which the range shown in region R2 is captured when the vehicle 9 is at position POS2 a predetermined distance from position POS1, generates a captured image PIC in which the range shown in region R3 is captured when the vehicle 9 is at position POS3 a predetermined distance from position POS2, generates a captured image PIC in which the range shown in region R4 is captured when the vehicle 9 is at position POS4 a predetermined distance from position POS3, and generates a captured image PIC in which the range shown in region R5 is captured when the vehicle 9 is at position POS5 a predetermined distance from position POS4. In this example, the path inference processing unit 25 infers the positions of the left and right lane markings 112L and 112R based on five captured images PIC obtained each time the vehicle 9 travels a predetermined distance. This predetermined distance may be, for example, approximately 8 m. In this example, the distance between positions POS1 and POS5 is 32 m (= 8 m × 4), which is longer than the length of the intersection CR12 in the direction of extension of the path 112. Therefore, the path inference processing unit 25 can infer the positions of the left and right lane markings 112L and 112R based on the multiple captured images PIC even in portions of the path where lane markings are not provided, thereby enabling the path position to be inferred.

[0022] The travel path inference processing unit 25 uses machine learning technology to infer the position of the travel path on which the vehicle 9 is traveling, based on the multiple captured images PIC. The processing by the travel path inference processing unit 25 uses a trained machine learning model generated by a machine learning device (not shown). The travel path inference processing unit 25 has a convolution processing unit 21, a storage unit 22, a long short term memory (LSTM) processing unit 23, and a fully connected processing unit 24.

[0023] The convolution processing unit 21 is configured to generate processed data DP by performing convolution processing based on the captured image PIC. The captured image PIC includes images of various objects, such as lane markings on the road surface, various guide signs on the road surface, street lights and traffic signals along the road, etc. The positions of these objects are related to the position of the road. The convolution processing unit 21 generates processed data DP by performing convolution processing based on the captured image PIC including images of such objects. The convolution processing unit 21 then supplies this processed data DP to the memory unit 22 and the LSTM processing unit 23.

[0024] The storage unit 22 is configured to store the processed data DP generated by the convolution processing unit 21. The storage unit 22 stores a plurality of pieces of processed data DP each generated by the convolution processing unit 21 based on a plurality of captured images PIC that are different from one another.

[0025] The LSTM processing unit 23 is configured to perform LSTM processing each time it receives processed data DP from the convolution processing unit 21, based on the processed data DP and the four processed data DP stored in the storage unit 22. Specifically, for example, when the LSTM processing unit 23 receives processed data DP based on the captured image PIC obtained when the vehicle 9 is at position POS5 in FIG. 4, it performs LSTM processing based on the processed data DP and the four processed data DP based on the captured image PIC obtained when the vehicle 9 is at positions POS1 to POS4.

[0026] The full connection processing unit 24 is configured to perform full connection processing based on the processing results of the LSTM processing unit 23, thereby generating roadway data DT indicating the position of the roadway on which the vehicle 9 is traveling. In this example, the roadway data DT includes data on the position of the left lane marking and data on the position of the right lane marking in a coordinate system based on the vehicle 9. In this example, the data on the position of the left lane marking and the data on the position of the right lane marking are represented as a sequence of points, as shown in FIG. 4. However, this is not limited to this, and instead, the roadway data DT may be data indicating the position of the centerline of the roadway, for example.

[0027] In this way, the driving path inference processing unit 25 generates driving path data DT by inferring the position of the driving path on which the vehicle 9 is driving based on the multiple captured images PIC supplied from the imaging device 10.

[0028] The control parameter generation unit 26 (FIG. 2) is configured to generate control parameters for controlling the traveling of the vehicle 9 based on the traveling road data DT. The control parameters include, for example, a parameter for a target steering angle. The control parameter generation unit 26 is configured to calculate the target steering angle of the vehicle 9 based on the traveling road data DT.

[0029] The driving assistance processing unit 27 is configured to provide driving assistance to the vehicle 9 based on the control parameters generated by the control parameter generating unit 26. For example, the driving assistance processing unit 27 performs steering control of the vehicle 9 based on the target steering angle included in the control parameters supplied from the control parameter generating unit 26, thereby controlling the vehicle 9 to continue traveling in the driving lane.

[0030] Here, the roadway inference processing unit 25 corresponds to a specific example of a "roadway inference device" in an embodiment of the present disclosure. The convolution processing unit 21, the storage unit 22, the LSTM processing unit 23, and the fully connected processing unit 24 correspond to a specific example of an "arithmetic circuit" in an embodiment of the present disclosure. The captured image PIC corresponds to a specific example of a "captured image" in an embodiment of the present disclosure. The processed data DP corresponds to a specific example of "processed data" in an embodiment of the present disclosure.

[0031] [Operation and Function] Next, the operation and function of the driving assistance device 1 of this embodiment will be described.

[0032] (Overall Operation Overview) First, the operation of the driving assistance device 1 will be described with reference to Fig. 2. The imaging device 10 generates a captured image PIC by capturing an image of the area ahead of the vehicle 9. The driving path inference processing unit 25 of the processing device 20 generates driving path data DT by inferring the position of the driving path on which the vehicle 9 is traveling, based on the multiple captured images PIC supplied from the imaging device 10. The control parameter generation unit 26 generates control parameters for controlling the driving of the vehicle 9, based on the driving path data DT. The driving assistance processing unit 27 provides driving assistance for the vehicle 9, based on the control parameters generated by the control parameter generation unit 26.

[0033] (Detailed Operation) The roadway inference processing unit 25 generates roadway data DT by inferring the position of the roadway based on the multiple captured images PIC supplied from the imaging device 10. This process will be described in detail below.

[0034] 5 shows an example of the processing of the travel path inference processing unit 25. The travel path inference processing unit 25 performs the following processing each time the vehicle 9 travels a predetermined distance (for example, 8 m).

[0035] First, the convolution processing unit 21 of the roadway inference processing unit 25 performs a convolution process A21 based on the captured image PIC to generate processed data DP (processed data DP5 in this example). Then, the convolution processing unit 21 supplies the processed data DP5 to the storage unit 22 and the LSTM processing unit 23.

[0036] The storage unit 22 of the travel path inference processing unit 25 stores the processed data DP5 supplied from the convolution processing unit 21. This storage unit 22 has already stored four pieces of processed data DP1 to DP4 that were previously supplied from the convolution processing unit 21. The processed data DP1 to DP4 are generated in the order of processed data DP1, processed data DP2, processed data DP3, and processed data DP4, and are supplied to the storage unit 22.

[0037] These five pieces of processed data DP1 to DP5 correspond to the five pieces of processed data DP generated based on the five captured images PIC at positions POS1 to POS5 in FIG. 4, for example.

[0038] The LSTM processing unit 23 of the roadway inference processing unit 25 performs the LSTM processing A23 based on these five pieces of processed data DP1 to DP5. This LSTM processing A23 includes five LSTM processing steps A231 to A235. Specifically, the LSTM processing unit 23 first performs the LSTM processing A231 based on the processed data DP1. Next, the LSTM processing unit 23 performs the LSTM processing A232 based on the processing result of the LSTM processing A231 and the processed data DP2. Next, the LSTM processing unit 23 performs the LSTM processing A233 based on the processing result of the LSTM processing A232 and the processed data DP3. Next, the LSTM processing unit 23 performs the LSTM processing A234 based on the processing result of the LSTM processing A233 and the processed data DP4. Next, the LSTM processing unit 23 performs an LSTM process A235 based on the processing result of the LSTM process A234 and the processed data DP5.

[0039] Then, the full connection processing unit 24 of the travel path inference processing unit 25 performs full connection processing A24 based on the processing result of LSTM processing A235 performed by the LSTM processing unit 23, thereby generating travel path data DT indicating the travel path on which the vehicle 9 is traveling. In this example, the travel path data DT includes data on the position of the left lane marking and data on the position of the right lane marking in a coordinate system based on the vehicle 9. In this example, the data on the position of the left lane marking and the data on the position of the right lane marking are data on the positions of point sequences, as shown in FIG. 4 .

[0040] This is the end of the processing of the roadway inference processing unit 25.

[0041] Based on the roadway data DT, the control parameter generating unit 26 generates control parameters for controlling the traveling of the vehicle 9. For example, the control parameter generating unit 26 calculates a target steering angle of the vehicle 9 based on the roadway data DT.

[0042] 6 and 7 show an example of the processing of the control parameter generating unit 26. In FIG.

[0043] 6 , the control parameter generation unit 26 calculates the distance CL between the reference point of the vehicle 9 and the left lane marking 113L based on the roadway data DT. Here, the reference point of the vehicle 9 is, for example, the position of the imaging device 10 on the vehicle 9. Similarly, the control parameter generation unit 26 calculates the distance CR between the reference point of the vehicle 9 and the right lane marking 113R based on the roadway data DT.

[0044] Further, for example, as shown in FIG. 6 , the control parameter generation unit 26 calculates the angle BL between the vehicle length direction of the vehicle 9 and the extension direction of the left lane marking 113L based on the roadway data DT. In FIG. 6 , a straight line (solid line) indicating the vehicle length direction of the vehicle 9 is translated to the position of the left lane marking 113L to draw a straight line (dashed line). The angle between this straight line (dashed line) and the left lane marking 113L is the angle BL. Similarly, the control parameter generation unit 26 calculates the angle BR between the vehicle length direction of the vehicle 9 and the extension direction of the right lane marking 113R based on the roadway data DT. In FIG. 6 , a straight line (solid line) indicating the vehicle length direction of the vehicle 9 is translated to the position of the right lane marking 113R to draw a straight line (dashed line). The angle between this straight line (dashed line) and the right lane marking 113R is the angle BR.

[0045] Further, for example, the control parameter generation unit 26 calculates the curvature AL of the left dividing line 114L and the curvature AR of the right dividing line 114R of the road on which the vehicle 9 is traveling based on the road data DT, as shown in Figure 7.

[0046] Then, the control parameter generation unit 26 calculates the target steering angle of the vehicle 9 based on the distances CL, CR, the angles BL, BR, and the curvatures AL, AR. Specifically, the control parameter generation unit 26 calculates the target steering angle θ of the vehicle 9 using, for example, the following equation. Here, Gp, Gi, Gd, and Gff are predetermined coefficients.

[0047] Then, the driving assistance processing unit 27 performs driving assistance for the vehicle 9 based on the control parameters generated by the control parameter generating unit 26. For example, the driving assistance processing unit 27 performs steering control of the vehicle 9 based on the target steering angle θ included in the control parameters supplied from the control parameter generating unit 26. In this way, the driving assistance processing unit 27 controls the vehicle 9 so that it continues to travel in the driving lane.

[0048] Here, the convolution process A21 corresponds to a specific example of "convolution process" in an embodiment of the present disclosure. The LSTM process A23 corresponds to a specific example of "time series process" in an embodiment of the present disclosure. The processed data DP5 corresponds to a specific example of "first processed data" in an embodiment of the present disclosure. The processed data DP1 to DP4 correspond to a specific example of "one or more second processed data" in an embodiment of the present disclosure. The full connection process A24 corresponds to a specific example of "estimation process" in an embodiment of the present disclosure.

[0049] In this way, the driving path inference processing unit 25 is capable of generating first processed data (e.g., processed data DP5) by performing convolution processing A21 based on the captured image PIC generated by the imaging device 10 that captures an image in front of the vehicle 9, and is capable of storing the first processed data (e.g., processed data DP5), and is capable of performing time series processing (LSTM processing A23) based on the first processed data (e.g., processed data DP5) and one or more pieces of second processed data (e.g., processed data DP1 to DP4) that are generated and stored by the convolution processing A21 based on one or more past captured images PIC, and is provided with an arithmetic circuit (convolution processing unit 21, memory unit 22, LSTM processing unit 23, and full connection processing unit 24) that is capable of inferring the position of the driving path on which the vehicle 9 is traveling by performing estimation processing (full connection processing A24) based on the processing results of the time series processing (LSTM processing A23). This allows the path inference processing unit 25 to infer the position of the path based on the multiple captured images PIC, even in a portion of the path where no dividing lines are provided, such as an intersection. As a result, the path inference processing unit 25 can appropriately infer the position of the path.

[0050] Furthermore, the roadway inference processing unit 25 performs the LSTM process A23 based on the first processed data (e.g., processed data DP5) and one or more second processed data (e.g., processed data DP1 to DP4) that are generated and stored by the convolution process A21 based on one or more past captured images PIC. That is, the roadway inference processing unit 25 does not directly process the five captured images PIC, but instead uses the processed data DP1 to DP5 that are currently being processed. This reduces the amount of calculation required for one processing run to generate the roadway data DT. As a result, the roadway data DT can be generated even by a device with limited computing power, such as an in-vehicle device.

[0051] Furthermore, the travel path inference processing unit 25 configures the one or more past captured images PIC and the captured images PIC to be a series of images captured at different times each time the vehicle 9 travels a predetermined distance. As a result, the travel path inference processing unit 25 can appropriately infer the position of the travel path based on the multiple captured images PIC, even in a section of the travel path without markings, such as an intersection, regardless of the travel speed of the vehicle 9. In other words, if the one or more past captured images PIC and the captured images PIC are acquired each time the vehicle 9 travels for a predetermined period of time, for example, the vehicle 9 may not be able to pass through a section without markings if the travel speed of the vehicle 9 is slow. In such a case, the travel path inference device cannot infer the position of the travel path. On the other hand, the travel path inference processing unit 25 configures the one or more past captured images PIC and the captured images PIC to be a series of images captured at different times each time the vehicle 9 travels a predetermined distance. Therefore, the travel path inference processing unit 25 can appropriately infer the position of the travel path based on a plurality of captured images PIC, regardless of the travel speed of the vehicle 9, for example.

[0052] Furthermore, the travel path inference processing unit 25 is configured to determine the position of the travel path as a position in a coordinate system based on the vehicle 9. This allows the travel path inference processing unit 25 to infer the position of the travel path in the coordinate system based on the vehicle 9, and to control the travel of the vehicle 9 in the coordinate system based on the vehicle 9 based on the inference result.

[0053] [Effects] As described above, in this embodiment, the vehicle navigation system is provided with a calculation circuit that can generate first processed data by performing convolution processing based on captured images generated by an imaging device that captures images of the area ahead of the vehicle, can store the first processed data, can perform time-series processing based on the first processed data and one or more pieces of second processed data that have been generated and stored by convolution processing based on one or more past captured images, and can infer the position of the road on which the vehicle is traveling by performing estimation processing based on the processing results of the time-series processing. This makes it possible to appropriately infer the position of the road.

[0054] In this embodiment, LSTM processing is performed based on the first processed data and one or more second processed data that are each generated and stored by convolution processing based on one or more past captured images, thereby reducing the amount of calculation required for one processing run to generate roadway data.

[0055] In this embodiment, the one or more past captured images and the captured images are a series of images captured at different times each time the vehicle travels a predetermined distance, thereby enabling the position of the road to be appropriately inferred.

[0056] In this embodiment, the position of the road is determined as a position in a coordinate system based on the vehicle, which allows the position of the road to be inferred in the coordinate system based on the vehicle, and the vehicle's travel can be controlled in the coordinate system based on this inference result.

[0057] [Modification] In the above embodiment, the roadway data DT includes data on the position of the left lane marking and data on the position of the right lane marking. In this example, the data on the position of the left lane marking and the data on the position of the right lane marking are data on the positions of point sequences, as shown in FIG. 4 , but this is not limited to this. Alternatively, for example, as shown in FIGS. 6 and 7 , the data on the position of the left lane marking may be the distance CL, the angle BL, and the curvature AL, and the data on the position of the right lane marking may be the distance CR, the angle BR, and the curvature AR. For example, the distance CL corresponds to the position of the left lane marking 113L in the coordinate system of the vehicle 9 as viewed from the reference point of the vehicle 9, the angle BL corresponds to the extension direction of the left lane marking 113L in the coordinate system of the vehicle 9 as viewed from the reference point of the vehicle 9, and the curvature AL is the curvature of the left lane marking 114L. Similarly, the distance CR corresponds to the position of the right lane marking 113R in the coordinate system of the vehicle 9 as viewed from the reference point of the vehicle 9, the angle BR corresponds to the extension direction of the right lane marking 113R in the coordinate system of the vehicle 9 as viewed from the reference point of the vehicle 9, and the curvature AR is the curvature of the right lane marking 114R. Therefore, even in this case, the driving path data DT indicates the position of the driving path on which the vehicle 9 is traveling. In this case, the driving assistance device 1 can omit the control parameter generation unit 26, and the driving assistance processing unit 27 can provide driving assistance for the vehicle 9 based on the driving path data DT generated by the driving path inference processing unit 25.

[0058] Although an example of an embodiment of the present disclosure has been described above with reference to the accompanying drawings, the present disclosure is by no means limited to the above embodiment. Those skilled in the art will understand that various modifications and variations can be made without departing from the scope defined by the appended claims. The present disclosure is intended to encompass such modifications and variations to the extent that they fall within the scope of the appended claims and their equivalents.

[0059] For example, in the above embodiment, the roadway inference processing unit 25 infers the position of the roadway based on five captured images PIC, but this is not limiting. Instead, the roadway inference processing unit 25 may infer the position of the roadway based on, for example, two or more and four or less captured images PIC, or may infer the position of the roadway based on six or more captured images PIC.

[0060] For example, in the above embodiment, an LSTM processing unit 23 is provided, but this is not limited to this, and instead, for example, a processing unit that performs other time series processing different from LSTM processing may be provided.

[0061] For example, in the above embodiment, a full connection processing unit 24 is provided, but this is not limited to this. Instead, for example, a processing unit that performs other estimation processing different from the full connection processing and that is capable of generating roadway data DT may be provided.

[0062] The effects described in this specification are merely examples, and the effects of the present disclosure are not limited to the effects described in this specification. Therefore, other effects may be obtained with respect to the present disclosure.

[0063] Furthermore, the present disclosure may take the following aspects.

[0064] (1) A travel path inference device comprising an arithmetic circuit capable of generating first processed data by performing convolution processing based on captured images generated by an imaging device that captures images ahead of a vehicle, capable of storing the first processed data, and capable of performing time series processing based on the first processed data and one or more second processed data that are generated and stored by the convolution processing based on one or more past captured images, and capable of inferring a position of a travel path on which the vehicle is traveling by performing estimation processing based on a processing result of the time series processing. (2) The one or more past captured images and the captured image are a series of images captured at different times, obtained each time the vehicle travels a predetermined distance. (3) The travel path inference device according to (1) or (2), wherein the position of the travel path is a position in a coordinate system based on the vehicle. (4) A vehicle comprising: an imaging device capable of generating an imaged image by imaging an area ahead of the vehicle; and an arithmetic circuit capable of generating first processed data by performing convolution processing based on the imaged image, capable of storing the first processed data, capable of performing time series processing based on the first processed data and one or more pieces of second processed data that have been generated and stored by the convolution processing based on one or more past imaged images, and capable of inferring the position of the road on which the vehicle is traveling by performing estimation processing based on the processing results of the time series processing.

[0065] The processing device 20 shown in FIG. 2 can be implemented by circuitry including at least one semiconductor integrated circuit, such as at least one processor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC), and / or at least one field-programmable gate array (FPGA). The at least one processor can be configured to perform all or a portion of the various functions of the processing device 20 shown in FIG. 2 by reading instructions from at least one non-transitory, tangible computer-readable medium. Such media can take various forms, including, but not limited to, various magnetic media such as hard disks, various optical media such as CDs or DVDs, and various semiconductor memories (i.e., semiconductor circuits) such as volatile or non-volatile memories. Volatile memories can include DRAM and SRAM. Non-volatile memories can include ROM and NVRAM. An ASIC is an integrated circuit (IC) specialized to perform all or a portion of the various functions of the processing device 20 shown in FIG. 2. An FPGA is an integrated circuit designed to be configurable after manufacture to perform all or a portion of the various functions of the processing device 20 shown in FIG. 2.

Claims

1. A travel path inference device comprising an arithmetic circuit capable of generating first processed data by performing convolution processing based on captured images generated by an imaging device that captures images ahead of a vehicle, capable of storing the first processed data, capable of performing time series processing based on the first processed data and one or more second processed data that have been generated and stored by the convolution processing based on one or more past captured images, and capable of inferring the position of the travel path on which the vehicle is traveling by performing estimation processing based on the processing results of the time series processing.

2. The driving path inference device according to claim 1, wherein the one or more past captured images and the captured image are a series of images captured at different times each time the vehicle travels a predetermined distance.

3. The travel path inference device according to claim 1, wherein the position of the travel path is a position in a coordinate system based on the vehicle.

4. A vehicle equipped with an imaging device capable of generating an image by capturing an image ahead of the vehicle; and an arithmetic circuit capable of generating first processed data by performing convolution processing based on the captured image, capable of storing the first processed data, capable of performing time-series processing based on the first processed data and one or more second processed data that have been generated and stored by the convolution processing based on one or more past captured images, and capable of inferring the position of the road on which the vehicle is traveling by performing estimation processing based on the processing results of the time-series processing.

Images