Driving path prediction device and vehicle

By processing multiple captured images using machine learning technology, driving path data is generated and vehicle driving is controlled, solving the problem of driving path prediction in areas without defined road markings and achieving stable driving control of vehicles at intersections and other locations.

CN123125636APending Publication Date: 2026-11-17SUBARU CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202480091088.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-23
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies struggle to accurately predict the location of a vehicle's path in areas without defined markings, such as intersections.

Method used

Machine learning techniques are employed to process multiple captured images using convolutional processing and Long Short-Term Memory (LSTM) networks to generate driving path data, infer the location of the vehicle's driving path, and generate control parameters through a combined processing approach to control the vehicle's movement.

Benefits of technology

Even in areas without defined lines, it can accurately predict the driving path, achieve stable vehicle control along the driving path, and adapt to different driving speeds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125636A_ABST
    Figure CN123125636A_ABST
Patent Text Reader

Abstract

A driving path estimation device according to one embodiment of the present invention includes a computational circuit that can generate first processing data by performing convolution processing on an image captured by a camera in front of a vehicle, and can store the first processing data. The circuit can perform time series processing on the first processing data and one or more second processing data generated and stored by using convolution processing based on one or more past captured images, and can estimate the position of the driving path of the vehicle by performing estimation processing based on the processing result of the time series processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a path prediction device capable of predicting the travel path of a vehicle, and a vehicle equipped with such a path prediction device. Background Technology

[0002] In vehicles, it is common practice to photograph the area in front of the vehicle and detect the driving path based on the photographic results. For example, Patent Document 1 discloses a technique for predicting the future route of a vehicle based on photographic results taken from the area in front of the vehicle.

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Publication No. 2019-509534 Summary of the Invention

[0004] A driving path estimation device according to one embodiment of the present invention includes a computational circuit. The computational circuit can generate first processed data by performing convolution processing on an image captured by a camera in front of the vehicle, and can store the first processed data. It can perform time series processing based on the first processed data and one or more second processed data generated and stored using convolution processing based on one or more past captured images, and can estimate the position of the driving path of the vehicle by performing estimation processing based on the processing result of the time series processing.

[0005] A vehicle according to one embodiment of the present invention includes a camera and a computational circuit. The camera generates a captured image by capturing a picture of the front of the vehicle. The computational circuit generates first processed data by performing convolution processing on the captured image and stores the first processed data. It performs time-series processing based on the first processed data and one or more second processed data generated and stored using convolution processing based on one or more past captured images. It performs estimation processing based on the processing results of the time-series processing to infer the position of the vehicle's current driving path. Attached Figure Description

[0006] The accompanying drawings are provided to further understand the invention and are incorporated in and constitute a part of this specification. The drawings illustrate one embodiment and, together with the description, serve to explain the principles of the invention.

[0007] Figure 1 This is an explanatory diagram showing a structural example of a vehicle equipped with a driving assistance device according to an embodiment of the present invention.

[0008] Figure 2 It is shown Figure 1 A block diagram of a structural example of a driving assistance device is shown.

[0009] Figure 3 This is an explanatory diagram illustrating an example of the operation of a device for predicting the location of a travel path, as shown in the reference example.

[0010] Figure 4 It is shown Figure 2 The diagram illustrates an example of an operation of the driving path prediction processing unit.

[0011] Figure 5 It is shown Figure 2 The diagram illustrates an example of the detailed operation of the driving path prediction processing unit.

[0012] Figure 6 It is shown Figure 2 The diagram illustrates an example of the parameters processed by the control parameter generation unit.

[0013] Figure 7 It is shown Figure 2 The diagram illustrates an example of other parameters processed by the control parameter generation unit. Detailed Implementation

[0014] A route estimation device that estimates the location of a travel path estimates, for example, based on markings set on the travel path. However, there may be sections of the travel path without markings, such as intersections. It is desirable for the route estimation device to estimate the location of the travel path appropriately even in such cases.

[0015] The goal is to provide a path prediction device and a vehicle capable of appropriately predicting the location of a driving path.

[0016] Hereinafter, some exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the following description represents a specific example of the present invention and should not be construed as limiting the invention. For example, elements including numerical values, shapes, materials, components, the positions of the components, and the connection methods of the components are merely examples and should not be construed as limiting the invention. Furthermore, in the following exemplary embodiments, structural elements not described in the independent claims based on the superior concept of the present invention are arbitrary and can be provided as needed. The drawings are schematic and are not intended to be illustrated at full size. Throughout this specification and the drawings, the same reference numerals are used to denote component elements having substantially the same function and substantially the same structure, and repeated descriptions are omitted. Additionally, structural elements not directly related to an embodiment of the present invention are not illustrated in the drawings.

[0017] <Implementation Method> [Structure Example] Figure 1 and Figure 2An example of the structure of a driving assistance device 1 having a driving path estimation device according to one embodiment is shown. The driving assistance device 1 is mounted on a vehicle 9 and configured to assist the driver in driving the vehicle 9. The driving assistance device 1 includes a camera 10 and a processing device 20.

[0018] The imaging device 10 is configured to generate an image PIC by taking a picture of the front of the vehicle 9. In this example, as... Figure 1 As shown, the imaging device 10 is located inside the vehicle 9, near the upper part of the windshield. In this example, the imaging device 10 is a so-called monocular camera that includes one image sensor. It should be noted that the invention is not limited thereto; instead, the imaging device 10 may also be, for example, a stereo camera including two image sensors. The imaging device 10 generates a series of captured images (PICs) by performing an imaging action at a predetermined frame rate (e.g., 60 fps). The imaging device 10 then supplies the image data of the generated series of captured images (PICs) to the processing device 20.

[0019] The processing unit 20 is configured to control the operation of the driving assistance device 1 by processing based on the captured image PIC. The processing unit 20 is configured using, for example, one or more processors, one or more memories, and performs processing by executing a program. The processing unit 20 includes a driving path prediction processing unit 25, a control parameter generation unit 26, and a driving assistance processing unit 27.

[0020] The driving path estimation processing unit 25 is configured to estimate the position of the driving path of the vehicle 9 based on multiple captured images (PICs) supplied from the capturing device 10, thereby generating driving path data DT. The driving path estimation processing unit 25 estimates the position of the driving path, for example, based on dividing lines set on the driving path. However, there may be sections in the driving path without dividing lines, such as intersections. Even in sections of the driving path without dividing lines, the driving path estimation processing unit 25 can estimate the position of the driving path based on multiple captured images (PICs).

[0021] The following is an example of how a device can infer the location of a driving path based on a single captured image PIC, rather than multiple captured image PICs.

[0022] Figure 3 This illustrates an example of a device that infers the location of a driving path based on a single captured image (PIC). In this example, a vehicle 8 equipped with the device travels along a driving path from... Figure 3 The vehicle travels from below towards above.

[0023] exist Figure 3In example (A), vehicle 8 is traveling on travel path 111 and is passing through intersection CR11 located on travel path 111. Left and right dividing lines 111L and 111R are provided on the portion of travel path 111 closest to and rear of intersection CR11. On the other hand, no dividing lines 111L and 111R are provided at intersection CR11. In this example, the dividing lines 111L and 111R closest to intersection CR11 and the dividing lines 111L and 111R rear of intersection CR11 intersect in the direction of extension of travel path 111. Figure 3 They are located at the same position on the horizontal axis of vehicle 8. The camera device of vehicle 8 is located on vehicle 8. Figure 3 A photographic image PIC is generated by taking a picture at position (A). The area shown as R11 is captured in this photographic image PIC. It should be noted that, for ease of explanation, the shape of this area R11 is shown as a rectangle, but it is actually a more complex shape. For example, based on the image of the dividing lines 111L and 111R contained in this photographic image PIC, vehicle 8 infers the position of the dividing line shown as a dashed line at intersection CR11. In this example, the inferred driving path matches the actual driving path 111.

[0024] exist Figure 3 In example (B), vehicle 8 is traveling on travel path 112 and is passing through intersection CR12 located on travel path 112. Left and right dividing lines 112L and 112R are provided on the portion of travel path 112 closest to and rear of intersection CR12. On the other hand, no dividing lines 112L and 112R are provided at intersection CR12. In this example, the dividing lines 112L and 112R closest to intersection CR12 intersect the dividing lines 112L and 112R rear of intersection CR12 in the direction of their intersection with the extending direction of travel path 112. Figure 3 They are located at different positions on the lateral side (in the middle). That is, the driving path 112 is offset in the direction intersecting the extension direction of the driving path 112. The imaging device of vehicle 8 is located on vehicle 8. Figure 3A photographic image (PIC) is generated by taking a picture at position (B) at a specific time. The PIC contains the area shown as R12. For example, vehicle 8 uses the dividing lines 112L and 112R contained in the PIC to infer the position of the dividing line shown as a dashed line at intersection CR12. In this example, the inferred driving path is inconsistent with the actual driving path 112. That is, because driving path 112 is offset in the direction intersecting the extension direction of driving path 112, vehicle 8 has difficulty correctly inferring the position of the driving path based on the PIC.

[0025] On the other hand, the driving path estimation processing unit 25 of this embodiment estimates the position of the driving path that the vehicle 9 is traveling on based on a plurality of captured image PICs supplied from the capturing device 10. Therefore, the driving path estimation processing unit 25 can estimate the position of the driving path based on a plurality of captured image PICs even in the part of the driving path where no dividing lines are provided.

[0026] Figure 4 An example of operation of the driving path prediction processing unit 25 is shown. In this example, vehicle 9... Figure 3 The driving path 112 shown in (B) is from Figure 4 The vehicle travels from below towards above.

[0027] In this example, the driving path estimation processing unit 25 estimates the position of the driving path based on five captured images (PICs) generated each time the vehicle 9 travels a predetermined distance. The imaging device 10 of the vehicle 9 generates a captured image (PIC) showing the area R1 at position POS1 where the vehicle 9 is located; at position POS2, after the vehicle 9 has traveled a predetermined distance from position POS1, it generates a captured image (PIC) showing the area R2; at position POS3, after the vehicle 9 has traveled a predetermined distance from position POS2, it generates a captured image (PIC) showing the area R3; at position POS4, after the vehicle 9 has traveled a predetermined distance from position POS3, it generates a captured image (PIC) showing the area R4; and at position POS5, after the vehicle 9 has traveled a predetermined distance from position POS4, it generates a captured image (PIC) showing the area R5. In this example, the driving path estimation processing unit 25 estimates the positions of the left and right dividing lines 112L and 112R based on the five captured images (PICs) obtained each time the vehicle 9 travels a predetermined distance. This predetermined distance can be set, for example, to about 8 meters. In this case, the distance between position POS1 and position POS5 is 32m (=8m×4). In this example, this distance is longer than the length of the intersection CR12 in the extension direction of the driving path 112. Therefore, even in the part of the driving path where no dividing lines are set, the driving path estimation processing unit 25 can estimate the positions of the left and right dividing lines 112L and 112R based on multiple captured images PIC. As a result, it can estimate the position of the driving path.

[0028] The driving path prediction processing unit 25 uses machine learning technology to predict the position of the driving path of vehicle 9 based on multiple captured image PICs. In the processing of the driving path prediction processing unit 25, a learned machine learning model generated by a machine learning device (not shown) is used. The driving path prediction processing unit 25 includes a convolution processing unit 21, a storage unit 22, an LSTM (Long Short Term Memory) processing unit 23, and a fully integrated processing unit 24.

[0029] The convolution processing unit 21 is configured to generate processing data DP by performing convolution processing based on captured images (PICs). Captured images (PICs) include, for example, images of various subjects such as road markings along a driving path, various directional signs on the road, streetlights along the driving path, and traffic signals. The positions of these subjects are related to the positions of the driving path. The convolution processing unit 21 generates processing data DP by performing convolution processing on captured images (PICs) including images of such subjects. Then, the convolution processing unit 21 supplies the processing data DP to the storage unit 22 and the LSTM processing unit 23.

[0030] The storage unit 22 is configured to store the processing data DP generated by the convolution processing unit 21. The storage unit 22 stores multiple processing data DPs generated by the convolution processing unit 21 based on multiple different captured images PIC.

[0031] The LSTM processing unit 23 is configured to perform LSTM processing based on the processing data DP supplied from the convolution processing unit 21 and the four processing data DPs stored in the storage unit 22. Specifically, for example, in... Figure 4 In this process, when the LSTM processing unit 23 is supplied with processing data DP based on the captured image PIC obtained when the vehicle 9 is at position POS5, LSTM processing is performed based on the processing data DP and four processing data DPs based on the captured image PIC obtained when the vehicle 9 is at positions POS1 to POS4.

[0032] The full-association processing unit 24 is configured to perform full-association processing based on the processing result of the LSTM processing unit 23, thereby generating driving path data DT representing the position of the driving path that vehicle 9 is currently traveling. In this example, the driving path data DT includes data on the position with respect to the left dividing line and data on the position with respect to the right dividing line in a coordinate system based on vehicle 9. In this example, as... Figure 4 As shown, the data regarding the position of the left dividing line and the data regarding the position of the right dividing line are represented by a point sequence. It should be noted that this is not a limitation; instead, the driving path data DT can also be, for example, data representing the position of the centerline of the driving path.

[0033] Thus, the driving path estimation processing unit 25 estimates the position of the driving path that the vehicle 9 is traveling on based on multiple captured images PIC supplied from the capturing device 10, thereby generating driving path data DT.

[0034] Control parameter generation unit 26 ( Figure 2 The system is configured to generate control parameters for controlling the driving of vehicle 9 based on driving path data DT. These control parameters include, for example, a target steering angle. The control parameter generation unit 26 calculates the target steering angle of vehicle 9 based on the driving path data DT.

[0035] The driver assistance processing unit 27 is configured to provide driver assistance to the vehicle 9 based on control parameters generated by the control parameter generation unit 26. For example, the driver assistance processing unit 27 performs steering control of the vehicle 9 based on the target steering angle included in the control parameters supplied from the control parameter generation unit 26, thereby controlling the vehicle 9 to continuously travel in the driving lane.

[0036] Here, the driving path prediction processing unit 25 corresponds to a specific example of a "driving path prediction device" in one embodiment of the present invention. The convolution processing unit 21, the storage unit 22, the LSTM processing unit 23, and the fully coupled processing unit 24 correspond to a specific example of an "operation loop" in one embodiment of the present invention. The captured image PIC corresponds to a specific example of an "captured image" in one embodiment of the present invention. The processed data DP corresponds to a specific example of a "processed data" in one embodiment of the present invention.

[0037] [Actions and Functions] Next, the operation and function of the driving assistance device 1 in this embodiment will be explained.

[0038] (Overall action summary) First, refer to Figure 2 The operation of the driving assistance device 1 will be explained. The imaging device 10 generates an image PIC by capturing images of the front of the vehicle 9. The driving path estimation processing unit 25 of the processing unit 20 generates driving path data DT by estimating the position of the driving path of the vehicle 9 based on multiple image PICs supplied from the imaging device 10. The control parameter generation unit 26 generates control parameters for controlling the driving of the vehicle 9 based on the driving path data DT. The driving assistance processing unit 27 performs driving assistance for the vehicle 9 based on the control parameters generated by the control parameter generation unit 26.

[0039] (Detailed actions) The driving path estimation processing unit 25 generates driving path data DT by estimating the location of the driving path based on multiple captured images PIC supplied from the imaging device 10. The processing will be described in detail below.

[0040] Figure 5 An example of the processing of the route prediction processing unit 25 is shown. Whenever the vehicle 9 travels a predetermined distance (e.g., 8m), the route prediction processing unit 25 performs the following processing.

[0041] First, the convolution processing unit 21 of the driving path prediction processing unit 25 generates processing data DP (in this example, processing data DP5) by performing convolution processing A21 based on the captured image PIC. Then, the convolution processing unit 21 supplies the processing data DP5 to the storage unit 22 and the LSTM processing unit 23.

[0042] The storage unit 22 of the path prediction processing unit 25 stores the processing data DP5 supplied from the convolution processing unit 21. This storage unit 22 already stores four processing data sets DP1 to DP4 previously supplied from the convolution processing unit 21. The processing data sets DP1 to DP4 are generated in the order of processing data DP1, processing data DP2, processing data DP3, and processing data DP4, and are supplied to the storage unit 22.

[0043] These five processed data DP1 to DP5 respectively correspond to, for example, based on... Figure 4 Five processing data points (DPs) are generated from five captured images (PICs) located at positions POS1 to POS5.

[0044] The LSTM processing unit 23 of the path prediction processing unit 25 performs LSTM processing A23 based on the five processed data DP1 to DP5. This LSTM processing A23 includes five LSTM processes A231 to A235. Specifically, first, the LSTM processing unit 23 performs LSTM processing A231 based on processed data DP1. Next, the LSTM processing unit 23 performs LSTM processing A232 based on the processing result of LSTM processing A231 and processed data DP2. Next, the LSTM processing unit 23 performs LSTM processing A233 based on the processing result of LSTM processing A232 and processed data DP3. Next, the LSTM processing unit 23 performs LSTM processing A234 based on the processing result of LSTM processing A233 and processed data DP4. Next, the LSTM processing unit 23 performs LSTM processing A235 based on the processing result of LSTM processing A234 and processed data DP5.

[0045] Then, the full-association processing unit 24 of the driving path prediction processing unit 25 performs full-association processing A24 based on the processing result of LSTM processing A235 performed by LSTM processing unit 23 to generate driving path data DT representing the driving path of vehicle 9. In this example, the driving path data DT includes data on the position of the vehicle 9 relative to the left dividing line and data on the position of the vehicle 9 relative to the right dividing line in a coordinate system based on vehicle 9. In this example, as... Figure 4 As shown, the data regarding the position of the left dividing line and the data regarding the position of the right dividing line are data regarding the position of the point sequence.

[0046] The processing of the driving path prediction processing unit 25 has ended.

[0047] The control parameter generation unit 26 generates control parameters for controlling the driving of the vehicle 9 based on the driving path data DT. For example, the control parameter generation unit 26 calculates the target steering angle of the vehicle 9 based on the driving path data DT.

[0048] Figure 6 , Figure 7 An example of the processing of the control parameter generation unit 26 is shown.

[0049] For example, such as Figure 6 As shown, the control parameter generation unit 26 calculates the distance CL between the reference point of vehicle 9 and the left dividing line 113L based on the driving path data DT. Here, the reference point of vehicle 9 is, for example, the position of the imaging device 10 in vehicle 9. Similarly, the control parameter generation unit 26 calculates the distance CR between the reference point of vehicle 9 and the right dividing line 113R based on the driving path data DT.

[0050] Additionally, for example, such as Figure 6 As shown, the control parameter generation unit 26 calculates the angle BL between the vehicle length direction of the vehicle 9 and the extension direction of the left dividing line 113L based on the driving path data DT. Figure 6 In this process, a straight line (dashed line) is drawn by moving a straight line (solid line) representing the vehicle length direction of vehicle 9 parallel to the position of the left dividing line 113L. The angle between this straight line (dashed line) and the left dividing line 113L is angle BL. Similarly, the control parameter generation unit 26 calculates the angle BR between the vehicle length direction of vehicle 9 and the extension direction of the right dividing line 113R based on the driving path data DT. Figure 6 In the diagram, a straight line (dashed line) is drawn by moving the straight line (solid line) representing the vehicle length direction of vehicle 9 parallel to the position of the right dividing line 113R. The angle between this straight line (dashed line) and the right dividing line 113R is angle BR.

[0051] Additionally, for example, such as Figure 7 As shown, the control parameter generation unit 26 calculates the curvature AL of the left dividing line 114L and the curvature AR of the right dividing line 114R of the driving path that the vehicle 9 is driving based on the driving path data DT.

[0052] Then, the control parameter generation unit 26 calculates the target steering angle of the vehicle 9 based on the distances CL and CR, the angles BL and BR, and the curvatures AL and AR. Specifically, the control parameter generation unit 26 calculates the target steering angle θ of the vehicle 9 using, for example, the following mathematical formula.

[0053]

Mathematical Formula 1

[0054] Here, Gp, Gi, Gd, and Gff are predetermined coefficients.

[0055] Then, the driver assistance processing unit 27 performs driver assistance for the vehicle 9 based on the control parameters generated by the control parameter generation unit 26. For example, the driver assistance processing unit 27 performs steering control of the vehicle 9 based on the target steering angle θ included in the control parameters supplied from the control parameter generation unit 26. In this way, the driver assistance processing unit 27 controls the vehicle 9 to continuously travel in the driving lane.

[0056] Here, convolution processing A21 corresponds to a specific example of "convolution processing" in one embodiment of the present invention. LSTM processing A23 corresponds to a specific example of "time series processing" in one embodiment of the present invention. Processed data DP5 corresponds to a specific example of "first processed data" in one embodiment of the present invention. Processed data DP1 to DP4 correspond to a specific example of "one or more second processed data" in one embodiment of the present invention. Full combination processing A24 corresponds to a specific example of "estimation processing" in one embodiment of the present invention.

[0057] Thus, the driving path estimation processing unit 25 includes a computational loop (convolution processing unit 21, storage unit 22, LSTM processing unit 23, and full associative processing unit 24). This computational loop can generate first processing data (e.g., processing data DP5) by performing convolution processing A21 on a captured image PIC generated by the imaging device 10 in front of the vehicle 9, and can store the first processing data (e.g., processing data DP5). It can also perform time-series processing (LSTM processing A23) based on the first processing data (e.g., processing data DP5) and one or more second processing data (e.g., processing data DP1 to DP4) generated and stored using convolution processing A21 based on one or more past captured images PIC. Finally, it can perform estimation processing (full associative processing A24) based on the processing result of time-series processing (LSTM processing A23) to estimate the position of the driving path of the vehicle 9. Therefore, in the driving path estimation processing unit 25, even in sections of the driving path without dividing lines, such as intersections, the position of the driving path can be estimated based on multiple captured image PICs. As a result, the location of the driving path can be appropriately predicted in the driving path prediction processing unit 25.

[0058] Furthermore, in the driving path prediction processing unit 25, LSTM processing A23 is performed based on first processed data (e.g., processed data DP5) and one or more second processed data (e.g., processed data DP1 to DP4) generated and stored respectively from one or more previously captured images PIC using convolution processing A21. That is, in the driving path prediction processing unit 25, for example, instead of performing direct processing based on five captured images PIC, processing is performed using data from the processing process, namely processed data DP1 to DP5. As a result, the computational load in one processing step for generating driving path data DT can be reduced. Consequently, even devices with limited computing power, such as vehicle-mounted devices, can generate driving path data DT.

[0059] Furthermore, in the driving path estimation processing unit 25, the previous one or more captured image PICs and captured image PICs are a series of images captured at different times each time the vehicle 9 travels a predetermined distance. Therefore, in the driving path estimation processing unit 25, for example, even in sections of the driving path where no dividing lines are set, such as intersections, the position of the driving path can be appropriately estimated based on the multiple captured image PICs regardless of the vehicle 9's speed. That is, assuming that the previous one or more captured image PICs and captured image PICs are acquired each time the vehicle 9 travels a predetermined distance, for example, if the vehicle 9 fails to pass through a section without dividing lines when its speed is slow, the driving path estimation device cannot estimate the position of the driving path in this case. On the other hand, in the driving path estimation processing unit 25, the previous one or more captured image PICs and captured image PICs are a series of images captured at different times each time the vehicle 9 travels a predetermined distance. Therefore, in the driving path estimation processing unit 25, for example, the position of the driving path can be appropriately estimated based on the multiple captured image PICs regardless of the vehicle 9's speed.

[0060] Furthermore, in the driving path estimation processing unit 25, the position of the driving path is the position in a coordinate system based on the vehicle 9. Therefore, the driving path estimation processing unit 25 can estimate the position of the driving path in the coordinate system based on the vehicle 9, and based on the estimation result, control the driving of the vehicle 9 in the coordinate system based on the vehicle 9.

[0061] [Effect] As described above, in this embodiment, a computational circuit is included. This circuit can generate first processed data by performing convolution processing on an image captured by a camera in front of the vehicle, and can store the first processed data. It can also perform time-series processing based on the first processed data and one or more second processed data generated and stored using convolution processing based on one or more past captured images. Furthermore, it can perform estimation processing based on the processing results of the time-series processing to infer the position of the vehicle's current driving path. Therefore, the position of the driving path can be appropriately inferred.

[0062] In this embodiment, LSTM processing is performed based on the first processing data and one or more second processing data generated and stored from one or more previously captured images using convolution processing. Therefore, the computational load in one processing step for generating driving path data can be suppressed.

[0063] In this embodiment, the previously captured images are a series of images taken at different times each time the vehicle travels a predetermined distance. Therefore, the location of the travel path can be appropriately inferred.

[0064] In this embodiment, the position of the driving path is the position in a coordinate system based on the vehicle. Therefore, it is possible to predict the position of the driving path in the coordinate system based on the vehicle, and based on the predicted result, the vehicle's driving can be controlled in the coordinate system based on the vehicle.

[0065] [Variation Example] In the above embodiment, the driving path data DT includes data regarding the position of the left dividing line and data regarding the position of the right dividing line. In this example, such as... Figure 4 As shown, the data regarding the position of the left dividing line and the data regarding the position of the right dividing line are data about the positions of the point sequence, but are not limited to this. Instead, for example, it could also be as follows: Figure 6 , 7As shown, the data for the position of the left dividing line are distance CL, angle BL, and curvature AL, while the data for the position of the right dividing line are distance CR, angle BR, and curvature AR. For example, distance CL corresponds to the position of the left dividing line 113L in the coordinate system of vehicle 9 as viewed from the reference point of vehicle 9, angle BL corresponds to the extension direction of the left dividing line 113L in the coordinate system of vehicle 9 as viewed from the reference point of vehicle 9, and curvature AL is the curvature of the left dividing line 114L. Similarly, distance CR corresponds to the position of the right dividing line 113R in the coordinate system of vehicle 9 as viewed from the reference point of vehicle 9, angle BR corresponds to the extension direction of the right dividing line 113R in the coordinate system of vehicle 9 as viewed from the reference point of vehicle 9, and curvature AR is the curvature of the right dividing line 114R. Therefore, even in this case, the driving path data DT indicates the position of the driving path that vehicle 9 is traveling on. In this case, the control parameter generation unit 26 can be omitted in the driving assistance device 1, and the driving assistance processing unit 27 can perform driving assistance for the vehicle 9 based on the driving path data DT generated by the driving path prediction processing unit 25.

[0066] While an embodiment of the present invention has been described above with reference to the accompanying drawings, the present invention is by no means limited to the above-described embodiment. It should be understood that various modifications and alterations can be made by those skilled in the art without departing from the scope defined by the appended claims. The present invention is intended to include various modifications and alterations, provided they fall within the scope of the appended claims and their equivalents.

[0067] For example, in the above embodiment, the driving path estimation processing unit 25 estimates the location of the driving path based on five captured image PICs, but the present invention is not limited thereto. Instead, the driving path estimation processing unit 25 may also estimate the location of the driving path based on two or more but less than four captured image PICs, or it may estimate the location of the driving path based on six or more captured image PICs.

[0068] For example, in the above embodiment, an LSTM processing unit 23 is provided, but it is not limited to this. Instead, for example, a processing unit that performs other time series processing different from LSTM processing may also be provided.

[0069] For example, in the above embodiment, a full combination processing unit 24 is provided, but it is not limited to this. Instead, for example, a processing unit that performs other estimation processing different from the full combination processing that can generate driving path data DT may also be provided.

[0070] The effects described in this specification are merely illustrative, and the effects of the invention are not limited to those described herein. Therefore, other effects can also be obtained with respect to the invention.

[0071] Furthermore, the present invention can be implemented in the following ways.

[0072] (1) A travel path prediction device, comprising a computational circuit, The computational loop can generate first processing data by performing convolution processing on images captured by a camera in front of the vehicle, and can store the first processing data. It can perform time series processing based on the first processing data and one or more second processing data generated and stored using the convolution processing based on one or more past images. It can perform estimation processing based on the processing results of the time series processing to infer the location of the vehicle's current driving path.

[0073] (2) The travel path estimation device based on the travel path recorded in (1), wherein, The aforementioned one or more past images and the images being captured are a series of images taken at different times each time the vehicle travels a predetermined distance.

[0074] (3) A travel path estimation device based on the travel path recorded in (1) or (2), wherein, The location of the driving path is the location in a coordinate system with the vehicle as the reference.

[0075] (4) A vehicle having: A camera device capable of generating an image by shooting the front of a vehicle; and The computational loop is capable of generating first processing data by performing convolution processing on the captured image, storing the first processing data, performing time series processing based on the first processing data and one or more second processing data generated and stored using the convolution processing based on one or more past captured images, and performing estimation processing based on the processing result of the time series processing to infer the location of the driving path of the vehicle.

[0076] Figure 2 The processing device 20 shown can be implemented by a circuit including at least one semiconductor integrated circuit, such as at least one processor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC), and / or at least one field-programmable gate array (FPGA). The at least one processor can be configured to execute instructions by reading instructions from at least one non-transitory and tangible computer-readable medium. Figure 2All or part of the various functions in the processing device 20 shown. Such media can take various forms, but are not limited to, including, various magnetic media such as hard disks, various optical media such as CDs or DVDs, and various semiconductor memories (i.e., semiconductor circuits) such as volatile or non-volatile memories. Volatile memories can include DRAM and SRAM. Non-volatile memories can include ROM and NVRAM. ASICs are specifically designed for executing... Figure 2 The processing device 20 shown contains all or some of the various functions of an integrated circuit (IC). An FPGA is designed to be configured after manufacturing to perform... Figure 2 Integrated circuits comprising all or some of the various functions in the processing device 20 shown.

Claims

1. A travel path prediction device, characterized in that, The driving path estimation device includes a computational circuit that can generate first processed data by performing convolution processing on images captured by a camera in front of the vehicle, and can store the first processed data. It can also perform time series processing based on the first processed data and one or more second processed data generated and stored using the convolution processing based on one or more past captured images. Furthermore, it can perform estimation processing based on the processing results of the time series processing to estimate the location of the driving path that the vehicle is currently traveling on.

2. The travel path prediction device according to claim 1, characterized in that, The aforementioned one or more past images and the images being captured are a series of images taken at different times each time the vehicle travels a predetermined distance.

3. The travel path prediction device according to claim 1, characterized in that, The location of the driving path is the location in a coordinate system with the vehicle as the reference.

4. A vehicle, characterized in that, have: A camera device capable of generating an image by shooting the front of a vehicle; and The computational loop is capable of generating first processing data by performing convolution processing on the captured image, storing the first processing data, performing time series processing based on the first processing data and one or more second processing data generated and stored using the convolution processing based on one or more past captured images, and performing estimation processing based on the processing result of the time series processing to infer the location of the driving path of the vehicle.

Citation Information

Patent Citations

  • System and method for estimating future path

    JP2019509534A