Processing device and processing method

Abstract

The present invention is for estimating the type of a travel path. This processing device comprises: a processing unit that sequentially generates data groups containing multiple types of data in a time series on the basis of an image 20 captured by a camera provided to a vehicle to capture the surroundings of the vehicle; and an estimation unit that estimates the type of a travel path 31 of the vehicle on the basis of a plurality of the data groups in a time series.

Description

[0001] [Document name] Statement

[0002] [Title of invention] Processing device and processing method

[0003] [Technical Field]

[0004]

[001] The present invention relates to a processing device and a processing method.

[0005] [Background technology]

[0006]

[002] In recent years, technologies have been proposed that assist drivers in driving by controlling all or part of the driver's driving operations. For example, as disclosed in Patent Document 1, a technology has been proposed for autonomous driving, in which a vehicle runs automatically without the driver's operation.

[0007] [Prior art documents]

[0008] [Patent documents]

[0009]

〇 0 0 3

[0010] [Patent Document 1] International Publication No. 2017 / 208781

[0011] Summary of the Invention

[0012] [Problem to be solved by the invention]

[0013]

[0004] However, various functions related to driving assistance and autonomous driving may be permitted only when the vehicle is traveling on a specific type of road. Therefore, a proposal for estimating the type of road is desired.

[0014]

[0005] In view of these problems, the present invention aims to provide a processing device and a processing method capable of estimating the type of roadway.

[0015] [Means for solving the problem]

[0016]

[0006] In order to solve the above problem, the processing device includes a processing unit that sequentially generates data groups containing multiple types of data in a time series based on images captured by a camera installed on the vehicle that captures images of the vehicle's surroundings, and an estimation unit that estimates the type of road the vehicle is traveling on based on the multiple time series data groups.

[0017]

[0007] In order to solve the above problem, in the processing method, a processing unit of the processing device sequentially generates data groups containing multiple types of data in chronological order based on images captured by a camera mounted on the vehicle that captures images of the vehicle's surroundings, and an estimation unit of the processing device estimates the type of road the vehicle is traveling on based on the multiple data groups in chronological order.

[0018] [Effects of the Invention]

[0019]

[0008] According to the present invention, it is possible to estimate the type of road.

[0020] [Brief explanation of the drawings]

[0021]

〇 0 0 9

[0022] [Figure 1] Schematic diagram showing the general configuration of a vehicle according to an embodiment of the present invention.

[0023] FIG. 2 is a block diagram showing an example of the functional configuration of a processing device according to an embodiment of the present invention.

[0024] [Figure 3] A figure showing an example of an image captured by a vehicle camera in an embodiment of the present invention.

[0025] [Figure 4] A flowchart showing an example of the processing flow performed by a processing device according to an embodiment of the present invention.

[0026] [Figure 5] A figure showing an example of multiple time-series data groups related to an embodiment of the present invention.

[0027] [Figure 6] A block diagram showing details of an example of the functional configuration of a processing device according to an embodiment of the present invention. [Form for carrying out the invention]

[0028]

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The dimensions, materials, and other specific values ​​shown in the embodiments are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In this specification and drawings, elements having substantially the same functions and configurations are designated by the same reference numerals to avoid redundant explanation, and elements not directly related to the present invention are not shown.

[0029]

[0011] <Vehicle Configuration> The configuration of a vehicle 10 according to an embodiment of the present invention will be described with reference to Figs. 1 and 2.

[0030]

[0012] Figure 1 is a schematic diagram showing the general configuration of a vehicle 10. As shown in Figure 1, the vehicle 10 includes a steering mechanism 11, a drive source 12, a braking device 13, a camera 14, a wheel speed sensor 15, and a processing device 16.

[0031]

[0013] The steering mechanism 11 is a mechanism that changes the steering angle of the vehicle 10. The steering angle of the vehicle 10 refers to the turning angle of the tires of the vehicle 10. The steering mechanism 11 includes a steering wheel 11a. The driver of the vehicle 10 can change the steering angle by performing a steering operation using the steering wheel 11a.

[0032]

[0014] The drive source 12 outputs a drive force that is transmitted to the drive wheels of the vehicle 10. Examples of the drive source 12 include an engine and an electric motor.

[0033]

[0015] The brake device 13 applies braking force to the wheels of the vehicle 10. For example, the brake device 13 includes a hydraulic control unit, which adjusts the hydraulic pressure of the brake fluid in the wheel cylinder, thereby adjusting the braking force applied to the wheel.

[0034]

[0016] Camera 14 captures images of the surroundings of vehicle 1O. For example, camera 14 is provided at the front of vehicle 1O and captures images of the area ahead of vehicle 1O. As will be described later, the images captured by camera 14 are used in processing to estimate the type of road on which vehicle 1O is traveling. However, the number and arrangement of cameras 14 are not limited to the example in FIG. 1. For example, the number of cameras 14 may be greater than that in the example in FIG. 1.

[0035]

[0017] Wheel speed sensors 15 are provided on each wheel to detect the wheel speed of each wheel.

[0036]

[0018] The processing device 16 performs various processes related to the vehicle 10. The processing device 16 includes a CPU (Central Processing Unit), which is an arithmetic processing device, a ROM (Read Only Memory), which is a memory element that stores programs used by the CPU, calculation parameters, etc., and a RAM (Random Access Memory), which is a memory element that temporarily stores parameters that change as the CPU executes. The processing device 16 may be, for example, a single device, or may be divided into multiple devices. When the processing device 16 is divided into multiple devices, the various functions described below may be shared among the multiple devices.

[0037]

[0019] Fig. 2 is a block diagram showing an example of the functional configuration of the processing device 16. As shown in Fig. 2, the processing device 16 includes, for example, an acquisition unit 16a, a control unit 16b, a generation unit 16c, an estimation unit 16d, and a storage unit 16e.

[0038]

[0020] The acquisition unit 16a acquires information from each device in the vehicle 10. For example, the acquisition unit 16a acquires information from the camera 14 and the wheel speed sensor 15. In this specification, the acquisition of information may include the extraction or generation of information (for example, calculation).

[0039]

[0021] In particular, the acquisition unit 16a can extract various data as secondary information from the images captured by the camera 14. As will be described later, the data extracted from the images captured by the camera 14 is stored in the storage unit 16e and used to estimate the type of road on which the vehicle 10 is traveling. Details of such data will be described later.

[0040]

[0022] The control unit 16b controls the operation of each device in the vehicle 10. For example, the control unit 16b controls the operation of the steering mechanism 11, the drive source 12, and the brake device 13.

[0041]

[0023] The control unit 16b can perform automatic driving by controlling, for example, the steering mechanism 11, the drive source 12, and the brake device 13. Automatic driving is a control that automatically drives the vehicle 10 without the driver's driving operation. For example, in automatic driving, the control unit 16b automatically drives the vehicle 10 along a route toward a destination specified by the driver. Note that automatic driving may be control that handles all driving operations by the driver, or control that handles only some of the driving operations. Note that the following mainly describes an example in which the vehicle to which the present invention is applied is a vehicle 10 that can perform automatic driving, but the present invention may also be applied to a vehicle 10 that can perform various driving assistance functions other than automatic driving.

[0042]

[0024] For example, during automatic driving, the control unit 16 b can automatically control the speed of the vehicle 10 by automatically controlling the drive source 12 and the brake device 13. Also, during automatic driving, the control unit 16 b can automatically control the steering angle of the vehicle 10 by automatically controlling the steering mechanism 11.

[0043]

[0025] In addition, in the case of autonomous driving, the control unit 16b may control the behavior of the vehicle 10 according to the surrounding traffic conditions, etc., using, for example, information about the surrounding environment of the vehicle 10 obtained from an image captured by the camera 14. Examples of the surrounding environment information include information related to the distance or direction to an object located around the vehicle 10 (for example, relative position, relative distance, relative speed, relative acceleration, etc.), or characteristics of the object located around the vehicle 10 (for example, the type of object, the shape of the object itself, a mark attached to the object, etc.).

[0044]

[0026] The driver of vehicle 10 can switch between, for example, an automated driving mode in which automated driving is performed and a manual driving mode in which automated driving is not performed and the driving of vehicle 10 is left to manual driving. For example, the driver can switch between these modes by performing a specific operation using an input device of vehicle 10.

[0045]

[0027] The generation unit 16c and the estimation unit 16d perform processing related to estimation of the type of road on which the vehicle 10 is traveling. The generation unit 16c reads data extracted by the acquisition unit 16a from the image captured by the camera 14 from the storage unit 16e, thereby sequentially generating data groups containing multiple types of data in chronological order. The estimation unit 16d estimates the type of road on which the vehicle 10 is traveling based on the multiple time-series data groups. Details of the processing performed by the generation unit 16c and the estimation unit 16d will be described later.

[0046]

[0028] The memory unit 16e stores various information. As will be described later, the information stored in the memory unit 16e is used in the process of estimating the type of road on which the vehicle 10 is traveling.

[0047]

[0029] As described above, in the processing device 16 according to this embodiment, the acquisition unit 16a and the generation unit 16c correspond to a processing unit that sequentially generates a data group including multiple types of data in time series based on an image captured by the camera 14.

[0048]

[0030] <Operation of Processing Apparatus> The operation of the processing apparatus 16 according to the embodiment of the present invention will be described with reference to Figs. 3 to 6.

[0049]

[0031] As described above, the control unit 16b can perform autonomous driving, which causes the vehicle 10 to travel automatically without any driving operation by the driver. Here, autonomous driving is permitted only when the vehicle 10 is traveling on a specific type of road (for example, a national expressway). Therefore, in order to determine whether autonomous driving can be performed, it is necessary to estimate the type of road. Note that the type of road can be, for example, a national expressway, a general national highway, a prefectural road, or a municipal road, as defined by the Road Act.

[0050] As will be described later, in this embodiment, the type of road on which the vehicle 10 is traveling can be estimated by processing by the generation unit 16 c and the estimation unit 16 d using images captured by the camera 14. Before describing the processing performed by the generation unit 16 c and the estimation unit 16 d, data obtained from images captured by the camera 14 will be described below.

[0051]

[0033] Fig. 3 is a diagram showing an example of an image 2O captured by the camera 14 of the vehicle 1O. The image 2O in Fig. 3 is captured by the camera 14 while the vehicle 1O is traveling on the travel path 31. For example, the camera 14 captures images at predetermined time intervals. Then, the acquisition unit 16a extracts various data from the captured image 2O as secondary information each time the camera 14 captures an image. The data extracted at predetermined time intervals in this manner is stored in the memory unit 16e.

[0052]

[0034] The acquisition unit 16a can extract various data, for example, by performing image processing on the image 20 captured by the camera 14.

[0053]

[0035] For example, the acquisition unit 16a extracts data of a road sign 32 shown in the image 20. The data of the road sign 32 is specifically data indicating the position and type of the road sign 32. The road sign 32 is a signboard installed beside the travel path 31. For example, the road sign 32 may include a guide sign, a warning sign, a regulatory sign, a directional sign, an auxiliary sign, etc. A guide sign is a signboard that displays information about a destination (for example, the name of the destination, the direction to the destination, or the distance to the destination). A warning sign is a signboard that displays information to be careful of when passing through (for example, the risk of falling rocks or the risk of animals jumping out). Regulatory signs are signs that indicate restrictions on passage (for example, no parking or maximum speeds). Directional signs are signs that indicate designated locations in road traffic (for example, parking areas or priority roads). Supplementary signs are signs that supplement other signs (for example, to indicate the applicable section or time of the sign's contents).

[0054]

[0036] The acquisition unit 16a extracts, for example, feature quantities (such as shape, color, and arrangement) of the road sign 32 shown in the image 20 using image processing, and extracts data of the road sign 32 using a prediction model for predicting the road sign 32 shown in the image 20 from the feature quantities. The prediction model is constructed by learning in advance using, for example, an existing machine learning algorithm.

[0055]

[0037] For example, the acquisition unit 16a extracts data of the moving object 33 shown in the image 20. The data of the moving object 33 is specifically data indicating the position and type of the moving object 33. Examples of the moving object 33 include a vehicle, a bicycle, and a pedestrian.

[0056]

[0038] The acquisition unit 16a extracts, for example, feature quantities (such as shape, color, and location) of the moving object 33 shown in the image 20 using image processing, and extracts data on the moving object 33 using a prediction model for predicting the moving object 33 shown in the image 20 from the feature quantities. The prediction model is constructed by learning in advance using, for example, an existing machine learning algorithm.

[0057]

[0039] For example, the acquisition unit 16a extracts data of the road markings 34 shown in the image 20. The data of the road markings 34 is specifically data indicating the position and type of the road markings 34. The road markings 34 are symbols or letters painted on the road surface of the travel path 31. For example, the road markings 34 may include regulatory markings and directional markings. Regulatory markings are markings that indicate restrictions on passage (for example, no turns, no overtaking, or maximum speeds). Directional markings are markings that indicate designated locations in road traffic (for example, crosswalks or lane boundaries).

[0058]

[0040] The acquisition unit 16a extracts, for example, feature quantities (such as shape, color, and arrangement) of the road markings 34 shown in the image 20 using image processing, and extracts data on the road markings 34 using a prediction model for predicting the road markings 34 shown in the image 20 from the feature quantities. The prediction model is constructed by learning in advance using, for example, an existing machine learning algorithm.

[0059]

[0041] For example, the acquisition unit 16a extracts data on the type of the road 31. Specifically, the data on the type of the road 31 is data indicating the type of the road 31. As described above, the type of the road 31 can be, for example, a national expressway, a general national highway, a prefectural road, or a municipal road, as defined by the Road Act.

[0060]

[0042] The acquisition unit 16a extracts feature quantities (such as shape, color, and arrangement) of the image 20 using image processing, and extracts data on the type of the road 31 using a prediction model for predicting the type of the road 31 from the feature quantities. The prediction model is constructed by learning in advance using an existing machine learning algorithm, for example.

[0061]

[0043] As described above, the type of the road 31 is estimated at each time point by the acquisition unit 16a using the image 20 and the prediction model. However, since it is difficult to accurately estimate the type of the road 31 only by the estimation process by the acquisition unit 16a, the type of the road 31 is estimated again by processing by the generation unit 16c and the estimation unit 16d, which will be described later.

[0062]

[0044] Below, we will explain in detail an example of processing by the generation unit 16c and the estimation unit 16d using the image 20 captured by the camera 14.

[0063]

[0045] Fig. 4 is a flowchart showing an example of the flow of processing performed by the processing device 16. Step S101 in Fig. 4 corresponds to the start of the processing flow shown in Fig. 4. The processing flow shown in Fig. 4 starts, for example, while the vehicle 10 is traveling.

[0064]

[0046] As described above, the acquisition unit 16a extracts various data as secondary information from the captured image 20 each time an image is captured by the camera 14. Therefore, while a NO determination is made in step S102 (described below) and step S102 is being repeated, the acquisition unit 16a repeatedly extracts various data from the image 20.

[0065]

[0047] When the processing flow shown in FIG. 4 starts, in step S102, the generation unit 16c determines whether the travel distance of the vehicle 10 has increased by a reference distance.

[0066]

[0048] As will be described later, in the example of Fig. 4, the processes of steps S103 to S105 (specifically, generation of a data group and estimation of the type of road 31) are performed each time the travel distance of vehicle 10 increases by a reference distance. That is, in step S102, generation unit 16c determines whether the travel distance of vehicle 10 has increased by a reference distance since the processes of steps S103 to S105 were last performed.

[0067]

[0049] As will be described later, in the example of Fig. 4, the type of the road 31 is estimated using a plurality of time-series data groups generated each time the travel distance of the vehicle 10 increases by a reference distance. Therefore, the reference distance is set to an appropriate distance that is not too long and not too short, for example, from the viewpoint of accurately estimating the type of the road 31.

[0068] For example, the generation unit 16 c can determine the travel distance of the vehicle 10 based on the detection result of the wheel speed sensor 15. Therefore, the generation unit 16 c can determine whether the travel distance of the vehicle 10 has increased by a reference distance based on the detection result of the wheel speed sensor 15.

[0069]

[0051] If it is determined that the mileage of vehicle 10 has not increased by the reference distance (step S102 / NO), step S102 is repeated. On the other hand, if it is determined that the mileage of vehicle 10 has increased by the reference distance (step S102 / YES), proceed to step S103.

[0070]

[0052] If the answer is YES in step S102, in step S103, the generation unit 16c creates a data group including multiple types of data.

[0071]

[0053] Specifically, the generation unit 16c generates a data group including multiple types of data by reading data extracted from the image 20 by the acquisition unit 16a and stored in the storage unit 16e. For example, the generation unit 16c generates a data group including various types of data most recently extracted by the acquisition unit 16a (specifically, data on road signs 32, data on moving objects 33, data on road markings 34, and data on the type of road 31).

[0072]

[0054] For specific types of data included in the generated data group, the generation unit 16c may generate the data using multiple pieces of data extracted by the acquisition unit 16a at each time point while the vehicle 10 travels the reference distance (i.e., from the time when YES was determined in the previous step S102 to the time when YES was determined in the current step S102).

[0073]

[0055] For example, it is difficult to determine whether the road sign 32 shown in the image 20 obtained before the vehicle 10 traveled the reference distance (i.e., the time when the previous step S102 was judged as YES) and the road sign 32 shown in the image 20 obtained at the current time (i.e., the time when the current step S102 was judged as YES) are the same sign by comparing the data extracted at the two times mentioned above.

[0074] Therefore, the generation unit 16c may use, for example, a plurality of pieces of data extracted by the acquisition unit 16a at each time point while the vehicle 10 travels a reference distance (e.g., taking into consideration the continuity of the plurality of pieces of data), to generate data including information indicating the likelihood of estimating the type of the road sign 32 shown in the image 20 currently obtained, as data of the road sign 32 included in the generated data group. Note that such data is generated, for example, using a prediction model for predicting the likelihood from the plurality of pieces of data extracted at each time point. The prediction model is constructed by learning in advance, for example, using an existing machine learning algorithm.

[0075]

[0057] Next, in step S104, the generation unit 16c stores the generated data group in the memory unit 16e.

[0076]

[0058] As described above, the generation unit 16c generates a data group each time the travel distance of the vehicle 10 increases by the reference distance, and stores the generated data group in the storage unit 16e. In this way, the generation unit 16c sequentially generates data groups in time series, and multiple time series data groups are stored in the storage unit 16e.

[0077]

[0059] Figure 5 is a diagram showing an example of a plurality of time-series data groups G1. For example, the plurality of data groups G1 shown in Figure 5 are stored in the storage unit 16e. In the example of Figure 5, the plurality of data groups G1 stored in the storage unit 16e are shown in a data structure in a data table format in which the type of data is classified by row and the time of data acquisition is classified by column. However, the plurality of data groups G1 stored in the storage unit 16e may be shown in a data structure in a format other than a data table format.

[0078]

[0060] In the example of Figure 5, data groups G1 generated at each time point T1, T2, T3-•-Tn are stored in the memory unit 16e. That is, n data groups G1 are stored in the memory unit 16e. The times T1, T2, T3-•-Tn are arranged in order of proximity to the current time. Each data group G1 includes data D1 of a road sign 32, data D2 of a moving object 33, data D3 of a road marking 34, and data D4 of the type of road 31.

[0079]

[0061] Of times T1, T2, T3 - Tn, time T! is the most recent. The data group G1 at time T1 includes data D1 of the road sign 32 generated at time T1, data D2 of the moving object 33 generated at time T1, data D3 of the road marking 34 generated at time T1, and data D4 of the type of road 31 generated at time T!.

[0080]

[0062] Of times T1, T2, T3-•-Tn, time T2 is the next most recent time after time T1. Between time T2 and time T1, vehicle 10 travels a reference distance. Data group G1 at time T2 includes data D1 of road sign 32 generated at time T2, data D2 of moving object 33 generated at time T2, data D3 of road marking 34 generated at time T2, and data D4 of the type of road 31 generated at time T2.

[0081]

[0063] Of times T1, T2, and T3-•-Tn, time T3 is the next most recent time after time T2. Between time T3 and time T2, vehicle 10 travels a reference distance. Data group G1 at time T3 includes data D1 of road sign 32 generated at time T3, data D2 of moving object 33 generated at time T3, data D3 of road marking 34 generated at time T3, and data D4 of the type of road 31 generated at time T3.

[0082]

[0064] Of times T1, T2, T3 - Tn, time Tn is the oldest. The data group G1 at time Tn includes data D1 of the road sign 32 generated at time Tn, data D2 of the moving object 33 generated at time Tn, data D3 of the road marking 34 generated at time Tn, and data D4 of the type of road 31 generated at time Tn.

[0083]

[0065] In step S!04, the generation unit 16c stores the data group G1 generated in step S103 in the memory unit 16e on a first-in, first-out basis. That is, the generation unit 16c deletes the oldest data group G1 stored in the memory unit 16e, and adds and stores the data group G1 generated in step S103 in the memory unit 16e. For example, in the example of FIG. 5, the generation unit 16c deletes the data group G1 stored in the memory unit 16e at time Tn, and adds and stores the most recently generated data group G1 in the memory unit 16e. As a result, the number of data groups G1 stored in the memory unit 16e is maintained at n. In this way, the memory unit 16e stores a predetermined number (n in the example of FIG. 5) of recently generated time-series data groups G1, and the time-series data groups G1 stored in the memory unit 16e are updated every time the travel distance of the vehicle 10 increases by a reference distance.

[0066] After step S104 in FIG. 3, in step S105, the estimation unit 16d estimates the type of the travel path 31 of the vehicle 10 based on the multiple time-series data groups G1, and the process returns to step S102.

[0084]

[0067] Specifically, the estimation unit 16d estimates the type of the vehicle 10's road 31 using a plurality of time-series data groups G1 stored in the memory unit 16e and updated each time the travel distance of the vehicle 10 increases by a reference distance, and a prediction model for predicting the type of the vehicle 10's road 31 from the plurality of data groups G1. The prediction model is constructed by learning in advance using, for example, an existing machine learning algorithm. For example, in the example of FIG. 5, the estimation unit 16d estimates the type of the vehicle 10's road 31 using n time-series data groups G1 stored in the memory unit 16e and updated in step S104.

[0085]

[0068] An example of the processing performed by the processing device 16 has been described above. Here, details of an example of the functional configuration of the processing device 16 will be described again with reference to FIG.

[0086]

[0069] Fig. 6 is a block diagram showing the details of an example functional configuration of the processing device 16. In Fig. 6, of the functional configuration of the processing device 16, an acquisition unit 16a, a generation unit 16c, and an estimation unit 16d are particularly extracted and shown, and other functional configurations are omitted.

[0087]

[0070] As shown in FIG. 6, the acquisition unit 16a includes, for example, a first acquisition unit 16a1, a second acquisition unit 16a2, a third acquisition unit 16a3, and a fourth acquisition unit 16a4.

[0088]

[0071] The first acquisition unit 16a1, the second acquisition unit 16a2, the third acquisition unit 16a3, and the fourth acquisition unit 16a4 extract various types of data as secondary information from the captured image 20 each time an image is captured by the camera 14. The first acquisition unit 16a1 acquires an image 20 from the camera 14 and extracts data on a road sign 32 that appears in the image 20. The second acquisition unit 16a2 acquires an image 20 from the camera 14 and extracts data on a moving object 33 that appears in the image 20. The third acquisition unit 16a3 acquires an image 20 from the camera 14 and extracts data on a road marking 34 that appears in the image 20. The fourth acquisition unit 16a4 acquires the image 20 from the camera 14 and extracts data on the type of the driving path 31.

[0089]

[0072] As shown in FIG. 6, the generation unit 16c includes, for example, a first generation unit 16c1, a second generation unit 16c2, a third generation unit 16c3, a fourth generation unit 16c4, a fifth generation unit 16c5, a sixth generation unit 16c6, a seventh generation unit 16c7, an eighth generation unit 16c8, and a ninth generation unit 16c9.

[0090]

[0073] The first generation unit 16c1 and the second generation unit 16c2 perform processing to output trigger signals to the third generation unit 16c3, the fourth generation unit 16c4, the fifth generation unit 16c5, and the sixth generation unit 16c6 each time the travel distance of the vehicle 10 increases by a reference distance. The first generation unit 16c1 calculates the travel distance of the vehicle 10 based on the detection result of the wheel speed sensor 15. When the second generation unit 16c2 determines that the travel distance of the vehicle 10 has increased by a reference distance based on the calculation result by the first generation unit 16c1, the second generation unit 16c2 outputs a trigger signal to the third generation unit 16c3, the fourth generation unit 16c4, the fifth generation unit 16c5, and the sixth generation unit 16c6.

[0091]

[0074] The third generation unit 16c3, the fourth generation unit 16c4, the fifth generation unit 16c5, and the sixth generation unit 16c6 read out data extracted from the image 20 by the acquisition unit 16a every time a trigger signal is input from the second generation unit 16c2. The third generation unit 16c3 reads out data of the road sign 32 extracted by the first acquisition unit 16a1 at each time point between the time the previous trigger signal was input and the time the current trigger signal is input (that is, while the vehicle 10 is traveling the reference distance), and outputs the data to the seventh generation unit 16c7. The fourth generation unit 16c4 reads out data of the moving object 33 most recently extracted by the second acquisition unit 16a2 and outputs it to the eighth generation unit 16c8. The fifth generation unit 16c5 reads out data of the road marking 34 most recently extracted by the third acquisition unit 16a3 and outputs it to the eighth generation unit 16c8. The sixth generation unit 16c6 reads out data on the type of the road 31 most recently extracted by the fourth acquisition unit 16a4 and outputs it to the eighth generation unit 16c8.

[0092]

[0075] The seventh generation unit 16c7 uses the data of the multiple road signs 32 read out by the third generation unit 16c3 to generate data including information indicating the likelihood of estimating the type of the road sign 32 shown in the currently obtained image 2〇 as data of the road sign 32 included in the generated data group G1. The data of the road sign 32 generated by the seventh generation unit 16c7 is output to the eighth generation unit 16c8.

[0093]

[0076] The eighth generation unit 16c8 stores a data group G1 including the data of the road sign 32 input from the seventh generation unit 16c7, the data of the moving object 33 input from the fourth generation unit 16c4, the data of the road marking 34 input from the fifth generation unit 16c5, and the data of the type of the travel path 31 input from the sixth generation unit 16c6 in the memory unit 16e by a first-in, first-out method.

[0094]

[0077] The ninth generation unit 16c9 converts the time-series data group G1 stored in the memory unit 16e into information that can be input into a prediction model used by the estimation unit 16d to predict the type of the road 31, and outputs the information to the estimation unit 16d. Then, the estimation unit 16d predicts the type of the road 31 using the time-series data group G1.

[0095]

[0078] As described above, in the processing device 16 according to this embodiment, the processing unit (in the above example, the acquisition unit 16a and the generation unit 16c) sequentially generates a data group G1 including multiple types of data in a time series based on the image 20 captured by the camera 14, and the estimation unit 16d estimates the type of the road 31 of the vehicle 10 based on the multiple time series data groups G1. As a result, the type of the road 31 can be estimated by understanding the transition of various information in a predetermined section of the road 31 including the current position of the vehicle 10 based on the multiple time series data groups G1. Therefore, for example, it is possible to appropriately determine whether various functions related to driving assistance, autonomous driving, etc. can be executed depending on the type of the road 31.

[0096]

[0079] In particular, according to this embodiment, the type of the road 31 can be estimated without using any sensor other than the camera 14. Therefore, the number of sensors installed in the vehicle 10 can be reduced, thereby reducing costs. Furthermore, for example, when positioning information transmitted from a GNSS (Global Navigation Satellite System) satellite is acquired by a sensor and the type of the road 31 is estimated using the positioning information, in a location where the positioning information cannot be acquired stably (for example, inside a tunnel or in an urban area), it may be difficult to estimate the type of the road 31, or the accuracy of estimating the type of the road 31 may decrease. On the other hand, in this embodiment, the type of the road 31 can be stably estimated regardless of the location.

[0097]

[0080] Furthermore, according to this embodiment, the type of the road 31 can be estimated without using map information. For example, when estimating the type of the road 31 using map information, if the user has not updated the map information, it may be difficult to estimate the type of the road 31, or the accuracy of estimating the type of the road 31 may decrease. On the other hand, in this embodiment, the type of the road 31 can be stably estimated regardless of the update status of the map information. Also, when estimating the type of the road 31 using map information, it may be necessary to obtain map information by communication. On the other hand, in this embodiment, the type of the road 31 can be estimated without using map information, so there is no need to prepare a communication means.

[0098]

[0081] Furthermore, according to this embodiment, the type of the road 31 can be estimated without using information on the driver's driving operation (for example, information on the time progression of acceleration / deceleration operations, etc.). For example, when estimating the type of the road 31 using information on the driver's driving operation, if an unexpected driving operation or a special driving operation to cope with a traffic jam is performed, the accuracy of estimating the type of the road 31 may decrease. On the other hand, in this embodiment, the type of the road 31 can be stably estimated regardless of the driver's driving operation.

[0099]

[0082] Furthermore, according to this embodiment, it is easy to estimate the type of the road 31 corresponding to each country that is the destination of the vehicle 10. For example, by preparing each prediction model used in the processing example described above for each country and simply changing it to correspond to the country of destination, the type of the road 31 can be estimated.

[0100]

[0083] The processing example of Fig. 4 has been described above as an example of processing performed by the processing device 16. However, the processing performed by the processing device 16 is not limited to the above processing example, and may be, for example, a processing obtained by appropriately modifying the above processing example.

[0101]

[0084] For example, in the above example, the data group G1 includes data D1 of road signs 32, data D2 of moving objects 33, data D3 of road markings 34, and data D4 of the type of road 31. However, the types of data included in the data group G1 are not limited to the above example. For example, any part of data D1, data D2, data D3, and data D4 may be omitted from the data group G1. Furthermore, for example, types of data other than data D1, data D2, data D3, and data D4 may be added to the data group G1. For example, data of stationary objects (such as toll gates on expressways and toll roads, traffic lights, and traffic cones (registered trademarks)) may be added to the data extracted by the acquisition unit 16 a and input to the generation unit 16 c, and this data may be included in the data group G 1. Note that the acquisition unit 16 a may extract data of stationary objects from the image 20, for example, by using features learned in advance.

[0102]

[0085] In the above example, the generation unit 16c generates the data group G1 sequentially in time series by generating the data group G1 each time the travel distance of the vehicle 10 increases by a reference distance. However, the generation unit 16c may generate the data group G1 sequentially in time series by a method other than the above method. For example, the generation unit 16c may generate the data group G1 sequentially in time series by generating the data group G1 each time the cumulative time that the vehicle 10 is traveling (i.e., the actual elapsed time minus the time that the vehicle 10 is stopped) increases by a reference time. The reference time is set to a length of time that allows it to be determined that the travel distance of the vehicle 10 has increased by approximately the reference distance.

[0103]

[0086] <Effects of the Processing Apparatus> The effects of the processing apparatus 16 according to the embodiment of the present invention will be described.

[0104]

[0087] The processing device 16 includes a processing unit (in the above example, an acquisition unit 16a and a generation unit 16c) that sequentially generates a data group G1 including multiple types of data in a time series based on an image 2O captured by a camera 14 installed on the vehicle 1O that captures images of the surroundings of the vehicle 1O, and an estimation unit 16d that estimates the type of the road 31 of the vehicle 1O based on the multiple time series data groups G1. As a result, the type of the road 31 can be estimated by understanding the transition of various information in a predetermined section of the road 31 including the current position of the vehicle 1O based on the multiple time series data groups G1. Therefore, for example, it is possible to appropriately determine whether various functions related to driving assistance, autonomous driving, etc. can be executed depending on the type of the road 31.

[0105]

[0088] Furthermore, according to this embodiment, the type of the road 31 can be estimated using a small number of sensors, thereby achieving the various effects described above. Furthermore, according to this embodiment, the type of the road 31 can be estimated without using map information, thereby achieving the various effects described above. Furthermore, according to this embodiment, the type of the road 31 can be estimated without using information on the driver's driving operation, thereby achieving the various effects described above. Furthermore, according to this embodiment, as described above, it is easy to estimate the type of the road 31 according to each country where the vehicle 10 is destined.

[0106]

[0089] Preferably, in the processing device 16, the data group G1 includes data D1 of the road sign 32. This makes it possible to focus on the time-series data D1 of the multiple road signs 32 and understand the transition of information about the road signs 32 in a predetermined section of the road 31 that includes the current position of the vehicle 10, thereby appropriately realizing the estimation of the type of the road 31.

[0107]

[0090] Preferably, in the processing device 16, the data group G1 includes data D2 of the moving body 33. This makes it possible to focus on the time-series data D2 of the multiple moving bodies 33 and grasp the transition of information about the moving body 33 in a predetermined section of the travel path 31 that includes the current position of the vehicle 10, thereby appropriately realizing estimation of the type of the travel path 31.

[0108]

[0091] Preferably, in the processing device 16, the data group G1 includes data D3 of the road markings 34. This makes it possible to focus on the time-series data D3 of the plurality of road markings 34 and understand the transition of information about the road markings 34 in a predetermined section of the road 31 that includes the current position of the vehicle 10, thereby appropriately realizing estimation of the type of the road 31.

[0109]

[0092] Preferably, in the processing device 16, the data group G1 includes data D4 on the type of the road 31. This makes it possible to estimate the type of the road 31 by taking into account the data D4 on the type of the road 31 included in the multiple time-series data groups G1. This improves the accuracy of estimating the type of the road 31.

[0110]

[0093] Preferably, in the processing device 16, the processing unit (in the above example, the generating unit 16c) generates the data group G1 each time the travel distance of the vehicle 10 increases by a reference distance, thereby sequentially generating the data group G1 in a time series. This makes it possible to appropriately grasp the transition of various information in a predetermined section of the travel path 31 including the current position of the vehicle 10 based on the multiple time series data groups G1, and therefore to appropriately estimate the type of the travel path 31.

[0111]

[0094] Although preferred embodiments of the present invention have been described above with reference to the accompanying drawings, it goes without saying that the present invention is not limited to the above-described embodiments, and that various modifications and alterations within the scope of the claims also fall within the technical scope of the present invention.

[0112]

[0095] For example, the processes described herein using flowcharts do not necessarily have to be performed in the order shown in the flowcharts. Some process steps may be performed in parallel. Additional process steps may also be employed, and some process steps may be omitted.

[0113]

[0096] For example, the series of processes performed by the processing device 16 described above may be realized by software, hardware, or a combination of software and hardware. The programs constituting the software are stored in advance in a storage medium provided inside or outside the information processing device.

[0114] [Explanation of symbols]

[0115] [ 0 0 9 7 ]

[0116] 1 〇 Vehicle 1 1 Steering mechanism

Claims

[Document name] Scope of claims

1. A processing device comprising: processing units (16a, 16c) that sequentially generate data groups (G1) including multiple types of data in time series based on images (20) captured by a camera (14) provided on a vehicle (10) that captures images of the surroundings of the vehicle (10); and an estimation unit (16d) that estimates the type of road (31) on which the vehicle (10) is traveling based on the multiple data groups (G1) in time series.

2. The processing device according to claim 1, wherein the data group (G1) includes data on road signs (32).

3. The processing device according to claim 1, wherein the data group (G 1 ) includes data of a moving body (3 3 ).

4. The processing device according to claim 1, wherein the data group (G1) includes data on road markings (34).

5. The processing device according to claim 1, wherein the data group (G1) includes data on the type of the travel path (31).

6. The processing device according to any one of claims 1 to 5, wherein the processing unit (16c) generates the data group (G1) each time the travel distance of the vehicle (10) increases by a reference distance, thereby generating the data group (G1) sequentially in time series.

7. A processing method in which a processing unit (16a, 16c) of a processing device (16) sequentially generates data groups (G1) including multiple types of data in time series based on images (20) captured by a camera (14) provided on a vehicle (10) for capturing images of the surroundings of the vehicle (10), and an estimation unit (16d) of the processing device (16) estimates the type of a road (31) on which the vehicle (10) is traveling based on the multiple time-series data groups (G1).

Images