Road surface condition prediction system and road surface condition prediction method

Abstract

This road surface condition prediction system comprises: a generation unit that generates image data corresponding to desired precipitation amount information and desired position information by using a trained model generated by performing learning processing using learning data in which precipitation amount information and position information are associated with image data including a road surface; and an output unit that outputs the image data generated by the generation unit as a road surface condition.

Description

Road surface condition prediction system and road surface condition prediction method

[0001] The present invention relates to a road surface condition prediction system and a road surface condition prediction method.

[0002] In recent years, vehicles have been equipped with various devices such as sensors and cameras, which are used to control vehicle driving and monitor the surrounding environment. Such information is used not only for vehicle driving control itself but also for providing information to the driver. In addition, vehicles are now connected to servers via networks to acquire various information and provide it to the driver.

[0003] For example, Patent Document 1 discloses a configuration in which an image captured by an on-board image capture unit is displayed so that the user can operate it, and the image is processed based on an instruction and then displayed.

[0004] Japanese Patent Application Publication No. 2023-19828

[0005] Meanwhile, environmental conditions, such as road conditions on which vehicles travel, change constantly depending on the climate, weather, and the vehicles traveling on them. Therefore, there is a demand for acquiring environmental information for any location in real time or at a specified time period. For example, in areas with heavy precipitation, there is a demand for acquiring information on locations where road conditions have deteriorated, making travel difficult, such as unpaved roads, and for using this information to select driving routes.

[0006] In recent years, image generation functions using machine learning have become widespread. These functions use previously acquired training data to learn and generate new images. There is a demand for such image generation functions to be used for vehicle driving assistance and for providing information to drivers.

[0007] The present invention has been devised in view of the above-mentioned problems, and aims to predict and provide road surface conditions at various locations based on precipitation information. However, the present invention is not limited to this purpose. Another object of the present invention is to achieve effects that cannot be obtained by conventional techniques, which are derived from the configurations shown in the below-described embodiments of the invention.

[0008] A road surface condition prediction system according to one embodiment of the present invention has the following configuration: the road surface condition prediction system includes: a generation unit that generates image data corresponding to desired precipitation information and desired location information by using a trained model generated by performing a training process using training data in which precipitation information, location information, and image data including a road surface are associated with each other; and an output unit that outputs the image data generated by the generation unit as road surface conditions.

[0009] A road surface condition prediction system according to another embodiment of the present invention has the following configuration: a storage unit that stores image data including a road surface, a reception unit that receives specification of conditions, an acquisition unit that acquires image data from the storage unit based on the conditions specified by the reception unit, a generation unit that generates new image data corresponding to desired precipitation information and the image data acquired by the acquisition unit using a trained model that generates new image data using precipitation information and image data as input, the trained model being generated by performing a learning process using training data in which precipitation information and image data including the road surface are associated, and an output unit that outputs the new image data generated by the generation unit as road surface conditions.

[0010] A road surface condition prediction method according to another embodiment of the present invention has the following configuration: the road surface condition prediction method includes: a generating step of generating image data corresponding to desired precipitation information and desired location information, using a trained model generated by performing a learning process using training data in which precipitation information, location information, and image data including a road surface are associated with each other; and an output step of outputting the image data generated in the generating step as road surface conditions.

[0011] A road surface condition prediction method according to another embodiment of the present invention has the following configuration: a receiving step of receiving specification of conditions, an acquisition step of acquiring image data from a storage unit that stores image data including a road surface based on the conditions specified in the receiving step, a generation step of generating new image data corresponding to desired precipitation information and the image data acquired by the acquisition unit using a trained model that generates new image data using precipitation information and image data as input, the trained model being generated by performing a learning process using training data in which precipitation information and image data including the road surface are associated, and an output step of outputting the new image data generated in the generation step as road surface conditions.

[0012] According to the present invention, it is possible to predict and provide road surface conditions at various locations based on precipitation information.

[0013] A diagram showing an example of a system configuration according to an embodiment of the present invention. A diagram showing an example of a functional configuration of a vehicle according to an embodiment of the present invention. A diagram showing an example of a configuration of a UI screen according to an embodiment of the present invention. A diagram showing an example of a configuration of a UI screen according to an embodiment of the present invention. A flowchart of processing according to an embodiment of the present invention. A schematic diagram for explaining the concept of a learning process according to an embodiment of the present invention. A schematic diagram for explaining the concept of a learning process according to an embodiment of the present invention. A schematic diagram for explaining the concept of a learning process according to an embodiment of the present invention.

[0014] A road surface condition prediction system and a road surface condition prediction method will be described as embodiments with reference to the drawings. The embodiments described below are merely examples, and are not intended to exclude various modifications or application of techniques not explicitly stated in the following embodiments. Each configuration of the present embodiment can be implemented with various modifications within the scope of the spirit thereof. Furthermore, they can be selected or combined as needed. Furthermore, in each drawing, the same components are assigned the same reference numerals to indicate correspondence.

[0015] 1 is a diagram showing an example of the overall configuration of a system according to this embodiment. The system includes a vehicle 100, a collection server 200, and a network 250. For simplicity of explanation, the system will be described using an example of one vehicle and one collection server 200, but multiple vehicles and multiple collection servers 200 may be provided, each configured to be able to communicate with the network 250.

[0016] Vehicle 100 includes a processing unit 110, a storage unit 120, a drive control unit 130, a driving and drive system 140, a camera 150, sensors 160, a communication unit 170, and a UI (User Interface) unit 180. A known method may be used to control the driving of vehicle 100. Furthermore, the system of vehicle 100 is not particularly limited, and vehicle 100 may be any of front-wheel drive, rear-wheel drive, and four-wheel drive.

[0017] The processing unit 110 is a processing unit that is responsible for overall control of the vehicle 100. The processing unit 110 may be configured with a processing device such as a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit). The processing unit 110 may also be configured with an ECU (Electronic Control Unit). The ECU may be configured as, for example, an LSI (Large-Scale Integration) device that integrates a microprocessor, or an electronic control unit configured as an embedded electronic device. The processing unit 110 performs various processes described below, for example, by reading and executing a program (not shown) stored in the storage unit 120.

[0018] The storage unit 120 is configured with a volatile / non-volatile storage device, and may be configured by combining, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), etc. The storage unit 120 provides and stores various data and programs in response to requests from the processing unit 110.

[0019] Drive control unit 130 controls driving drive system 140 based on commands from processing unit 110 and driver operations. Driving drive system 140 performs operations for driving vehicle 100 based on instructions from drive control unit 130. Note that known configurations may be used for the control related to the driving of vehicle 100 and the structure of the drive system, and detailed description thereof will be omitted here.

[0020] The camera 150 is an imaging device for acquiring images of the surroundings of the vehicle 100. The imaging settings and imaging range of the camera 150 are not particularly limited and may be set arbitrarily. For example, the imaging range may include the front, rear, and sides of the vehicle 100.

[0021] The sensors 160 include various sensors used for driving control, status detection, and periphery monitoring of the vehicle 100. For example, the sensors may include a gyro sensor, a rotation sensor, a brake sensor, an accelerator sensor, a steering angle sensor, a vehicle speed sensor, a position sensor, a yaw rate sensor, etc. These sensors may be provided in common with those used for driving control of the vehicle 100.

[0022] The communication unit 170 is an interface for communicating with an external network. The communication unit 170 may be compatible with, for example, short-range wireless communication or Internet communication. The UI unit 180 is a user interface for passengers of the vehicle 100. The UI unit 180 may be configured to include, for example, a speaker (not shown) for outputting audio, a display (not shown) for displaying a screen, and an operation unit (not shown) for receiving operations from the passengers.

[0023] The collection server 200 is a server device configured to be able to communicate with a plurality of vehicles 100 via a network 250. The collection server 200 may be configured as a cloud-based or on-premise server. The collection server 200 includes a processing unit 210, a storage unit 220, a communication unit 230, and an interface unit 240.

[0024] The processing unit 210 executes various functions described below by reading and executing various programs and data stored in the storage unit 220. The processing unit 210 may be configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), or an FPGA (Field Programmable Gate Array). In particular, when performing the learning process described below, it is preferable that the CPU and the GPU work together to perform the process.

[0025] The storage unit 220 is a storage area for storing and holding various data, and may be composed of, for example, a non-volatile storage area such as a ROM or HDD, or a volatile storage area such as a RAM, etc. The storage unit 220 may, for example, have separate storage areas for use by the CPU and GPU.

[0026] The communication unit 230 communicates with an external device (in this example, the vehicle 100) via a wired or wireless network 250, transmitting and receiving various data and signals. The communication method used by the communication unit 230 is not particularly limited, and may be compatible with a plurality of communication methods. For example, a wide area network (WAN), a local area network (LAN), power line communication, short-range wireless communication (e.g., Bluetooth (registered trademark)), etc. may be used.

[0027] The interface unit 240 may be a user interface for a user such as an administrator, or may be an interface for transmitting and receiving data to and from an external device (not shown).

[0028] [Functional Configuration] Fig. 2 is a diagram for explaining an example of a functional configuration relating to a data acquisition method for the vehicle 100 according to this embodiment. This function is provided by the processing unit 110 of the vehicle 100 shown in Fig. 1. The processing unit 110 is configured to include a determination processing unit 201, an image extraction unit 202, and a recording control unit 203.

[0029] In this embodiment, the processing unit 110 can acquire peripheral images using the camera 150 at predetermined timing. Furthermore, the processing unit 110 can derive and use various information based on various signals detected by the sensors 160. In the data acquisition method according to this embodiment, gyro values ​​(vibration information), the rotation values ​​of each tire, the amount of braking operation, the amount of steering operation, and the vehicle speed are used. For convenience, these pieces of information are referred to as "driving information." The driving information may be acquired directly from the sensors 160, or may be derived based on signals acquired by the sensors 160. Furthermore, the types of driving information listed here are merely examples, and other information may also be used as input.

[0030] The determination processing unit 201 uses the input gyro value to derive road surface information during travel, which in this case is a roughness value. The roughness value may be, for example, the well-known IRI (International Roughness Index). The roughness value may be defined by classifying the roughness into a plurality of predefined roughness levels. The determination processing unit 201 also derives a slip value for the vehicle 100 based on the comparison result of the rotation values ​​of each tire. The road surface roughness value and the slip value may be derived using well-known methods, and detailed description thereof will be omitted here.

[0031] Next, the determination processing unit 201 performs a determination process using threshold values ​​for the brake operation amount, the steering operation amount (steering angle), and the vehicle speed. Here, an example is shown in which the determination is made in the order of the brake operation amount, the steering operation amount, and the vehicle speed, but the order is not particularly limited. In addition, it is sufficient to use at least one of these, and any one of them may be omitted. The threshold values ​​Th for each are ｂ , Th ｈ , Th ｓ is defined in advance and stored in the storage unit 120 so as to be able to be referenced.

[0032] If the input brake operation amount, steering operation amount, and vehicle speed are all smaller than the thresholds, the determination processing unit 201 compares the already derived unevenness value and slip value with predetermined set values. At this time, the unevenness value and slip value may be average values ​​of a predetermined number of sampling results. In this embodiment, the predetermined set values ​​for the unevenness value and slip value are defined in advance in a table or the like in accordance with precipitation information. Different values ​​are used depending on the amount of precipitation at the location where the vehicle 100 is located. If either the unevenness value or the slip value exceeds the predetermined set value, the determination processing unit 201 generates and outputs a trigger signal. The determination processing unit 201 also notifies the recording control unit 203 of various information used in the determination, such as the unevenness value and slip value.

[0033] When a trigger signal is generated, the image extraction unit 202 identifies image data starting from that point in time. For example, while the vehicle 100 is traveling, surrounding images are continuously captured by the camera 150 and stored in the storage unit 120. The image extraction unit 202 then acquires image data from the time when the trigger signal was generated, or from a certain period going back from that time, or from a certain period of time after that time. The image data acquired here may be a still image or a moving image.

[0034] The recording control unit 203 associates the image data extracted by the image extraction unit 202 in response to the generation of a trigger signal with information received from the determination processing unit 201 (hereinafter referred to as "information tag") and stores the image data as a file in the storage unit 120. At this time, the recording control unit 203 may further acquire information such as date and time information (timestamp) at the time the trigger signal was generated, location information of the vehicle 100, precipitation amount information, and vehicle type, and assign it as the information tag 204. Note that this information may be information previously stored in the processing unit 110 of the vehicle 100, or may be information acquired via the sensors 160, the communication unit 170, etc. The vehicle type may be defined based on the size, classification, weight, dimensions, etc. of the vehicle.

[0035] Thereafter, the files stored in the storage unit 120 are transmitted to the collection server 200 via the network 250 at an arbitrary timing. The arbitrary timing here may be set at a predetermined time interval, or may be transmitted when a predetermined event occurs. For example, the arbitrary timing may be transmitted when a certain number of files have been recorded in the storage unit 120 of the vehicle 100. Alternatively, the collection server 200 may issue a query to the vehicle 100, and the files may be provided to the collection server 200 in response to the query.

[0036] [Usage Example 1] A usage example of the files collected as described above will be described using Figures 3 and 4. Here, an example of a function displayed on the UI unit 180 mounted on the vehicle 100 will be described. As described above, the collection server 200 according to this embodiment collects multiple files from multiple vehicles 100 traveling at various locations and is ready for use.

[0037] FIG. 3 is a diagram showing an example of the configuration of a UI screen 300 of a navigation system that presents a driving route and road conditions on a map. An icon 301 indicates the current position of the vehicle. An icon 302 indicates the location of the destination. In this case, road information 305 is shown on the map. The road information is defined based on the above-mentioned file. In this example, the road surface condition is determined based on the unevenness value and slippage value included in the information tag associated with the file, and the driving ease is classified into four categories. In FIG. 3, four types of lines are used: black lines, coarsely hatched lines, dotted lines, and finely hatched lines. Of these, the finely hatched lines indicate the lowest driving ease, i.e., the locations where driving by the vehicle is estimated to be the most difficult, and an icon 303 and a current image 304 of the location are displayed.

[0038] A recommended route 306 is presented based on the information regarding the ease of driving. Such ease of driving may vary depending on the vehicle's performance, type, etc., and may be derived taking such information into consideration. In the above example, the ease of driving is divided into four levels, but two or more levels are sufficient. In the example of FIG. 3, the image 304 shows only the location with the lowest ease of driving, but the system may be configured to display current images for other locations as well.

[0039] FIG. 4 shows an example of the configuration of a UI screen 400 in a navigation system that compares and displays a current image 401 and a past image 402. The current image 401 corresponds to an image captured by the camera 150 at the location where the vehicle 100 is located. The past image 402 is displayed by searching for a corresponding file from files stored in the collection server 200 based on the current location of the vehicle 100, the amount of precipitation, the time of day, the vehicle type, and the like, and providing the file to the vehicle 100. The search uses the information tag 204 associated with the file. The UI screen 400 may be displayed based on, for example, an instruction from the driver of the vehicle 100, or based on the driving state of the vehicle 100. The driving state may be, for example, a case where a road surface roughness value equal to or greater than a predetermined threshold is detected, or a case where the amount of precipitation at the driving position exceeds a predetermined threshold.

[0040] When providing information to the vehicle 100 using the collected files, the collection server 200 may generate the information to be provided after further collecting various information from other servers, etc. For example, precipitation information, weather information, road traffic information, etc. for the location where the vehicle 100 is scheduled to travel may be collected and combined with the information to be provided.

[0041] [Processing Flow] Figure 5 is a flowchart of the data acquisition process according to this embodiment. The process shown in Figure 5 is realized, for example, by the processing unit 110 of the vehicle 100 reading and executing various data and programs stored in the storage unit 120. This processing flow may be started when the vehicle 100 starts traveling, or may be started based on a user instruction. For ease of explanation, the processing unit 110 will be collectively described as the processing entity.

[0042] In S501, the processing unit 110 acquires the current location and precipitation information for the current location. The current location can be acquired using a position sensor, and the precipitation information can be acquired by inquiring to a predetermined server using the communication unit 170.

[0043] In S502, the processing unit 110 sets the setting values ​​for the subsequent road surface condition information and slippage information based on the precipitation amount information acquired in S501. In this embodiment, the setting values ​​may be set by referring to a predefined table.

[0044] In S503, the processing unit 110 acquires an image of the surroundings of the vehicle 100 via the camera 150 and stores the image in the storage unit 120. The image may be acquired as a moving image or a still image. It is assumed that the surrounding image is continuously captured by the camera 150.

[0045] In S504, processing unit 110 acquires driving information, which includes gyro values ​​(vibration information), rotation values ​​of each tire, brake operation amount, steering operation amount, and vehicle speed.

[0046] In S505, the processing unit 110 derives road surface condition information using the driving information acquired in S504. The road surface condition information in this embodiment corresponds to a road surface unevenness value derived based on a gyro value.

[0047] In S506, processing unit 110 uses the driving information acquired in S504 to derive slippage information of vehicle 100. The slippage information corresponds to a slippage value of vehicle 100, which is derived based on the difference in rotation speed of each tire provided on vehicle 100.

[0048] In S507, the processing unit 110 determines whether the brake operation amount acquired in S504 is equal to or smaller than the threshold value Th ｂ It is determined whether the brake operation amount is equal to or greater than the threshold value Th. ｂ If the brake operation amount is equal to or greater than the threshold value Th ｂ If it is smaller (NO in S507), the process of processing unit 110 proceeds to S508.

[0049] In S508, the processing unit 110 calculates the steering wheel operation amount acquired in S504 based on the threshold Th ｈ It is determined whether the steering wheel operation amount is equal to or greater than the threshold value Th. ｈ If the steering wheel operation amount is equal to or greater than the threshold value Th ｈ If it is smaller (NO in S508), the process of processing unit 110 proceeds to S509.

[0050] In S509, the processing unit 110 determines whether the speed acquired in S504 is equal to or greater than the threshold value Th ｓ It is determined whether the speed is equal to or greater than the threshold value Th ｓ If the speed is equal to or greater than the threshold value Th ｓ If it is smaller (NO at S509), processing unit 110 proceeds to S510.

[0051] In S510, the processing unit 110 compares the road surface condition information derived in S505 and the slippage information derived in S506 with the predetermined set value set in S502. The road surface condition information (unevenness value) and the slippage information (slippage value) here may be average values ​​of a predetermined number of samples. In this embodiment, if at least one of the unevenness value and the slippage value exceeds the predetermined set value, the processing unit 110 generates a trigger signal.

[0052] In S511, the processing unit 110 determines whether or not both the unevenness value and the slippage value are within the set values ​​as a result of the comparison process in S510. That is, it determines whether or not a trigger signal has been generated. If both the unevenness value and the slippage value are within the predetermined set values ​​(YES in S511), the processing by the processing unit 110 proceeds to S513. On the other hand, if at least one of them exceeds the set value (NO in S511), the processing by the processing unit 110 proceeds to S512.

[0053] In S512, the processing unit 110 executes file recording processing based on the comparison result. Specifically, if a trigger signal has been generated, image data corresponding to the timing of generation of the trigger signal is extracted from the captured image acquired in S503. Furthermore, the processing unit 110 acquires an information tag corresponding to the timing of generation of the trigger signal. Then, the processing unit 110 associates the extracted image data with the information tag and records them as a file in the storage unit 120. Then, the processing of the processing unit 110 proceeds to S513.

[0054] In S513, processing unit 110 determines whether or not to end the data acquisition process. The data acquisition process may be ended, for example, based on a user instruction or when vehicle 100 stops traveling. If the data acquisition process is to be ended (YES in S513), this processing flow ends. On the other hand, if the data acquisition process is not to be ended (NO in S513), the process returns to S501 and repeats the process.

[0055] Thereafter, the vehicle 100 transmits the recorded file to the collection server 200 at any timing, thereby providing the file.

[0056] As described above, this embodiment makes it possible to appropriately extract target information from information acquired in a vehicle and acquire data. Furthermore, it makes it possible to share the acquired data among multiple devices.

[0057] [Use Example 2] In the above, an example has been described in which a shared file is used as part of the route navigation function of the IVI device installed in the vehicle 100. Another use example of the shared file in this embodiment will be described. Here, an example is shown in which collected files are used in a learning process.

[0058] Note that "learning" or "machine learning" here refers to generating a "trained model" by performing learning using training data and an arbitrary learning algorithm. A trained model is updated as needed as learning progresses using multiple pieces of training data, and its output changes even for the same input. Therefore, a trained model is not limited to a specific state at any point in time. Here, a model used in learning is referred to as a "trained model," and a learning model that has undergone a certain level of learning is referred to as a "trained model."

[0059] Any algorithm may be used as the learning algorithm depending on the function to be provided. For example, a convolutional neural network (CNN) may be used. The convolutional neural network includes multiple convolutional layers, multiple pooling layers, and multiple fully connected layers. The configuration of the convolutional neural network is not particularly limited, and the number of layers and the configuration may be set arbitrarily. Furthermore, the learning algorithm is not limited to a CNN, and other learning algorithms such as an autoencoder may be used.

[0060] Specific examples of "training data" will be described later, but the configuration may vary depending on the learning algorithm used. Training data may include training data used for the training itself, verification data used to verify a trained model, and test data used to test a trained model. In the following description, the term "training data" will be used to refer collectively to data related to training, and the term "training data" will be used to refer to data used when performing the training itself. It is not intended to clearly classify training data, verification data, and test data included in training data; for example, depending on the methods of training, verification, and testing, all training data may also be training data.

[0061] Generally, a learning process is broadly divided into a learning phase in which a learning process is performed using a predetermined learning algorithm, and an estimation phase in which estimation is performed using a trained model generated by the learning process. The learning phase and the estimation phase may be implemented in different devices or may be implemented in the same device. The learning process is generally expected to impose a high processing load. Therefore, it is preferable that the device corresponding to the learning phase and the device corresponding to the estimation phase are configured as separate devices, and the device performing the estimation phase appropriately acquires and uses the trained model generated.

[0062] For example, a learning server (not shown) may be configured to perform the learning process in the learning phase. The trained model obtained as a result of the learning process may then be provided to a service server (not shown) that executes the estimation phase. The service server may then use the trained model to provide a desired function to the vehicle 100. Alternatively, the vehicle 100 may obtain the trained model from the learning server and provide a predetermined function by performing the operation of the estimation phase.

[0063] A usage example using the learning process will be described with reference to FIGS. 6A and 6B. Here, an example of image generation will be described. FIG. 6A is a diagram for explaining the concept of the learning phase. FIG. 6B is a diagram for explaining the concept of the estimation phase. In this example, the collection server 200 is described as functioning as a learning server and performing the learning process. The vehicle 100 is described as acquiring the trained model generated by the learning server, executing the estimation phase, and providing a predetermined function (past images).

[0064] In this embodiment, the above-described file is used as the learning data 600. Therefore, the learning data 600 is associated with image data 601 and an information tag 602. Note that instead of using the above-described file as is, the learning data 600 may be used after applying predetermined preprocessing to each data in the file. When the collection server 200 and the learning server are configured separately, the learning data may be shared by providing the files collected by the collection server 200 to the learning server.

[0065] As shown in FIG. 6A , in the learning phase, the learning server performs a learning process 603 using learning data 600 and a predetermined learning algorithm to generate a trained model. In this example, an information tag 602 included in the learning data 600 is input to the learning model, and a predicted image 604 is output. Then, a comparison process 605 is performed between image data 601 corresponding to the input information tag 602 and the output predicted image 604, and the parameters of the learning model are adjusted so that these are similar to each other, thereby performing the learning process. This process is repeated to generate a trained model.

[0066] 6B , when performing image generation processing 611 in the estimation phase, the vehicle 100 acquires an information tag 610 and inputs it into the trained model. Information corresponding to the information tag 610 may be acquired from the sensors 160 while traveling as described above, or may be specified by the user. Then, as a result of the image generation processing 611, a predicted image 612 corresponding to the information tag 610 is obtained.

[0067] The predicted image 612 obtained as described above is an image predicted based on the location, precipitation amount, vehicle type, date and time, etc., indicated by the information tag 610. By presenting such a predicted image 612 to the user, the user can understand the estimated road conditions under the conditions indicated by the information tag 610 without actually driving to the location, and can use this as a reference when operating the vehicle or determining a route. In addition, the user can understand the predicted situation at the destination at the predicted arrival time at the destination.

[0068] Another use example using the learning process will be described using FIGS. 7A and 7B. Here, an example of image analysis will be described. FIG. 7A is a diagram for explaining the concept of the learning phase. FIG. 7B is a diagram for explaining the concept of the estimation phase. Here, the description will be given assuming that the collection server 200 functions as a learning server and performs the learning process. The description will be given assuming that the vehicle 100 acquires the trained model generated by the learning server, executes the estimation phase, and provides a predetermined function (image analysis).

[0069] In this embodiment, the above-described file is used as the training data 700. Therefore, the training data 700 is associated with an information tag 701 and image data 702. Note that instead of using the above-described file as is, the training data 700 may be used after applying predetermined preprocessing to each data item in the file.

[0070] As shown in FIG. 7A , in the learning phase, the learning server performs a learning process 703 using learning data 700 and a predetermined learning algorithm to generate a trained model. In this example, image data 702 included in the learning data 700 is input to the learning model, and analysis information 704 is output. Then, a comparison process 705 is performed between an information tag 701 corresponding to the input image data 702 and the output analysis information 704, and the parameters of the learning model are adjusted so that they are similar to each other, thereby performing the learning process. This process is repeated to generate a trained model.

[0071] As shown in FIG. 7B , when performing image analysis processing 711 in the estimation phase, vehicle 100 acquires image data 710 and inputs it into the trained model. Information corresponding to image data 710 may be acquired from camera 150 while traveling, as described above. Alternatively, a user may specify a predetermined condition (e.g., the value of an item included in an information tag) to acquire past image data that meets the condition from a database of already recorded files (not shown). Then, analysis information 712 of image data 710 is obtained as a result of image analysis processing 711.

[0072] The analysis information 712 obtained as described above is road surface condition information (unevenness value) and slippage information (slippage value) predicted based on the image data 710. By presenting such analysis information 712 to the user, the user can understand the road surface condition estimated based on the current image, and can use it as a reference when operating the vehicle or determining a route. At this time, the estimated road surface condition information may be plotted on a map as shown in FIG. 3, or a warning message or the like may be displayed to alert the user while driving.

[0073] Some vehicles are capable of running in various driving modes. Examples of driving modes include, for example, a two-wheel drive mode, a four-wheel drive mode, and other driving modes that switch between different driving forces. Other examples include a normal mode for paved roads and an off-road mode for unpaved roads, which are modes tailored to road surface conditions. The driving modes are not limited to these and may include other driving modes. Furthermore, the driving modes are not limited to driving modes, and may also include correction or modification of control parameters used in a driving mode. Vehicles capable of switching between multiple driving modes may be configured to adjust the driving mode and driving parameters based on estimated road surface condition information. For example, driving modes may be predefined in association with road surface condition information. Such switching may be performed automatically by a control unit of vehicle 100, or may be configured to present the target driving mode to the driver and then accept a switching instruction.

[0074] 6A and 7A show examples of configurations in which one of image data and information tags is used as input and the other is used for comparison processing for the training data, but this is not limited to this. For example, a configuration may be adopted in which the image data and part of the information tags are used as input to a training model, and comparison processing is performed using the output of the training model and the remaining training tags. In a trained model generated in this manner, the output may be image data or analytical information. Note that the combination of input and output is arbitrary, and the input and output of the trained model in the estimation phase are also specified corresponding to the relationship between the input and output of the training model during the training process.

[0075] For example, the model may be trained to generate and output new image data using image data and arbitrary information (e.g., precipitation information, location information, and time information) as input. Furthermore, the input to the trained model obtained as a result of this training process may be image data searched based on predetermined conditions from a database of past files and information specified by the user. The input image data and the generated image data (predicted image) may then be displayed in association with each other.

[0076] As described above, this embodiment allows information that can be shared among multiple vehicles to be collected from the vehicles. This information can then be used to further improve vehicle functionality and user convenience.

[0077] <Other Embodiments> Furthermore, in the present invention, a program or application for realizing the functions of one or more of the above-described embodiments can be supplied to a system or device using a network or a storage medium, etc., and one or more processors in a computer of the system or device can read and execute the program.

[0078] As such, the present invention is not limited to the above-described embodiments, and the present invention also contemplates the mutual combination of the various components of the embodiments, as well as modifications and applications by those skilled in the art based on the description in the specification and well-known techniques, and these modifications and applications are included in the scope of protection sought.

[0079] As described above, this specification discloses the following: (1) A road surface condition prediction system having: a generation unit (e.g., 110) that generates image data corresponding to desired precipitation information and desired location information using a trained model generated by performing a training process using training data in which precipitation information, location information, and image data including road surfaces are associated; and an output unit (e.g., 180) that outputs the image data generated by the generation unit as road surface conditions. This configuration makes it possible to predict and provide road surface conditions at various points according to precipitation information.

[0080] (2) A road surface condition prediction system having: a storage unit (e.g., 220) that stores image data including road surfaces; a reception unit (e.g., 230, 240) that receives designation of conditions; an acquisition unit (e.g., 210) that acquires image data from the storage unit based on the conditions specified by the reception unit; a generation unit (e.g., 210) that generates new image data corresponding to desired precipitation information and the image data acquired by the acquisition unit using a trained model that generates new image data using precipitation information and image data as input, the trained model being generated by performing a learning process using training data in which precipitation information and image data including the road surface are associated; and an output unit (e.g., 210, 230, 240) that outputs the new image data generated by the generation unit as road surface conditions. With this configuration, it is possible to predict and provide road surface conditions at various points according to precipitation information.

[0081] (3) The road surface condition prediction system according to (1) or (2), wherein the learning data further includes time information, and the learning process further uses the time information. With this configuration, it is possible to generate image data that predicts road surface conditions based on the time information.

[0082] (4) The road surface condition prediction system according to (1), wherein the output unit displays the image data generated by the generation unit in association with the position on the map indicated by the desired position information. With this configuration, a predicted image of the desired position is generated and mapped on the map, thereby improving visibility and convenience for the user.

[0083] (5) In the road surface condition prediction system according to (2), the output unit displays the image data acquired by the acquisition unit and the new image data generated by the generation unit in association with each other. With this configuration, it is possible to improve user convenience by displaying the past image data and the predicted image in comparison with each other.

[0084] (6) In a vehicle capable of driving by switching between a plurality of driving modes, the road surface condition output by the output unit is used to switch between the plurality of driving modes according to the road surface condition. With this configuration, it is possible to switch the vehicle's driving mode based on the predicted road surface condition, thereby improving convenience for the driver in operating the vehicle.

[0085] (7) A road surface condition prediction method comprising: a generating step of generating image data corresponding to desired precipitation information and desired location information using a trained model generated by performing a learning process using training data in which precipitation information, location information, and image data including a road surface are associated; and an output step of outputting the image data generated in the generating step as road surface conditions. With this configuration, it is possible to predict and provide the road surface conditions at each point according to the precipitation information.

[0086] (8) A road surface condition prediction method comprising: a receiving step of receiving designation of conditions; an acquisition step of acquiring image data from a storage unit that stores image data including road surfaces based on the conditions designated in the receiving step; a generation step of generating new image data corresponding to desired precipitation information and the image data acquired in the acquisition step using a trained model that generates new image data using precipitation information and image data as input, the trained model being generated by performing a learning process using training data in which precipitation information and image data including the road surface are associated; and an output step of outputting the new image data generated in the generation step as road surface conditions. With this configuration, it is possible to predict and provide the road surface conditions at each point according to precipitation information.

[0087] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0088] The present invention is applicable to the manufacturing industry of vehicle or in-vehicle IVI (In-Vehicle Infotainment) devices, as well as to the manufacturing industry of control devices mounted on vehicles.

[0089] REFERENCE SIGNS LIST 100 Vehicle 110 Processing section 120 Storage section 130 Drive control section 140 Traveling drive system 150 Camera 160 Sensors 170 Communication section 180 UI (User Interface) section 200 Collection server 210 Processing section 220 Storage section 230 Communication section 240 Interface section 250 Network

Claims

1. A road surface condition prediction system having: a generation unit that generates image data corresponding to desired precipitation information and desired location information using a trained model generated by performing a learning process using training data in which precipitation information, location information, and image data including road surfaces are associated; and an output unit that outputs the image data generated by the generation unit as road surface conditions.

2. A road surface condition prediction system having: a memory unit that stores image data including road surfaces; a reception unit that receives specified conditions; an acquisition unit that acquires image data from the memory unit based on the conditions specified by the reception unit; a generation unit that generates new image data corresponding to desired precipitation information and the image data acquired by the acquisition unit using a trained model that generates new image data using precipitation information and image data as input, the trained model being generated by performing a learning process using training data in which precipitation information and image data including the road surface are associated; and an output unit that outputs the new image data generated by the generation unit as road surface conditions.

3. A road surface condition prediction system according to claim 1 or 2, wherein the learning data further includes time information, and the learning process further uses the time information.

4. A road surface condition prediction system as described in claim 1, wherein the output unit displays the image data generated by the generation unit in association with the position on the map indicated by the desired position information.

5. A road surface condition prediction system according to claim 2, wherein the output unit displays the image data acquired by the acquisition unit in association with the new image data generated by the generation unit.

6. A road surface condition prediction system as described in claim 1 or 2, in a vehicle capable of switching between multiple driving modes, wherein the road surface conditions output by the output unit are used to switch between the multiple driving modes in accordance with the road surface conditions.

7. A road surface condition prediction method comprising: a generation step of generating image data corresponding to desired precipitation information and desired location information using a trained model generated by performing a learning process using training data in which precipitation information, location information, and image data including the road surface are associated; and an output step of outputting the image data generated in the generation step as road surface conditions.

8. A road surface condition prediction method comprising: a receiving step for receiving specified conditions; an acquisition step for acquiring image data from a memory unit that stores image data including road surfaces based on the conditions specified in the receiving step; a generation step for generating new image data corresponding to desired precipitation information and the image data acquired in the acquisition step using a trained model that generates new image data using precipitation information and image data as input, the trained model being generated by performing a learning process using training data in which precipitation information and image data including the road surface are associated; and an output step for outputting the new image data generated in the generation step as road surface conditions.

Images