Road image presentation device and image generation model

Abstract

To provide a road image presentation device, a road image presentation method, a road image presentation program, a storage medium, and an image generation model which can present an appropriate road image according to environmental conditions such as weather.SOLUTION: According to input based on an original image as an image obtained by imaging an object road in a non-snow covered state, weather information including at least the amount of snow cover in a periphery of the object road, and traffic volume information indicating the volume of traffic of the object road, an image generation model generates a snow cover image as an image of the object road in a snow covered state according to the amount of the snow cover and the volume of the traffic.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a road image presentation device and an image generation model. [Background technology]

[0002] There is known a technology for presenting an image captured by an in-vehicle camera to a user. For example, Patent Document 1 discloses a distribution system including a distribution server that distributes image data in response to a request from a user. In the distribution system, image data captured at various times and in various environments and corresponding shooting location information and shooting environment information are stored in a storage server, and when a distribution request for image data is received, image data corresponding to the shooting location conditions and shooting environment conditions specified by the distribution request is searched for and distributed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-83034 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described distribution system, one of the problems is that if image data corresponding to the specified shooting position conditions and shooting environment conditions is not stored in the storage server, it is not possible to distribute images in response to a user request.

[0005] The present invention has been made in consideration of the above-mentioned points, and one of its objectives is to provide a road image presentation device and an image generation model that are capable of presenting appropriate road images according to environmental conditions such as weather. [Means for solving the problem]

[0006] The invention described in claim 1 is an image generation model that generates a snow image, which is an image of a target road with snow corresponding to the amount of snow and the traffic volume, in response to inputs based on an original image, which is an image of a target road in a state where there is no snow, weather information including at least the amount of snow around the target road, and traffic volume information indicating the traffic volume on the target road.

[0007] The invention described in claim 5 is a road image presentation device characterized by including: a reception unit for receiving a road designation; an environmental condition acquisition unit for acquiring environmental conditions including at least weather conditions including the amount of snowfall and traffic conditions indicating traffic volume; an image acquisition unit for acquiring as a presentation image a generated image generated by inputting an image of a road designated by the road designation in a snow-free state and the environmental conditions acquired by the environmental condition acquisition unit into an image generation model that generates a snow-covered image, which is an image of the target road in a state where there is snowfall corresponding to the amount of snowfall and the traffic volume, in response to input based on an original image, which is an image of a target road photographed in a snow-free state, weather information including at least the amount of snowfall around the target road, and traffic volume information indicating the traffic volume on the target road; and a presentation control unit that controls the presentation of the presentation image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing an overview of a road image presentation system according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram showing a configuration of a front seat portion of an automobile according to an embodiment. [Diagram 3] 1 is a block diagram showing an example of a configuration of an in-vehicle device according to an embodiment; [Figure 4] FIG. 4 is a diagram illustrating an example of information transmitted from the in-vehicle device to the server according to the embodiment. [Diagram 5] FIG. 2 is a block diagram showing an example of a configuration of a server device according to an embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of an image generation model according to an embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of an image generation model according to an embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a method for constructing an image generation model according to an embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a method for constructing an image generation model according to an embodiment. [Figure 10] 4 is a flowchart showing an example of a routine executed by the in-vehicle device according to the embodiment. [Figure 11] 5 is a flowchart illustrating an example of a routine executed by the server device according to the embodiment. [Figure 12] FIG. 4 is a diagram showing an example of a road image presented in the embodiment. [Figure 13] FIG. 4 is a diagram showing an example of a road image presented in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description and accompanying drawings, substantially the same or equivalent parts are designated by the same reference numerals. EXAMPLES

[0010] The configuration of a road image presentation system 100 including a road image presentation device according to an embodiment will be described with reference to the accompanying drawings.

[0011] Fig. 1 shows an outline of the configuration of a road image presentation system 100. As shown in Fig. 1, the road image presentation system 100 includes an in-vehicle device 10 as a road image presentation device, and a server 40. Note that Fig. 1 shows a case where the in-vehicle device 10 is mounted on an automobile M.

[0012] The in-vehicle device 10 and the server 40 can transmit and receive data to and from each other via a network NW using a communication protocol such as TCP / IP or UDP / IP. The network NW can be constructed, for example, by a mobile communication network, wireless communication such as Wi-Fi (registered trademark), and Internet communication including wired communication.

[0013] The in-vehicle device 10 can perform control to request the server 40 to provide road images, which are images of roads specified by the user, such as roads near a destination, and to obtain the images from the server 40 and present them to the user. This request may include environmental information, such as information on the weather, time, etc.

[0014] The server 40 stores past road images, which are images captured for each road, and transmits road images to the vehicle-mounted device 10 in response to a request from the vehicle-mounted device 10. Specifically, for example, information indicating a road specified by the user is sent as a request from the vehicle-mounted device 10, and the server 40 transmits a road image of the road to the vehicle-mounted device 10.

[0015] In addition, if the server 40 holds past road images for a road specified in response to a request from the vehicle-mounted device 10 that match the environmental conditions such as weather and time included in the request, the server 40 transmits the past road images to the vehicle-mounted device 10.

[0016] If the server 40 does not have any past road images for a road specified in response to a request from the vehicle-mounted device 10 that match the environmental conditions such as weather and time included in the request, the server 40 generates a road image in response to the request using the road image of the specified road and the environmental conditions included in the request and transmits the generated road image to the vehicle-mounted device 10.

[0017] In the following, in this embodiment, the in-vehicle device 10 is a car navigation device. In addition, in this embodiment, the in-vehicle device 10 is a terminal device of a so-called cloud-type car navigation device that receives a destination to which the user wants guidance from the user, transmits the destination to the server 40, and the server 40 generates a route to the destination.

[0018] Fig. 2 is a perspective view showing the vicinity of the front seat of the automobile M equipped with the in-vehicle device 10. Fig. 2 shows, as an example of installation, a case in which the in-vehicle device 10 is installed inside a dashboard DB of the front seat of the automobile M.

[0019] The touch panel 11 is, for example, a touch panel monitor that combines a display such as a liquid crystal display capable of displaying images with a touch pad. The touch panel 11 is disposed, for example, in a center console of the dashboard DB. The touch panel 11 may be disposed in a location that is visible to the driver and within the driver's reach. For example, the touch panel 11 may be attached on the dashboard DB.

[0020] The touch panel 11 can perform screen display based on the control of the in-vehicle device 10. For example, car navigation guidance is displayed on the touch panel 11. For example, a road image of a road designated by the user is displayed on the touch panel 11. For example, a road image according to environmental conditions such as the current weather on a road near a destination designated by the user or the weather at the time of expected arrival is displayed on the touch panel 11.

[0021] Furthermore, the touch panel 11 can transmit a signal representing an input operation to the touch panel 11 received from a user to the in-vehicle device 10. For example, operations related to a car navigation function, such as setting a destination, can be performed via the touch panel 11. Furthermore, operations related to the display of road images, such as designating a road when displaying a road image, can be performed via the touch panel 11.

[0022] The speaker 13 is provided, for example, on the interior side of an A-pillar AP. The speaker 13 is capable of emitting sounds such as music and voices under the control of the in-vehicle device 10. For example, the speaker 13 emits voice guidance for car navigation.

[0023] The microphone 15 is a microphone device that receives sounds inside the vehicle, and is disposed, for example, on the dashboard DB. The microphone 15 may be disposed anywhere, such as near the rearview mirror RM, as long as it can receive sounds inside the vehicle. For example, an operation of a car navigation system or an operation for displaying a road image may be performed by voice via the microphone 15.

[0024] The GPS receiver 17 is a device that receives signals (GPS signals) from GPS (Global Positioning System) satellites. The GPS receiver 17 is arranged, for example, on a dashboard DB. The GPS receiver 17 may be arranged anywhere as long as it can receive GPS signals. The GPS receiver 17 is capable of transmitting the received GPS signals to the in-vehicle device 10. The in-vehicle device 10 acquires current position information of the automobile M using the GPS signals.

[0025] The exterior camera 19 is an imaging device that captures an image ahead of the automobile M. In this embodiment, the exterior camera 19 is attached to the inside of the windshield FG near the rearview mirror RM so that the image capture direction is the front. Note that the location of the exterior camera 19 is not limited to this, and the exterior camera 19 may be attached to the dashboard DB, for example.

[0026] For example, a road image captured by the exterior camera 19 may be transmitted to the server 40 and stored in the server 40. In this case, the automobile M plays the role of a probe vehicle that collects road images. Note that the automobile M may be equipped with a camera that captures road images, for example, a camera that captures images of the side or rear of the automobile M.

[0027] 3 is a block diagram showing the configuration of the in-vehicle device 10. For example, the in-vehicle device 10 is a device in which an input unit 25, a storage unit 27, a control unit 29, a communication unit 31, and an output unit 33 cooperate with each other via a system bus 23.

[0028] The automobile M is also equipped with an acceleration sensor 21. The acceleration sensor 21 is capable of measuring the acceleration of the automobile M and outputting a signal indicating the measured acceleration. The acceleration sensor 21 is a sensor capable of detecting acceleration in the traveling direction of the automobile M, i.e., the front-rear direction, as viewed from above the automobile M. The acceleration sensor is also capable of detecting, for example, acceleration in the lateral direction (width direction) perpendicular to the traveling direction of the automobile M.

[0029] The input unit 25 is an interface unit that connects the in-vehicle device 10 to the touch panel 11, the microphone 1, the GPS receiver 17, the outside-vehicle camera 19, and the acceleration sensor 21 so that they can communicate with each other.

[0030] The in-vehicle device 10 can receive a signal indicating an input operation performed on the touch pad of the touch panel 11 via the input unit 25. For example, the in-vehicle device 10 can accept, via the input unit 25, a setting input of a destination for car navigation performed by the user on the touch panel 11.

[0031] The in-vehicle device 10 can acquire, via the input unit 25, voice data of the voice picked up by the microphone 15. For example, the in-vehicle device 10 can accept, via the input unit 25, an input operation by voice made by a user.

[0032] The in-vehicle device 10 can receive a GPS signal from the GPS receiver 17 via the input unit 25 and obtain information on the current position of the in-vehicle device 10 from the GPS signal. The in-vehicle device 10 can also obtain an image captured by the exterior camera 19 via the input unit 25.

[0033] The in-vehicle device 10 can receive a signal indicating the acceleration measured by the acceleration sensor 21 via the input unit 25. The in-vehicle device 10 acquires the acceleration of the automobile M based on, for example, the acceleration indicated by the sensor signal of the acceleration sensor 21. The in-vehicle device 10 may acquire current position information of the automobile M based on, for example, the acceleration indicated by the sensor signal of the acceleration sensor 21 in addition to the GPS signal from the GPS receiver 17.

[0034] The storage unit 27 is a storage device configured by, for example, a hard disk drive, a solid state drive (SSD), a flash memory, etc. The storage unit 27 stores various programs executed in the in-vehicle device 10, such as an operating system and software for terminals. The various programs include a program for executing a process related to the presentation of road images in the in-vehicle device 10.

[0035] The various programs may be acquired, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices. In other words, the various programs stored in the storage unit 27 can be transmitted via a network, and can be recorded on a computer-readable recording medium and transferred.

[0036] The storage unit 27 also stores map information including road maps. The map information is used, for example, for displaying directions in car navigation systems.

[0037] The control unit 29 is configured with a CPU (Central Processing Unit) 29A, a ROM (Read Only Memory) 29B, a RAM (Random Access Memory) 29C, etc., and functions as a computer. The CPU 29A reads out and executes various programs stored in the ROM 29B and the storage unit 27 to realize various functions.

[0038] In this embodiment, the control unit 29 performs functions such as obtaining road images stored in the server 40 or road images generated based on the stored road images from the server 40 and displaying them on the touch panel 11 to present them to the user, and car navigation functions.

[0039] The communication unit 31 is a communication device that transmits and receives data to and from external devices in accordance with instructions from the control unit 29. The communication unit 31 is, for example, a network interface card (NIC) for connecting the in-vehicle device 10 to the network NW. The communication unit 31 is connected to the above-mentioned network NW, and transmits and receives various data to and from the server 40. The control unit 29 transmits information indicating road designations and environmental conditions to the server 40 via the communication unit 31.

[0040] The output unit 33 is an output interface unit for the touch panel 11 and the speaker 13. The control unit 29 can transmit an image signal to the touch panel 11 via the output unit 33 to cause it to display. The control unit 29 can also transmit an audio signal to the speaker 13 to cause it to output audio. The control unit 29 causes the touch panel 11 to display a road image received from the server 40 via the output unit 33.

[0041] [Road image display device functions] The following describes the functions of the control unit 29 of the in-vehicle device 10. The control unit 29 acquires an image of a road specified by the user from the server 40. The control unit 29 also acquires a road image taken or generated under environmental conditions, such as weather, specified by the user from the server 40. Alternatively, the control unit 29 automatically acquires, for example, the current environmental conditions, such as weather, of the specified road, and acquires from the server 40 a road image taken or generated under the acquired environmental conditions.

[0042] The control unit 29 receives an input operation for designating a road for which a road image is to be displayed, for example, via the touch panel 11. The control unit 29 functions as a reception unit that receives the designation of the road.

[0043] Furthermore, the control unit 29 acquires environmental conditions, which are conditions under which the road image to be displayed was captured. The environmental conditions include at least weather. For example, the control unit 29 acquires current weather information around the specified road as the environmental conditions from an external server that provides weather information. The environmental conditions may also include traffic conditions indicating traffic volume. For example, the control unit 29 acquires current traffic information of the specified road as the environmental conditions from an external server that provides traffic information.

[0044] The environmental conditions may also include time conditions. The time conditions are, for example, time conditions that affect the appearance of the road image. For example, the time period when the road image was taken can be used as a time condition, and the brightness of the outdoors when the road image was acquired can be specified by dividing a day into time periods such as morning, noon, evening, and night.

[0045] For example, when the time when the user specifies a road is during the daytime, the control unit 29 acquires the daytime as an environmental condition, and acquires an image taken during the daytime or an image generated as an image taken during the daytime from the server 40. The control unit 29 may also acquire, for example, the time specified by the user as a time condition, or when the user sets a destination, may calculate an estimated time of arrival at the destination and acquire the estimated time of arrival as a time condition. The control unit 29 functions as an environmental condition acquisition unit that acquires environmental conditions including weather conditions.

[0046] In this embodiment, when the control unit 29 accepts the designation of a road and acquires the environmental conditions, it transmits information indicating the designated road and information indicating the environmental conditions to the server 40, thereby requesting a road image.

[0047] Fig. 4 is a diagram showing road image request information DS1, which is an example of road image request information transmitted by the control unit 29 of the in-vehicle device 10 to the server 40. As shown in Fig. 4, the road image request information DS1 includes road designation information associated with an in-vehicle device ID, which is an identifier for identifying the in-vehicle device 10. The road designation information is, for example, a road link ID for identifying a road link. The road designation information may be, for example, information indicating a point on a road.

[0048] The road image request information DS1 further includes environmental information. The environmental information includes at least weather, and may further include traffic conditions and time conditions indicating traffic volume. The traffic volume may be expressed, for example, by the number of vehicles passing per unit time, or may be expressed by a numerical value indicating the degree of congestion or congestion. The time conditions may be, for example, a time period, a time, or a date.

[0049] In the example shown in Figure 4, the road image request information DS1 includes weather conditions such as weather (i.e., sky condition), amount of snowfall, and temperature, and in addition to the weather information, it also includes time zone information indicating traffic volume and whether it is day or night or the time of day, etc.

[0050] After transmitting the road image request information, the control unit 29 receives from the server 40 a road image stored in the server 40 or a road image generated in the server 40, and acquires the road image as a presentation image to be presented on the touch panel 11, thereby functioning as an image acquisition unit. The server 40 stores images (hereinafter also referred to as past images) acquired under various environmental conditions for each of a plurality of roads.

[0051] If a matching image that matches the road specification and environmental conditions included in the transmitted road image request information, i.e., a past image taken of the road specified by the road specification and taken under the environmental conditions included in the road image request information (i.e., under the environmental conditions acquired by the control unit 29 as an environmental condition acquisition unit), is stored in the server 40, the control unit 29 acquires the matching image sent from the server 40 as the presentation image.

[0052] If the suitable image is not stored in the server 40, the control unit 29 acquires, as the presentation image, a generated image sent from the server 40, which is generated in the server 40 based on an image of the road specified by the road specification and the environmental conditions acquired by the environmental condition acquisition unit.

[0053] The control unit 29 functions as a presentation control unit by causing the touch panel 11 to display the presentation image acquired from the server 40.

[0054] 5 is a block diagram showing the configuration of the server 40. For example, the server 40 is a device in which a large-capacity storage device 43, a control unit 45, and a communication unit 47 cooperate with each other via a system bus 41.

[0055] The server 40 has the function of receiving road image request information including a road designation and environmental conditions from the vehicle-mounted device 10, and transmitting to the vehicle-mounted device 10 an image stored in the server 40 or an image generated based on an image stored in the server 40, depending on the road designation and environmental conditions.

[0056] The server 40 also has a function of receiving current location information of the automobile M and information of a destination set by the user of the in-vehicle device 10 from the in-vehicle device 10, and generating a route to the destination based on the current location information and the destination information.

[0057] The mass storage device 43 is configured, for example, by a hard disk device and an SSD (solid state drive), and stores various programs such as an operating system and software for the server 40.

[0058] The large-capacity storage device 43 also includes a road image database (road image DB in FIG. 5) 43A. The road image database 43A stores images of each road in association with each road.

[0059] In this specification, a road image in which each road is photographed under a predetermined environmental condition is called a reference image. In this embodiment, the reference image is typically a road image photographed in daytime in clear weather (a photographed reference image). In other words, the reference image is a road image photographed under environmental conditions with good visibility. Alternatively, the reference image may be a road image generated as a road image in daytime in clear weather (a generated reference image). In other words, the reference image may be a road image generated as an image under environmental conditions with good visibility. In this embodiment, the road image database 43A further includes road images other than the reference image, i.e., road images photographed under various environmental conditions other than the predetermined environmental conditions.

[0060] For example, each of the road images stored in the road image database 43A may be associated with a road link, or may be associated with a point on a road or a point on a map.

[0061] The large-capacity storage device 43 also includes an image generation model storage unit 43 B. In this embodiment, a plurality of image generation models are stored in the image generation model storage unit 43 B.

[0062] The image generation model storage unit 43B stores a characteristic image generation model, which is an image generation model that has been machine-learned to output a road image in which a reference image and environmental conditions are inputted and the reference image is processed so that the image has characteristics corresponding to the environmental conditions.

[0063] [Feature image generation model] Fig. 6 is a diagram showing a snow cover image generation model M1, which is an example of a feature image generation model. The snow cover image generation model M1 shown in Fig. 6 includes a neural network constructed by deep learning. The neural network is composed of an input layer, two or more intermediate layers (hidden layers), and an output layer.

[0064] For example, the input layer of the snow cover image generation model M1 receives reference image data and, as values ​​indicating environmental information, values ​​indicating the amount of snow and the amount of traffic.

[0065] When input is made to the input layer, the output layer outputs road image data having characteristics corresponding to the environmental information input to the input layer. In the example shown in Fig. 6, the output layer outputs road image data that has been image-processed so that the reference image input to the input layer has characteristics corresponding to the amount of snowfall and traffic volume. For example, the output layer outputs image data of a road image that shows the presence of snowfall corresponding to the input amount of snowfall on a road included in the reference image, and includes ruts in the snow-covered part on the road corresponding to the input traffic volume.

[0066] The snow cover image generation model M1 is, for example, a trained model constructed by machine learning using supervised learning.

[0067] In this case, when learning the snow-covered image generation model M1, for example, a snow-covered image of an arbitrary road in a snowy state, information indicating the environmental conditions when the snow-covered image was taken, and a non-snow-covered image of the arbitrary road in a state where there is no snow are used as a learning data set. Here, the information indicating the environmental conditions includes the amount of snow around the arbitrary road, and the traffic volume per predetermined time on the road when the snow-covered image was taken. Then, the snow-covered image is designated as the correct answer data, and learning is performed while updating the parameters of the neural network by, for example, the backpropagation method so that the output data generated by applying image processing to the non-snow-covered image input to the model has the visual characteristics of the snow-covered image that is the correct answer data (i.e., the snow-covered state according to the environmental conditions). For example, the image data output from the model is compared with the snow-covered image that is the correct answer data to calculate the error, and the weighting of each parameter of the neural network is changed according to the magnitude of the error. The output from the model and the weighting are repeatedly changed to reduce the error.

[0068] Furthermore, a snow image generation model M1 can be constructed by taking multiple snow images with different amounts of snow or traffic for one road and performing the above-mentioned learning process using each of the multiple snow images as correct answer data, or by performing the above-mentioned learning process for each of a number of different roads.

[0069] The learning method for the snow cover image generation model M1 is not limited to the above-described supervised learning, and other known learning methods may be used.

[0070] In other words, the snow-covered image generation model M1 as an image generation model is a trained model that uses as training datasets a first snow-covered image, which is an image of an arbitrary first road photographed when there is snow, weather information including the amount of snow at least around the first road at the time the first snow-covered image was photographed and traffic volume information indicating the traffic volume on the first road, and a first non-snow-covered image, which is an image of the first road photographed when there is no snow, and has been machine-learned so that the output generated by applying image processing to the first non-snow-covered image has the visual characteristics of the first snow-covered image.

[0071] In addition, in Figure 6, an example has been described in which a reference image and environmental information are input into the snow-covered image generation model M1 to generate a snow-covered image corresponding to the environmental information based on the reference image. However, the image used as the basis for generating a snow-covered image (also called the original image) is not limited to this, and for example, a road image taken under rainy or cloudy environmental conditions (a non-snow-covered image other than the reference image) can also be input to the snow-covered image generation model M1.

[0072] In other words, the snow image generation model M1 is an image generation model that generates a snow image, which is an image of the target road with snow corresponding to the amount of snow and traffic volume, in response to inputs based on an original image, which is an image of the target road in a snow-free state, weather information including at least the amount of snow around the target road, and traffic volume information indicating the traffic volume on the target road.

[0073] In addition, although an example of inputting the amount of snowfall and traffic volume as environmental conditions has been described, the environmental conditions input to the snowfall image generation model M1 may further include weather conditions other than the amount of snowfall, such as information indicating the weather conditions, such as sunny, rainy, cloudy, etc., and information indicating the temperature, or temporal conditions, such as information indicating the time of day or time period.

[0074] In addition, as a feature image generation model, a rain image generation model that generates a road image in a rainy state based on a reference image and environmental conditions including the amount of precipitation may be constructed and stored in the image generation model storage unit 43B, in addition to the snowfall image generation model M1.

[0075] Also, for example, a cloudy image generation model that generates a road image in a cloudy state based on a reference image and environmental conditions including cloud cover may be constructed and stored in the image generation model storage unit 43B.

[0076] [Reference image generation model] The image generation model storage unit 43B also stores a reference image generation model that outputs a reference image having characteristics according to the predetermined environmental conditions when a road image captured under conditions other than the predetermined environmental conditions is input. In this example, the predetermined environment is assumed to be sunny weather and daytime.

[0077] Fig. 7 is a diagram showing a reference image generation model M2, which is an example of a reference image generation model. The reference image generation model M2 shown in Fig. 7 is a neural network constructed by deep learning. The neural network is composed of an input layer, two or more intermediate layers (hidden layers), and an output layer.

[0078] For example, snow-covered image data is input to the input layer of the image generation model M2 as a road image captured under conditions other than a specified environment. When the input is made to the input layer, reference image data having characteristics corresponding to the specified environmental conditions (sunny daytime) is output to the output layer.

[0079] In the example shown in FIG. 7, the snowfall image data input to the input layer is cleared and an image showing the road surface in clear weather is output as the reference image data.

[0080] The reference image generation model M2 is, for example, a trained model constructed by machine learning using supervised learning.

[0081] In this case, when learning the reference image generation model M2, for example, a reference image of an arbitrary road without snow actually photographed under the environmental conditions of the reference image and a snow-covered image of the same road as the arbitrary road photographed under snow-covered conditions are used as a learning data set. Then, the reference image is designated as the correct answer data, and learning is performed while updating the parameters of the neural network by the backpropagation method so that the output data generated by applying image processing to the snow-covered image input to the model has the visual characteristics of the reference image, which is the correct answer data (i.e., no snow, clear weather, and photographed during the daytime). Furthermore, by performing the above-mentioned learning for each of a large number of different roads, the reference image generation model M2 can be constructed.

[0082] The learning method for the reference image generation model M2 is not limited to the above-described supervised learning, and other known learning methods may be used.

[0083] In addition, in Figure 7, an example has been described in which a snow-covered image is input into the reference image generation model M2, and a reference image corresponding to a daytime time period when there is no snow, the weather is sunny, and it is based on the snow-covered image, but images other than snow-covered images, for example, road images taken under environmental conditions such as rain or cloudy weather, can also be input to the reference image generation model M2.

[0084] In other words, the reference image generation model M2 as a non-snow-covered image generation model is a trained model that has been machine-learned using a second non-snow-covered image, which is an image of an arbitrary second road when there is no snow, and a second snow-covered image, which is an image of the second road when there is snow, as training data sets, and performs image processing on the second snow-covered image so that the output generated has the visual characteristics of the second non-snow-covered image.

[0085] [Image generation model using CycleGAN] In this embodiment, the snow-covered image generation model M1 as the first image generation model and the reference image generation model M2 as the second image generation model are machine-learned using a cycle generative adversarial network (CycleGAN), and the accuracy of generating road images having characteristics according to environmental conditions is further improved. For example, the snow-covered image generation model M1 can generate an image having characteristics such as snow appropriately even for roads having shapes different from those of the roads in the road image used for learning.

[0086] CycleGAN uses a discriminator D1 for the reference image generation model M2 and a discriminator D2 for the snow-covered image generation model M1.

[0087] Fig. 8 is a block diagram showing a schematic diagram of a GAN configuration for optimizing a reference image generation model M2. As shown in Fig. 8 and as described in Fig. 6, when environmental conditions EC1 and reference image data BM1 are input to the snow-covered image generation model M1, snow-covered image data SM1 is output. When the output snow-covered image data SM1 is input to the reference image generation model M2, reference image data BM2 is output, as described in Fig. 7.

[0088] The first classifier D1 compares the reference image data BM1 with the reference image data BM2, and classifies whether the reference image data BM2 is the reference image data BM1 that is the original image. That is, the first classifier D1 performs a genuine / fake determination.

[0089] 8, the first classifier D1 and the reference image generation model M2 are trained to compete with each other by updating the parameters of the first classifier D1 and the reference image generation model M2 so that the classification accuracy of the first classifier D1 is improved and the accuracy of reproduction of the features of the reference image data BM1 by the reference image generation model M2 is increased. This reduces the difference in features between the reference image data BM1 and the reference image data BM2, in other words, the features of the reference image data BM2 become closer to the features of the reference image data BM1.

[0090] Fig. 9 is a block diagram showing a schematic diagram of a GAN configuration for optimizing a snow-covered image generation model M1. As shown in Fig. 9 and as described in Fig. 7, when snow-covered image data SM2 is input to the reference image generation model M2, reference image data BM3 is output. When the output reference image data BM3 and environmental conditions EC2 are input to the snow-covered image generation model M1, snow-covered image data SM3 is output, as described in Fig. 6.

[0091] The second classifier D2 compares the snow-covered image data SM2 with the snow-covered image data SM3, and classifies whether the snow-covered image data SM3 is the original snow-covered image data SM2 or not. That is, the second classifier D2 performs a true / false determination.

[0092] 9, in order to improve the classification accuracy of the second classifier D2 and increase the reproduction accuracy of the features of the snow-covered image data SM2, the parameters of the second classifier D2 and the snow-covered image generation model M1 are updated, and the second classifier D2 and the snow-covered image generation model M1 learn to compete with each other. This reduces the difference in features between the snow-covered image data SM2 and the snow-covered image data SM3, in other words, the features of the snow-covered image data SM3 become closer to the features of the snow-covered image data SM2.

[0093] The accuracy of the snow cover image generation model M1 and the reference image generation model M2 is improved by repeatedly performing the optimization of Fig. 8 and the optimization of Fig. 9. Each of the snow cover image generation model M1 and the reference image generation model M2 can be said to be a trained model that has been machine-learned by a cycle generative adversarial network (CycleGAN) formed including the snow cover image generation model M1 and the reference image generation model M2.

[0094] 5, the large-capacity storage device 43 includes a map information database (not shown) in which map information including road maps is stored. The map information is used, for example, for route generation in the navigation function. The road images in the road image DB 43A may be stored in association with the map information in the map information database.

[0095] The control unit 45 is configured with a CPU (Central Processing Unit) 45A, a ROM (Read Only Memory) 45B, a RAM (Random Access Memory) 45C, etc., and functions as a computer. The CPU 45A reads out and executes various programs stored in the ROM 45B and the large-capacity storage device 43 to realize various functions.

[0096] The communication unit 47 is connected to the above-mentioned network NW, and transmits and receives various data to and from the in-vehicle device 10. For example, the control unit 45 receives road image request information including a road designation and environmental conditions from the in-vehicle device 10 via the communication unit 47.

[0097] If a matching image is stored in the road image database 43A, the control unit 45 transmits the matching image to the vehicle-mounted device 10, the matching image being an image of a road specified by the road specification included in the road image request information received from the vehicle-mounted device 10 and being a road image captured under the environmental conditions included in the road image request information.

[0098] If a suitable image is not stored in the road image database 43A, the control unit 45 transmits to the in-vehicle device 10 a generated image that corresponds to the road specified by the road specification included in the road image request information and that is generated to correspond to the environmental conditions included in the road image request information.

[0099] For example, when a reference image of a specified road captured in daytime in fine weather is stored in road image database 43A and no suitable image is stored, control unit 45 transmits a generated image generated based on the reference image of the specified road and the received environmental conditions to vehicle-mounted device 10. For example, control unit 45 inputs the reference image, the amount of snowfall, and the traffic volume to snowfall image generation model M1, and transmits the output snowfall image to vehicle-mounted device 10.

[0100] For example, when the road image database 43A does not store a reference image taken in daytime hours when the weather is clear for a specified road, and no suitable image is stored, the control unit 45 generates a reference image based on an image other than the reference image stored in the road image database 43A for the specified road, and transmits a generated image based on the generated reference image and the received environmental conditions to the in-vehicle device 10.

[0101] For example, the control unit 45 inputs an image of rainy weather stored in the road image database 43A into the reference image generation model M2, inputs the output reference image and the received environmental conditions into the snow-covered image generation model M1, and transmits the output snow-covered image to the in-vehicle device 10.

[0102] [Control routine] A control routine executed in the road image display system 100 of this embodiment will now be described.

[0103] 10 is a flowchart showing a control routine RT1, which is an example of a control routine executed by the control unit 29 of the in-vehicle device 10. The control unit 29 starts the control routine RT1, for example, when an input operation by a user to instruct displaying a road image is accepted.

[0104] When the control routine RT1 is started, the control unit 29 determines whether or not a road designation has been accepted (step S101). In step S101, for example, when the control unit 29 has accepted an input operation to designate a road or a point on a road via the touch panel 11, the control unit 29 determines that a road designation has been accepted.

[0105] In step S101, if it is determined that a road designation has not been accepted (step S101: NO), the control unit 29 repeatedly executes step S101 and determines again whether or not a road designation has been accepted.

[0106] In step S101, when it is determined that a road designation has been accepted (step S101: YES), the control unit 29 acquires environmental conditions (step S102). As described above, the environmental conditions include at least weather conditions and may further include traffic conditions or time conditions, and indicate the environmental conditions that should be the shooting conditions for the road image that the user wants to check.

[0107] In step S102, for example, the control unit 29 acquires current meteorological information around the road specified in step S101 as the weather conditions. In step S102, for example, traffic conditions including the current traffic volume of the specified road may be acquired from road traffic information or the like. In addition, at least a part of the environmental conditions may be designated by the user, and the control unit 29 may acquire the environmental conditions input by the user.

[0108] After executing step S102, the control unit 29 transmits information including the road designation acquired in step S101 and the environmental condition acquired in step S102 to the server 40 as road image request information (step S103).

[0109] After executing step S103, the control unit 29 judges whether or not a road image has been received from the server 40 (step S104). In step S104, for example, a previous image of the road designated by the road designation and stored in the server 40 or a generated image generated in the server 40 may be received.

[0110] In step S104, if it is determined that a road image has not been received (step S104: NO), the control unit 29 repeats step S104 to determine again whether or not a road image has been received.

[0111] In step S105, if it is determined that the road image has been received (step S104: YES), the control unit 29 causes the touch panel 11 to display the received road image (step S105).

[0112] After executing step S105, the control unit 29 ends the control routine RT1.

[0113] According to the control routine RT1, for example, a road image reflecting environmental conditions such as current weather conditions around a specified road can be presented. For example, the user can check the current road condition by being presented with such a road image.

[0114] In step S102 of the control routine RT1, the control unit 29 may acquire future environmental conditions for the specified road. For example, when a future date and time is specified by the user, weather forecast information for the specified date and time around the specified road may be acquired as weather conditions, and further, the expected traffic volume of the specified road at the specified date and time may be acquired as traffic conditions. This allows, for example, the user to check future road conditions, such as the estimated time of arrival at the destination, by images.

[0115] The weather forecast information may include, for example, forecast information on snowfall. Furthermore, the future environmental conditions may include forecast information on traffic volume. The traffic volume forecast information may be acquired, for example, based on statistical data on traffic volume. Furthermore, the traffic volume forecast information may be acquired in association with the amount of snowfall. For example, from data on past snowfall, a date and time during the same time period as the scheduled arrival time when the amount of snowfall was close to the predicted value of the amount of snowfall may be identified, and the statistical value of the traffic volume at that date and time may be used as the predicted traffic volume.

[0116] 11 is a flowchart showing a control routine RT2, which is an example of a control routine executed by the control unit 45 of the server 40. For example, when the server 40 is powered on, the control unit 45 repeatedly executes the control routine RT2.

[0117] When the control routine RT2 is started, the control unit 45 determines whether or not image request information has arrived (step S201). In step S201, the control unit 45 determines that image request information has arrived when information including a road designation and environmental conditions, such as those illustrated in FIG. 4, is received from the in-vehicle device 10.

[0118] In step S201, when the control unit 45 determines that image request information has not arrived even after a predetermined time has elapsed (step S201: NO), it ends the control routine RT2 and starts a new control routine RT2.

[0119] In step S201, when the control unit 45 determines that image request information has arrived (step S201: YES), it searches the road image database 43A of the large-capacity storage device 43 to determine whether a suitable image that was taken of the road specified by the road specification received from the vehicle-mounted device 10 and was taken under the received environmental conditions is stored (step S202).

[0120] In step S202, if it is determined that a matching image is stored (step S202: YES), the control unit 45 transmits the matching image to the in-vehicle device 10 (step S203).

[0121] In step S202, if it is determined that a suitable image is not stored (step S202: NO), the control unit 45 generates a presentation image, which is a road image to be presented on the in-vehicle device 10, based on the image of the road specified by the road specification received from the in-vehicle device 10 and the received environmental conditions (step S204).

[0122] In step S204, for example, the control unit 45 inputs a reference image in which the specified road is captured and the received environmental conditions into an image generation model, thereby generating a road image having characteristics according to the environmental conditions.

[0123] In step S204, for example, the control unit 45 can input a reference image of a specified road and the received environmental conditions including the amount of snow and traffic volume into the snow image generation model M1, thereby generating a snow image of a state in which there is snow according to the amount of snow and traffic volume.

[0124] For example, by inputting a reference image BM as shown in Fig. 12 and environmental conditions including the amount of snow and traffic volume into the snow cover image generation model M1, it is possible to generate a snow cover image SM having features such as the degree of ruts W depending on the traffic volume in addition to the snow cover state depending on the amount of snow, as shown in Fig. 13. In this way, when there is snow, it is possible to generate an image showing the snow cover state including ruts depending on the traffic volume by including the traffic volume in the input parameters.

[0125] In step S204, for example, if the control unit 45 does not store a reference image of the specified road in the road image database 43A, it inputs an image other than the reference image in which the specified road was photographed into the reference image generation model M2, and inputs the output reference image and environmental conditions into the snow-covered image generation model M1 or another feature image generation model, to generate a road image having features according to the environmental conditions.

[0126] In step S204, instead of or in addition to image generation using an image generation model, a road image may be generated by simply adding rain or clouds to an image of the road specified by the road specification based on environmental conditions.

[0127] After executing step S204, control unit 45 transmits the generated image generated in step S204 to in-vehicle device 10 (step 205). The generated image received by in-vehicle device 10 is presented to the user via touch panel 11.

[0128] After executing step S205, the control unit 45 ends the control routine RT2.

[0129] As described in detail above, the road image presentation device of this embodiment is characterized by including: a reception unit that receives a road specification; an environmental condition acquisition unit that acquires environmental conditions including weather conditions; an image acquisition unit that acquires, as a presentation image, a suitable image that is a past image of the road specified by the road specification and that was taken under the environmental conditions acquired by the environmental condition acquisition unit, if a suitable image is stored in the memory unit that stores a plurality of past images of roads; and, if a suitable image is not stored, acquires, as a presentation image, an image of the road specified by the road specification and a generated image generated based on the acquired environmental conditions; and a presentation control unit that controls the presentation of the presentation image.

[0130] With this configuration, even if there is no image that matches the specified road and environmental conditions among the previously captured images, it is possible to obtain and present an image generated in the server.

[0131] In addition, the snow cover image generation model of this embodiment is an image generation model that generates a snow cover image, which is an image of a target road with snow cover corresponding to the amount of snow cover and traffic volume, in response to inputs based on an original image, which is an image of a target road in a state where there is no snow cover, weather information including at least the amount of snow cover around the target road, and traffic volume information indicating the traffic volume on the target road.

[0132] According to the image generation model, it is possible to generate a snow image that reproduces the condition of road ruts according to the traffic volume in addition to the amount of snowfall. Since the condition of road ruts greatly affects the ease of driving a vehicle on a snowy road, a user who is presented with such a snow image can check the condition of the road at the destination and make necessary preparations.

[0133] Therefore, according to this embodiment, it is possible to provide a road image presentation device, a road image presentation method, a road image presentation program, a storage medium, and an image generation model that are capable of presenting appropriate road images according to environmental conditions such as weather.

[0134] For example, a user can see the current or future road conditions at their destination and make preparations as necessary.

[0135] In the above embodiment, an example has been described in which images taken under various environmental conditions other than the reference image are stored in the road image database of the server 40, but the present invention is not limited to this, and only reference images taken under predetermined conditions may be stored in the road image database. This can reduce the amount of data of road images to be stored in advance. In this case, for example, images of environmental conditions other than the predetermined conditions may be generated as necessary. Also, for example, reference images may be presented for environmental conditions other than snowfall.

[0136] In order to reduce the amount of road image data to be stored, for example, road images may be deleted after a predetermined time has elapsed so that only real-time images are stored in the road image database, or the number of images to be stored may be limited so that only the most recent images are stored. The road image database of the server 40 may be stored in an external server.

[0137] The configurations of the in-vehicle device 10, the server 40, the routines, the image generation model, and the like in the above-described embodiment are merely examples, and can be appropriately selected, combined, or modified depending on the application, etc.

[0138] The in-vehicle device may have all or part of the functions of the server 40 according to the above embodiment. Alternatively, the server may have part of the functions of the in-vehicle device 10. For example, the control unit of the in-vehicle device 10 may have at least part of the image generation model that the server 40 has in the above embodiment, and may generate a road image using the image generation model based on a road image received from the server 40 as a presentation image. Also, for example, the control unit of the server 40 may have at least part of the function of acquiring environmental conditions that the in-vehicle device 10 has in the above embodiment, and when a user operates the in-vehicle device 10 to specify a road and a future date and time, may acquire weather forecast information for the specified date and time around the specified road and the expected traffic volume of the specified road.

[0139] In the above embodiment, the in-vehicle device 10 is a navigation device, but the present invention is not limited to this. The in-vehicle device 10 may be, for example, a drive recorder. The in-vehicle device 10 is not limited to being mounted on a vehicle, and may be a terminal device such as a smartphone, a tablet, a PC, or a wearable device. [Explanation of symbols]

[0140] 10 Onboard equipment 11 Touch Panel 13 Speaker 15. Mike 17 GPS receiver 19 Exterior camera 21 Acceleration sensor 25 Input section 27 Memory section 29 Control Unit 31 Communications Department 33 Output section 40 Servers 43 Mass storage 43A Road Image Database 43B Image generation model memory unit 45 Control section 47 Communications Department

Claims

1. An image generation model that generates a snow image, which is an image of a target road with snow corresponding to the amount of snow and the traffic volume, in response to inputs based on an original image, which is an image of a target road in a state where there is no snow, weather information including at least the amount of snow around the target road, and traffic volume information indicating the traffic volume on the target road.

2. The image generation model according to claim 1, wherein the image generation model uses as training datasets a first snow-covered image, which is an image of an arbitrary first road when there is snow, weather information including at least the amount of snow around the first road at the time the first snow-covered image was captured and traffic volume information indicating the traffic volume on the first road, and a first non-snow-covered image, which is an image of the first road when there is no snow, and is a trained model that has been machine-learned so that an output generated by applying image processing to the first non-snow-covered image has the visual features of the first snow-covered image.

3. The image generation model according to claim 1, wherein the image generation model is a trained model that has been machine-learned using a cycle generative adversarial network (CycleGAN) formed including a non-snow-covered image generation model that is a trained model in which a second non-snow-covered image, which is an image of an arbitrary second road in a state without snow, and a second snow-covered image taken in a state where the second road is covered with snow, are used as training data sets, and an output generated by applying image processing to the second snow-covered image has the visual features of the second non-snow-covered image.

4. 4. The image generation model according to claim 1, wherein the snow-covered image includes road ruts that occur in snow-covered areas and correspond to the traffic volume.

5. a reception unit for receiving a designation of a road; an environmental condition acquisition unit that acquires environmental conditions including at least weather conditions including an amount of snowfall and traffic conditions indicating a traffic volume; an image acquisition unit that acquires, as a presentation image, a generated image generated by inputting an image of a road designated by road designation that is free of snow and environmental conditions acquired by the environmental condition acquisition unit into an image generation model that generates a snow-covered image, which is an image of the target road in a state where there is snow corresponding to the amount of snow and the traffic volume, in response to input based on an original image, which is an image of a target road in a snow-free state, weather information including at least the amount of snow around the target road, and traffic volume information indicating the traffic volume on the target road; a presentation control unit that controls the presentation of the presentation image; A road image presentation device comprising:

Images