Information processing device, control method, program, and storage medium

Abstract

A server device 3 functions as an information processing device, and includes an acquisition means, an image extraction means, and a display control means. The acquisition means acquires moving body information including the current position, moving direction, and moving speed of a moving body. On the basis of the current position, moving direction, and moving speed, the image extraction means extracts a past scenery image corresponding to the surrounding scenery at the current position of the moving body from past images that are images captured in the past. The display control means displays the past scenery image on a terminal device existing outside the moving body.

Description

Information processing device, control method, program, and storage medium

[0001] The present invention relates to a system for displaying an image related to the location of an in-vehicle device.

[0002] Conventionally, there have been known technologies for enabling a conversation between a driver of a vehicle and a person outside the vehicle. For example, Patent Document 1 discloses a technology in which an in-vehicle device and an external terminal communicate with each other to mediate a voice call between the driver of the vehicle and an operator of the external terminal, and the in-vehicle device transmits front image data representing an image of the area in front of the vehicle to the external terminal.

[0003] JP 2016-213791 A

[0004] With the above-described method, for example, if image transmission from the in-vehicle device is not performed or if image transmission from the in-vehicle device is interrupted, it becomes impossible to check the scenery ahead of the vehicle on the external terminal.

[0005] In view of the above-mentioned problems, one object of the present invention is to provide an information processing device that can display a scenery image corresponding to an external scenery from a moving body on an external terminal.

[0006] The claimed invention is characterized in that it is an information processing device comprising: an acquisition means for acquiring mobile body information including the current position, moving direction, and moving speed of the mobile body; an image extraction means for extracting a past scenery image corresponding to the surrounding scenery at the current position of the mobile body from past images, which are images taken in the past, based on the current position, the moving direction, and the moving speed; and a display control means for displaying the past scenery image on a terminal device located outside the mobile body.

[0007] The claimed invention is a control method executed by a computer, comprising: an acquisition process for acquiring moving body information including the current position, moving direction, and moving speed of the moving body; an image extraction process for extracting a past scenery image corresponding to the surrounding scenery at the current position of the moving body from past images that are images taken in the past based on the current position, the moving direction, and the moving speed; and a display control process for displaying the past scenery image on a terminal device located outside the moving body.

[0008] The claimed invention is a program executed by a computer, comprising: an acquisition means for acquiring mobile body information including the current position, direction of movement, and speed of movement of the mobile body; an image extraction means for extracting a past scenery image corresponding to the surrounding scenery at the current position of the mobile body from past images, which are images taken in the past, based on the current position, direction of movement, and speed of movement; and a program that causes the computer to function as a display control means for displaying the past scenery image on a terminal device located outside the mobile body.

[0009] 1 shows an example of the configuration of a driving call system according to an embodiment. FIG. 1 shows an example of a schematic configuration of an in-vehicle device. FIG. 2 shows an example of a schematic configuration of an external terminal. FIG. 3 shows an example of a schematic configuration of a server device. FIG. 4 shows an example of a flowchart of a past scenery image display process. FIG. 5 shows another example of a flowchart of a past scenery image display process. FIG. 6 shows another example of a flowchart of a past scenery image display process. FIG. 7 shows an example of a display on an external terminal. FIG. 8 is a diagram for explaining a method of estimating an interpolation position showing an example of a display on an external terminal in a second modified example. FIG. 9 is a diagram for explaining the flow until each image is displayed, and the locations of delays.

[0010] In one preferred embodiment of the present invention, an information processing device comprises an acquisition means for acquiring mobile body information including the current position, direction of movement, and speed of movement of the mobile body, an image extraction means for extracting a past scenery image corresponding to the surrounding scenery at the current position of the mobile body from past images, which are images taken in the past, based on the current position, direction of movement, and speed of movement, and a display control means for displaying the past scenery image on a terminal device located outside the mobile body.

[0011] The information processing device can display past scenery images corresponding to the scenery around the moving object on a terminal device located outside the moving object, thereby allowing the user to understand where the moving object is traveling.

[0012] In one aspect of the information processing device, the image extraction means identifies a corrected position in the moving direction from the current position according to the moving speed, and extracts the past image associated with the corrected position as the past scenery image. This allows the information processing device to appropriately display the past scenery image on the terminal device.

[0013] In another aspect of the information processing device, the image extraction means identifies, as the corrected position, a position that is a distance from the current position in the direction of movement that corresponds to the distance traveled at the moving speed over a predetermined time. In a preferred example, the predetermined time is set based on an update interval of the moving object information. This allows the information processing device to display an interpolated image on the terminal device during the period from when the moving object information is acquired until the next moving object information is acquired.

[0014] In another aspect of the information processing device, the predetermined time is set based on the time required for the image extraction means to perform processing to extract the past scenery image, thereby allowing the information processing device to correct a delay in displaying the past scenery image.

[0015] In another aspect of the information processing device, the acquisition means acquires a current scenery image obtained by photographing an external scenery from the moving body at the current location, and includes a display image determination means for determining an image to be displayed on the terminal device from the current scenery image and the past scenery image, and the display control means causes the terminal device to display the image determined by the determination means. This allows the information processing device to appropriately display the current scenery image or the past scenery image on the terminal device.

[0016] In another aspect of the information processing device, the predetermined time is set based on a difference between a past image display time, which is a time required from when the current position of the moving object reaches a first position until the past scenery image corresponding to the first position is displayed on the terminal device, and a current image display time, which is a time required from when the current position of the moving object reaches the first position until the current scenery image captured from the moving object at the first position is displayed on the terminal device. This allows the information processing device to correct a delay in displaying the past scenery image.

[0017] In another aspect of the information processing device, the information processing device is used in a system that mediates communication between a passenger of a moving body and a user who uses the terminal device outside the moving body, thereby enabling the user to communicate with the passenger of the moving body while viewing a landscape image corresponding to the scenery around the moving body.

[0018] In another preferred embodiment of the present invention, a control method executed by a computer includes: an acquisition step of acquiring mobile object information including a current position, a moving direction, and a moving speed of the mobile object; an image extraction step of extracting a past scenery image corresponding to a surrounding scenery at the current position of the mobile object from past images that are images taken in the past based on the current position, the moving direction, and the moving speed; and a display control step of displaying the past scenery image on a terminal device located outside the mobile object. By executing this control method, an information processing device can display the past scenery image corresponding to the surrounding scenery of the mobile object on a terminal device located outside the mobile object.

[0019] In yet another embodiment of the present invention, a computer-executable program causes the computer to function as an acquisition means for acquiring mobile object information including a current position, a moving direction, and a moving speed of the mobile object; an image extraction means for extracting a past scenery image corresponding to a surrounding scenery at the current position of the mobile object from past images that are images captured in the past based on the current position, the moving direction, and the moving speed; and a display control means for displaying the past scenery image on a terminal device located outside the mobile object. By executing this program, the computer of the information processing device can display the past scenery image corresponding to a surrounding scenery of the mobile object on a terminal device located outside the mobile object. Preferably, the program is stored in a storage medium.

[0020] Preferred embodiments of the present invention will now be described with reference to the drawings.

[0021] <System Configuration> [Overall Configuration] Fig. 1 shows an example of the configuration of a driving call system according to Example 1. The driving call system includes an in-vehicle device 1 installed in a vehicle, an external terminal 2 used by a person outside the vehicle, and a server device 3. Data communication is performed between the in-vehicle device 1 and the server device 3, and between the external terminal 2 and the server device 3, via a communication network 4 such as the Internet or a dedicated communication network.

[0022] The vehicle-mounted device 1 is a terminal device that travels with the vehicle and communicates data with the server device 3 via the communication network 4. Therefore, the vehicle-mounted device 1 can also be considered a communication device that travels with the vehicle. Hereinafter, the vehicle equipped with the vehicle-mounted device 1 will also be referred to as the "target vehicle." During a voice call between the driver of the target vehicle and the user of the external terminal 2, the vehicle-mounted device 1 transmits voice data intended for the external terminal 2 and scenic images (moving images) captured outside the target vehicle to the server device 3. The vehicle-mounted device 1 also transmits mobile object information to the server device 3 every predetermined time T1. The mobile object information includes the current location, moving speed, and moving direction of the target vehicle. The vehicle-mounted device 1 may be an in-vehicle device installed or attached to the vehicle, or may be a mobile terminal, such as a smartphone, that is brought into the vehicle and used. The vehicle-mounted device 1 may also be incorporated into the vehicle.

[0023] The external terminal 2 is a terminal device operated by a person outside the target vehicle, and performs data communication with the server device 3 via the communication network 4. The external terminal 2 transmits voice data intended for the in-vehicle device 1 to the server device 3 during a voice call between the driver of the target vehicle and the user of the external terminal 2. Furthermore, during the voice call, the external terminal 2 receives and displays display data including scenic images or past scenic images from the server device 3. This allows the user of the external terminal 2 to understand where the target vehicle is traveling. The external terminal 2 is, for example, a mobile terminal such as a smartphone. The external terminal 2 is an example of a "terminal device."

[0024] The server device 3 relays data necessary for voice communication between the in-vehicle device 1 and the external terminal 2. The server device 3 also determines, under predetermined conditions described below, whether to transmit a scenic image to the external terminal 2 or a past scenic image corresponding to the scenic image to the external terminal 2. The server device 3 then generates display data including the scenic image or the past scenic image and transmits it to the external terminal 2. Here, a past scenic image refers to a past scenic image captured before the scenic image was captured in real time, and a past scenic image corresponding to a scenic image refers to a past scenic image corresponding to the capture position or capture direction of the scenic image. The server device 3 can extract a past scenic image corresponding to a scenic image based on the moving object information received from the in-vehicle device 1.

[0025] In addition, even when a voice call is not in progress, the server device 3 may display an image including a landscape image or a past landscape image on the display unit of the external terminal 2 in response to a request from the external terminal 2, thereby allowing the user of the external terminal 2 to unilaterally check where the target vehicle is traveling.

[0026] 2 shows an example of a schematic configuration of the on-board device 1. The on-board device 1 mainly includes a communication unit 11, a storage unit 12, an input unit 13, a control unit 14, a sensor group 15, a display unit 16, and a sound output unit 17. The elements within the on-board device 1 are connected to each other via a bus line 10.

[0027] The communication unit 11 performs data communication with other terminals under the control of the control unit 14. For example, the communication unit 11 may receive map data for updating a map DB (Database) 5 from a map management server (not shown).

[0028] The storage unit 12 is configured with various types of memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and non-volatile memory (including a hard disk drive, flash memory, etc.). The storage unit 12 stores programs for the vehicle-mounted device 1 to execute predetermined processes. The programs may include application programs for making calls. The storage unit 12 is also used as a working memory for the control unit 14. The programs executed by the vehicle-mounted device 1 may be stored in a storage medium other than the storage unit 12.

[0029] The storage unit 12 also stores a map DB 5. The map DB 5 stores various data necessary for route guidance. The map DB 5 contains data necessary for displaying a map based on a predetermined position, such as the current position of the target vehicle. The map DB 5 is a database that includes, for example, road data that represents a road network using a combination of nodes and links, and facility data that indicates facilities that are candidates for destinations, stopovers, or landmarks. The map DB 5 may be updated based on information received by the communication unit 11 from a map management server under the control of the control unit 14.

[0030] The input unit 13 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 16 is a display, etc. that displays information under the control of the control unit 14. The sound output unit 17 is a speaker, etc. that outputs sound under the control of the control unit 14.

[0031] The control unit 14 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and controls the entire vehicle-mounted device 1 .

[0032] The processing performed by the control unit 14 is not limited to being realized by software programs, but may be realized by any combination of hardware, firmware, and software. The processing performed by the control unit 14 may also be realized by a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, the program executed by the control unit 14 in this embodiment may be realized by using this integrated circuit.

[0033] The sensor group 15 includes various sensors that sense the state of the target vehicle or the environment outside the vehicle. The sensor group 15 includes an exterior camera 51, a driver camera 52, and a vehicle behavior detector 53.

[0034] The exterior camera 51 is one or more cameras that capture images of the area outside the target vehicle, such as the area in front of the target vehicle, and generates images (also called "scenery images") captured at predetermined time intervals. The driver camera 52 is a camera installed so that the driver's face is included in the capture range, and generates images (also called "driver images") captured at predetermined time intervals.

[0035] The vehicle behavior detector 53 generates detection signals indicating the behavior of the target vehicle, such as the current position, vehicle speed, acceleration, steering angle, etc. The vehicle behavior detector 53 includes, for example, a Global Navigation Satellite System (GNSS) receiver, a gyro sensor, an Inertial Measurement Unit (IMU), a vehicle speed sensor, an acceleration sensor, and a steering angle sensor.

[0036] In addition, the sensor group 15 may include various external sensors (including cameras, lidars, radars, ultrasonic sensors, infrared sensors, sonar, etc.) and internal sensors in addition to the exterior vehicle camera 51, the driver camera 52, and the vehicle behavior detector 53.

[0037] The configuration of the in-vehicle device 1 shown in FIG. 2 is an example, and various modifications may be made to the configuration shown in FIG. 2 . For example, instead of storing the map DB 5 in the memory unit 12, the control unit 14 may receive map information from a map management server (not shown) via the communication unit 11. Furthermore, the in-vehicle device 1 may not have the map DB 5, and the server device 3 may perform processing based on the map DB 5. In another example, at least one of the input unit 13, the display unit 16, and the sound output unit 17 may be provided in the target vehicle as an external device of the in-vehicle device 1, and may supply the generated signals to the in-vehicle device 1. Furthermore, at least some of the sensors in the sensor group 15 may be sensors mounted on the target vehicle. In this case, the in-vehicle device 1 may acquire information output by sensors mounted on the target vehicle from the target vehicle based on a communication protocol such as CAN (Controller Area Network).

[0038] 3 shows an example of the schematic configuration of the exterior terminal 2. The exterior terminal 2 mainly includes a communication unit 21, a storage unit 22, an input unit 23, a control unit 24, a sensor group 25, a display unit 26, and a sound output unit 27. The elements within the exterior terminal 2 are connected to each other via a bus line 20.

[0039] The communication unit 21 performs data communication with other terminals under the control of the control unit 24 .

[0040] The storage unit 22 is configured with various types of memory such as RAM, ROM, and non-volatile memory. The storage unit 22 stores programs for the external terminal 2 to execute predetermined processes. The above-mentioned programs may include application programs for communicating with the driver and displaying information related to driving the target vehicle of the in-vehicle device 1 (including displaying maps and various captured images) when communication with the in-vehicle device 1 is established via the server device 3. The storage unit 22 is also used as a working memory for the control unit 24. The programs executed by the external terminal 2 may be stored in a storage medium other than the storage unit 22.

[0041] The input unit 23 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 26 is a display, etc. that displays information based on the control of the control unit 24. The sound output unit 27 is a speaker, etc. that outputs sound based on the control of the control unit 24. The sensor group 25 includes an internal sensor that senses the state of the exterior terminal 2 and an external sensor that senses the state of the outside world of the exterior terminal 2.

[0042] The control unit 24 includes a CPU, a GPU, and the like, and controls the entire exterior terminal 2. Note that the configuration of the exterior terminal 2 shown in Fig. 3 is an example, and various modifications may be made to the configuration shown in Fig. 3.

[0043] 4 shows an example of a schematic configuration of the server device 3. The server device 3 mainly includes a communication unit 31, a storage unit 32, and a control unit 33. The elements within the server device 3 are connected to each other via a bus line 30.

[0044] The communication unit 31 performs data communication with external devices such as the in-vehicle device 1 and the external terminal 2 under the control of the control unit 33 .

[0045] The storage unit 32 is configured with various types of memory such as RAM, ROM, and non-volatile memory (including a hard disk drive, a flash memory, etc.). The storage unit 32 stores programs for the server device 3 to execute predetermined processes. The above-mentioned programs may include an application program for mediating calls, an application program for causing the external terminal 2 to display maps, scenic images, etc. The storage unit 32 is also used as a working memory for the control unit 33. The programs executed by the server device 3 may be stored in a storage medium other than the storage unit 32.

[0046] The storage unit 32 also stores a map DB 5 and a past image library 6. Past scenery images are stored in advance in the past image library 6. The past scenery images are stored in the past image library 6 in association with information such as the shooting position including latitude and longitude and lane information, the shooting direction, the shooting date and time, and the weather at the time of shooting.

[0047] The control unit 33 includes a CPU, a GPU, etc., and controls the entire server device 3. The control unit 33 also executes programs stored in the storage unit 32 to perform extraction processing of past scenic images and display control processing. Note that the configuration of the server device 3 shown in Fig. 4 is an example, and various changes may be made to the configuration shown in Fig. 4.

[0048] [Process for Extracting Past Scenic Images] Next, the process for extracting past scenic images will be described. The server device 3 extracts past scenic images corresponding to scenic images of the target vehicle from the past image library 6 based on the moving object information (current position, moving direction, and moving speed) of the target vehicle received from the onboard device 1. Specifically, the server device 3 extracts from the past image library 6 past scenic images that were captured at a position within a predetermined range from the current position (latitude and longitude) of the target vehicle and in the same direction as the moving direction of the target vehicle. Note that the server device 3 may extract past scenic images based on information such as the current date and time (season and time zone) and current weather in addition to the moving object information.

[0049] [Display Control Process] Next, the display control process will be described. The server device 3 determines whether to transmit a scenic image or a past scenic image to the external terminal 2 based on predetermined conditions described below. If the server device 3 determines to transmit a scenic image to the external terminal 2, it transmits the scenic image received from the in-vehicle device 1 to the external terminal 2. On the other hand, if the server device 3 determines to transmit a past scenic image to the external terminal 2, it executes the above-mentioned process of extracting past scenic images and transmits the extracted past scenic image to the external terminal 2. The predetermined conditions will be described below with specific examples.

[0050] (First Example) While a vehicle is traveling, communication is likely to become unstable due to the effects of fading, etc., which may result in delays or interruptions in the transmission of scenic images, or a decrease in the image quality of the scenic images. Therefore, in the first example, the communication quality of the communication between the in-vehicle device 1 and the server device 3 is used as the predetermined condition. If the communication quality is stable, the server device 3 transmits the scenic images received from the in-vehicle device 1 to the external terminal 2. On the other hand, if the communication quality is unstable, the server device 3 transmits past scenic images to the external terminal 2.

[0051] The server device 3 can use the communication error rate as an index of communication quality. For example, when the communication error rate for communication of audio data or display data is equal to or greater than a predetermined threshold TH1, the server device 3 determines that the communication quality is unstable. On the other hand, when the communication error rate is less than the predetermined threshold TH1, the server device 3 determines that the communication quality is stable.

[0052] Here, the error rate is, for example, a packet loss rate. The server device 3 monitors the sequence numbers of packets received from the vehicle-mounted device 1 and detects missing packets. The server device 3 calculates the packet loss rate based on the number of successfully received packets and the number of missing packets. The error rate is, for example, the communication disconnection time per unit time. The server device 3 records the communication disconnection time that occurs between the establishment of communication with the vehicle-mounted device 1 and the elapse of a predetermined time, and calculates the communication disconnection time per unit time. Furthermore, if TCP is used as the transport layer protocol of the OSI reference model for sending and receiving audio data, the server device 3 may use the packet retransmission rate as the error rate.

[0053] It is desirable for the in-vehicle device 1, the external terminal 2, and the server device 3 to use highly reliable TCP for sending and receiving voice data and mobile object information, and to use UDP for sending and receiving display data that requires real-time performance, such as landscape images and past landscape images, but the combination of protocols to be used is not limited to the above. For example, UDP may be used for sending and receiving voice data and mobile object information.

[0054] Furthermore, the above-described methods for calculating the communication quality index and the communication error rate are merely examples, and the present invention is not limited to these.

[0055] 5 is a flowchart of the process in the first example. First, the server device 3 establishes communication between the in-vehicle device 1 and the external terminal 2 (step S101). The server device 3 exchanges information (e.g., communication address information) necessary to establish communication between the in-vehicle device 1 and the external terminal 2, and also performs processing necessary to perform data communication between the in-vehicle device 1 and the external terminal 2 (including authentication processing for the in-vehicle device 1 and the external terminal 2). Once communication between the in-vehicle device 1 and the external terminal 2 is established, the driver initiates a call with the user of the external terminal 2. Furthermore, the in-vehicle device 1 transmits a scenic image to the external terminal 2 via the server device 3.

[0056] Next, the server device 3 calculates the communication error rate between the in-vehicle device 1 and the server device 3 and determines whether the communication error rate is equal to or greater than a predetermined threshold (step S102). If the communication error rate is equal to or greater than the predetermined threshold TH1 (step S102: Yes), the server device 3 determines that the communication quality is unstable and stops transmission of the scenic image received from the in-vehicle device 1 (step S104). Then, the server device 3 extracts past scenic images based on the mobile object information received from the in-vehicle device 1 and transmits them to the external terminal 2 (step S105). If the communication error rate is less than the predetermined threshold TH1 (step S102: No), the server device 3 determines that the communication quality is stable and transmits the scenic image received from the in-vehicle device 1 to the external terminal 2 (step S103).

[0057] Next, the server device 3 determines whether the call between the driver and the user of the external terminal 2 has ended (step S106). If the server device 3 determines that the call has ended (step S106; Yes), the server device 3 ends the processing of the flowchart. On the other hand, if the server device 3 determines that the call has not ended (step S106; No), the server device 3 returns the processing to step S101.

[0058] In the first example, when the communication quality of the in-vehicle device 1 is unstable, the server device 3 can cause the external terminal 2 to display a past scenic image corresponding to the scenic image. This allows the user of the external terminal 2 to understand the shape of the road on which the target vehicle is traveling even when the communication quality of the in-vehicle device 1 is unstable, and enables the user to provide route guidance to the driver.

[0059] (Second Example) In a second example, a selection instruction by a user is used as the predetermined condition. For example, the user of the external terminal 2 operates a selection button displayed on the external terminal 2 to select whether to display a scenic image or a past scenic image, and transmits the instruction to the server device 3. If a scenic image is selected, the server device 3 transmits the scenic image from the in-vehicle device 1 to the external terminal 2. On the other hand, if a past scenic image is selected, the server device 3 transmits the past scenic image to the external terminal 2. The user of the external terminal 2 may also specify conditions such as the date and time (season and time of day) when the past scenic image was captured and the weather at the time of capture. In this case, the server device 3 extracts the past scenic image based on the mobile object information and the specified conditions, and transmits it to the external terminal 2.

[0060] 6 is a flowchart of the process in the second example. Step S201 and steps S203 to S206 are the same as the process in step S101 and steps S103 to S106 in the first example shown in FIG. 5, so a description thereof will be omitted.

[0061] The server device 3 determines whether the user of the external terminal 2 has selected a past scenic image (step S202). If the user has selected a past scenic image (step S202: Yes), the server device 3 proceeds to step S204. On the other hand, if the user has not selected a past scenic image but has selected a scenic image (step S202: No), the server device 3 proceeds to step S203.

[0062] In the above example, the user of the external terminal 2 performs the selection operation, but instead, the driver may operate the in-vehicle device 1 to send to the server device 3 a selection instruction for the image to be displayed on the external terminal 2.

[0063] In the second example, the display can be switched between a scenic image and a past scenic image at a timing desired by the user. This allows the user to provide route guidance to the driver by switching the display from the scenic image to a past scenic image when the user feels that visibility is poor due to fog or other reasons. Furthermore, when the user wants to see the scenery from the car window in a different season or time of day than the current one, the user can view past scenic images taken in a specified season or time of day.

[0064] (Third Example) In a third example, the predetermined condition is the visibility of the scenery ahead. If the server device 3 determines that the visibility of the scenery ahead is good, it transmits a scenery image from the in-vehicle device 1 to the external terminal 2. On the other hand, if the server device 3 determines that the visibility of the scenery ahead is poor, it transmits a past scenery image to the external terminal 2.

[0065] The server device 3 can use, for example, visibility distance (hereinafter also referred to as "visibility") as an index of visibility. For example, the server device 3 estimates visibility from a landscape image received from the vehicle-mounted device 1 using a pre-prepared image recognition model or the like, and determines whether the visibility of the landscape ahead is good or not. Note that visibility may be expressed as a distance (km) or as a visibility class. This image recognition model may be a model that has been trained in advance using training data in which a large number of landscape images are labeled with visibility.

[0066] The server device 3 may also use weather, time of day, or a combination of these as an indicator of visibility. For example, the server device 3 acquires weather information, including weather, from a weather system that provides weather information. The server device 3 then determines the visibility of the scenery ahead based on the current location of the target vehicle acquired from the in-vehicle device 1 and the weather information. The server device 3 determines that the visibility of the scenery ahead is good if the weather in the area where the target vehicle is traveling is not bad. On the other hand, the server device 3 determines that the visibility of the scenery ahead is poor if the weather in the area where the target vehicle is traveling is bad. The server device 3 also determines the visibility of the scenery ahead based on the time of day when the target vehicle is traveling. If the time of day is daytime, the server device 3 determines that the visibility of the scenery ahead is good. On the other hand, if the time of day is nighttime, the server device 3 determines that the visibility of the scenery ahead is poor.

[0067] The indicators of visibility of the scenery ahead shown above are merely examples, and the present invention is not limited to these.

[0068] 7 is a flowchart of the process in the third example. Step S301 and steps S303 to S306 are the same as the process in step S101 and steps S103 to S106 in the first example shown in FIG. 5, so a description thereof will be omitted.

[0069] The server device 3 determines whether the visibility of the scenery ahead of the target vehicle is good (step S302). If the server device 3 determines that the visibility of the scenery ahead is good (step S302: Yes), the server device 3 proceeds to step S303. On the other hand, if the server device 3 determines that the visibility of the scenery ahead is poor (step S302: No), the server device 3 proceeds to step S304.

[0070] In the third example, the server device 3 determines the visibility of the scenery ahead and switches the display on the external terminal 2 according to the determination result, thereby eliminating the need for the user to perform a switching operation.

[0071] (Fourth Example) In the fourth example, the predetermined condition is the amount of communication data of the vehicle-mounted device 1. In the fourth example, it is assumed that the vehicle-mounted device 1 has a contract with a carrier that provides communication services and has subscribed to a communication plan. It is assumed that the communication plan specifies a limit on the amount of communication data for a predetermined period and communication fees.

[0072] For example, the vehicle-mounted device 1 calculates the amount of available communication data based on the current amount of communication data consumed and the communication data limit value. When the amount of available communication data falls below a predetermined threshold TH2, the vehicle-mounted device 1 notifies the server device 3 and stops transmitting scenic images. Upon receiving the notification from the vehicle-mounted device 1, the server device 3 starts transmitting past scenic images to the external terminal 2.

[0073] 8 is a flowchart of the process in the fourth example. Step S401 and steps S403 to S406 are the same as the process in step S101 and steps S103 to S106 in the first example shown in FIG. 5, so a description thereof will be omitted.

[0074] The server device 3 determines whether the amount of available communication data has fallen below a predetermined threshold TH2 based on the notification from the vehicle-mounted device 1 (step S402). If the server device 3 receives a notification from the vehicle-mounted device 1, i.e., if the amount of available communication data has fallen below the predetermined threshold TH2 (step S402: Yes), the server device 3 proceeds to step S404. On the other hand, if the server device 3 does not receive a notification from the vehicle-mounted device 1, i.e., if the amount of available communication data is not less than the predetermined threshold TH2 (step S402: No), the server device 3 proceeds to step S403.

[0075] In the fourth example, the server device 3 can allow the user of the external terminal 2 to understand the shape of the road on which the target vehicle is traveling, while saving the amount of communication data of the in-vehicle device 1 .

[0076] [Display Example] Next, a display example of the external terminal 2 will be described.

[0077] 9A shows an example of a scenic image display mode. In FIG. 9A, an image display field 61, a map display field 62, a switching button 71, and a location mark 72 are displayed on the display screen of the external terminal 2.

[0078] The image display field 61 displays the latest scenery image (moving image) captured by the exterior camera 51 of the target vehicle. The map display field 62 displays map information of the area around the current location of the target vehicle. The map display field 62 includes a position mark 72 indicating the current location of the target vehicle. The switch button 71 is a button for selecting whether to display a scenery image or a past scenery image. The user of the exterior terminal 2 can switch between displaying a scenery image and a past scenery image by pressing the switch button 71.

[0079] (Past Scenic Image Display Mode) Fig. 9(B) is an example of a display of a past scenic image. In Fig. 9(B), an image display field 61a, a map display field 62, a switching button 71, a position mark 72, and a past mark 73 are displayed on the display screen of the external terminal 2. The image display field 61a displays a past scenic image corresponding to the scenic image of the target vehicle. The image display field 61a also includes a past mark 73 indicating that a past scenic image is being displayed.

[0080] In this manner, in the display example of the external terminal 2, the server device 3 allows the user of the external terminal 2 to understand the shape of the road on which the target vehicle is traveling by displaying a scenic image or a past scenic image on the external terminal 2. This allows the user of the external terminal 2 to provide route guidance to the driver, for example.

[0081] [Modifications] Next, suitable modifications of the above-described embodiment will be described. The following modifications may be applied to the above-described embodiment in combination.

[0082] (Variant 1) When sending a landscape image to the external terminal 2, the vehicle-mounted unit 1 and the external terminal 2 may send and receive audio data and landscape images without going through the server device 3, and may go through the server device 3 only when sending a past landscape image to the external terminal 2.

[0083] As an example, when communication quality is used as a predetermined condition in the display control process of the server device 3, the external terminal 2 calculates the error rate of communication between the onboard device 1 and the external terminal 2 and transmits the calculated error rate to the server device 3. The server device 3 then determines whether the communication quality is stable based on the communication error rate received from the external terminal 2. If the server device 3 determines that the communication quality is unstable, it notifies the onboard device 1 and starts transmitting past scenic images to the external terminal 2. Upon receiving the notification from the server device 3, the onboard device 1 stops transmitting the scenic images. Alternatively, the external terminal 2 may calculate the error rate of communication between the onboard device 1 and the external terminal 2 and determine whether the communication quality is stable. In this embodiment, if the external terminal 2 determines that the communication quality is unstable, it notifies the onboard device 1 and the server device 3. Then, upon receiving the notification from the external terminal 2, the server device 3 starts transmitting past scenic images, and upon receiving the notification from the external terminal 2, the onboard device 1 stops transmitting the scenic images.

[0084] As another example, in the display control process of the server device 3, if the predetermined condition is that a switching operation has been received from the in-vehicle device 1 or the external terminal 2, the in-vehicle device 1 or the external terminal 2 that has received the switching operation from the scenic image to the past scenic image transmits a switching request to the server device 3. The server device 3 then starts transmitting the past scenic image to the external terminal 2 and requests the in-vehicle device 1 to stop transmitting the scenic image. Alternatively, the in-vehicle device 1 that has received the switching operation from the scenic image to the past scenic image may transmit a switching request to the external terminal 2, and the external terminal 2 may transmit a request to transmit the past scenic image to the server device 3. In this configuration, when the in-vehicle device 1 receives the switching operation, the server device 3 receives the request to transmit the past scenic image from the external terminal 2, starts transmitting the past scenic image to the external terminal 2, and requests the in-vehicle device 1 to stop transmitting the scenic image. On the other hand, when the external terminal 2 receives the switching operation, the external terminal 2 transmits a request to transmit past scenery images to the server device 3, receives the past scenery images from the server device 3, and requests the in-vehicle device 1 to stop transmitting scenery images.

[0085] As another example, in the display control process of the server device 3, if the predetermined condition is the visibility of the scenery ahead, the in-vehicle device 1 or the external terminal 2 determines the visibility of the scenery ahead, and if it determines that the visibility of the scenery ahead is poor, transmits the determination result to the server device 3. The server device 3 then notifies the in-vehicle device 1 based on the determination result and starts transmitting past scenery images to the external terminal 2. Upon receiving the notification from the server device 3, the in-vehicle device 1 stops transmitting scenery images. Alternatively, if the in-vehicle device 1 determines that the visibility of the scenery ahead is poor, the external terminal 2 may transmit the determination result to the external terminal 2. In this embodiment, upon receiving the determination result from the in-vehicle device 1 that the visibility of the scenery ahead is poor, the external terminal 2 transmits a request to transmit past scenery images to the server device 3, receives past scenery images from the server device 3, and requests the in-vehicle device 1 to stop transmitting scenery images.

[0086] As another example, in the display control process of the server device 3, if the predetermined condition is the amount of communication data available to the in-vehicle device 1, the server device 3 determines whether the amount of communication data available to the in-vehicle device 1 is less than a predetermined threshold TH2. If it is determined that the amount of communication data available is less than the threshold TH2, the in-vehicle device 1 transmits the determination result to the server device 3 and stops transmitting scenic images. Then, the server device 3 starts transmitting past scenic images to the external terminal 2. Alternatively, if the in-vehicle device 1 determines that the amount of communication data available is less than the predetermined threshold TH2, the server device 3 may transmit the determination result to the external terminal 2. In this embodiment, when the external terminal 2 receives from the in-vehicle device 1 the determination result that the amount of communication data available is less than the predetermined threshold TH2, the server device 3 transmits a past scenic image transmission request to the server device 3, receives the past scenic image from the server device 3, and requests the in-vehicle device 1 to stop transmitting scenic images.

[0087] (Variant 2) In the above embodiment, the server device 3 transmits either a landscape image or a past landscape image to the external terminal 2, but the server device 3 may also display the landscape image and the past landscape image side by side based on the user's selection.

[0088] FIG. 10 shows a display example of the external terminal 2 according to the second modification. In FIG. 10 , an image display field 61b, a map display field 62, a switching button 71, a location mark 72, a current mark 74, and a past mark 75 are displayed on the display screen of the external terminal 2. The image display field 61b displays a landscape image and a past landscape image side by side. The image display field 61b includes a current mark 74 indicating that the image is a landscape image and a past mark 75 indicating that the image is a past landscape image. Note that the past mark 75 only needs to indicate that the displayed image is a past landscape image, and in FIG. 10 , the past mark 75 displays the year in which the past landscape image was captured. With a display such as that in FIG. 10 , the user of the external terminal 2 can enjoy comparing current and past landscapes.

[0089] (Variant 3) The server device 3 may extract one or more interpolated images from the past image library 6 to be inserted between the past landscape image corresponding to the currently received mobile object information and the past landscape image corresponding to the next mobile object information to be received, and display them on the off-vehicle terminal 2.

[0090] Specifically, the server device 3 estimates a position (hereinafter also referred to as an "interpolated position") interpolated between the current position of the target vehicle and the next position of the target vehicle at every predetermined time T2. Then, the server device 3 extracts a past scenery image corresponding to the interpolated position from the past image library 6. The server device 3 displays the extracted past scenery image as an interpolated image on the off-vehicle terminal 2. Here, the predetermined time T2 is a predetermined numerical value that is set to be greater than 0 and smaller than the update interval (predetermined time T1) of the mobile object information. Furthermore, the next position of the target vehicle is the positioning position of the mobile object information to be received next time. Furthermore, the interpolated position is an example of a "corrected position."

[0091] FIG. 11 is a diagram illustrating a method for estimating an interpolated position. The server device 3 calculates a distance D based on a predetermined time T2 and the current traveling speed of the target vehicle. The server device 3 then estimates a position where the target vehicle has moved the distance D from its current position in the current traveling direction as an interpolated position 1. The server device 3 extracts a past scenery image corresponding to the interpolated position 1 from the past image library 6 and displays it on the off-vehicle terminal 2. The server device 3 then estimates a position where the target vehicle has moved the distance D from the interpolated position 1 in the current traveling direction as an interpolated position 2. The server device 3 extracts a past scenery image corresponding to the interpolated position 2 from the past image library 6 and displays it on the off-vehicle terminal 2. In this way, the server device 3 extracts an interpolated image at every predetermined time T2 until the next time the server receives moving object information, and displays it on the off-vehicle terminal 2.

[0092] This allows the server device 3 to display the past scenery images on the external terminal 2 as smooth moving images.

[0093] (Modification 4) The server device 3 may extract past scenic images taking into account processing delays and communication delays.

[0094] 12A and 12B are diagrams illustrating the process for displaying each image and the locations of delays. FIG. 12A shows the process for displaying a past scenery image. In FIG. 12A, locations that require time (hereinafter also referred to as "delay") are indicated by dotted rectangles, and locations that do not require time (i.e., have little impact on the overall processing time) are indicated by solid rectangles. As can be seen from FIG. 12A, delays occur in the calculation of the position by the in-vehicle device 1, the transmission of moving object information from the in-vehicle device 1 to the server device 3, and the extraction of the past scenery image by the server device 3 until the past scenery image is displayed on the external terminal 2.

[0095] 12(B) shows the flow up to the display of a scenic image. In FIG. 12(B), as in FIG. 12(A), delayed portions are indicated by dotted rectangles, and portions that do not require time are indicated by solid rectangles. As can be seen from FIG. 12(B), a delay occurs in the encoding of the scenic image (moving image) by the in-vehicle device 1 until the scenic image is displayed on the external terminal 2.

[0096] 12A and 12B show that, compared to landscape images, there are many places where delays occur in the past landscape images before they are displayed on the external terminal 2. Therefore, the display timing of the past landscape images on the external terminal 2 is delayed compared to the display timing of the landscape images. Therefore, the server device 3 corrects the delays described above in order to match the display timing of the past landscape images with the display timing of the landscape images.

[0097] Specifically, first, the server device 3 calculates the difference between the display delay time of the past scenic image and the display delay time of the scenic image. Note that the display delay time of the past scenic image includes a position coordinate calculation processing delay V1, a communication delay W1, and a past scenic image extraction processing delay X1, as shown in FIG. 12A. Also, the display delay time of the scenic image includes an encoding processing delay V2, as shown in FIG. 12B. Then, the server device 3 calculates the difference in display delay time based on the following formula: Difference in display delay time=(V1-V2)+W1+X1 (1)

[0098] Here, the server device 3 applies a theoretical design value for the position coordinate calculation process as the position coordinate calculation process delay V1, and a theoretical design value for the encoding process as the encoding process delay V2. The server device 3 also applies a difference in time between the timestamp assigned by the vehicle-mounted device 1 when transmitting the mobile object information and the time the server device 3 receives the mobile object information as the communication delay W1. The server device 3 also applies a theoretical design value for the longest time of the past scenery image extraction process as the past scenery image extraction process delay X1.

[0099] Next, in the process of extracting past scenery images, the server device 3 regards a position that is a distance (travel speed x difference in display delay time) from the current position in the travel direction as the current position, and extracts past scenery images corresponding to that position from the past image library 6. If a route is set in advance, the server device 3 regards a position (travel speed x difference in display delay time) from the current position along the route as the current position. Then, the server device 3 causes the external terminal 2 to display the extracted past scenery images.

[0100] In this way, according to this modified example, the timing of displaying the past scenery image on the external terminal 2 can be matched to the timing of displaying the scenery image, so that, for example, when the display switches between the past scenery image and the scenery image, the positional shift of the scenery can be reduced.

[0101] Instead of the difference in display delay time, the server device 3 may use the difference (hereinafter also referred to as the "display time difference") between the time required from when the vehicle carrying the on-board device 1 moves and reaches a first position until a past scenic image corresponding to the first position is displayed on the off-board terminal 2, and the time required from when the vehicle carrying the on-board device 1 moves and reaches the first position until a scenic image captured by the on-board device 1 at the first position is displayed on the off-board terminal 2. In this case, the server device 3 regards a position that is a distance (travel speed x display time difference) in the movement direction from the current position as the current position, and extracts a past scenic image corresponding to that position.

[0102] Alternatively, the server device 3 may correct the delay by using only the display delay time of the past scenery image instead of the difference in the display delay time. In this case, the server device 3 considers a position that is a distance from the current position in the moving direction equal to "moving speed x display delay time of the past scenery image" as the current position, and extracts a past scenery image corresponding to that position. The display delay time of the past scenery image is an example of "the time required for the process of extracting the past scenery image." This allows the server device 3 to display, on the off-vehicle terminal 2, a past scenery image that corresponds to the scenery currently being viewed by the driver of the target vehicle.

[0103] The delay correction methods described above are merely examples, and the present invention is not limited to these.

[0104] As described above, the server device 3 functions as an information processing device and includes an acquisition means, an image extraction means, and a display control means. The acquisition means acquires mobile object information including the current position, moving direction, and moving speed of the mobile object. The image extraction means extracts a past scenery image corresponding to the surrounding scenery at the current position of the mobile object from past images, which are images captured in the past, based on the current position, moving direction, and moving speed. The display control means displays the past scenery image on a terminal device located outside the mobile object. This allows the user of the external terminal 2 to understand where the mobile object is traveling.

[0105] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a control unit, which is a computer, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., floppy disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory).

[0106] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference.

[0107] REFERENCE SIGNS LIST 1 In-vehicle device 2 External terminal 3 Server device 4 Communication network 5 Map DB 11, 21, 31 Communication unit 12, 22, 32 Storage unit 13 Input unit 14, 24, 33 Control unit 15, 25 Sensor group 16, 26 Display unit 17, 27 Sound output unit

Claims

1. An acquisition means for acquiring information about a moving object, including its current position, direction of movement, and speed of movement. Image extraction means that identifies a position obtained by moving from the current position in the direction of movement by the distance traveled in a predetermined time at the aforementioned moving speed as a corrected position, and extracts the past image associated with the corrected position from past images, which are images taken in the past, as a past landscape image. A display control means for displaying the aforementioned past landscape image on a terminal device located outside the mobile body, An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the predetermined time is set based on the update interval of the mobile information.

3. The information processing apparatus according to claim 1, wherein the predetermined time is set based on the time required for the image extraction means to extract the past landscape image.

4. The acquisition means acquires a current landscape image taken from the moving object at the current location, The terminal device is equipped with a display image determination means that determines an image to be displayed on the terminal device from among the current landscape image and the past landscape image. The information processing apparatus according to claim 1, wherein the display control means causes the image determined by the display image determination means to be displayed on the terminal device.

5. The information processing apparatus according to claim 4, wherein the predetermined time is set based on the difference between the past image display time required from the time the current position of the moving body reaches the first position until the past landscape image corresponding to the first position is displayed on the terminal device, and the current image display time required from the time the current position of the moving body reaches the first position until the current landscape image taken from the moving body at the first position is displayed on the terminal device.

6. An information processing device according to any one of claims 1 to 5, used in a system that mediates communication between the occupants of a mobile vehicle and a user using the terminal device outside the mobile vehicle.

7. A control method performed by a computer, An acquisition step to acquire information about a moving object, including its current position, direction of movement, and speed of movement, The image extraction process involves identifying a position that is moved from the current position in the direction of movement by the distance traveled in a predetermined time at the aforementioned movement speed as a corrected position, and extracting the past image associated with the corrected position from past images, which are images taken in the past, as a past landscape image. A display control step that causes the aforementioned past landscape image to be displayed on a terminal device located outside the mobile body, A control method having

8. A program that is executed by a computer, An acquisition means for acquiring information about a moving object, including its current position, direction of movement, and speed of movement. Image extraction means that identifies a position obtained by moving from the current position in the direction of movement by the distance traveled in a predetermined time at the aforementioned moving speed as a corrected position, and extracts the past image associated with the corrected position from past images, which are images taken in the past, as a past landscape image. A program that causes the computer to function as a display control means for displaying the aforementioned past landscape images on a terminal device located outside the mobile body.

9. A storage medium characterized by storing the program described in claim 8.