Information processing device, control method, program, and storage medium

Abstract

To provide an information processing device capable of suitably mediating the communication between an occupant of a mobile body and a user existing outside the mobile body.SOLUTION: An in-vehicle device 1, an off-vehicle terminal 2, or a server device 5 is an information processing device used in a system mediating the communication between an occupant of a vehicle being a mobile body and a user existing outside the vehicle. The information processing device comprises acquisition means, processing means, and display control means. The acquisition means acquires a peripheral captured image obtained by capturing the periphery of the vehicle from the vehicle. The processing means subjects a predetermined non-stationary object to image processing among the objects existing in the peripheral captured image. The display control means displays the peripheral captured image having been subjected to image processing on the terminal device used by the user.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to a system for mediating communication between a passenger of a vehicle and a user outside the vehicle. [Background technology]

[0002] Conventionally, there are known technologies for enabling a conversation between a vehicle occupant and a person outside the vehicle. For example, Patent Document 1 discloses a conversation providing system that enables a conversation between a vehicle driver and a conversation partner by communicating with an in-vehicle system via a communication network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-138277 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a system in which communication between the occupants and people outside the vehicle is carried out, delays due to communication delays and processing delays can affect communication between the occupants and people outside the vehicle.

[0005] In view of the above-mentioned problems, one object of the present invention is to provide an information processing device that can preferably mediate communication between occupants of a moving body and a user outside the moving body. [Means for solving the problem]

[0006] The invention described in claim 1 is An information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body, comprising: An acquisition means for acquiring a surrounding image obtained by capturing an image of the surroundings of the moving object; a processing means for performing image processing on a predetermined non-stationary object among objects present in the peripheral photographed image; a display control means for displaying the processed peripheral photographed image on a terminal device used by the user; The present invention is characterized by having the following.

[0007] The invention described in claim 9 is as follows: A control method executed by an information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body, comprising: An acquisition step of acquiring a surrounding image obtained by capturing an image of the surroundings of the moving object; a processing step of performing image processing on a predetermined non-stationary object among objects present in the peripheral photographed image; a display control step of displaying the processed peripheral photographed image on a terminal device used by the user; The present invention is characterized by having the following.

[0008] The invention described in claim 10 is as follows: A program executed by a computer of an information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body, comprising: An acquisition means for acquiring a surrounding image obtained by capturing an image of the surroundings of the moving object; a processing means for performing image processing on a predetermined non-stationary object among objects present in the peripheral photographed image; A display control means for displaying the processed peripheral photographed image on a terminal device used by the user. The present invention is characterized in that it is a program that causes the computer to function as the [Brief description of the drawings]

[0009] [Figure 1] 1 shows an example of the configuration of a driving call system. [Diagram 2] 2 shows an example of a schematic configuration of an in-vehicle device. [Diagram 3] 2 shows an example of a schematic configuration of an external terminal. [Figure 4] 1A is a diagram visualizing a delay time that occurs in a driving communication system; FIG. 1B is a diagram showing a specific example of a vehicle position Pa, a vehicle position Pb, and a vehicle position Pc in the vicinity of an intersection 90; [Diagram 5] (A) shows a surrounding image captured by an on-board device at a vehicle position Pa just before the intersection, and (B) shows a surrounding image captured by an on-board device at a vehicle position Pc approaching the intersection. [Figure 6] FIG. 2 shows a functional block diagram of a control unit of the in-vehicle device. [Figure 7] 1 is a diagram showing a predicted shooting area on a surrounding captured image obtained at a vehicle position Pa just before an intersection. [Figure 8] 8 is a first example of a display image generated from the surrounding captured image shown in FIG. 7. [Figure 9] FIG. 8 is a second example of a display image generated from the surrounding captured image shown in FIG. [Figure 10] 4 is an example of a flowchart showing a procedure of a process executed by an in-vehicle device. [Figure 11] FIG. 13 shows an example of a functional block diagram of a control unit in a modified example. [Figure 12] (A) shows an image clearly indicating the predicted shooting area of ​​the displayed image when there is no steering operation, and (B) shows an image clearly indicating the predicted shooting area of ​​the displayed image when the steering wheel is turned clockwise. [Figure 13] 13 shows a configuration example of a driving call system according to a second modification. [Figure 14] 2 illustrates an example of a schematic configuration of a server device. [Figure 15] FIG. 13 is an example of a functional block diagram of a control unit of an external terminal in a third modified example. [Figure 16] 5 is a flowchart illustrating an example of a procedure of a process executed by an external terminal. [Figure 17] 13 is an example of a display image in Modification Example 5. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In one preferred embodiment of the present invention, an information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body includes an acquisition unit that acquires a peripheral image of the moving body, a processing unit that applies image processing to a predetermined non-stationary object among objects present in the peripheral image, and a display control unit that causes the peripheral image that has been subjected to the image processing to be displayed on a terminal device used by the user. With this aspect, the information processing device can suitably display on the terminal device a display image in which the image processing has been applied to the non-stationary object, taking into account a delay (time lag) that occurs in the system.

[0011] In one aspect of the information processing device, the information processing device further includes a detection unit for detecting the non-stationary object included in the peripheral captured image, and a state prediction unit for predicting a state of the non-stationary object after a predicted time of a delay occurring in the communication, and the processing unit processes the image based on the result of the predicted state. With this aspect, the information processing device can display on the terminal device a display image that has been processed by taking into account the state of the non-stationary object after a delay occurring in the system.

[0012] In another aspect of the information processing device, the state prediction means predicts, as the state, at least a position of the non-stationary object after the predicted time of the delay, and the processing means performs the image processing based on the predicted result of the position. With this aspect, the information processing device can display on the terminal device a display image that has been processed by taking into account the position of the non-stationary object after a delay caused in the system.

[0013] In another aspect of the information processing device, the information processing device further includes a determination unit that determines whether the non-stationary object is present within a shooting range of a shooting unit that generates the peripheral shot image after the predicted delay time based on the position prediction result, and the processing unit performs the image processing based on the result of the determination. With this aspect, the information processing device can display on the terminal device a display image in which the non-stationary object has been subjected to image processing based on whether the non-stationary object is present within a shooting range after a delay caused in the system.

[0014] In another aspect of the information processing device, the processing means processes the image so that the non-stationary object determined not to exist within the shooting range is made invisible. With this aspect, the information processing device can prevent unnecessary information from being presented to the user of the terminal device by displaying on the terminal device a display image in which the non-stationary object that does not need to be considered in driving is made invisible.

[0015] In another aspect of the information processing device, the processing means processes the image so as to highlight the non-stationary object determined to be present within the shooting range. With this aspect, the information processing device causes the terminal device to display a display image highlighting the non-stationary object that requires attention while driving, and can draw the user's attention to the non-stationary object.

[0016] In another aspect of the information processing device, the information processing device further includes a steering information acquisition unit that acquires steering information related to a steering operation of the moving object, and the determination unit performs the determination based on the steering information and the position prediction result. With this aspect, even if a steering operation is performed, the information processing device can accurately determine whether or not a non-stationary object is present within the shooting range after a delay caused by the non-stationary object in the system.

[0017] In another aspect of the information processing device, when the non-stationary object is a traffic light, the state prediction means predicts a signal display of the traffic light after the predicted delay time, and the processing means performs the image processing so as to display information on the predicted result of the signal display in association with the traffic light. With this aspect, the information processing device can cause a display image that accurately represents the signal display of the traffic light after a delay occurring in the system to be displayed on the terminal device.

[0018] In another preferred embodiment of the present invention, a control method is executed by an information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body, the control method including: an acquisition step of acquiring a peripheral photographed image of the moving body, a processing step of performing image processing on a predetermined non-stationary object among objects present in the peripheral photographed image, and a display control step of displaying the peripheral photographed image that has been subjected to the image processing on a terminal device used by the user. By executing this control method, the information processing device can suitably generate a display image to be displayed on the terminal device.

[0019] In another preferred embodiment of the present invention, a program is executed by a computer of an information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body, and causes the computer to function as an acquisition means for acquiring a peripheral photographed image of the moving body, a processing means for performing image processing on a predetermined non-stationary object among objects present in the peripheral photographed image, and a display control means for displaying the peripheral photographed image that has been subjected to the image processing on a terminal device used by the user. By executing this program, the computer of the information processing device can preferably generate a display image to be displayed on the terminal device. Preferably, the program is stored in a storage medium. EXAMPLES

[0020] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0021] (1) System Configuration FIG. 1 shows a configuration example of a driving call system according to an embodiment. The driving call system has an in-vehicle device 1 provided in a vehicle and an external terminal 2 used by a person outside the vehicle. In the driving call system, the in-vehicle device 1 and the external terminal 2 communicate with each other to mediate a voice call between the driver of the vehicle and the user of the external terminal 2, and the user of the external terminal 2 talks to the driver of the vehicle while checking various information obtained from the vehicle through the external terminal 2, and issues instructions such as warnings. In this case, in this embodiment, the in-vehicle device 1 appropriately adjusts the image displayed on the external terminal 2 to reduce the effect of delays (including processing delays and communication delays) that occur in the driving call system in communication between users.

[0022] The vehicle-mounted device 1 moves with the vehicle and performs processing for realizing a call between the driver of the vehicle and the user of the external terminal 2. Hereinafter, the vehicle equipped with the vehicle-mounted device 1 is also called the "target vehicle". The vehicle-mounted device 1 performs data communication with the external terminal 2 via a communication network 3 such as the Internet or a dedicated communication network. Data exchanged between the vehicle-mounted device 1 and the external terminal 2 includes an image to be displayed on the external terminal 2 (also called a "display image"), voice data generated during a call between the driver of the target vehicle and the user of the external terminal 2, and display data other than the display image for displaying information about the target vehicle or the driver. In this embodiment, the vehicle-mounted device 1 generates a display image based on an image (also called a "surrounding image") captured from the target vehicle of the surroundings of the target vehicle. At this time, the vehicle-mounted device 1 generates a display image that accurately represents the state of an object present within the capture range after a delay caused by the driving call system in order to prevent a malfunction caused by a delay caused by the driving call system.

[0023] The vehicle-mounted device 1 may be a navigation device installed in a target vehicle and providing route guidance to a set destination, or may be a mobile terminal such as a smartphone. The vehicle-mounted device 1 may also be incorporated in the target vehicle. The vehicle-mounted device 1 is an example of an "information processing device." The target vehicle is an example of a "mobile body."

[0024] The external terminal 2 is a terminal operated by a person outside the target vehicle, and performs data communication with the in-vehicle device 1 via the communication network 3. The external terminal 2 is, for example, a mobile terminal such as a smartphone. For example, the external terminal 2 establishes communication with the in-vehicle device 1 while the target vehicle is being driven, and exchanges voice data with the in-vehicle device 1 for a call between the driver and the user of the external terminal 2. In addition, the external terminal 2 receives display data from the in-vehicle device 1 during a call, and displays images based on the surrounding captured images based on the display data. The above-mentioned voice data and display data received by the external terminal 2 from the in-vehicle device 1 are hereinafter also referred to as "output data". In addition, the voice data, etc. transmitted by the external terminal 2 to the in-vehicle device 1 are also referred to as "response data". The external terminal 2 is an example of a "terminal device".

[0025] The communication between the in-vehicle unit 1 and the external terminal 2 may be realized by being relayed by a server device (not shown). Even when the in-vehicle unit 1 and the external terminal 2 establish direct communication, the in-vehicle unit 1 and / or the external terminal 2 may exchange information (e.g., communication address information) required for the in-vehicle unit 1 and the external terminal 2 to establish communication with the server device. In these cases, the server device performs processes required for the in-vehicle unit 1 and the external terminal 2 to perform data communication (including authentication processes for the in-vehicle unit 1 and the external terminal 2).

[0026] (2) Device configuration 2 shows an example of a schematic configuration of the vehicle-mounted device 1. The vehicle-mounted device 1 mainly includes a communication unit 11, a storage unit 12, an input unit 13, a control unit 14, a sensor group 15, a display unit 16, and a sound output unit 17. The elements in the vehicle-mounted device 1 are connected to each other via a bus line 10.

[0027] The communication unit 11 performs data communication with other terminals under the control of the control unit 14. The communication unit 11 may receive, for example, map data for updating a map DB (DataBase) 4 from a map management server (not shown).

[0028] The storage unit 12 is configured with various types of memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a non-volatile memory (including a hard disk drive, a flash memory, etc.). The storage unit 12 stores programs for the in-vehicle device 1 to execute predetermined processing. The above-mentioned programs may include an application program for making a call, and an application program for causing the external terminal 2 to display a map or an image captured in the target vehicle. The storage unit 12 is also used as a working memory for the control unit 14. The programs executed by the in-vehicle device 1 may be stored in a storage medium other than the storage unit 12.

[0029] The storage unit 12 also stores a map DB4. Various data necessary for route guidance are recorded in the map DB4. The map DB4 is data necessary for map display based on a predetermined position such as the current position of the target vehicle. The map DB4 is a database including, for example, road data that represents a road network by a combination of nodes and links, and facility data that indicates facilities that are candidates for a destination, a stop-off point, or a landmark. The map DB4 may be updated based on information received by the communication unit 11 from a map management server under the control of the control unit 14.

[0030] The input unit 13 is a button, a touch panel, a remote controller, a voice input device, etc. for the user to operate. The display unit 16 is a display or the like that displays information based on the control of the control unit 14. The sound output unit 17 is a speaker or the like that outputs sound based on the control of the control unit 14.

[0031] The sensor group 15 includes various sensors that perform sensing related to the state of the target vehicle or the environment outside the vehicle. The sensor group 15 includes an exterior camera 51, a driver camera 52, and a vehicle behavior detector 53.

[0032] The exterior camera 51 is one or more cameras that capture images of the outside of the target vehicle, such as the area in front of the target vehicle, and generates surrounding images captured at a predetermined time interval. The surrounding images may be a composite image of images output by multiple exterior cameras 51, or may be an image cut out from a predetermined range of an image captured by a camera (including a composite image of multiple cameras).

[0033] The driver's camera 52 is a camera provided so as to include the driver's face within its imaging range, and generates images captured at predetermined time intervals.

[0034] The vehicle behavior detector 53 generates detection signals indicating the behavior of the target vehicle, such as the current position, vehicle speed, acceleration, steering angle, etc. The vehicle behavior detector 53 includes, for example, a Global Navigation Satellite System (GNSS) receiver, a gyro sensor, an Inertial Measurement Unit (IMU), a vehicle speed sensor, an acceleration sensor, a steering angle sensor, and the like.

[0035] In addition, the sensor group 15 may include various external sensors (including cameras, lidars, radars, ultrasonic sensors, infrared sensors, sonar, etc.) and internal sensors in addition to the exterior camera 51, the driver camera 52, and the vehicle behavior detector 53.

[0036] The control unit 14 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire vehicle-mounted device 1. The control unit 14 functions as an "acquisition means," a "processing means," a "display control means," a computer that executes a program, etc.

[0037] The processing executed by the control unit 14 is not limited to being realized by software using a program, but may be realized by any combination of hardware, firmware, and software. The processing executed by the control unit 14 may be realized by using an integrated circuit programmable by a user, such as a field-programmable gate array (FPGA) or a microcomputer. In this case, the program executed by the control unit 14 in this embodiment may be realized by using this integrated circuit.

[0038] The configuration of the vehicle-mounted device 1 shown in FIG. 2 is an example, and various modifications may be made to the configuration shown in FIG. 2. For example, instead of storing the map DB 4 in the memory unit 12, the control unit 14 may receive map information from a map management server (not shown) via the communication unit 11. In another example, at least one of the input unit 13, the display unit 16, and the sound output unit 17 may be provided in the target vehicle as an external device of the vehicle-mounted device 1, and may supply the generated signal to the vehicle-mounted device 1. Also, at least some of the sensors in the sensor group 15 may be sensors provided in the target vehicle. In this case, the vehicle-mounted device 1 may acquire information output by a sensor provided in the target vehicle from the target vehicle based on a communication protocol such as CAN (Controller Area Network).

[0039] 3 shows an example of a schematic configuration of the external terminal 2. The external terminal 2 mainly includes a communication unit 21, a storage unit 22, an input unit 23, a control unit 24, a sensor group 25, a display unit 26, and a sound output unit 27. The elements in the external terminal 2 are connected to each other via a bus line 20.

[0040] The communication unit 21 performs data communication with other terminals under the control of the control unit 24. The storage unit 22 is composed of various types of memory such as RAM, ROM, and non-volatile memory. The storage unit 22 stores programs for the external terminal 2 to execute predetermined processes. The above-mentioned programs may include application programs for calling the driver and displaying information related to the driving of the target vehicle of the in-vehicle device 1 (including displaying maps and various images) when communication with the in-vehicle device 1 is established. The storage unit 22 is used as a working memory for the control unit 24. The programs executed by the external terminal 2 may be stored in a storage medium other than the storage unit 22.

[0041] The input unit 23 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 26 is a display or the like that displays information based on the control of the control unit 24. The sound output unit 27 is a speaker or the like that outputs sound based on the control of the control unit 24. The sensor group 25 includes an internal sensor that senses the state of the exterior terminal 2 and an external sensor that senses the state of the external world of the exterior terminal 2.

[0042] The control unit 24 includes a CPU, a GPU, and the like, and controls the entire vehicle exterior terminal 2. Note that the configuration of the vehicle exterior terminal 2 shown in Fig. 3 is an example, and various modifications may be made to the configuration shown in Fig. 3.

[0043] (3) System Delays Explained Next, a delay (also called "system delay") that occurs in the driving call system will be described. In this embodiment, the system delay corresponds to the time length (excluding the user's reaction time) from capturing the surrounding captured image to outputting the response data from the vehicle-mounted device 1 when the vehicle-mounted device 1 outputs response data from the vehicle-mounted device 2 in response to the output from the vehicle-mounted device 2 based on the surrounding captured image.

[0044] 4(A) is a diagram visualizing the delay that occurs in the driving call system from the generation of the surrounding captured image to the output of the response data by the vehicle-mounted device 1. As shown in FIG. 4(A), the vehicle-mounted device 1 generates an encoding delay "X1" required for encoding the surrounding captured image output by the exterior shooting camera 51, and a decoding delay "Z2" required for decoding the response data (audio data) received from the exterior terminal 2 to output it by the sound output unit 17. In addition, in the communication between the vehicle-mounted device 1 and the exterior terminal 2, a communication delay "Y1" required for transmitting the output data from the vehicle-mounted device 1 to the exterior terminal 2, and a communication delay "Y2" required for transmitting the response data from the exterior terminal 2 to the vehicle-mounted device 1 occur. Furthermore, in the external terminal 2, a decoding delay "Z1" occurs which is required for decoding the output data received by the external terminal 2 from the vehicle-mounted unit 1 so as to be output by the display unit 26 and the sound output unit 27, and an encoding delay "X2" occurs which is required for encoding the data generated by the audio input device (such as a microphone) of the input unit 23.

[0045] For ease of explanation, the position of the target vehicle at the time when the vehicle exterior photographing camera 51 photographs the surroundings image (i.e., the photographing position of the surroundings image photographed by the vehicle exterior photographing camera 51) is denoted as "vehicle position Pa." The position of the target vehicle at the time when the output data based on the generated surroundings photographed image is output by the display unit 26 and the sound output unit 27 is displayed as "vehicle position Pb," and the position of the target vehicle at the time when the response data generated by the input unit 23 in response to the output of the output data is output by the sound output unit 17 is displayed as "vehicle position Pc."

[0046] FIG. 4B shows a specific example of a vehicle position Pa, a vehicle position Pb, and a vehicle position Pc in the vicinity of the intersection 90.

[0047] In this case, in the transition from the vehicle position Pa at the time of image capture to the vehicle position Pb at the time of image display, an encoding delay X1, a communication delay Y1, and a decoding delay Z1 occur, and the total time of these delays is X1+Y1+Z1 It becomes.

[0048] On the other hand, in the transition from the vehicle position Pb at the time of image display to the vehicle position Pc at the time of response output, an encoding delay X2, a communication delay Y2, and a decoding delay Z2 occur, and the total time of these delays is X2+Y2+Z2 It becomes.

[0049] The system delay corresponds to the time length from the time the image is captured to the time the response is output, i.e., the time length it takes for the target vehicle to travel from vehicle position Pa to vehicle position Pc, and is the sum of the above-mentioned delays: X1+Y1+Z1+X2+Y2+Z2

[0050] The user of the external terminal 2 will be required to give instructions to the driver of the target vehicle by viewing an image of the surroundings (i.e., an image taken from behind the distance from vehicle position Pa to vehicle position Pc) that is effectively delayed by the above-mentioned system delay from the current state of the target vehicle being driven by the driver, which may result in communication problems.

[0051] As a specific example, a case will be considered in which the user of the external terminal 2 instructs the target vehicle to turn right at the intersection 90 in the example of FIG. 4(B).

[0052] In this case, the user of the external terminal 2 looks at the display image based on the surrounding captured image captured at the vehicle position Pa and makes an utterance instructing the target vehicle to turn right at the intersection 90. At the time of this utterance (image display time), the target vehicle has actually already reached vehicle position Pb. Then, response data corresponding to the above utterance is transmitted from the external terminal 2 to the vehicle-mounted device 1, and at the time of response output when the vehicle-mounted device 1 outputs the above utterance, the target vehicle has reached vehicle position Pc. However, at vehicle position Pc, the target vehicle has already approached the intersection 90 and is unable to turn right, resulting in a communication failure caused by a system delay between the user of the vehicle-mounted device 1 and the user of the external terminal 2.

[0053] 5A shows an image 60A, which is a surrounding captured image generated by the vehicle-mounted device 1 at a vehicle position Pa just before the intersection 90. FIG 5B shows an image 60C, which is a surrounding captured image generated by the vehicle-mounted device 1 at a vehicle position Pc approaching the intersection 90.

[0054] 5(A) corresponds to an image visually recognized by the user of the external terminal 2 immediately before issuing a right turn instruction in the specific example described above. According to the image 60A, the target vehicle is in a position where it can turn right at the intersection 90. Therefore, the user of the external terminal 2 gives an utterance instructing the target vehicle to turn right at the intersection 90.

[0055] On the other hand, image 60C shown in Fig. 5(B) corresponds to the view ahead of the target vehicle in the above specific example at the time of response output when the driver hears a right turn instruction from the user of the external terminal 2. In this case, the target vehicle is approaching the intersection 90 as shown in image 60C shown in Fig. 5(B), so the driver cannot drive the target vehicle in accordance with the instruction of the user of the external terminal 2.

[0056] (4) Generate display images Next, the generation of a display image by the vehicle-mounted device 1 will be described. In summary, the vehicle-mounted device 1 detects a non-stationary object included in a surrounding captured image, and generates a display image by applying image processing to the detected non-stationary object. In this way, the vehicle-mounted device 1 suitably generates a display image by processing the surrounding captured image so that the non-stationary object matches the state after the system delay.

[0057] Here, a "non-stationary object" is an object (i.e., a specific type of object) that is predetermined as an object whose state can change within a short period of time, and may be a stationary object that displays a display that needs to be referred to while driving, or a moving object that requires attention while driving. Examples of stationary non-stationary objects include traffic lights. Examples of moving non-stationary objects include pedestrians and cars (including bicycles).

[0058] Fig. 6 is an example of a functional block diagram of the control unit 14 of the vehicle-mounted device 1. The control unit 14 functionally includes a surrounding captured image acquisition unit 41, a speed information acquisition unit 42, a delay time information acquisition unit 43, a display image generation unit 44, and a transmission control unit 45. Note that in Fig. 6, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to that shown in Fig. 6. The same applies to other functional block diagrams described later.

[0059] The surrounding captured image acquisition unit 41 sequentially acquires surrounding captured images captured at a predetermined time interval by the exterior vehicle image capturing camera 51. The surrounding captured image acquisition unit 41 supplies the acquired surrounding captured images to the display image generation unit 44. The surrounding captured image acquisition unit 41 is an example of an "acquisition means".

[0060] The speed information acquisition unit 42 acquires vehicle speed information of the target vehicle. In this case, for example, the speed information acquisition unit 42 generates vehicle speed information indicating the speed of the target vehicle based on an output signal of a vehicle speed sensor or the like included in the vehicle behavior detector 53. The speed information acquisition unit 42 supplies the acquired vehicle information to the display image generation unit 44.

[0061] The delay time information acquisition unit 43 generates delay time information indicating a predicted system delay time (also simply referred to as "delay time") and supplies the generated delay time information to the display image generation unit 44. Here, the delay time corresponds to the total time (X1+Y1+Z1+X2+Y2+Z2) of each delay time in FIG. 5(A). For example, the delay time information acquisition unit 43 stores the generated delay time information in the storage unit 13 or the like and acquires the delay time information from the storage unit 13. In this case, the delay time information acquisition unit 43 may generate the delay time information and store it in the storage unit 13 immediately after the in-vehicle device 1 and the external terminal 2 establish communication, or may generate or update the delay time information at predetermined time intervals while the in-vehicle device 1 and the external terminal 2 establish communication.

[0062] Here, a specific example of a method for generating delay time information will be described.

[0063] In a first specific example, the in-vehicle device 1 and the external terminal 2 transmit timestamps to each other, calculate the difference between the timestamp received by each device and the timestamp generated at the time of reception, and the difference calculated by the external terminal 2 is received by the in-vehicle device 1, thereby calculating the sum of the above-mentioned differences as the delay time. Note that image or audio data may be transmitted together with the timestamp, and the processing time for encoding and decoding the image or audio data may be added to the above-mentioned difference.

[0064] In the second specific example, the vehicle-mounted device 1 transmits sound data indicating a specific sound (preferably inaudible range) to the external terminal 2 as dummy output data to cause the external terminal 2 to output the sound, and receives sound data generated by the input unit 23 of the external terminal 2 from the sound output by the external terminal 2 as response data from the external terminal 2. Then, the vehicle-mounted device 1 measures the time from the start of encoding the dummy output data to the end of decoding the response data as the delay time. The measured value may be shared between the vehicle-mounted device 1 and the external terminal 2. In addition, when the measurement based on the second specific example cannot be performed due to the influence of the echo canceller, for example, the external terminal 2 may set the function of the echo canceller to off during measurement. In another example, the vehicle-mounted device 1 may set the specific sound to be output to a sound of a specific frequency to which the echo canceller is not applied.

[0065] The vehicle-mounted device 1 may receive delay time information generated by the external terminal 2 from the external terminal 2. Alternatively, the vehicle-mounted device 1 and the external terminal 2 may each generate delay time information, and an average value of the delay times indicated by the delay time information may be used.

[0066] The display image generation unit 44 generates a display image based on the surrounding captured image acquired from the surrounding captured image acquisition unit 41, the speed information acquired from the speed information acquisition unit 42, and the delay time information acquired from the delay time information acquisition unit 43, and supplies the generated display image to the transmission control unit 45.

[0067] In this case, the display image generating unit 44 executes a process of detecting image areas of non-stationary objects in the surrounding captured images (also called "non-stationary object detection process"), a process of predicting the state of each non-stationary object after a delay time has elapsed (also called "state prediction process"), a process of determining whether each non-stationary object is present within the shooting range of the exterior shooting camera 51 after a delay time has elapsed (also called "continuous shooting determination process"), and a process of processing each non-stationary object in the surrounding captured images (also called "image processing process"). The display image generating unit 44 is an example of a "processing means".

[0068] In the non-stationary object detection process, the display image generating unit 44 recognizes the image area and type (class) of the non-stationary object included in the surrounding captured image based on the surrounding captured image. In this case, the display image generating unit 44 may recognize the image area and type of the non-stationary object included in the surrounding captured image based on any image recognition (pattern recognition) technology.

[0069] For example, the display image generating unit 44 may detect a non-stationary object from the surrounding captured image using a deep learning model that has been machine-learned. In this case, the deep learning model is a model that has been machine-learned to output an inference result indicating the image area and type of a non-stationary object in an input image when the image is input, and model information of the deep learning model (including learned parameters) is stored in advance in the storage unit 13 or the like. Then, the display image generating unit 44 inputs the surrounding captured image supplied from the surrounding captured image acquiring unit 41 to the deep learning model, thereby acquiring the inference result output by the deep learning model (i.e., information indicating the image area and type of a non-stationary object). This deep learning model may be, for example, a deep learning model having any architecture used for semantic segmentation, instance segmentation, or the like.

[0070] In the state prediction process, the display image generating unit 44 estimates the state of each non-stationary object after the delay time has elapsed. For example, the display image generating unit 44 detects an image area of ​​each non-stationary object from each of a predetermined number of time-series surrounding captured images obtained immediately before, and estimates the position of each non-stationary object after the delay time has elapsed based on the time-series detection results. In this case, for example, the display image generating unit 44 may specify the three-dimensional position of the non-stationary object from the time-series surrounding captured images based on a three-dimensional restoration technique such as SfM (Structue from Motion). In another example, the display image generating unit 44 may estimate the position of each non-stationary object in the image after the delay time has elapsed using any tracking technique such as a Kalman filter. In yet another example, the display image generating unit 44 may calculate the relative moving speed of each non-stationary object with respect to the target vehicle from the time-series detection results, and estimate the position of each non-stationary object after the delay time has elapsed based on the calculated moving speed and the speed information of the target vehicle. In addition, when the non-stationary object is a stationary object, the display image generating unit 44 may consider the non-stationary object to be present at a fixed position without performing the above-mentioned position estimation. The position of each non-stationary object predicted in the state prediction process is also simply called a "predicted position."

[0071] Furthermore, if the non-stationary object is a traffic light, the display image generating unit 44 predicts the signal display (display color) of the traffic light after the delay time has elapsed. In this case, the display image generating unit 44 acquires information on the signal display of the traffic light present in front of the target vehicle (e.g., waiting time for a red light) using, for example, a traffic light information utilization driving support system (TSPS: Traffic Signal Prediction Systems). The display image generating unit 44 receives information sent from a roadside transmitter / receiver or optical beacon on the road, for example, by an ETC2.0 in-vehicle device or optical beacon. Then, the display image generating unit 44 predicts the signal display of the traffic light after the delay time has elapsed based on the received information.

[0072] In the continuous shooting determination process, the display image generating unit 44 determines whether or not each non-stationary object is present within the shooting range of the outside-vehicle shooting camera 51 after the delay time has elapsed. Hereinafter, a non-stationary object determined to be present within the shooting range of the outside-vehicle shooting camera 51 after the delay time has elapsed is referred to as a "continuously present object," and a non-stationary object determined to be present outside the shooting range of the outside-vehicle shooting camera 51 after the delay time has elapsed is referred to as a "non-continuously present object." A non-continuously present object is an object that is likely to no longer be present in front of the target vehicle when the user of the outside-vehicle terminal 2 gives an instruction while looking at the surrounding captured image displayed on the outside-vehicle terminal 2, at the time the user's voice is output to the target vehicle.

[0073] In this case, for example, the display image generating unit 44 identifies a rectangular area (also called a "predicted shooting area") on the surrounding captured image that is predicted to be captured by the exterior vehicle capturing camera 51 after the delay time has elapsed, based on the speed information and delay time information. Then, the display image generating unit 44 determines that a non-stationary object whose predicted position exists within the predicted shooting area is a continuously existing object. On the other hand, the display image generating unit 44 determines that a non-stationary object whose predicted position exists outside the predicted shooting area is a non-continuously existing object.

[0074] Here, a method for determining the predicted shooting area will be explained in more detail. For example, the display image generating unit 44 identifies a pixel that is a reference point from the surrounding captured image, and sets the predicted shooting area centered on the identified pixel that is the reference point. Typically, the reference point is a vanishing point identified from the surrounding captured image based on any image processing technology. Alternatively, when the vehicle exterior capturing camera 51 is installed with its capturing direction facing the front of the target vehicle, the center point in the vertical and horizontal directions of the surrounding captured image may be set as the reference point. At this time, the display image generating unit 44 sets the size (vertical width and horizontal width) of the above-mentioned predicted shooting area to be smaller as the distance obtained by multiplying the delay time indicated by the delay time information by the speed indicated by the speed information is longer. In this case, for example, correspondence information such as a lookup table or formula indicating the correspondence between the speed of the target vehicle, the delay time, and the size to be set is stored in advance in the storage unit 12, etc., and the display image generating unit 44 refers to the correspondence information and determines the above-mentioned size from the speed and delay time of the target vehicle. The aspect ratio of the predicted shooting area may be a default value.

[0075] Instead of making a judgment regarding a continuously existing object and a non-continuously existing object based on a predicted shooting area set in the surrounding captured image, the display image generating unit 44 may specify the shooting range of the outside-vehicle shooting camera 51 after the delay time has elapsed on a three-dimensional coordinate system, and judge whether each non-stationary object is a continuously existing object or a non-continuously existing object based on the presence or absence of the predicted position of each non-stationary object in the specified shooting range. In this case, for example, the display image generating unit 44 restores the position and attitude of the outside-vehicle shooting camera 51 in a three-dimensional coordinate system by any three-dimensional restoration technology such as SfM using the time-series surrounding captured images, and then predicts the position and attitude of the outside-vehicle shooting camera 51 after the delay time has elapsed based on the speed information and delay time information. Then, the display image generating unit 44 specifies the shooting range of the outside-vehicle shooting camera 51 in the three-dimensional coordinate system based on the generated prediction result, and judges whether the predicted position of each non-stationary object exists in the specified shooting range.

[0076] In the image processing, the display image generating unit 44 processes the non-stationary object in the peripheral photographed image based on the prediction result of the state prediction processing and the determination result of the continuous photographing determination processing. For example, the display image generating unit 44 processes the image area of ​​the non-continuously existing object in the peripheral photographed image by erasing or applying a mosaic so that the non-continuously existing object cannot be visually recognized by the user in the display image. In this case, the display image generating unit 44 may erase the non-continuously existing object using any image processing technique based on the peripheral photographed image in time series obtained immediately before. Alternatively, the display image generating unit 44 may cut out only a part of the set shooting prediction area in the peripheral photographed image and set the cut-out image as the display image in order to erase the non-continuously existing object from the display image.

[0077] On the other hand, the display image generating unit 44 may regard the image area of ​​the continuously present object in the surrounding captured image as an object requiring continuous attention, and may highlight it by using a border effect or the like.

[0078] Furthermore, in the image processing, when the non-stationary object is a traffic light, the display image generating unit 44 may superimpose an image showing the traffic light display after a delay time has elapsed (also called a "predicted traffic light display image") on the image area of ​​the traffic light or its surrounding area based on the result of the state prediction processing. The predicted traffic light display image is an example of "information relating to the prediction result of the traffic light display."

[0079] The transmission control unit 45 transmits the output data including the display image supplied from the display image generation unit 44 to the external terminal 2 via the communication unit 11. In this case, the transmission control unit 45 includes the voice data generated by the input unit 13 in the output data. The transmission control unit 45 is an example of a "display control means".

[0080] 7 is a diagram showing a predicted photographing area 65 on an image 60A, which is a surrounding photographed image obtained at a vehicle position Pa just before an intersection 90 shown in FIG. 5(A). A traffic light 80, pedestrians 81 and 84, and other vehicles 82 and 83 exist as non-stationary objects in the image 60A.

[0081] In this example, when the image 60A shown in Fig. 5(A) is obtained as a surrounding captured image at the vehicle position Pa, the vehicle-mounted device 1 sets a predicted capture area 65 based on speed information measured at the vehicle position Pa and delay time information stored in the storage unit 13, etc. In addition, the vehicle-mounted device 1 calculates predicted positions of a traffic light 80, pedestrians 81 and 84, and other vehicles 82 and 83, which are non-stationary objects, and determines whether these predicted positions are included in the predicted capture area 65.

[0082] FIG. 8 is a first example of a display image generated from the surrounding captured image shown in FIG. 7. In the first example, the vehicle-mounted device 1 determines that the pedestrian 84 and the other vehicle 83 are non-continuously existing objects because the predicted positions of the pedestrian 84 and the other vehicle 83 are outside the predicted capture area 65, and performs image processing to erase the pedestrian 84 and the other vehicle 83 from the display image. On the other hand, the vehicle-mounted device 1 determines that the traffic light 80, the pedestrian 81, and the other vehicle 82 are continuous-existing objects because the predicted positions of the traffic light 80, the pedestrian 81, and the other vehicle 82 are inside the predicted capture area 65, and displays outer frames 90A, 91, and 92 for highlighting these image areas. In addition, the vehicle-mounted device 1 predicts the traffic light display after the delay time of the traffic light 80 has elapsed, and displays a predicted traffic light display image 90B showing the predicted result of the traffic light display in association with the traffic light 80.

[0083] According to the first example, the in-vehicle unit 1 can cause the external terminal 2 to display a display image that takes into account the state after the delay time has elapsed, so that the user of the external terminal 2 can give accurate instructions to the driver based on the state of the unsteady object after the delay time has elapsed.

[0084] Fig. 9 is a second example of a display image generated from the peripheral captured image shown in Fig. 7. In the first example, the vehicle-mounted device 1 applies mosaic to the image areas of the other vehicle 83 and the pedestrian 84 that have been determined to be non-continuously existing objects, thereby providing mosaic areas 93 and 94, respectively. In this way, the vehicle-mounted device 1 generates a display image in which the image areas of the non-stationary objects that are non-continuously existing objects are applied with mosaic, thereby making it possible to prevent the user of the external terminal 2 from viewing objects that do not need to be considered after the delay time has elapsed.

[0085] Fig. 10 is an example of a flowchart showing the procedure of processing executed by the vehicle-mounted device 1. The vehicle-mounted device 1 executes the processing of the flowchart shown in Fig. 10 when communication with the external terminal 2 is established and a call is started.

[0086] First, the vehicle-mounted device 1 acquires a surrounding image output by the exterior camera 51 (step S101). Furthermore, the vehicle-mounted device 1 acquires speed information of the target vehicle based on information output by the vehicle behavior detector 53, and acquires delay time information stored in the storage unit 13 or generated by an arbitrary method (step S102).

[0087] Then, the vehicle-mounted device 1 generates a display image based on the surrounding captured image acquired in step S101 and the speed information and delay time information acquired in step S102 (step S103). In this case, the vehicle-mounted device 1 sequentially executes the above-mentioned non-stationary object detection process, state prediction process, continuous shooting determination process, and image processing process to generate the display image.

[0088] Then, the in-vehicle device 1 transmits output data including the display image to the external terminal 2. Also, the in-vehicle device 1 receives response data transmitted from the external terminal 2 in response to the output data, and performs output (such as audio output) based on the response data (step S104).

[0089] Then, the vehicle-mounted device 1 determines whether the call has ended (step S105). If the vehicle-mounted device 1 determines that the call has ended (step S105; Yes), the vehicle-mounted device 1 ends the process of the flowchart. On the other hand, if the vehicle-mounted device 1 determines that the call has not ended (step S105; No), the vehicle-mounted device 1 returns the process to step S101.

[0090] (5) Variations Next, preferred modifications of the above-mentioned embodiment will be described. The following modifications may be combined and applied to the above-mentioned embodiment.

[0091] (Variation 1) The vehicle-mounted device 1 may generate a display image based on steering operation information of the target vehicle. When a steering operation is performed, the orientation of the target vehicle (i.e., the direction of the driver's line of sight) changes during a system delay in response to the steering operation. Therefore, the vehicle-mounted device 1 according to this modification generates a display image that takes into consideration such a change in the orientation of the target vehicle that occurs during a system delay.

[0092] Fig. 11 shows an example of a functional block diagram of the control unit 14 in the modified example. Functionally, the control unit 14 has a surrounding captured image acquisition unit 41, a speed information acquisition unit 42, a delay time information acquisition unit 43, a display image generation unit 44, a transmission control unit 45, and a steering information acquisition unit 46. The processes executed by the surrounding captured image acquisition unit 41, the speed information acquisition unit 42, the delay time information acquisition unit 43, the display image generation unit 44, and the transmission control unit 45 are similar to the processes described using Fig. 6, and the description thereof will be omitted as appropriate.

[0093] The steering information acquisition unit 46 acquires steering information representing the steering operation direction and operation amount (more specifically, the operation amount per unit time). For example, the steering information acquisition unit 46 acquires the steering information from the target vehicle using any communication protocol such as CAN. Then, the steering information acquisition unit 46 supplies the acquired steering information to the display image generation unit 44.

[0094] The display image generating unit 44 generates a display image based on the surrounding captured image provided by the surrounding captured image acquiring unit 41, the speed information provided by the speed information acquiring unit 42, the delay time information provided by the delay time information acquiring unit 43, and the steering information provided by the steering information acquiring unit 46. In this case, the display image generating unit 44 executes a non-stationary object detection process and a state prediction process, which are the same processes as those in the above-mentioned embodiment. In addition, in the continuous shooting determination process, the display image generating unit 44 specifies the predicted shooting range of the outside-vehicle shooting camera 51 after the delay time has elapsed based on the surrounding captured image, the speed information, the delay time information, and the steering information, and determines whether the object is a continuously present object or a non-continuously present object based on the specified shooting range. A specific example of a method for specifying the shooting range of the outside-vehicle shooting camera 51 after the delay time has elapsed will be described later. Then, the display image generating unit 44 executes an image processing process, which is the same process as those in the above-mentioned embodiment, based on the prediction result of the state prediction process and the determination result of the continuous shooting determination process, to generate a display image by processing the surrounding captured image.

[0095] Fig. 12(A) shows an image 60A (see Fig. 7) clearly showing a predicted shooting area 65M of a display image when the steering operation amount indicated by the steering information is 0 (when there is no steering operation). Fig. 12(B) shows an image 60A clearly showing a predicted shooting area 65R of a display image when the steering operation amount indicated by the steering information is a predetermined amount in the rightward operation direction (i.e., when the steering wheel is turned clockwise).

[0096] As shown in Fig. 12(A), when there is no steering operation, the vehicle-mounted device 1 sets a shooting prediction area 65M that coincides with the shooting prediction area 65 in Fig. 7 that does not take steering information into account. Then, the vehicle-mounted device 1 determines that non-stationary objects (traffic lights 80, pedestrians 81, other vehicles 82) whose predicted positions exist within the shooting prediction area 65M are continuous-existing objects, highlights these non-stationary objects, and determines that non-stationary objects whose predicted positions exist outside the shooting prediction area 65M are non-continuous-existing objects, and erases these non-stationary objects. The shooting prediction area when there is no steering operation (i.e., the shooting prediction area 65M) is also called the "standard shooting prediction area".

[0097] On the other hand, when the steering operation amount indicated by the steering information is to the right, the vehicle-mounted device 1 sets a shooting prediction area 65R that is moved to the right from the standard shooting prediction area 65M by an amount corresponding to the operation amount. In this case, the vehicle-mounted device 1 determines the size of the shooting prediction area 65R to be set in the surrounding shooting image based on the speed information and the delay time information, and determines the position (center position) of the shooting prediction area 65R based on the vanishing point of the surrounding shooting image and the steering information. For example, correspondence information such as an equation or a lookup table indicating the correspondence between the steering operation amount and the in-image movement amount from the standard shooting prediction area is stored in advance in the storage unit 12, and the vehicle-mounted device 1 sets the shooting prediction area 65R by moving the standard shooting prediction area by the in-image movement amount determined based on the correspondence information and the steering information. The display image generated in this way is an image that is approximate to the surrounding shooting image acquired after the delay time has elapsed. Then, the vehicle-mounted device 1 determines that the non-stationary objects (traffic light 80, pedestrians 81 and 84, other vehicle 82) whose predicted positions are present within the predicted shooting area 65R are continuous-present objects, and highlights these non-stationary objects. On the other hand, the vehicle-mounted device 1 determines that the non-stationary objects whose predicted positions are present outside the predicted shooting area 65R are non-continuously present objects, and erases these non-stationary objects. Thus, in the example shown in FIG. 12(B), the pedestrian 84 is determined to be a continuous-present object due to the operation of the steering wheel.

[0098] Similarly, when the steering operation amount indicated by the steering information is to the left, the vehicle-mounted device 1 sets a predicted shooting area that is moved to the left from the standard predicted shooting area 65M by an amount corresponding to the operation amount.

[0099] In this modification, the vehicle-mounted device 1 may generate a display image based on angular velocity information indicating a change in the angular velocity of the target vehicle instead of the steering operation information of the target vehicle. That is, the steering information acquisition unit 46 may acquire angular velocity information indicating a change in the angular velocity of the target vehicle as steering information related to the steering operation. Specifically, the vehicle-mounted device 1 acquires angular velocity information from the output of a gyro sensor provided in the vehicle behavior detector 53. Then, the imaging prediction area may be set in the same manner as in the case where the steering operation direction and operation amount are acquired as the steering information.

[0100] According to this modification, the vehicle-mounted device 1 can suitably generate a display image that takes into consideration a change in the orientation of the target vehicle caused by a steering operation that occurs during a system delay.

[0101] (Variation 2) At least a part of the processing executed by the vehicle-mounted device 1 may be executed by a server device that performs data communication with the vehicle-mounted device 1 and the external terminal 2.

[0102] 13 shows a configuration example of a driving call system according to Modification 2. The driving call system includes an in-vehicle device 1A, an external terminal 2, and a server device 5. The in-vehicle device 1A and the server device 5, and the external terminal 2 and the server device 5, respectively, perform data communication via a communication network 3.

[0103] The vehicle-mounted device 1A has the same configuration (see FIG. 2) as the vehicle-mounted device 1 described in the above embodiment. When the server device 5 performs processing based on the map DB 4, the vehicle-mounted device 1A does not need to have the map DB 4. The vehicle-mounted device 1A transmits to the server device 5 an upload signal including information output by the sensor group 15, input information input by the input unit 13, and the like.

[0104] The server device 5 relays data necessary for a voice call between the in-vehicle device 1A and the external terminal 2. During a voice call, the server device 5 generates output data including a display image based on an upload signal including a surrounding captured image received from the in-vehicle device 1A, and transmits the generated output data to the external terminal 2. The server device 5 also receives response data generated by the external terminal 2, and transmits the received response data to the in-vehicle device 1A. The time required for the server device 5 to execute the above-mentioned relay-related process is included in the communication delay Y1 shown in FIG. 4 in the case of relaying output data supplied from the in-vehicle device 1 to the external terminal 2, and is included in the communication delay Y2 shown in FIG. 4 in the case of relaying response data supplied from the external terminal 2 to the in-vehicle device 1.

[0105] 14 shows an example of a schematic configuration of the server device 5. The server device 5 mainly includes a communication unit 41, a storage unit 42, and a control unit 44. The elements in the server device 5 are connected to each other via a bus line 40.

[0106] The communication unit 41 performs data communication with external devices such as the in-vehicle unit 1A and the external terminal 2 under the control of the control unit 44. The storage unit 42 is composed of various types of memory such as RAM, ROM, and non-volatile memory (including a hard disk drive, flash memory, etc.). The storage unit 42 stores programs for the server device 5 to execute predetermined processes. The storage unit 42 also includes a map DB4. The control unit 44 includes a CPU, a GPU, etc., and controls the entire server device 5. The control unit 44 also executes the programs stored in the storage unit 42 to perform processes required for displaying a display image on the external terminal 2 while taking into account system delay.

[0107] In this way, even when the server device 5 executes the processing required for displaying a display image that takes into account the system delay on the external terminal 2, the driving call system displays a display image that takes into account the system delay on the external terminal 2, and the user of the external terminal 2 can convey appropriate warnings and instructions with reduced effects of time lag to the user of the in-vehicle unit 1. In this modification, the server device 5 is an example of an "information processing device."

[0108] (Variation 3) The generation of the display image may be executed by the external terminal 2 instead of the in-vehicle device 1. In this case, the in-vehicle device 1 transmits the output data including the surrounding captured image before the display image is generated to the external terminal 2, and the external terminal 2 generates the display image from the surrounding captured image included in the output data. The output data includes at least the speed information of the target vehicle in addition to the display image. In this modification, the external terminal 2 is an example of an "information processing device."

[0109] 15 is an example of a functional block diagram of the control unit 24 of the external terminal 2 in Modification 3. The control unit 24 functionally has a surrounding captured image acquisition unit 41A, a speed information acquisition unit 42A, a delay time information acquisition unit 43A, a display image generation unit 44A, an output control unit 45A, and a transmission control unit 45B.

[0110] The surrounding captured image acquisition unit 41A receives output data transmitted by the vehicle-mounted device 1 via the communication unit 21, and sequentially acquires the surrounding captured images included in the output data. The surrounding captured image acquisition unit 41A supplies the acquired surrounding captured images to the display image generation unit 44A.

[0111] The speed information acquisition unit 42A acquires the speed information of the target vehicle contained in the output data received by the communication unit 21. The speed information acquisition unit 42A supplies the acquired vehicle information to the display image generation unit 44A.

[0112] The delay time information acquisition unit 43A generates delay time information indicating the delay time, and supplies the generated delay time information to the display image generation unit 44. Here, the delay time corresponds to the total time (X1+Y1+Z1+X2+Y2+Z2) of each delay time in FIG. 5(A). The delay time information acquisition process executed by the delay time information acquisition unit 43A is the same as the process executed by the delay time information acquisition unit 43 in FIG. 6. The delay time information acquisition unit 43A may acquire the delay time information from the vehicle-mounted device 1, or may acquire delay time information stored in advance in the storage unit 22.

[0113] The display image generating unit 44A generates a display image based on the surrounding captured image acquired from the surrounding captured image acquiring unit 41A, the speed information acquired from the speed information acquiring unit 42A, and the delay time information acquired from the delay time information acquiring unit 43A, and supplies the generated display image to the output control unit 45A. In this case, the display image generating unit 44A sequentially executes a non-stationary object detection process, a state prediction process, a continuous shooting determination process, and an image processing process. The display image generating process executed by the display image generating unit 44A is the same as the process executed by the display image generating unit 44 in FIG. 6.

[0114] The output control unit 45A displays the display image supplied from the display image generation unit 44A on the display unit 26. The output control unit 45A also outputs information based on output data other than the display image received from the vehicle-mounted device 1 via the communication unit 21 on the display unit 26 and / or the sound output unit 27. The transmission control unit 45B transmits input information such as voice data generated by the input unit 23 to the vehicle-mounted device 1 as response data via the communication unit 21.

[0115] Fig. 16 is an example of a flowchart showing a procedure of processing executed by the external terminal 2. The external terminal 2 executes the processing of the flowchart shown in Fig. 16 when communication with the in-vehicle device 1 is established and a call is started.

[0116] First, the external terminal 2 receives output data including a captured image of the surroundings and speed information of the target vehicle from the vehicle-mounted device 1 (step S201). Furthermore, the external terminal 2 acquires delay time information (step S202).

[0117] Then, the external terminal 2 generates a display image based on the surrounding captured image and speed information acquired in step S201 and the delay time information acquired in step S202 (step S203). In this case, the external terminal 2 sequentially executes a non-stationary object detection process, a state prediction process, a continuous capture determination process, and an image processing process to generate a display image.

[0118] Then, the external terminal 2 controls the display of the display image generated in step S203 and the output of information other than the display image (step S204). In addition, in step S204, the external terminal 2 transmits response data based on the input by the input unit 23 to the in-vehicle device 1.

[0119] Next, the external terminal 2 determines whether the call has ended (step S205). If the external terminal 2 determines that the call has ended (step S205; Yes), the process of the flowchart ends. On the other hand, if the external terminal 2 determines that the call has not ended (step S205; No), the process returns to step S201.

[0120] As described above, the external terminal 2 in this modified example generates a display image in which image processing is applied to the image area of ​​a non-stationary object that is a non-continuously existing object, thereby preventing the user of the external terminal 2 from viewing objects that do not need to be considered after the delay time has elapsed.

[0121] (Variation 4) In the block configuration of FIG. 6, the delay time information acquisition unit 43 is not an essential component, and the display image generation unit 44 may generate a display image based on a surrounding captured image and speed information.

[0122] In this case, for example, the display image generating unit 44 regards a fixed predetermined time as a delay time, and generates a display image based on the predetermined time and the moving speed. In this case, for example, correspondence information such as a lookup table or a formula showing the correspondence between the speed of the target vehicle when the delay time is regarded as a predetermined time and the size of the shooting prediction area is stored in advance in the storage unit 12, etc., and the display image generating unit 44 may refer to the correspondence information and determine the size of the shooting prediction area from the speed information of the target vehicle. The above-mentioned correspondence information may be model information (including learned parameters) for configuring a machine learning model (including a deep learning model and a statistical model) that has learned the relationship between the speed of the vehicle and the shooting prediction area to be set in the surrounding shooting image.

[0123] (Variation 5) The vehicle-mounted device 1 may process the image region of the non-stationary object by erasing or mosaic, regardless of whether the non-stationary object is a continuous-existence object or a non-continuous-existence object.

[0124] Fig. 17 is an example of a display image generated by the vehicle-mounted device 1 in this modified example. When the vehicle-mounted device 1 obtains an image 60A, which is a surrounding captured image obtained at the vehicle position Pa just before the intersection 90 shown in Fig. 5(A), the vehicle-mounted device 1 detects image areas of non-stationary objects (traffic lights 80, pedestrians 81, 84, and other vehicles 82, 83) in the surrounding captured image, and applies mosaic processing to the image areas. In this way, the vehicle-mounted device 1 in this modified example performs non-stationary object detection processing and image processing, but does not perform state prediction processing and continuous shooting determination processing.

[0125] Traffic lights and moving objects in the surrounding captured images are subject to system delays and are therefore unreliable information for the user. In consideration of the above, in this modified example, the in-vehicle device 1 generates a display image in which the surrounding captured images are processed so that unsteady objects are not visible, thereby enabling the user of the external terminal 2 to receive guidance without confusion.

[0126] As described above, the in-vehicle device 1, the external terminal 2, or the server device 5 is an information processing device used in a system that mediates communication between an occupant of a vehicle, which is a moving body, and a user present outside the vehicle, and includes an acquisition means, a processing means, and a display control means. The acquisition means acquires a surrounding captured image obtained by capturing an image of the surroundings of the vehicle from the vehicle (which may include the in-vehicle device 1). The processing means applies image processing to a predetermined non-stationary object among objects present in the surrounding captured image. The display control means causes the surrounding captured image that has been subjected to image processing to be displayed on a terminal device used by the user. This makes it possible to suitably generate a display image that takes into account delays that occur in the system.

[0127] In each of the above-mentioned embodiments, the program can be stored using various types of non-transitory computer readable media and supplied to a control unit, which is a computer. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory).

[0128] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0129] 1, 1A in-vehicle unit 2. External terminal 3. Communication Network 4. Map DB 5. Server equipment 11, 21, 41 Communications Department 12, 22, 42 storage section 13 Input section 14, 24, 44 Control section 15, 25 Sensor group 16, 26 Display section 17, 27 Sound output section

Claims

1. An information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body, comprising: An acquisition means for acquiring a surrounding image obtained by capturing an image of the surroundings of the moving object; a processing means for performing image processing on a predetermined non-stationary object among objects present in the peripheral photographed image; a display control means for displaying the processed peripheral photographed image on a terminal device used by the user; An information processing device having the above configuration.

2. A detection means for detecting the non-stationary object included in the surrounding photographed image; a state prediction means for predicting a state of the non-stationary object after a predicted time of delay occurring in the communication; and The information processing apparatus according to claim 1 , wherein the processing means processes the image based on a result of the prediction of the state.

3. The state prediction means predicts, as the state, at least a position of the non-stationary object after the predicted time of the delay; The information processing apparatus according to claim 2 , wherein the processing means processes the image based on a result of the position prediction.

4. The method further includes a determination unit that determines whether or not the non-stationary object is present within a shooting range of an image capturing unit that generates the peripheral image based on the position prediction result, The information processing apparatus according to claim 2 , wherein the processing means processes the image based on a result of the determination.

5. The information processing apparatus according to claim 4 , wherein the processing means processes the image so that the non-stationary object determined not to exist within the shooting range becomes invisible.

6. The information processing apparatus according to claim 4 , wherein the processing means performs the image processing so as to highlight the non-stationary object determined to be present within the shooting range.

7. a steering information acquisition means for acquiring steering information related to a steering operation of the moving body, 7. The information processing apparatus according to claim 4, wherein the determining means makes the determination based on the steering information and the predicted position.

8. The state prediction means predicts a signal display of the traffic light after the predicted delay time when the non-stationary object is a traffic light, The information processing device according to claim 2, wherein the processing means performs the image processing so as to display information regarding the predicted result of the traffic light display in association with the traffic light.

9. A control method executed by an information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body, comprising: An acquisition step of acquiring a surrounding image obtained by capturing an image of the surroundings of the moving object; a processing step of performing image processing on a predetermined non-stationary object among objects present in the peripheral photographed image; a display control step of displaying the processed peripheral photographed image on a terminal device used by the user; The control method includes:

10. A program executed by a computer of an information processing device used in a system that mediates communication between a passenger of a moving body and a user outside the moving body, comprising: An acquisition means for acquiring a surrounding image obtained by capturing an image of the surroundings of the moving object; a processing means for performing image processing on a predetermined non-stationary object among objects present in the peripheral photographed image; A display control means for displaying the processed peripheral photographed image on a terminal device used by the user. A program for causing the computer to function as a

11. A storage medium storing the program according to claim 10.

Images