Information processing device and information processing method

Abstract

The present invention acquires surrounding information for the surrounding environment of a vehicle, sets a virtual viewpoint position ahead in the travel direction of the vehicle, uses the acquired surrounding information to generate virtual surrounding information representing the surrounding environment of the virtual viewpoint position which has been set, transmits a first control signal which causes an external device positioned in a location separated from the vehicle to output the generated virtual surrounding information, receives driving operation information which has been output by the external device and which pertains to a driving operation of the vehicle, generates, on the basis of the received driving operation information, driving assistance information for assisting driving of the vehicle, and generates a second control signal which causes an output device inside the vehicle to output the driving assistance information.

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] A vehicle remote support system is known in which, when autonomous driving of a vehicle is difficult, images of the vehicle's surroundings are transmitted to a remote control center, and an operator at the control center remotely controls the vehicle while viewing the surrounding images displayed on an image display unit (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-003890 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology of Patent Document 1, the operator himself operates the vehicle while watching the driving conditions around the vehicle displayed on the image display unit, so the vehicle is operated without delay in response to changes in the driving conditions. However, when the operator remotely supports the manual driving by the driver on board the vehicle, a time lag occurs between when the operator provides remote support and when the driver drives in accordance with the remote support, resulting in the following problems. In the technology of Patent Document 1, the operator provides remote support while watching the driving conditions around the vehicle from the position of the vehicle, so even if the driver follows the remote support, the driver cannot drive in response to changes in the driving conditions without delay.

[0005] The problem to be solved by the present invention is to provide an information processing device and an information processing method that enable a driver to follow remote assistance and drive in response to changes in driving conditions without delay. [Means for solving the problem]

[0006] The present invention solves the above problems by acquiring peripheral information of the vehicle's surrounding environment, setting a virtual viewpoint position ahead in the vehicle's direction of travel, using the peripheral information to generate virtual peripheral information representing the surrounding environment of the virtual viewpoint position, transmitting a first control signal to output the generated virtual peripheral information to an external device located away from the vehicle, receiving driving operation information related to the driving operation of the vehicle output from the external device, generating driving assistance information to assist in driving the vehicle based on the received driving operation information, and generating a second control signal to output the driving assistance information to an output device located within the vehicle. Effect of the Invention

[0007] According to the present invention, the driver can follow remote assistance to drive in response to changes in the driving conditions without delay. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing device according to the present embodiment. [Figure 2A] FIG. 2A is a diagram showing an example of the surrounding environment of a vehicle. [Figure 2B] FIG. 2B is a diagram showing an example of displaying the positional relationship between a virtual vehicle and vehicles around the virtual vehicle. [Diagram 3] FIG. 3 is a flowchart showing an example of the procedure of the information processing method in the information processing device of this embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of the procedure of the information processing method in the information processing device of this embodiment. [Diagram 5] FIG. 5 is a flowchart showing an example of a procedure of an information processing method in the information processing device, the vehicle, and the terminal device according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An embodiment of an information processing device according to the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of a configuration of an information processing system 10 including an information processing device 100 according to the present embodiment. As shown in FIG. 1, in the present embodiment, the information processing system 10 includes an information processing device 100, a vehicle 200, and a terminal device 300. The information processing device 100 is a server that communicates with the vehicle 200 and the terminal device 300 and can exchange information. The information processing device 100 transmits driving support information for supporting a first user who manually drives the vehicle 200 to the vehicle 200, and transmits information of the surrounding environment at a position ahead of the vehicle 200 in the traveling direction to the terminal device 300. The terminal device 300 is located in a remote space away from the vehicle 200. For example, the remote space is a control center that manages multiple vehicles. In the control center, the terminal device 300 outputs information of the surrounding environment at a position ahead of the vehicle 200 to the second user via the second information output device 340.

[0010] The second user is an operator at a control center, and while checking information on the surrounding environment in a position ahead of the vehicle 200, provides the first user with driving support information on the driving operation of the vehicle 200. For example, one method of providing driving support information to the first user is to display a virtual vehicle image representing a virtual vehicle traveling in front of the vehicle 200 on the windshield of the vehicle 200 so as to be superimposed on the actual scene outside the vehicle. The second user leads the vehicle 200 by operating the virtual vehicle, and provides driving guidance to the first user by the movement of the virtual vehicle. Since the second user operates the virtual vehicle in response to changes in the driving situation while watching the surrounding environment at a position ahead of the position of the vehicle 200 in the traveling direction, the second user can provide the first user with driving content required at a position ahead of the traveling direction of the vehicle 200. On the other hand, the first user operates the vehicle 200 while watching the movement of the virtual vehicle displayed on the windshield of the vehicle 200. As a result, the first user can perform driving that responds to changes in the driving situation without delay by following the movement of the virtual vehicle. In the present embodiment, the driving support information is not limited to a virtual vehicle image, and may be voice guidance by the second user. In the present embodiment, the information processing device 100 may be applied to a device mounted on the vehicle 200 or the terminal device 300.

[0011] The information processing device 100 includes a controller 110 and a communication device 120. The controller 110 includes a computer having hardware and software, and this computer includes a ROM (Read Only Memory) storing a program, a CPU (Central Processing Unit) executing the program stored in the ROM, and a RAM (Random Access Memory) functioning as an accessible storage device. Note that, as the operating circuit, instead of or together with the CPU, an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like can be used.

[0012] The controller 110 includes at least a peripheral information acquisition unit 111, a virtual viewpoint position setting unit 112, an image generation unit 113, a voice information generation unit 114, a transmission unit 115, a reception unit 116, and a driving support information control unit 117 as functional blocks, and executes each function by cooperation between software for implementing each function or executing each process and hardware. Specifically, the controller 110 first acquires peripheral information of the surrounding environment of the vehicle 200, and sets a virtual viewpoint position ahead in the traveling direction of the vehicle 200. Next, the controller 110 uses the peripheral information to generate virtual peripheral information representing the surrounding environment of the virtual viewpoint position, and transmits a control signal to the terminal device 300 via the second information output device 340 to output the virtual peripheral information to the second user. Then, the controller 110 receives the driving operation information input to the input device 320, and generates a control signal to output the driving support information generated based on the driving operation information. Specifically, the controller 110 generates a control signal for displaying a virtual vehicle image via the first information output device 240 so as to be superimposed on the actual scene of the surrounding environment of the vehicle 200. Note that in this embodiment, the functions of the controller 110 are divided into seven blocks, and the functions of each functional block are described, but the functions of the controller 110 do not necessarily need to be divided into seven blocks, and may be divided into six or less functional blocks, or eight or more functional blocks.

[0013] The surrounding information acquisition unit 111 acquires surrounding information of the surrounding environment of the vehicle 200. The surrounding information includes a surrounding image captured of the surrounding environment of the vehicle 200, audio information in the surrounding environment of the vehicle 200, and information on the distance and direction of objects around the vehicle 200. Specifically, the surrounding information acquisition unit 111 acquires a surrounding image of the vehicle 200 captured by the captured image acquisition unit 222 of the vehicle 200. The surrounding information acquisition unit 111 may acquire a surrounding image of the vehicle 200 captured by a sensor group of another vehicle existing around the vehicle 200. The surrounding information acquisition unit 111 may acquire a surrounding image of the vehicle 200 captured by an infrastructure sensor. The infrastructure sensor is a device that is set along the road on which the vehicle 200 is traveling and detects environmental information around the infrastructure sensor. In addition, the surrounding information acquisition unit 111 is not limited to acquiring both the surrounding image and the audio information, and may only acquire the surrounding image.

[0014] Furthermore, the surrounding information acquisition unit 111 acquires audio information in the surrounding environment of the vehicle 200. Specifically, the surrounding information acquisition unit 111 acquires audio information around the vehicle 200 acquired by an external audio acquisition unit 223 of the vehicle 200. The surrounding information acquisition unit 111 may acquire audio information from another vehicle present around the vehicle 200, detected by a sensor group of the other vehicle. The surrounding information acquisition unit 111 may acquire audio information around the vehicle 200 detected by an infrastructure sensor.

[0015] Moreover, the surrounding information acquisition unit 111 acquires information on the distance and direction of an object present around the vehicle 200. Specifically, the surrounding information acquisition unit 111 acquires information on the distance and direction of an object around the vehicle 200 detected by the object detection unit 224 of the vehicle 200. The object includes other vehicles, bicycles, pedestrians, traffic restrictions, construction sites, and the like present around the vehicle 200. The distance and direction of the object are the distance between the vehicle 200 and the object and the direction of the object relative to the vehicle 200. The surrounding information acquisition unit 111 may acquire information on the distance and direction of the object detected by a sensor group of the other vehicle from another vehicle present around the vehicle 200. The surrounding information acquisition unit 111 may acquire information on the distance and direction of the object around the vehicle 200 detected by an infrastructure sensor.

[0016] The virtual viewpoint position setting unit 112 sets a virtual viewpoint position ahead in the traveling direction of the vehicle 200. The virtual viewpoint position is a virtual viewpoint position set in the surrounding environment of the vehicle 200. The virtual viewpoint position is the position of the virtual vehicle in the surrounding environment of the vehicle 200. For example, the virtual viewpoint position is a position corresponding to the position of the driver's seat of the virtual vehicle. The second user can see the surrounding environment of the virtual vehicle from the virtual viewpoint position. The virtual viewpoint position is set as a coordinate position in the real space based on the position of the vehicle 200. In this embodiment, when the first user who is the driver of the vehicle 200 starts the virtual leading vehicle system, the virtual viewpoint position setting unit 112 sets the virtual viewpoint position to an initial setting position. The initial setting position is a predefined position, for example, a position ahead in the traveling direction of the vehicle 200 and a first predetermined distance away from the vehicle 200.

[0017] Moreover, the virtual viewpoint position setting unit 112 updates the virtual viewpoint position based on the movement instruction information. The movement instruction information is information for moving the virtual vehicle by the second user. The movement instruction information includes information on the steering, accelerator, and brake operations of the virtual vehicle. The movement instruction information is output to the controller 110 via the terminal device 300 at a constant cycle. For example, the movement instruction information is information on a steering operation output by the second user performing a steering operation. Also, for example, when the second user continues to maintain the vehicle speed of the virtual vehicle at a constant speed, the movement instruction information is information on an accelerator operation for maintaining the vehicle speed at a constant speed. Also, the movement instruction information may be information on designation of a position through which the virtual vehicle passes. For example, when the second user designates positions through which the virtual vehicle passes as a plurality of points on a map showing the surroundings of the virtual vehicle, information on the positions designated as the points is output as the movement instruction information. The virtual viewpoint position setting unit 112 updates the virtual viewpoint position corresponding to the position of the virtual vehicle based on the movement instruction information for the virtual vehicle. Specifically, the virtual viewpoint position setting unit 112 calculates the position of the virtual viewpoint position after a predetermined time (for example, after one second) based on the position of the virtual viewpoint position and the movement instruction information, and updates the virtual viewpoint position to the calculated virtual viewpoint position after the predetermined time. For example, the virtual viewpoint position setting unit 112 first calculates the movement distance and movement direction after the predetermined time based on the vehicle speed information and steering angle information of the vehicle 200 acquired as the movement instruction information. Next, the virtual viewpoint position setting unit 112 calculates a position moved by the movement distance in the movement direction based on the current virtual viewpoint position as the virtual viewpoint position after the predetermined time. Then, the virtual viewpoint position setting unit 112 updates the virtual viewpoint position to the calculated virtual viewpoint position after the predetermined time.

[0018] Moreover, the virtual viewpoint position setting unit 112 resets the virtual viewpoint position when a predetermined condition is satisfied. In this embodiment, in parallel with the control for updating the virtual viewpoint position based on the movement instruction information by the second user described above, a control for determining whether the predetermined condition is satisfied for resetting the virtual viewpoint position is executed. Specifically, the virtual viewpoint position setting unit 112 executes a determination as to whether the predetermined condition is satisfied at a constant cycle, and resets the virtual viewpoint position when it is determined that the predetermined condition is satisfied. Then, after the resetting, the virtual viewpoint position setting unit 112 sets the reset virtual viewpoint position as the current virtual viewpoint position and continues the control for updating the virtual viewpoint position. For example, when a predetermined condition is satisfied, the virtual viewpoint position setting unit 112 resets the virtual viewpoint position to a position ahead in the traveling direction of the vehicle 200 and at a first predetermined distance from the vehicle 200. When the virtual viewpoint position is reset by the virtual viewpoint position setting unit 112, virtual surrounding information to be output to the second user is generated based on the reset virtual viewpoint position, and the position of the virtual vehicle image displayed to the first user is updated. In addition, in this embodiment, when resetting, the virtual viewpoint position is set at a position away from vehicle 200 by a first predetermined distance, but when resetting, the position may be set to a distance other than the first predetermined distance, not limited to the same first predetermined distance as when initially set.

[0019] The predetermined conditions for resetting the virtual viewpoint position determined by the virtual viewpoint position setting unit 112 are specifically when the virtual viewpoint position is separated from the vehicle 200 by a second predetermined distance or more, when the virtual viewpoint position does not exist on the road on which the vehicle 200 is traveling, and when the driver inputs command information related to resetting the virtual viewpoint position. For example, since the second user can operate the virtual vehicle independently of the movement of the vehicle 200, it may happen that the vehicle 200 cannot catch up with the movement of the virtual vehicle. If the distance between the virtual vehicle and the vehicle 200 increases, the driving of the virtual vehicle may not be helpful to the driver of the vehicle 200. Therefore, the virtual viewpoint position setting unit 112 determines whether the virtual viewpoint position, that is, the position of the virtual vehicle, is separated from the vehicle 200 by a second predetermined distance or more, and when it is determined that the virtual viewpoint position is separated from the vehicle 200 by the second predetermined distance or more, resets the virtual viewpoint position to a position within a first predetermined distance from the vehicle 200. Specifically, the virtual viewpoint position setting unit 112 calculates the distance between the vehicle 200 and the virtual viewpoint position based on the position information of the vehicle 200 and the position information of the virtual viewpoint position, and determines whether or not the distance between the vehicle 200 and the virtual viewpoint position is equal to or greater than a second predetermined distance. When it is determined that the distance between the vehicle 200 and the virtual viewpoint position is equal to or greater than the second predetermined distance, the virtual viewpoint position setting unit 112 resets the virtual viewpoint position to a position that is a first predetermined distance away from the vehicle 200, ahead of the vehicle 200 in the traveling direction. Furthermore, when it is determined that the distance between the vehicle 200 and the virtual viewpoint position is equal to or greater than the second predetermined distance, the virtual viewpoint position setting unit 112 may limit the moving speed of the virtual viewpoint position so that the virtual viewpoint position is within the first predetermined distance from the vehicle 200. Specifically, the virtual viewpoint position setting unit 112 outputs to the terminal device 300 a speed limiting signal that limits the speed that can be input by the second user to a speed lower than a predetermined speed. For example, when the vehicle 200 is traveling at 50 km / h, the virtual viewpoint position setting unit 112 limits the vehicle speed that can be input by the second user to a speed lower than 50 km / h. In this embodiment, the first predetermined distance and the second predetermined distance are each a value set in advance. For example, the first predetermined distance is set to a distance shorter than the second predetermined distance.

[0020] In addition, in this embodiment, the virtual viewpoint position setting unit 112 resets the virtual viewpoint position when the virtual viewpoint position does not exist on the road on which the vehicle 200 is traveling. This is because, even if the virtual vehicle is located within a second predetermined distance from the vehicle 200, if the virtual vehicle is traveling on a road different from the road on which the vehicle 200 is traveling, the driving of the virtual vehicle is not helpful to the driver of the vehicle 200. For example, when a traveling route of the vehicle 200 is set, a road that is not on the traveling route corresponds to a different road. Therefore, the virtual viewpoint position setting unit 112 determines whether or not the virtual viewpoint position exists on a road different from the road on which the vehicle 200 is traveling, and resets the virtual viewpoint position to the initial setting position when it is determined that the virtual viewpoint position exists on a road different from the road on which the vehicle 200 is traveling. Specifically, the virtual viewpoint position setting unit 112 determines whether or not the virtual viewpoint position exists on a road different from the road on which the vehicle 200 is traveling, based on the map information, the position information of the vehicle 200, and the position information of the virtual viewpoint position. For example, the virtual viewpoint position setting unit 112 acquires road link information from the map information, and judges whether or not the link on which the vehicle 200 is located is different from the link on which the virtual viewpoint position exists. If it is judged that the link is different, the virtual viewpoint position setting unit 112 judges that the virtual viewpoint position exists on a road different from the road on which the vehicle 200 is traveling, and resets the virtual viewpoint position to a position ahead in the traveling direction of the vehicle 200 and away from the vehicle 200 by a first predetermined distance.

[0021] Moreover, the virtual viewpoint position setting unit 112 resets the virtual viewpoint position when the driver inputs command information related to the resetting of the virtual viewpoint position. This is because, when the virtual vehicle moves away while the vehicle 200 is traveling, the driver may wish to reset the position of the virtual vehicle. Therefore, the virtual viewpoint position setting unit 112 determines whether or not the driver inputs information related to the resetting of the position of the virtual vehicle, and resets the virtual viewpoint position to the initial setting position when it is determined that the driver inputs information related to the resetting of the position of the virtual vehicle. Specifically, when the virtual viewpoint position setting unit 112 receives command information related to the resetting of the virtual viewpoint position from the vehicle 200 via the communication device 120, it determines that the driver inputs information related to the resetting of the position of the virtual vehicle. Then, when it is determined that the driver inputs information related to the resetting of the position of the virtual vehicle, the virtual viewpoint position setting unit 112 resets the virtual viewpoint position to a position ahead in the traveling direction of the vehicle 200 and at a first predetermined distance from the vehicle 200.

[0022] The image generating unit 113 generates a virtual surrounding image representing the surrounding environment of the virtual viewpoint position. That is, the virtual surrounding image is an image representing the surrounding environment at the position of the virtual vehicle. The second user can visually recognize the surrounding environment as if he / she were sitting in the driver's seat of the virtual vehicle by displaying the virtual surrounding image. The image generating unit 113 first acquires three-dimensional space information of the surrounding environment based on the vehicle 200, calculates a virtual viewpoint position in the three-dimensional space of the surrounding environment, and generates a virtual surrounding image at the virtual viewpoint position using the surrounding image of the vehicle 200. The three-dimensional space information is acquired as three-dimensional space information of the surrounding environment based on the position information of the vehicle 200 and three-dimensional map information. In addition, the three-dimensional space information may be generated in real time by estimating the position of the vehicle 200 and generating a three-dimensional environmental map related to the surrounding environment of the vehicle 200 using SLAM based on sensor information acquired by the sensor group 220 of the vehicle 200. In addition, the image generating unit 113 may generate a virtual surrounding image at the virtual viewpoint position by a light field rendering technique. Specifically, the image generation unit 113 acquires light ray information of the surrounding environment of the vehicle 200 from surrounding images of the surrounding environment of the vehicle 200 captured by cameras from multiple viewpoints, and generates a virtual surrounding image by synthesizing an image at the virtual viewpoint position based on the light ray information that reaches the virtual viewpoint position.

[0023] The image generating unit 113 may generate an overhead image that shows the surrounding environment of the virtual viewpoint position from an overhead perspective. The image generating unit 113 generates an overhead image that shows the positional relationship between the virtual viewpoint position and the object from the distance between the virtual viewpoint position and the object around the virtual viewpoint position and the direction of the object's position relative to the virtual viewpoint position, which are acquired by the surrounding information acquiring unit 111. FIG. 2A is an example showing the surrounding environment of the vehicle 200. FIG. 2B is a diagram showing an example of an overhead image showing the surrounding environment of the virtual vehicle in the surrounding environment of the vehicle 200 shown in FIG. 2A. That is, FIG. 2B shows an overhead image generated based on information that detects vehicles traveling around the vehicle 200. As shown in FIG. 2A, vehicles V1, V2, and V3 are traveling around the vehicle 200. The first user sees the virtual vehicle traveling in a forward position in the traveling direction of the vehicle 200. In the example of FIG. 2A, vehicle 200 detects vehicles V1, V2, and V3. FIG. 2B shows an image displayed to a second user who drives a virtual vehicle, in which the virtual vehicle and vehicles V2 and V3 traveling in front and to the sides of the virtual vehicle are displayed. By looking at the overhead image, the second user can grasp the positional relationship between the virtual vehicle and surrounding objects. The overhead image may be an image including the position of vehicle 200 and the position of the virtual vehicle. The overhead image may be generated by viewpoint conversion using an image of the surroundings of vehicle 200.

[0024] The audio information generating unit 114 generates virtual ambient sound. The virtual ambient sound is a virtual sound heard from the surrounding environment at a virtual viewpoint position. The audio information generating unit 114 sets the position of the virtual sound source to correspond to the position of the sound source in the surrounding environment based on the audio information of the surrounding environment of the vehicle 200 acquired by the external audio acquiring unit 223, and performs audio signal processing of the virtual sound so as to localize a sound image of the virtual sound at the position of the virtual sound source. The audio information generating unit 114 outputs the audio information generated by the audio signal processing to the communication device 120. Also, in this embodiment, the audio information generating unit 114 may not generate the virtual ambient sound, and only the image generating unit 113 may generate the virtual image.

[0025] The transmission unit 115 transmits a control signal for outputting the information generated by the image generation unit 113 and the audio information generation unit 114 to the vehicle 200 and the terminal device 300 via the communication device 120. Specifically, the transmission unit 115 transmits a control signal for outputting virtual peripheral information including the virtual peripheral image and virtual peripheral audio information to the second information output device 340 to the terminal communication device 330. In addition, the transmission unit 115 transmits a control signal for outputting the virtual vehicle image generated by the image generation unit 113 and audio information related to driving guidance input by the second user to the first information output device 240 to the vehicle 200.

[0026] The receiving unit 116 receives driving operation information input to the input device 320 by the second user. The driving operation information is information related to the driving operation of the vehicle 200. Specifically, the driving operation information includes movement instruction information for moving the position of the virtual vehicle (virtual viewpoint position) by the second user. That is, the second user provides information necessary for the driving operation of the vehicle 200 by moving the virtual vehicle displayed to the first user. For example, when there is an obstacle on the left side, the second user can inform the first user that a driving operation to move the vehicle 200 to the right is necessary by moving the virtual vehicle to the right. In addition, the driving operation information may include device control information related to the control of the in-vehicle device of the virtual vehicle. For example, the second user provides the first user with information that a driving operation to stop the vehicle 200 is necessary by inputting device control information to turn on the brake lamp. In addition, the receiving unit 116 receives, as the driving operation information, voice information related to driving guidance by the second user from the terminal device 300.

[0027] The driving support information control unit 117 generates driving support information for supporting the driving of the vehicle 200, and generates a control signal for outputting the driving support information to the first user via the first information output device 240. Specifically, first, the driving support information control unit 117 generates a virtual vehicle image. The virtual vehicle image is configured, for example, by a 3D object of the vehicle. The virtual vehicle image includes images of in-vehicle equipment such as steering lamps and brake lamps. When the driver of the vehicle 200 activates the virtual leading vehicle system, the driving support information control unit 117 generates a virtual vehicle image. In addition, the driving support information control unit 117 generates an image in which the in-vehicle equipment of the virtual vehicle is operating based on the equipment control information. For example, when equipment control information for turning on the brake lamps is input, the driving support information control unit 117 generates an image of the virtual vehicle with the brake lamps turned on. In addition, in this embodiment, the driving support information control unit 117 may generate a virtual image of another object, not limited to an image of the virtual vehicle. For example, the driving support information control unit 117 may generate a virtual image of an arrow indicating the driving direction of the vehicle 200.

[0028] Next, the driving support information control unit 117 generates a control signal for displaying the virtual vehicle image so as to be superimposed on the virtual viewpoint position set by the virtual viewpoint position setting unit 112. Setting of the virtual viewpoint position includes setting to an initial setting position, updating of the virtual viewpoint position, and resetting of the virtual viewpoint position. In addition, when a virtual vehicle image representing a virtual vehicle in which an in-vehicle device is operating is generated based on the device control information, the driving support information control unit 117 generates a control signal for displaying the virtual vehicle image so as to be superimposed on the virtual viewpoint position. In addition, the driving support information control unit 117 may generate voice information related to driving support based on voice information of the second user received by the receiving unit 116. When the driving support information control unit 117 generates the voice information related to driving support, it generates a control signal for outputting the voice information from the voice output device 242 of the first information output device 240. In addition, when there is an obstacle between the vehicle 200 and the virtual viewpoint position that blocks the virtual vehicle from the field of view of the first user, the driving support information control unit 117 generates a control signal for displaying an image of the obstacle so as to be superimposed on the virtual vehicle image. The obstacle includes, for example, another vehicle, a building, etc. Specifically, the driving support information control unit 117 determines whether or not an obstacle exists between the vehicle 200 and the virtual viewpoint position based on the front-rear relationship between the virtual viewpoint position and the obstacle estimated from the surrounding image including the depth information. Then, when it is determined that an obstacle exists between the vehicle 200 and the virtual viewpoint position, the driving support information control unit 117 generates a control signal for displaying an image of the obstacle so as to be superimposed on the virtual vehicle image at the position of the obstacle.

[0029] The communication device 120 communicates with the in-vehicle communication device 230 of the vehicle 200 and the terminal communication device 330 of the terminal device 300, and transmits and receives information. Specifically, the communication device 120 transmits virtual surrounding information to the terminal device 300, and receives driving operation information from the terminal device 300. In addition, the communication device 120 receives surrounding information of the vehicle 200 from the in-vehicle communication device 230, and transmits driving assistance information to the in-vehicle communication device 230.

[0030] Next, the vehicle 200 will be described. The vehicle 200 includes an in-vehicle controller 210, a sensor group 220, an in-vehicle communication device 230, a first information output device 240, an input device 250, and a map database. The in-vehicle controller 210 controls the functions of each device and sensor of the vehicle 200. Specifically, the in-vehicle controller 210 acquires sensor information from the sensor group 220 and outputs it to the in-vehicle communication device 230. In addition, the in-vehicle controller 210 outputs driving assistance information transmitted from the information processing device 100 to the first information output device 240.

[0031] The sensor group 220 acquires information on the status of the surrounding environment of the vehicle. The sensor group 220 includes a position information acquisition unit 221, a captured image acquisition unit 222, an external sound acquisition unit 223, an object detection unit 224, and a map information acquisition unit 225. The position information acquisition unit 221 detects the current position of the vehicle 200. The position information acquisition unit 221 is, for example, a GPS, and acquires the position information of the vehicle by receiving radio waves transmitted from a plurality of communication satellites with a receiver. The position information acquisition unit 221 can also detect changes in the position information of the vehicle by periodically receiving radio waves transmitted from a plurality of satellite communications. The captured image acquisition unit 222 captures an image of the driving environment around the vehicle 200. The captured image acquisition unit 222 is a well-known imaging device that includes an imaging element such as a so-called CCD or CMOS, an optical lens, an optical filter, a peripheral electronic circuit such as a power supply, and captures an image using the imaging element. The captured image acquisition unit 222 is arranged to capture an image of the surrounding environment of the vehicle 200. For example, the captured image acquisition unit 222 is a plurality of cameras arranged on the front, rear, left and right sides of the vehicle 200, and captures scenery outside the vehicle in various directions. The captured image acquisition unit 222 may acquire a surrounding image including depth information. The captured image acquisition unit 222 repeatedly captures the surrounding image at a predetermined cycle. The captured image acquisition unit 222 transmits the captured captured image to the in-vehicle controller 210.

[0032] The external sound acquisition unit 223 collects sounds in the surrounding environment of the vehicle 200. The external sound acquisition unit 223 is, for example, a microphone array in which a plurality of microphones are set on the outside of the vehicle. The external sound acquisition unit 223 acquires spatial information of the sound and estimates the direction of the sound source. The external sound acquisition unit 223 transmits surrounding sound information, which is obtained by converting the collected surrounding sound into an audio signal, to the in-vehicle controller 210. The object detection unit 224 detects objects around the vehicle 200. The object detection unit 224 irradiates, for example, millimeter waves or ultrasonic waves around the vehicle 200 to scan a predetermined surrounding range and detects objects present around the vehicle 200. The objects include other vehicles, bicycles, pedestrians, traffic restrictions, construction sites, and the like present around the vehicle 200. The object detection unit 224 detects the relative position (orientation) between the object and the vehicle 200 and the distance between the object and the vehicle 200. The map information acquisition unit 225 acquires map information including the surrounding environment of the vehicle 200. Specifically, the map information acquisition unit 225 acquires map information including the surrounding environment of the vehicle 200 from the map database 260 based on the position information of the vehicle 200 acquired by the position information acquisition unit 221. The map information acquisition unit 225 may acquire map information from an external map database. The map information acquired by the map information acquisition unit 225 is output to the in-vehicle controller 210.

[0033] The in-vehicle communication device 230 communicates with the communication device 120 to exchange information. Specifically, the in-vehicle communication device 230 receives driving assistance information from the communication device 120, and transmits sensor information and map information to the communication device 120.

[0034] The first information output device 240 outputs image information and audio information to the first user. The first information output device 240 includes an image display device 241 and an audio output device 242. The first information output device 240 is, for example, an AR head mounted display. The first user can view an augmented reality (AR) in which an image of a virtual vehicle is superimposed on a real scene via the first information output device 240. The image display device 241 displays a virtual vehicle image so as to be superimposed on a virtual viewpoint position on a real scene of the surrounding environment of the vehicle 200. The image display device 241 is, for example, a transmissive display. The image display device 241 is not limited to a display of the AR head mounted display, and may be the windshield of the vehicle 200. The audio output device 242 outputs audio information of the second user. The audio output device 242 is, for example, a headphone provided in the AR head mounted display, or a speaker mounted on the vehicle 200.

[0035] The input device 250 is a device operated by the first user, and is used by the first user to reset the virtual viewpoint position to the periphery of the vehicle 200. The input device 250 is, for example, a device that allows input by manual operation, such as a touch panel arranged on a display, a switch button, or a joystick. The first user can input instruction information to the input device 250 to reset the position of the virtual vehicle to the periphery of the vehicle 200.

[0036] The map database 260 stores map information including road information and the like in advance. The map database 260 includes, for example, two-dimensional map information and three-dimensional map information. The three-dimensional map information is map information including three-dimensional information such as land elevation and building height. The three-dimensional map information may be information including a three-dimensional shape model of the terrain or buildings on the map, or information including a model in which an image of the terrain or buildings is added as texture to the shape model.

[0037] The terminal device 300 is a device used by the second user in the remote space, and outputs images and sounds representing the surrounding environment at the position of the virtual vehicle. The terminal device 300 may be installed in the remote space, for example. The terminal device 300 includes a terminal controller 310, an input device 320, a terminal communication device 330, and a second information output device 340. The terminal controller 310 controls the functions of each device of the terminal device 300. Specifically, when voice information of the second user is input to the input device 320, the terminal controller 310 outputs a control signal to the terminal communication device 330 to transmit the voice information to the information processing device 100. In addition, the terminal controller 310 outputs a control signal to cause the second information output device 340 to output the information received by the terminal communication device 330 to the second user.

[0038] The input device 320 is a device operated by the second user, and is used by the second user to input driving operation information. Specifically, the input device 320 is a control device for a virtual vehicle, and includes switches corresponding to, for example, a steering wheel, an accelerator, a brake, and a lamp switch. The control device may have the shape and structure of a steering wheel or an accelerator pedal of a general vehicle, or may use a joystick or the like. When the second user operates the control device, the operation content is input to the input device 320 as driving operation information. The driving operation information includes movement instruction information for moving the virtual vehicle and device control information for controlling the in-vehicle equipment of the virtual vehicle. For example, when the second user operates the steering wheel of the control device, steering angle information is input to the input device 320 as movement instruction information for the virtual vehicle. In addition, when the second user operates an in-vehicle equipment such as a brake lamp, the operation information of the in-vehicle equipment is input to the input device 320 as device control information. In addition, the input device 320 may receive input of voice information as driving operation information. For example, the input device 320 includes a microphone that collects the voice of the second user. When the second user speaks, the input device 320 inputs voice information of the second user.

[0039] In this embodiment, the operation of the virtual vehicle by the second user is limited to operations that are mechanically and physically executable in the actual vehicle. For example, operations such as instantaneously changing the vehicle speed of the virtual vehicle from 40 km to 60 km, and instantaneously moving the position of the virtual vehicle to an adjacent lane are restricted. This allows the virtual vehicle to move within a mechanically and physically possible range. Furthermore, the operation of the virtual vehicle by the second user is limited to operations that comply with traffic regulations. Furthermore, when the input device 320 receives a speed limit signal from the controller 110, it limits the vehicle speed that can be input by the second user to a predetermined speed or less.

[0040] The terminal communication device 330 communicates with the information processing device 100 and the in-vehicle communication device 230 of the vehicle 200 to exchange information. Specifically, the terminal communication device 330 receives virtual surrounding information from the information processing device 100 and transmits driving operation information to the information processing device 100.

[0041] The second information output device 340 is a device that outputs information on the surrounding environment at a virtual viewpoint position to the second user. The second information output device 340 is, for example, a glasses-type or goggles-type VR head-mounted display, and is worn on the head of the second user. The second user can view a virtual reality (VR) in which the surrounding environment at the position of the virtual vehicle is displayed via the terminal device 300. The second information output device 340 outputs a virtual surrounding image that represents the surrounding environment at the virtual viewpoint position and virtual surrounding audio information that represents the surrounding audio at the virtual viewpoint position. This allows the second user to see a view as if he / she were riding in the virtual vehicle, and can hear audio as if he / she was riding in the virtual vehicle. The second information output device 340 includes an image display device 341 and an audio output device 342. The image display device 341 is, for example, a non-transparent display. The audio output device 342 is, for example, a headphone or a speaker. The image display device 341 displays the virtual surrounding image generated by the image generation unit 113. This allows the second user to see the surrounding environment of the virtual vehicle at the virtual viewpoint position through the display on which the virtual surroundings image is displayed. The audio output device 342 outputs virtual surroundings audio information. This allows the second user to hear the audio around the virtual vehicle at the virtual viewpoint position through headphones.

[0042] Next, the procedure of the information processing method in the information processing device according to the present embodiment will be described with reference to FIG. 3 and FIG. 4. FIG. 3 is a flowchart showing the procedure of the information processing method in the information processing device according to the present embodiment. FIG. 4 is a flowchart showing a control flow for resetting the virtual viewpoint position, which is the position of the virtual vehicle image, and shows the control flow in step S10 in FIG. 3. In this embodiment, when the first user of the vehicle 200 starts the virtual leading vehicle system and starts using it, the information processing according to the present embodiment starts from step S1. In addition, in this embodiment, the control flow in FIG. 4 is a control executed after step S9, but is not limited to this, and the control flow in FIG. 4 may be executed between each step in FIG. 3. In this case, the update control of the virtual viewpoint position in FIG. 3 and the determination control related to the reset control of the virtual viewpoint position in FIG. 4 are executed in parallel.

[0043] In step S1, the controller 110 sets the virtual viewpoint position to an initial setting position. The initial setting position is a position ahead in the traveling direction of the vehicle 200 and a first predetermined distance away from the vehicle 200. In step S2, the controller 110 generates a virtual vehicle image representing the virtual vehicle. In step S3, the controller 110 transmits a control signal to the vehicle 200 to cause the image display device 241 to display the virtual vehicle image so as to be superimposed on the virtual viewpoint position on the real scenery of the surrounding environment of the vehicle 200.

[0044] In step S4, the controller 110 acquires peripheral information of the surrounding environment of the vehicle 200. Specifically, the controller 110 acquires a peripheral image and peripheral sound of the surrounding environment of the vehicle 200 detected by the sensor group 220 of the vehicle 200. In step S5, the controller 110 generates a virtual peripheral image and virtual peripheral sound based on the peripheral information acquired in step S4. In step S6, the controller 110 outputs the virtual peripheral image and virtual peripheral sound generated in step S5 to the terminal device 300.

[0045] In step S7, the controller 110 receives the driving operation information transmitted from the terminal device 300. Specifically, the controller 110 receives, as the driving operation information, the position of the virtual vehicle, i.e., the movement instruction information of the virtual viewpoint position and the device control information of the in-vehicle device of the virtual vehicle. The driving operation information is information inputted to the input device 320 of the terminal device 300 by the second user. In step S8, the controller 110 updates the virtual viewpoint position based on the movement instruction information received as the driving operation information in step S7. In step S9, the controller 110 transmits a control signal for displaying the virtual vehicle image at the updated virtual viewpoint position. That is, the controller 110 displays the position of the virtual vehicle image so as to move in accordance with the movement of the virtual viewpoint position. Thereby, the virtual vehicle image displayed to the first user can be moved in accordance with the operation of the virtual vehicle by the second user, and the first user sees as if the virtual vehicle is running in front of the vehicle 200.

[0046] In step S10, the controller 110 performs control for resetting the virtual viewpoint position. After the control in step S9, the controller 110 executes the control flow shown in FIG.

[0047] First, in step S101, the controller 110 acquires the distance between the virtual viewpoint position and the vehicle 200. Specifically, the controller 110 acquires the distance between the virtual viewpoint position and the vehicle 200 from the position information of the virtual viewpoint position and the position information of the vehicle 200. In step S102, the controller 110 determines whether the virtual viewpoint position is separated from the vehicle 200 by a second predetermined distance or more. Specifically, the controller 110 determines whether the distance between the virtual viewpoint position acquired in step S101 and the vehicle 200 is the second predetermined distance or more. If it is determined that the virtual viewpoint position is separated from the vehicle 200 by the second predetermined distance or more, the process proceeds to step S106. If it is not determined that the virtual viewpoint position is separated from the vehicle 200 by the second predetermined distance or more, the process proceeds to step S103.

[0048] In step S103, the controller 110 acquires the virtual viewpoint position and the position of the vehicle on the map. Specifically, the controller 110 identifies on which road on the map the virtual viewpoint position and the vehicle 200 are present, based on the map information acquired from the map database 260, and acquires the virtual viewpoint position and the position of the vehicle on the map. In step S104, the controller 110 determines whether or not the virtual viewpoint position is present on a road different from the road on which the vehicle 200 is traveling. If it is determined that the virtual viewpoint position is present on a road different from the road on which the vehicle 200 is traveling, the process proceeds to step S106. If it is not determined that the virtual viewpoint position is present on a road different from the road on which the vehicle 200 is traveling, the process proceeds to step S105.

[0049] In step S105, the controller 110 determines whether or not the first user has reset the virtual viewpoint position. Specifically, when the controller 110 receives virtual viewpoint position reset command information transmitted from the terminal device 300, the controller 110 determines that the first user has reset the virtual viewpoint position. When it is determined that the first user has reset the virtual viewpoint position, the process proceeds to step S106. When it is not determined that the first user has reset the virtual viewpoint position, the process proceeds to step S11 shown in FIG. 3.

[0050] In step S106, the controller 110 resets the virtual viewpoint position to a position within a first predetermined distance from the vehicle 200. After the resetting, the controller 110 executes the control flow from step S4 shown in FIG.

[0051] In step S11, the controller 110 judges whether the virtual leading car system has been terminated. If it is judged that the virtual leading car system has been terminated, the flow is terminated. If it is not judged that the virtual leading car system has been terminated, the flow returns to step S4, and the flow is repeated thereafter. In addition, in this embodiment, while the virtual leading car system is running, the control flow is repeatedly executed at a constant cycle. Specifically, the controller 110 acquires peripheral information of the vehicle 200 at a constant cycle, and updates the virtual peripheral information based on the acquired peripheral information. This makes it possible to update the peripheral environment of the virtual vehicle in response to the peripheral environment that changes in real time as the vehicle 200 travels. Similarly, the position of the virtual vehicle is updated in real time in response to the operation of the second user with respect to the virtual vehicle image displayed to the first user.

[0052] Next, the procedure of the information processing method in the information processing device 100, the vehicle 200, and the terminal device 300 according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the procedure of the information processing method according to this embodiment. In this embodiment, when the first user of the vehicle 200 starts the virtual lead vehicle system and starts using it, the information processing according to this embodiment starts from step S21.

[0053] In step S21, the information processing device 100 sets the virtual viewpoint position to an initial setting position. In step S22, the information processing device 100 generates a virtual vehicle image. In step S23, the information processing device 100 transmits to the vehicle 200 a control signal for causing the first information output device 240 to display the virtual vehicle image so as to superimpose the virtual vehicle image at the virtual viewpoint position on the real scenery of the surrounding environment of the vehicle 200. In step S24, the vehicle 200 receives the control signal transmitted in step S23 from the information processing device 100. In step S25, the vehicle 200 displays the virtual vehicle image to the first user via the first information output device 240.

[0054] In step S26, the vehicle 200 detects peripheral information of the surrounding environment of the vehicle 200 by the sensor group 220. In step S27, the vehicle 200 transmits the peripheral information detected in step S26 to the information processing device 100. In step S28, the information processing device 100 receives the peripheral information from the vehicle 200. In step S29, the information processing device 100 generates virtual peripheral information including a virtual peripheral image and a virtual peripheral sound. In step S30, the information processing device 100 transmits the virtual peripheral information generated in step S29 to the terminal device 300. In step S31, the terminal device 300 receives the virtual peripheral information from the information processing device 100. In step S32, the terminal device 300 outputs the virtual peripheral information to the second user via the second information output device 340.

[0055] In step S33, the terminal device 300 acquires driving operation information input by the second user to the input device 320. In step S34, the terminal device 300 transmits the driving operation information acquired in step S33 to the information processing device 100. In step S35, the information processing device 100 receives the driving operation information from the terminal device 300. In step S36, the information processing device 100 updates the virtual viewpoint position based on the movement instruction information received as the driving operation information in step S35.

[0056] In step S37, the information processing device 100 transmits to the vehicle 200 a control signal for displaying a virtual vehicle image at the updated virtual viewpoint position. That is, the information processing device 100 transmits a control signal for displaying the virtual vehicle image so as to move the position of the virtual vehicle image in accordance with the movement of the virtual viewpoint position. In step S38, the vehicle 200 receives the control signal transmitted from the information processing device 100 in step S37. In step S39, the vehicle 200 displays the virtual vehicle image on the first information output device so as to superimpose the virtual vehicle image at the updated virtual viewpoint position. Thereby, the virtual vehicle image displayed to the first user can be moved in response to the operation of the virtual vehicle by the second user, and the first user sees as if the virtual vehicle is running in front of the vehicle 200.

[0057] In step S40, the information processing device 100 judges whether the virtual leading vehicle system has been terminated. If it is judged that the virtual leading vehicle system has been terminated, the flow is terminated. If it is not judged that the virtual leading vehicle system has been terminated, the flow returns to step S26, and the flow is repeated thereafter.

[0058] As described above, in this embodiment, peripheral information of the vehicle's surrounding environment is acquired, a virtual viewpoint position is set in front of the vehicle in the traveling direction, the acquired peripheral information is used to generate virtual peripheral information representing the surrounding environment of the set virtual viewpoint position, a first control signal is transmitted to an external device located at a location distant from the vehicle to display the generated virtual peripheral information, driving operation information related to the driving operation of the vehicle output from the external device is received, driving assistance information for assisting the driving of the vehicle is generated based on the received driving operation information, and a second control signal is generated to output the driving assistance information to an output device in the vehicle. This allows the driver to follow the remote assistance and perform driving in response to changes in the driving situation without delay.

[0059] In this embodiment, a surrounding image of the vehicle's surrounding environment is acquired, a virtual surrounding image representing the surrounding environment at the set virtual viewpoint position is generated using the acquired surrounding image, and a control signal for displaying the generated virtual surrounding image on an external device is transmitted as a first control signal. This allows an operator who inputs driving operation information to the external device to input driving operation information while viewing the surrounding environment at the virtual viewpoint position ahead in the traveling direction of the vehicle.

[0060] In addition, in this embodiment, position information indicating the position of the vehicle is acquired, a virtual viewpoint position in the surrounding environment of the vehicle is calculated based on the position of the vehicle, and a virtual surrounding image is generated using the surrounding image based on the virtual viewpoint position. This makes it possible to generate a virtual surrounding image of the virtual viewpoint position using the surrounding image based on the positional relationship in the surrounding environment based on the position of the host vehicle.

[0061] In this embodiment, the surrounding image is acquired from at least one of the sensor information acquired by the vehicle, the sensor information acquired by other vehicles, and the sensor information acquired by infrastructure sensors installed along the roadway, thereby making it possible to acquire surrounding images of areas that are blind spots and cannot be acquired by the sensor of the vehicle itself.

[0062] In this embodiment, the system acquires position information indicating the position of the vehicle, and transmits a third control signal to cause an external device to display the vehicle position information and the virtual viewpoint position information, thereby enabling an operator who inputs driving operation information to the external device to input driving operation information while viewing the positional relationship between the vehicle and the virtual viewpoint position.

[0063] In the present embodiment, a virtual vehicle image showing a virtual vehicle is generated, a control signal for causing an output device to display the virtual vehicle image so that the virtual vehicle image is superimposed at a virtual viewpoint position on an actual scene of the environment surrounding the vehicle is generated as a second control signal, and the second control signal is transmitted to the output device. This enables the driver to drive in response to changes in the driving situation without delay while watching the virtual vehicle ahead in the traveling direction of the vehicle.

[0064] In the present embodiment, the driving operation information includes movement instruction information for moving the virtual viewpoint position, and the virtual viewpoint position is moved by the movement instruction information. This allows the driver to drive in response to changes in the driving situation without delay while watching the movement of the virtual vehicle displayed at the virtual viewpoint position.

[0065] In addition, in this embodiment, device control information for controlling the in-vehicle devices of the virtual vehicle is received as the driving operation information, and a virtual vehicle image showing the virtual vehicle in which the in-vehicle devices have been controlled is generated based on the device control information. This allows the driver to drive in response to changes in the driving situation without delay while watching the operating status of the in-vehicle devices of the virtual vehicle.

[0066] In this embodiment, it is determined whether the distance from the vehicle to the virtual viewpoint position is equal to or greater than a predetermined distance, and if it is determined that the distance is equal to or greater than the predetermined distance, the virtual viewpoint position is reset to a position where the distance from the vehicle to the virtual viewpoint position is shorter than the predetermined distance. This makes it possible to return the virtual viewpoint position to the vicinity of the vehicle even if the virtual viewpoint position has moved away from the vehicle.

[0067] In addition, in this embodiment, the driving operation information is received as voice information related to driving guidance for the vehicle, a voice control signal is generated to cause an output device to output the voice information, and the voice control signal is transmitted to the output device. This allows the driver to drive in response to changes in the driving situation without delay while listening to the driving guidance.

[0068] In the present embodiment, the driving operation information regarding the driving operation at the time when the vehicle reaches the virtual viewpoint position is received, so that the driver can grasp the driving operation at the position ahead in the traveling direction of the vehicle.

[0069] The above-described embodiments are described for the purpose of facilitating understanding of the present invention, and are not described for the purpose of limiting the present invention. Therefore, the elements disclosed in the above embodiments are intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of symbols]

[0070] 10. Information processing systems 100...Information processing device 110…Controller 111…Regional Information Acquisition Department 112...Virtual viewpoint position setting unit 113...Image generation unit 114...Speech information generation unit 115...Transmitter 116…Receiver 117... Driving support information control unit 120...Communication equipment 200…Vehicle 210…In-vehicle controller 220...Sensor group 230...In-vehicle communication device 240...first information output device 250…Input device 260…Map database 300...Terminal device 310...Terminal controller 320...Input device 330...Terminal communication device 340...Second information output device

Claims

1. A surrounding information acquisition unit that acquires surrounding information of the surrounding environment of the vehicle; a virtual viewpoint position setting unit that sets a virtual viewpoint position ahead in a traveling direction of the vehicle; a virtual information generation unit that generates virtual surrounding information representing a surrounding environment of the virtual viewpoint position set by the virtual viewpoint position setting unit, using the surrounding information acquired by the surrounding information acquisition unit; a transmitter that transmits a first control signal to an external device located away from the vehicle, the first control signal being used to output the virtual surroundings information generated by the virtual information generator to an external device located away from the vehicle; A receiving unit that receives driving operation information related to a driving operation of the vehicle output from the external device; a driving assistance information control unit that generates driving assistance information for assisting driving of the vehicle based on the driving operation information received by the receiving unit, and generates a second control signal that outputs the driving assistance information to an output device within the vehicle.

2. The surrounding information acquisition unit acquires a surrounding image of a surrounding environment of the vehicle, the virtual information generation unit generates a virtual surroundings image representing a surrounding environment of the virtual viewpoint position set by the virtual viewpoint position setting unit, using the surroundings image acquired by the surroundings information acquisition unit; The information processing apparatus according to claim 1 , wherein the transmission unit transmits, as the first control signal, a control signal for causing the external device to display the virtual surroundings image generated by the virtual information generation unit.

3. a location information acquisition unit that acquires location information indicating a location of the vehicle, The virtual information generation unit Calculating the virtual viewpoint position in the surrounding environment of the vehicle relative to the position of the vehicle; The information processing apparatus according to claim 2 , wherein the virtual surroundings image is generated using the surroundings image based on the virtual viewpoint position.

4. 4. The information processing device according to claim 2 or 3, wherein the surrounding information acquisition unit acquires the surrounding image from at least one of sensor information acquired by the vehicle, sensor information acquired by other vehicles, and sensor information acquired by infrastructure sensors installed along the roadway.

5. a location information acquisition unit that acquires location information indicating a location of the vehicle, 5. The information processing apparatus according to claim 1, wherein the transmission unit transmits a third control signal for causing the external device to display the position information of the vehicle and the position information of the virtual viewpoint position.

6. a virtual image generating unit that generates a virtual vehicle image showing the virtual vehicle, the driving assistance information control unit generates, as the second control signal, a control signal for causing the output device to display the virtual vehicle image such that the virtual vehicle image is superimposed at the virtual viewpoint position on an actual scene of a surrounding environment of the vehicle; 6. The information processing apparatus according to claim 1, wherein the transmission section transmits the second control signal to the output device.

7. The receiving unit receives, as the driving operation information, device control information for controlling an in-vehicle device of the virtual vehicle, The information processing apparatus according to claim 6 , wherein the virtual image generating unit generates the virtual vehicle image showing the virtual vehicle in which the in-vehicle device has been controlled based on the device control information.

8. The receiving unit receives, as the driving operation information, movement instruction information for moving the virtual viewpoint position, The information processing apparatus according to claim 6 , wherein the virtual viewpoint position setting section moves the virtual viewpoint position in response to the movement instruction information.

9. The virtual viewpoint position setting unit determining whether or not a distance from the vehicle to the virtual viewpoint position is equal to or greater than a predetermined distance; The information processing apparatus according to claim 8 , wherein, when it is determined that the distance is equal to or greater than the predetermined distance, the virtual viewpoint position is reset to a position where the distance becomes shorter than the predetermined distance.

10. The receiving unit receives, as the driving operation information, voice information regarding driving guidance for the vehicle, The driving support information control unit generates a voice control signal for causing the output device to output the voice information, 10. The information processing apparatus according to claim 1, wherein the transmission section transmits the audio control signal to the output device.

11. The information processing device according to claim 1 , wherein the receiving unit receives the driving operation information related to a driving operation at a time when the vehicle reaches the virtual viewpoint position.

12. Obtaining surrounding information about the vehicle's surrounding environment; A virtual viewpoint position is set in front of the vehicle in the traveling direction; generating virtual surrounding information representing a surrounding environment of the set virtual viewpoint position using the acquired surrounding information; Transmitting a first control signal to an external device located away from the vehicle to display the generated virtual surroundings information; Receive driving operation information regarding the driving operation of the vehicle output from the external device; An information processing method that generates driving assistance information for assisting driving of the vehicle based on the received driving operation information, and generates a second control signal for outputting the driving assistance information to an output device within the vehicle.

Images