Simulator system and simulation method

Abstract

To provide a technique for evaluating a remote operation system and a remote automatic operation system under an actual travel environment of an actual vehicle.SOLUTION: A simulator system includes: a simulator section for realizing an actual vehicle being a real vehicle arranged in a real world, a virtual world for reproducing the real world, and a virtual vehicle being a virtual vehicle arranged in the virtual world; a vehicle control section for generating control data in order to travel the virtual vehicle; a communication section for realizing communication between the simulator section and the vehicle control section; and an output section for outputting a movement locus of the actual vehicle and a movement locus of the virtual vehicle. The simulator section is mounted on the actual vehicle. The vehicle control section generates control data for the virtual vehicle to travel by following the actual vehicle, so as to transmit the data to the simulator section.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a simulator system and a simulation method. [Background technology]

[0002] In recent years, research and development of next-generation mobile communications, Beyond 5G (B5G) and 6G, has been actively conducted. The use of such next-generation mobile communications may enable previously unrealizable connected services, such as remote driving systems and remote automated driving systems. Here, a remote driving system refers to a system in which a driver in a remote location manually controls a vehicle from an external location based on video and location information transmitted from the vehicle. A remote automated driving system refers to a system in which a cloud or edge server performs automated driving based on video and location information transmitted from the vehicle. Hereinafter, services that control target devices such as vehicles via a network, such as remote driving systems and remote automated driving systems, or services that provide information used by target devices, are collectively referred to as "next-generation connected services."

[0003] For example, Patent Document 1 discloses a remote driving system that includes a moving object that is the target of remote operation by a remote operator and a remote operator terminal on the remote operator side. For example, Non-Patent Document 1 discloses a remote driving system that uses a real vehicle. "Real vehicle" means an actual vehicle placed in the real world. For example, Non-Patent Document 2 discloses communication evaluation for a remote driving system that uses a virtual vehicle. "Virtual vehicle" means a vehicle realized in a virtual world, in other words, a virtual vehicle placed in the virtual world. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-169715 [Non-patent literature]

[0005] [Non-Patent Document 1] Ericsson, Deutsche Telekom and Vay, "Ericsson, Deutsche Telekom and Vay show live teledrive technology demo with 5G", [online], February 22, 2023, [searched on April 3, 2024], Internet <URL:https: / / www.ericsson.com / en / news / 2023 / 2 / ericsson-dt-and-vay-demo-5g-teledriving> [Non-Patent Document 2] Stefan Neumeier, Ermias Andargie Walelgne, Vaibhav Bajpai, Jorg Ott, and Christian Facchi, "Measuring the Feasibility of Teleoperated Driving in Mobile Networks", [online], June 19, 2019, [searched on April 3, 2024], Internet <URL:https: / / ieeexplore.ieee.org / document / <8784466> [Summary of the Invention] [Problems to be Solved by the Invention] <000​​​The above-mentioned next-generation connected services require constant communication between the vehicle and a control device (such as a driver or an edge server) that generates control data for controlling the vehicle. Therefore, next-generation connected services require network connectivity and stability to enable communication between the vehicle and the control device. Here, "network connectivity" refers to the ease of connecting to a network, i.e., reliability, and "high connectivity" refers to high reliability, i.e., resistance to failures. Furthermore, "network stability" refers to the stability of communication quality in the network, and "high stability" refers to high communication quality.

[0007] The technology described in Patent Document 1 uses a simulator installed on the remote operator's terminal to correct for information loss due to network communication delays and communication interruptions, thereby reducing the sense of discomfort felt by the remote driver during operation. However, the technology described in Patent Document 1 does not take into consideration the evaluation of the network that realizes communication between the vehicle and the control device. The technology described in Non-Patent Document 1 focuses on creating a remote driving system using an actual vehicle, and the communication environment of the network that realizes communication between the vehicle and the control device is limited, i.e., it is a communication environment under ideal conditions that is not realistic. The technology described in Non-Patent Document 2 assumes communication conditions (communication delay, throughput) of the network that realizes communication between the vehicle and the control device. However, in actual communication, there are situations where jitter causes block noise in the camera image, or where instantaneous fluctuations in throughput can degrade video quality, resulting in image quality that is not suitable for remote driving. Therefore, it is not possible to determine whether a remote driving system (or remote automated driving system) is feasible based on communication conditions alone. These issues are not limited to cases where vehicles, such as remote driving systems and remote automated driving systems, are used as target devices for connected services, but are common to any devices, such as robots, drones, etc. Furthermore, these issues are not limited to cases where connected services are built using next-generation mobile communications such as B5G and 6G, but are common to any cases where connected services are built using any communication system.

[0008] The present invention has been made to solve at least some of the above-mentioned problems, and aims to provide technology that can evaluate remote driving systems and remote automated driving systems in the actual driving environment of an actual vehicle. [Means for solving the problem]

[0009] The present invention has been made to solve at least part of the above-mentioned problems, and can be realized in the following aspects.

[0010] (1) According to one aspect of the present invention, there is provided a simulator system comprising: a simulator unit that realizes a real vehicle that is an actual vehicle located in a real world, a virtual world that reproduces the real world, and a virtual vehicle that is a virtual vehicle located in the virtual world, a vehicle control unit that generates control data for driving the virtual vehicle, a communication unit that realizes communication between the simulator unit and the vehicle control unit, and an output unit that outputs a movement trajectory of the real vehicle and a movement trajectory of the virtual vehicle, the simulator unit being mounted on the real vehicle, and the vehicle control unit generating the control data for driving the virtual vehicle to follow the real vehicle and transmitting the control data to the simulator unit.

[0011] According to this configuration, the virtual vehicle realized by the simulator unit travels based on control data transmitted from a remote vehicle control unit. Because the simulator unit is installed in a real vehicle, the control data for driving the virtual vehicle can be said to be data transmitted in the actual driving environment of the real vehicle, in other words, in the network environment actually used by the real vehicle. The vehicle control unit generates control data that causes the virtual vehicle to travel in a manner following the real vehicle, specifically, control data that causes the virtual vehicle to follow the real vehicle in the virtual world. Therefore, by comparing the movement trajectory of the real vehicle output by the output unit with the movement trajectory of the virtual vehicle, system users can evaluate next-generation connected services in the actual driving environment of the real vehicle. Furthermore, by comparing the movement trajectory of the real vehicle output by the output unit with the movement trajectory of the virtual vehicle, system users can also evaluate the connectivity and stability of the network that realizes communication between the simulator unit (i.e., the real vehicle) and the vehicle control unit (i.e., the control device that generates the control data for controlling the vehicle).

[0012] (2) In the simulator system of the above form, the real vehicle has a position acquisition unit that acquires real vehicle position information indicating the current position of the real vehicle, and the position acquisition unit transmits the acquired real vehicle position information to the vehicle control unit, the simulator unit includes a virtual sensor that acquires information about the virtual vehicle, including virtual vehicle position information indicating the current position of the virtual vehicle, and transmits sensor information acquired by the virtual sensor to the vehicle control unit, and the vehicle control unit may generate the control data using the real vehicle position information and the sensor information. According to this configuration, the vehicle control unit can easily generate control data for the virtual vehicle to travel following the actual vehicle, using the actual vehicle position information and the sensor information.

[0013] (3) In the simulator system of the above form, the vehicle control unit may set a target position in the virtual world, which is a position in the virtual world that corresponds to the actual vehicle position information, and automatically generate the control data for driving the virtual vehicle from the current position of the virtual vehicle identified by the virtual vehicle position information of the sensor information to the target position. According to this configuration, the vehicle control unit automatically generates control data for driving the virtual vehicle from the current position of the virtual vehicle identified by the virtual vehicle position information of the sensor information to the target position. Therefore, this configuration provides a simulator system capable of evaluating remote automated driving systems.

[0014] (4) In the simulator system of the above form, the sensor information further includes an image of a front camera of the virtual vehicle, and the vehicle control unit includes a display unit that allows the driver to view the image and an operation unit that accepts operations of the driver on the virtual vehicle, and the display unit may display an image of the front camera image of the sensor information superimposed with a marker indicating a target position in the virtual world, which is a position in the virtual world that corresponds to the actual vehicle position information, and generate the control data according to the content of the operation obtained from the operation unit. According to this configuration, the vehicle control unit displays an image on the display unit in which a marker indicating the target position is superimposed on the image of the front camera of the sensor information, and generates control data according to the operation content acquired from the operation unit. Therefore, according to this configuration, a simulator system capable of evaluating a remote driving system can be provided.

[0015] (5) In the simulator system of the above form, the actual vehicle may further include a recognition unit that recognizes objects around the actual vehicle in the real world, and the simulator unit may transmit to the vehicle control unit, instead of an image from the front camera, an image in which an object that reproduces the object recognized by the recognition unit is superimposed on the image from the front camera. According to this configuration, the simulator unit transmits to the vehicle control unit an image in which an object reproducing an object recognized by the recognition unit is superimposed on an image captured by the front camera. Therefore, the vehicle control unit can display on the display unit an image in which an object reproducing an object around the actual vehicle in the real world and a landmark indicating a target position are superimposed on the image captured by the front camera. As a result, according to this configuration, the driver of the virtual vehicle can perform driving operations on the operation unit that are closer to reality while recognizing objects around the actual vehicle in the real world (e.g., people on the road, oncoming vehicles, etc.).

[0016] (6) In the simulator system of the above form, the communication unit may include a first communication unit that realizes communication between the simulator unit and the vehicle control unit using a first communication means, a second communication unit that realizes communication between the simulator unit and the vehicle control unit using a second communication means different from the first communication unit, and a communication control unit that controls multiplexing of communication using the first communication unit and the second communication unit. According to this configuration, the communication unit includes a first communication unit, a second communication unit, and a communication control unit that controls multiplexing of communication using the first communication unit and the second communication unit. Therefore, according to this configuration, communication between the simulator unit (i.e., the actual vehicle) and the vehicle control unit (i.e., the control device that generates control data for controlling the vehicle) can be multiplexed.

[0017] (7) In the simulator system of the above aspect, the output unit may be an evaluation unit that compares the movement trajectory of the actual vehicle with the movement trajectory of the virtual vehicle and evaluates the degree of deviation between the two movement trajectories. According to this configuration, the evaluation unit compares the movement trajectory of the actual vehicle with the movement trajectory of the virtual vehicle, thereby making it possible to evaluate the degree of deviation between the two movement trajectories.

[0018] The present invention can be realized in various forms, for example, in the form of a simulator device, an information processing device, a vehicle, a drone, a simulator system, a method executed in an information processing device to realize the functions of each of these devices and systems, a computer program to realize the functions of each of these devices and systems, a server device for distributing the computer program, a non-transitory storage medium on which the computer program is stored, etc. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating an example of a simulator system. [Figure 2] FIG. 10 is a sequence diagram illustrating an example of a simulation processing procedure. [Figure 3] FIG. 3 is a diagram illustrating steps S40 and S42 in FIG. 2. [Figure 4] FIG. 10 is a diagram illustrating an example of a simulator system according to a second embodiment. [Figure 5] FIG. 10 is a diagram showing an example of an image displayed on the display device in step d3. [Figure 6] FIG. 10 is a diagram illustrating an example of a simulator system according to a third embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a simulator system according to a fourth embodiment. [Figure 8] FIG. 10 is a diagram showing an example of an image displayed on the display device in step d3. [Figure 9]FIG. 10 is a diagram illustrating an example of a simulator system according to a fifth embodiment. [Figure 10] FIG. 13 is a diagram illustrating an example of a simulator system according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] First Embodiment FIG. 1 is a diagram illustrating an example of a simulator system 1000. The simulator system 1000 is a system that provides next-generation connected services to passengers of a real vehicle C using a server 20 on a network. The term "next-generation connected service" refers to a service that controls the real vehicle C via a network, or a service that provides some information used by the real vehicle C or the passengers of the real vehicle C to some target device (not limited to the real vehicle C) via a network. In the simulator system 1000 of the first embodiment, a remote automated driving system is an example of the next-generation connected service. The remote automated driving system is a system in which the server 20 automatically drives the real vehicle C based on video, location information, etc. transmitted from the real vehicle C.

[0021] As shown in FIG. 1, the simulator system 1000 includes a real vehicle C and a server 20. The real vehicle C is an actual vehicle placed in the real world. The real vehicle C is placed in any environment in which the real vehicle C actually travels. The server 20 is a server device placed on a network. The server 20 is placed in a location physically separated from the real vehicle C. The real vehicle C and the server 20 communicate with each other via communication units (vehicle-side communication unit 120, server-side communication unit 220). The on-board device 10 and position acquisition unit 11 of the real vehicle C correspond to a "first information processing device." The server 20 corresponds to a "second information processing device."

[0022] The real vehicle C includes an on-board device 10 and a position acquisition unit 11. The position acquisition unit 11 acquires position information indicating the current position of the real vehicle C. The position acquisition unit 11 can acquire the position information by receiving radio waves transmitted from artificial satellites that make up the GPS (Global Positioning System) or QZSS (Quasi-Zenith Satellite System) using a receiving device built into the position acquisition unit 11. Hereinafter, the position information of the real vehicle C acquired by the position acquisition unit 11 will also be referred to as "real vehicle position information." The real vehicle position information is also used as the initial position of the virtual vehicle and the destination of the virtual vehicle. For this reason, it is preferable that the position acquisition unit 11 be a module that supports RTK positioning, which can determine a more accurate position.

[0023] The in-vehicle device 10 includes a simulator unit 110, a vehicle-side communication unit 120, and an evaluation unit 130. The in-vehicle device 10 is an information processing device including a central processing unit (CPU), a read-only memory (ROM), a random access memory (RAM), and other memories (not shown). The in-vehicle device 10 may be configured using an application specific integrated circuit (ASIC).

[0024] The simulator unit 110 is a functional unit that realizes a virtual world that reproduces the real world and a virtual vehicle (hereinafter also referred to as a "virtual vehicle") placed in the virtual world. As shown in FIG. 1, the simulator unit 110 is mounted on the in-vehicle device 10 of the actual vehicle C. The simulator unit 110 has a virtual sensor 111, a virtual actuator 112, and a 3D city environment model 113.

[0025] The virtual sensor 111 virtually realizes one or more sensors that the virtual vehicle has. Examples of sensors realized by the virtual sensor 111 include the following a1 to a3. (a1) Virtual vehicle identifier (a2) GPS sensor or QZSS sensor (a3) Orientation sensor Hereinafter, the values ​​a1 to a3 acquired by the virtual sensor 111 will also be referred to as "sensor information." In this embodiment, the sensor information includes a virtual vehicle ID acquired by the sensor a1, virtual vehicle position information acquired by the sensor a2, and the orientation of the virtual vehicle acquired by the sensor a3. The sensor information is merely an example and may include other information.

[0026] The virtual actuator 112 virtually realizes one or more actuators that the virtual vehicle has. Examples of actuators realized by the virtual actuator 112 include the following b1 to b3. (b1) Steering wheel (b2) Axel (b3) Brake

[0027] The city environment 3D model 113 is a three-dimensional model of a virtual world in which a virtual vehicle is placed. In the city environment 3D model 113, a virtual world corresponding to the city block in which the actual vehicle C is placed is constructed. The city environment 3D model 113 includes a road model and a feature model that exists around the road. The feature means all natural and artificial objects on the ground, including buildings, houses, mountains, rivers, etc. Note that the city environment 3D model 113 does not include pedestrians, vehicles, etc. that are not included in the map.

[0028] Of the simulator unit 110, the virtual sensor 111 and virtual actuator 112 can be configured as a real-time simulator (emulator) built using CARLA. CARLA is a driving control simulator based on Unreal Engine. In addition to the sensors exemplified in a1 to a3 above, CARLA is equipped with various sensor models used in autonomous driving, such as LiDAR, cameras, and depth cameras, and also supports integration with RoS and Autoware. CARLA allows simulations to be performed by giving instructions to the simulator world using Python scripts. Of the simulator unit 110, the city environment 3D model 113 can be built by importing a map in OpenDrive format into CARLA. Note that the simulator unit 110 may be configured using CRSIM instead of CARLA.

[0029] The vehicle-side communication unit 120 realizes communication between the actual vehicle C and the server 20. Specifically, the vehicle-side communication unit 120 is a functional unit that realizes communication between functional units (simulator unit 110, position acquisition unit 11) on the actual vehicle C side and functional units (vehicle control unit 210) on the server 20 side. In this embodiment, the vehicle-side communication unit 120 performs communication in accordance with 5G or LTE, which are mobile communications. Any communication method can be used for the vehicle-side communication unit 120. For example, the vehicle-side communication unit 120 may be a functional unit that communicates in accordance with Beyond 5G (B5G), which is next-generation mobile communications, a functional unit that communicates in accordance with 6G, or a functional unit that communicates in accordance with satellite communications. The vehicle-side communication unit 120 may be a communication unit that communicates in accordance with wireless communications such as WiFi.

[0030] The evaluation unit 130 compares the movement trajectory of the real vehicle C with the movement trajectory of the virtual vehicle, evaluates the degree of separation between the two movement trajectories, and outputs the evaluation result. Details will be described later. The movement trajectory of the real vehicle C is a trajectory connecting real vehicle position information repeatedly acquired by the position acquisition unit 11. The movement trajectory of the virtual vehicle is a trajectory connecting virtual vehicle position information repeatedly acquired by the virtual sensor 111 (sensor a2). Note that the output of the evaluation result by the evaluation unit 130 corresponds to "output of the movement trajectory of the real vehicle C and the movement trajectory of the virtual vehicle."

[0031] The server 20 includes a vehicle control unit 210 and a server-side communication unit 220. The server 20 is a server device configured to include a central processing unit (CPU), read-only memory (ROM), random access memory (RAM), and memory (not shown). The server 20 may be a cloud server in which multiple server devices arranged on a network work together to realize the functions of the server 20 (described later). The server 20 may also be an edge server installed within a communication network.

[0032] The vehicle control unit 210 generates control data for running the virtual vehicle by remote automatic driving control. The control data is information for operating the virtual actuator 112 of the simulator unit 110 of the actual vehicle C. In this embodiment, the control data includes the following c1 to c3. (c1) Steering angle information of the steering wheel (c2) Accelerator opening information (c3) Brake opening information

[0033] The server-side communication unit 220 realizes communication between the server 20 and the actual vehicle C. Specifically, the server-side communication unit 220 is a functional unit that realizes communication between a functional unit (vehicle control unit 210) on the server 20 side and a functional unit (simulator unit 110, position acquisition unit 11) on the actual vehicle C side. The server-side communication unit 220 is configured using the same communication method as the vehicle-side communication unit 120 of the in-vehicle device 10. The server-side communication unit 220 of the server 20 and the vehicle-side communication unit 120 of the in-vehicle device 10 are also collectively referred to as "communication units."

[0034] Fig. 2 is a sequence diagram showing an example of a simulation processing procedure executed in the simulator system 1000. The processing in Fig. 2 is started in response to the engine start of the actual vehicle C shown in step S10. Instead of starting the engine of the actual vehicle C, the processing in Fig. 2 may be started by starting a predetermined application provided by the in-vehicle device 10 or by operating an operation unit provided on the in-vehicle device 10.

[0035] In step S12, the in-vehicle device 10 activates each functional unit therein. In step S14, the position acquisition unit 11 of the real vehicle C acquires the current position of the real vehicle C and transmits the real vehicle position information to the simulator unit 110. In step S16, the simulator unit 110 sets the initial position of the virtual vehicle. Specifically, the simulator unit 110 identifies a position in the virtual world that corresponds to the real vehicle position information acquired in step S14. The simulator unit 110 places the virtual vehicle at the identified position in the virtual world. In step S16, the position of the real vehicle C in the real world before traveling and the position of the virtual vehicle in the virtual world before traveling are aligned (logically the same position). For example, if the real vehicle C is located in front of an XX intersection in the real world, the virtual vehicle is located in front of the XX intersection in the virtual world realized by the city environment 3D model 113.

[0036] In step S20, the real vehicle C travels on an actual road in the real world. In step S22, the position acquisition unit 11 of the real vehicle C acquires the current position of the real vehicle C and transmits the real vehicle position information to the vehicle control unit 210 of the server 20. Specifically, the real vehicle position information transmitted from the position acquisition unit 11 is transmitted from the vehicle-side communication unit 120 of the in-vehicle device 10 to the server 20, received by the server-side communication unit 220 of the server 20, and transmitted to the vehicle control unit 210. In the subsequent steps, communication between the real vehicle C and the in-vehicle device 10 and the server 20 is also realized via the communication units (the vehicle-side communication unit 120, the server-side communication unit 220). This is represented by multiple arrows lined up in a straight line in FIG. 2.

[0037] In step S24, the position acquisition unit 11 of the actual vehicle C transmits the same actual vehicle position information as in step S22 to the evaluation unit 130. The evaluation unit 130 stores the actual vehicle position information acquired in step S24. In step S26, the simulator unit 110 of the actual vehicle C transmits the sensor information (virtual vehicle ID, virtual vehicle position information, and virtual vehicle orientation) acquired by the virtual sensor 111 to the vehicle control unit 210 of the server 20.

[0038] In step S30, the vehicle control unit 210 of the server 20 generates control data. Specifically, the vehicle control unit 210 sets the current position and orientation of the virtual vehicle from the sensor information acquired in step S26, and sets the target position of the virtual vehicle from the real vehicle position information acquired in step S22. The target position of the virtual vehicle means a position in the virtual world corresponding to the real vehicle position information. The vehicle control unit 210 automatically generates control data for driving the virtual vehicle from the set current position to the target position using PID control. The control data generated in step S30 includes the above-mentioned information c1 to c3. In this way, the vehicle control unit 210 generates control data with the real vehicle position information as the target position. Therefore, the generated control data is data for the virtual vehicle to drive while following the real vehicle C, in other words, data for the virtual vehicle to track the position of the real vehicle C in the virtual world. In step S32, the vehicle control unit 210 of the server 20 transmits the generated control data to the simulator unit 110 of the in-vehicle device 10.

[0039] In step S34, the simulator unit 110 of the real vehicle C causes the virtual vehicle to run based on the control data acquired from the server 20. Specifically, the simulator unit 110 uses the control data acquired from the server 20 to drive the virtual actuator 112. As a result, the virtual vehicle runs in the virtual world according to the content specified by the control data. In step S36, the simulator unit 110 of the real vehicle C transmits sensor information acquired by the virtual sensor 111 to the evaluation unit 130. The evaluation unit 130 stores the virtual vehicle position information from the sensor information acquired in step S36.

[0040] In step S40, the evaluation unit 130 compares the actual vehicle position information acquired in step S24 with the virtual vehicle position information acquired in step S36. In step S42, the evaluation unit 130 determines whether the difference between the actual vehicle position information and the virtual vehicle position information exceeds a predetermined threshold as a result of the comparison in step S40. If the difference exceeds the predetermined threshold, the evaluation unit 130 transmits the actual vehicle position information to the simulator unit 110 and causes the simulator unit 110 to correct the position of the virtual vehicle. Specifically, the simulator unit 110 resets the position of the virtual vehicle and performs the same process as in step S16. The predetermined threshold is set in advance by a user of the simulator system 1000 and stored in the in-vehicle device 10. The predetermined threshold can be set arbitrarily and may be changeable by the user.

[0041] The actual vehicle C and the server 20 repeatedly execute the processing described in steps S20 to S42. As a result, the running of the actual vehicle C (S20), the following of the virtual vehicle (S22 to S36), and the comparison of the position information of the actual vehicle C and the virtual vehicle (S40) are repeatedly executed. The processing described in steps S20 to S42 is repeated until an arbitrary termination condition is met. The termination condition can be set arbitrarily. The termination condition can be, for example, stopping the engine of the actual vehicle C, ending a predetermined application provided by the in-vehicle device 10, operating an operation unit provided on the in-vehicle device 10, etc.

[0042] In step S50, the evaluation unit 130 of the actual vehicle C evaluates the actual vehicle position information and the virtual vehicle position information stored through repeated processing. Specifically, the evaluation unit 130 calculates the distance between the actual vehicle position information and the virtual vehicle position information acquired in the same processing cycle using a well-known evaluation method, and assigns a higher evaluation to the smaller the distance and a lower evaluation to the larger the distance. Examples of well-known evaluation methods for calculating the distance between data include Euclidean distance, Manhattan distance, and Minkowski distance. Thereafter, the evaluation unit 130 outputs the evaluation result and then terminates the processing. The evaluation result output by the evaluation unit 130 may be the calculated distance itself, may be a score, or may be a rating such as low, medium, or high.

[0043] FIG. 3 is a diagram illustrating steps S40 and S42 in FIG. 2. FIG. 3 shows a map VMP in which a certain intersection in the real world is superimposed on the same intersection in the virtual world. On the map VMP, the time-series changes in the actual vehicle position information LC1 to LC7 are indicated by black circles, and the time-series changes in the virtual vehicle position information LV1 to LV7 are indicated by white circles. The actual vehicle position information and the virtual vehicle position information with the same last digit indicate information acquired in the same processing cycle in the simulation processing. For example, the actual vehicle position information LC1 and the virtual vehicle position information LV1 are information acquired in the first processing cycle.

[0044] In the simulation process, the control data generated by the vehicle control unit 210 is data for the virtual vehicle to travel following the real vehicle C. For this reason, as shown in Fig. 3, the virtual vehicle position information LV1 to LV3 follows the real vehicle position information LC1 to LC3 with a slight delay, even when communication is being performed satisfactorily. Here, since the real vehicle C travels through an actual city block, the communication environment between the real vehicle C and the server 20 is not constant, but changes due to the presence of obstacles, the strength of radio waves at that location, increases and decreases in communication traffic, etc.

[0045] If communication delays or interruptions occur due to such changes in the communication environment, delays or loss of information occurs in the information transmitted and received between the real vehicle C and the server 20 during the simulation process. As a result, as shown by LC4, LV4, and LC5, LV5 in FIG. 3, a discrepancy may occur between the real vehicle position information and the virtual vehicle position information. In step S40 of the simulation process, the evaluation unit 130 calculates the distance L4 between the real vehicle position information LC4 and the virtual vehicle position information LV4, and the distance L5 between the real vehicle position information LC5 and the virtual vehicle position information LV5. If these distances L4 and L5 (deviations) exceed a predetermined threshold, the evaluation unit 130 corrects the position of the virtual vehicle (step S42). As a result, as shown in the dashed circle in FIG. 3, in the next processing cycle, the position of the virtual vehicle is corrected so that the real vehicle position information LC6 and the virtual vehicle position information LV6 are approximately the same position. In this way, the subsequent simulation can be continued even if communication delays or interruptions occur.

[0046] As described above, according to the simulator system 1000 of the first embodiment, the virtual vehicle realized by the simulator unit 110 travels based on control data transmitted from the vehicle control unit 210 located at a remote location. Because the simulator unit 110 is mounted on the real vehicle C, the control data for driving the virtual vehicle is data transmitted in the actual driving environment of the real vehicle C, in other words, in the network environment actually used by the real vehicle C. The vehicle control unit 210 generates control data for driving the virtual vehicle while following the real vehicle C, specifically, control data for driving the virtual vehicle in the virtual world following the real vehicle C in the real world. Therefore, by comparing the movement trajectories LC1 to LC7 of the real vehicle C output by the evaluation unit 130 (output unit) with the movement trajectories LV1 to LV7 of the virtual vehicle, users of the simulator system 1000 can evaluate next-generation connected services in the actual driving environment of the real vehicle C. Furthermore, by comparing the movement trajectories LC1 to LC7 of the actual vehicle output by the evaluation unit 130 with the movement trajectories LV1 to LV7 of the virtual vehicle, the user of the simulator system 1000 can also evaluate the connectivity and stability of the network that realizes communication between the simulator unit 110 (i.e., the actual vehicle C) and the vehicle control unit 210 (i.e., the server 20 that generates control data for controlling the vehicle).

[0047] Furthermore, according to the simulator system 1000 of the first embodiment, the vehicle control unit 210 can easily generate control data for the virtual vehicle to travel following the real vehicle C, using the real vehicle position information and the sensor information. Furthermore, according to the simulator system 1000 of the first embodiment, the vehicle control unit 210 automatically generates control data for causing the virtual vehicle to travel from the current position of the virtual vehicle identified by the virtual vehicle position information of the sensor information to a target position. Therefore, the simulator system 1000 can provide a simulator system capable of evaluating a remote automated driving system.

[0048] Furthermore, the simulator system 1000 of the first embodiment includes the evaluation unit 130, and therefore, can evaluate the degree of discrepancy between the movement trajectories LC1 to LC7 of the actual vehicle C and the movement trajectories LV1 to LV7 of the virtual vehicle by comparing them with the evaluation unit 130. As a result, the simulator system 1000 can improve the usability of the system.

[0049] Second Embodiment 4 is a diagram illustrating an example of a simulator system 1000A according to the second embodiment. The simulator system 1000A according to the second embodiment is a remote driving system as a next-generation connected service. The remote driving system is a system in which a driver P, who is located remotely from the actual vehicle CA, manually controls the actual vehicle CA from the outside (via a network) based on video images, position information, and the like transmitted from the actual vehicle CA.

[0050] The simulator system 1000A of the second embodiment includes an in-vehicle device 10A instead of the in-vehicle device 10 and a server 20A instead of the server 20 in the configuration described in the first embodiment. The in-vehicle device 10A includes a simulator unit 110A instead of the simulator unit 110. The simulator unit 110A includes a virtual sensor 111A instead of the virtual sensor 111. The virtual sensor 111A virtually realizes the next sensor a4 in addition to the sensors a1 to a3 described in the first embodiment. (a4) Front camera mounted on the virtual vehicle

[0051] The server 20A includes a vehicle control unit 210A instead of the vehicle control unit 210. The vehicle control unit 210A performs control different from that of the vehicle control unit 210 described in the first embodiment. The vehicle control unit 210A further includes a display device 213 and a control device 214. The display device 213 is a device that allows the driver P to visually recognize an image. The image includes a video. Examples of the display device 213 include a liquid crystal display and a head-mounted display. The display device 213 corresponds to the "display unit." The control device 214 includes various devices that accept operations from the driver P regarding the virtual vehicle. The control device 214 of this embodiment includes a steering device for steering, an accelerator pedal for accelerator operation, and a brake pedal for braking. The configuration of the control device 214 can be modified in various ways, and may be a controller-type device, a virtually configured input device using an image, or a voice input device. The control device 214 corresponds to the "operation unit."

[0052] In step S26 of the simulation process (FIG. 2), the simulator unit 110A transmits the sensor information (a1 to a4) acquired by the virtual sensor 111A to the vehicle control unit 210A. In step S30, the vehicle control unit 210A generates control data by the following steps d1 to d5, instead of the steps described in the first embodiment. Steps of the simulation process other than those described above are the same as those in the first embodiment. (d1) The vehicle control unit 210A identifies the target position of the virtual vehicle from the actual vehicle position information acquired in step S22. (d2) Vehicle control unit 210A generates an image in which a mark indicating the target position identified in step d1 is superimposed on the image from the front camera out of the sensor information acquired in step S26. (d3) The vehicle control unit 210A displays the image generated in step d2 on the display device 213. (d4) The vehicle control unit 210A acquires the details of the operation performed on the control device 214 by the driver P looking at the display device 213. Specifically, the vehicle control unit 210A acquires steering angle information performed on the steering device, accelerator pedal opening information, and brake pedal opening information. (d5) The vehicle control unit 210A generates control data using the operation details acquired in step d4.

[0053] FIG. 5 is a diagram showing an example of image IM displayed on display device 213 in step d3. As shown in FIG. 5, image IM viewed by driver P displays landmarks LM1 and LM2 set from real vehicle position information relative to the scenery UE ahead of the virtual vehicle. Mark LM1 is a landmark in a certain processing cycle, and mark LM2 is a landmark in the next processing cycle. Image IM includes line segments LL1 and LL2. Line segments LL1 and LL2 may be omitted. Line segment LL1 is a link connecting the current position of the virtual vehicle and landmark LM1. Line segment LL2 is a link connecting landmark LM1 and landmark LM2.

[0054] As described above, the configuration of the simulator system 1000A can be modified in various ways, and may be configured to simulate a remote driving system. In the second embodiment, an example corresponding only to a remote driving system has been described. However, the vehicle control unit 210A may be configured to switch between the processing of step S30 described in the first embodiment and the processing of step S30 described in the second embodiment. In this way, the user can simulate either the remote automatic driving system or the remote driving system of their choice.

[0055] The simulator system 1000A of the second embodiment can also achieve the same effects as those of the first embodiment. Moreover, according to the simulator system 1000A of the second embodiment, the vehicle control unit 210A displays an image IM on the display device 213 (display unit) in which marks LM1 and LM2 indicating the target position are superimposed on an image of the front camera of the sensor information, and generates control data according to the content of the operation acquired from the control device 214 (operation unit). Therefore, the simulator system 1000A can provide a simulator system capable of evaluating a remote driving system.

[0056] Third Embodiment 6 is a diagram illustrating an example of a simulator system 1000B according to the third embodiment. The simulator system 1000B according to the third embodiment is a system that employs a remote driving system similar to that of the second embodiment as a next-generation connected service, and further has multiplexed communication units.

[0057] The simulator system 1000B of the third embodiment includes an in-vehicle device 10B instead of the in-vehicle device 10A and a server 20B instead of the server 20A in the configuration described in the second embodiment. The in-vehicle device 10B includes vehicle-side communication units 120B1 to 120B3 instead of the vehicle-side communication unit 120, and further includes a vehicle-side communication control unit 150. The vehicle-side communication unit 120B1 is a functional unit that realizes communication using a first communication means. The vehicle-side communication unit 120B2 is a functional unit that realizes communication using a second communication means different from the first communication means. The vehicle-side communication unit 120B3 is a functional unit that realizes communication using a third communication means different from the first and second communication means. The first to third communication means can be determined arbitrarily as long as they are different from each other. For example, the first to third communication means may be mobile communications with different communication carriers. For example, each of the communication means may conform to a different standard, such as a first communication means being a functional unit that communicates in accordance with next-generation mobile communications, a second communication means being a functional unit that communicates in accordance with satellite communications, and a third communication means being a functional unit that communicates in accordance with wireless communications.

[0058] The vehicle-side communication control unit 150 is a functional unit that controls communication multiplexing on the actual vehicle CB side. Specifically, in the simulation process (FIG. 2), the vehicle-side communication control unit 150 duplicates information transmitted from the actual vehicle CB and transmits the duplicated information to each of the vehicle-side communication units 120B1-120B3. In the simulation process (FIG. 2), the vehicle-side communication control unit 150 transmits only the first-arrived information among the information received by each of the vehicle-side communication units 120B1-120B3 from the server 20B to the other functional units of the in-vehicle device 10B.

[0059] Server 20B includes server-side communication units 220B1-220B3 instead of server-side communication unit 220, and further includes a server-side communication control unit 250. Server-side communication unit 220B1 is configured using the same communication method as vehicle-side communication unit 120B1. Server-side communication unit 220B2 is configured using the same communication method as vehicle-side communication unit 120B2. Server-side communication unit 220B3 is configured using the same communication method as vehicle-side communication unit 120B3. Server-side communication control unit 250 is a functional unit that controls communication multiplexing on the server 20B side. Specifically, in the simulation process (FIG. 2), server-side communication control unit 250 duplicates information transmitted from server 20B and transmits the duplicated information to each of server-side communication units 220B1-220B3. In the simulation process (FIG. 2), the server-side communication control unit 250 transmits only the first-arrived information among the information received from the actual vehicle CB by each of the server-side communication units 220B1 to 220B3 to the other functional units of the server 20B.

[0060] The vehicle-side communication unit 120B1 and the server-side communication unit 220B1 are also referred to as the "first communication unit." The vehicle-side communication unit 120B2 and the server-side communication unit 220B2 are also referred to as the "second communication unit." The vehicle-side communication unit 120B3 and the server-side communication unit 220B3 are also referred to as the "third communication unit." The vehicle-side communication control unit 150 and the server-side communication control unit 250 are also referred to as the "communication control unit."

[0061] As described above, the configuration of the simulator system 1000B can be modified in various ways, and it may have multiplexed communication units. The example in FIG. 6 illustrates three communication units. However, the simulator system 1000B may be configured to include any number of communication units, such as two or more. The vehicle-side communication units 120B1-120B3 are assumed to copy all information transmitted from the actual vehicle CB. Similarly, the server-side communication units 220B1-220B3 are assumed to copy all information transmitted from the server 20B. However, the vehicle-side communication units 120B1-120B3 and the server-side communication units 220B1-220B3 may copy only a portion of the information exchanged between the actual vehicle CB and the server 20B. The information to be copied is determined based on the priority of the information. For example, the actual vehicle position information, the front camera image, and the control data have a relatively high priority. Note that the front camera image may have a low resolution. For example, the orientation of the virtual vehicle and the high-resolution front camera image have a relatively low priority.

[0062] The simulator system 1000B of the third embodiment can also achieve the same effects as those of the first and second embodiments. Furthermore, according to the simulator system 1000B of the third embodiment, the communication unit includes first communication units 120B1 and 220B1, second communication units 120B2 and 220B2, and communication control units 150 and 250 that control multiplexing of communication using the first communication units 120B1 and 220B1 and the second communication units 120B2 and 220B2. Therefore, according to the simulator system 1000B, communication between the simulator unit 110A (i.e., the actual vehicle CB) and the vehicle control unit 210A (i.e., the server 20B that generates control data for controlling the actual vehicle CB) can be multiplexed.

[0063] <Fourth embodiment> 7 is a diagram illustrating an example of a simulator system 1000C according to the fourth embodiment. The simulator system 1000C according to the fourth embodiment employs a remote driving system similar to that of the second embodiment as a next-generation connected service, and is also a system capable of displaying information about the surroundings of the actual vehicle CC on an image displayed on a display device 213.

[0064] In the simulator system 1000C of the fourth embodiment, the actual vehicle CC further includes a recognition unit 12. The recognition unit 12 is a functional unit that recognizes objects around the actual vehicle CC in the real world. The recognition unit 12 can be configured by a LiDAR or a front camera that captures a front image of the actual vehicle CC. In FIG. 7, for convenience of illustration, the recognition unit 12 is written as "ER12." Before step S26 of the simulation process (FIG. 2), the recognition unit 12 transmits data obtained by the recognition unit 12 to the simulator unit 110C.

[0065] A simulator system 1000C of the fourth embodiment has an in-vehicle device 10C instead of the in-vehicle device 10A in the configuration described in the second embodiment. The in-vehicle device 10C has a simulator unit 110C instead of the simulator unit 110A. In step S26 of the simulation process (FIG. 2), the simulator unit 110C uses data acquired from the recognition unit 12 to generate an image in which an object reproducing an object recognized by the recognition unit 12 is superimposed on an image (a4 in the second embodiment) taken by a front camera mounted on the virtual vehicle. In step S26, the simulator unit 110C transmits the generated image instead of the image taken by the front camera of the virtual vehicle. Other processing by the simulator unit 110C is similar to that in the second embodiment.

[0066] FIG. 8 is a diagram showing an example of the image IMC displayed on the display device 213 in step d3. As a result of the above-described processing by the simulator unit 110C, the image IMC includes objects OB1 to OB3 in addition to the scenery UE ahead of the virtual vehicle and landmarks LM1 and LM2. The objects OB1 to OB3 are images that reproduce objects around the actual vehicle CC traveling through a city block in the real world. The objects OB1 and OB2 are pedestrians. The object OB3 is an oncoming vehicle. Thus, the "objects" recognized by the recognition unit 12 include people, vehicles, and the like. In the illustrated example, the objects OB1 to OB3 have shapes that are somewhat similar to objects in the real world. However, the objects OB1 to OB3 may have simplified shapes such as polygons. According to the image IMC in FIG. 8, the driver P can perform driving operations while referring to the image IMC that more closely resembles the actual traveling environment of the actual vehicle CC.

[0067] As described above, the configuration of the simulator system 1000C can be modified in various ways, and may be configured so that the driver P can visually recognize the environment around the real vehicle CC recognized by the recognition unit 12. In the example of FIG. 8, the simulator unit 110C generates an image in which objects OB1 to OB3 representing objects are superimposed on an image UE captured by a front camera mounted on a virtual vehicle. The simulator unit 110C may change the image UE captured by the front camera mounted on the virtual vehicle depending on the environment around the real vehicle CC other than the objects around the real vehicle CC. For example, if the real vehicle CC is traveling in the rain, the simulator unit 110C may change the image UE captured by the front camera to a rainy day. For example, if the real vehicle CC is traveling at night, the simulator unit 110C may change the image UE captured by the front camera to a night view. Rain and night are merely examples, and the simulator unit 110C may also take weather and time factors into consideration.

[0068] The simulator system 1000C of the fourth embodiment can also achieve the same effects as those of the first and second embodiments. Furthermore, according to the simulator system 1000C of the fourth embodiment, the simulator unit 110C transmits to the vehicle control unit 210A an image in which objects OB1 to OB3, which are reproductions of objects recognized by the recognition unit 12, are superimposed on an image captured by the front camera. Therefore, the vehicle control unit 210A can display on the display device 213 an image IMC in which objects OB1 to OB3, which are reproductions of objects around the actual vehicle in the real world, and markers LM1 and LM2, which indicate target positions, are superimposed on the image captured by the front camera. As a result, according to the simulator system 1000C, the driver P of the virtual vehicle can perform driving operations that are closer to reality with the control device 214 while recognizing objects around the actual vehicle in the real world (e.g., people near the road, oncoming vehicles, etc.).

[0069] Fifth Embodiment 9 is a diagram illustrating an example of a simulator system 1000D according to a fifth embodiment. The simulator system 1000D according to the fifth embodiment employs the same remote automated driving system as that of the first embodiment as a next-generation connected service. The simulator system 1000D according to the fifth embodiment includes an in-vehicle device 10D instead of the in-vehicle device 10 in the configuration described in the first embodiment. The in-vehicle device 10D includes an output unit 130D instead of the evaluation unit 130.

[0070] The output unit 130D does not execute steps S40 and S42 of the simulation process (FIG. 2). In step S50, the output unit 130D outputs the movement trajectories LC1 to LC7 of the actual vehicle CD and the movement trajectories LV1 to LV7 of the virtual vehicle, and then ends the process. The output destination of the movement trajectories can be determined arbitrarily. The output unit 130D may output the movement trajectories to a display device such as a display. The output unit 130D may output the movement trajectories to an arbitrary device on the network by transmitting them via the network. The output unit 130D may output the movement trajectories to a memory within the in-vehicle device 10 or attached to the in-vehicle device 10.

[0071] As described above, the configuration of the simulator system 1000D can be modified in various ways, and the simulator system 1000D may not include the evaluation unit 130 but may include the output unit 130D. Although the simulator system 1000D is exemplified as a system that realizes a remote automated driving system, the output unit 130D may also be included in a system that realizes a remote driving system. The simulator system 1000D of the fifth embodiment can also achieve the same effects as the first embodiment.

[0072] Sixth Embodiment FIG. 10 is a diagram illustrating an example of a simulator system 1000E according to a sixth embodiment. The simulator system 1000E according to the sixth embodiment employs the same remote automated driving system as that according to the first embodiment as a next-generation connected service. The simulator system 1000E according to the sixth embodiment includes an in-vehicle device 10E instead of the in-vehicle device 10 and a server 20E instead of the server 20 in the configuration described in the first embodiment. The in-vehicle device 10E does not include the evaluation unit 130 described in the first embodiment. The server 20E further includes an evaluation unit 230 in addition to the units described in the first embodiment. The function of the evaluation unit 230 of the server 20E is the same as that of the evaluation unit 130 described in the first embodiment.

[0073] As described above, the configuration of the simulator system 1000E can be modified in various ways, and functions corresponding to the evaluation unit and the output unit (fifth embodiment) may be provided on the server 20E side. The simulator system 1000E of the sixth embodiment can also achieve the same effects as those of the first embodiment.

[0074] <Modification of this embodiment> The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit of the present invention. For example, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. In addition, for example, the following modifications are also possible.

[0075] [Variation 1] In the above embodiment, an example of the configuration of the simulator system 1000, 1000A to 1000E has been shown. However, the configuration of the simulator system 1000 is merely an example, and any configuration can be adopted. For example, the simulator system 1000 may include other functional units not described above. For example, in the simulator system 1000, an automobile is exemplified as the actual vehicle C, but any target device can be used instead of the actual vehicle C. The target device can be any device that is to be controlled by the server 20 (or to which information is provided), such as a robot, a drone, or a smartphone.

[0076] [Variation 2] In the above embodiment, an example of the configuration of the actual vehicles C, CA to CE, and the in-vehicle devices 10, 10A to 10E has been described. However, the configurations of the actual vehicles C, CA to CE, and the in-vehicle devices 10, 10A to 10E are merely examples, and any configuration may be adopted. For example, the position acquisition unit 11 may be built into the in-vehicle device 10. For example, the in-vehicle device 10 may include other functional units not described above. For example, the in-vehicle device 10 may include a map corresponding to the city environment 3D model 113. In this case, the map may be used for route guidance for the actual vehicle C. Furthermore, the map may be used for driving assistance of the virtual vehicle by the simulator unit 110.

[0077] [Variation 3] In the above embodiment, an example of the processing procedure of the simulation processing (FIG. 2) has been described. However, the processing procedure of the simulation processing can be modified in various ways. In the simulation processing, the execution order of the above-described steps may be changed. In the simulation processing, other steps not described above may also be executed. In the simulation processing, some of the above-described steps may be omitted. For example, in step S50, the evaluation unit 130 may compare the movement trajectory of the virtual vehicle under wired communication instead of or in addition to the movement trajectory of the actual vehicle C. In other words, the evaluation unit 130 may compare the movement trajectory of the virtual vehicle obtained by executing the simulation processing in a non-wired communication environment (mobile communication, wireless communication, etc.) with the movement trajectory of the virtual vehicle obtained by executing the simulation processing in a wired communication environment in which communication interruptions are less likely to occur.

[0078] [Variation 4] The configurations of the simulator systems 1000, 1000A to 1000E of the first to sixth embodiments and the configurations of the simulator systems 1000, 1000A to 1000E of the first to third modifications may be combined as appropriate. For example, the communication multiplexing described in the third embodiment may be performed on the simulator system 1000 of the first embodiment. For example, the simulator systems 1000A to 1000C described in the second to fourth embodiments may be configured to include the output unit 130D described in the fifth embodiment. For example, the simulator systems 1000A to 1000D described in the second to fifth embodiments may be configured to include an evaluation unit (or output unit) in the server 20E.

[0079] This aspect has been described above based on embodiments and modifications. However, the above-described embodiments are intended to facilitate understanding of this aspect and are not intended to limit this aspect. This aspect may be modified or improved without departing from the spirit and scope of the claims, and equivalents thereof are included in this aspect. Furthermore, if a technical feature is not described as essential in this specification, it may be deleted as appropriate.

[0080] The present invention can also be realized in the following forms. [Application example 1] 1. A simulator system, comprising: Real vehicles are real vehicles placed in the real world, a simulator unit that realizes a virtual world that reproduces the real world and a virtual vehicle that is a virtual vehicle placed in the virtual world; a vehicle control unit that generates control data for running the virtual vehicle; a communication unit that realizes communication between the simulator unit and the vehicle control unit; an output unit that outputs a movement trajectory of the actual vehicle and a movement trajectory of the virtual vehicle; Equipped with the simulator unit is mounted on the actual vehicle, The vehicle control unit generates the control data for causing the virtual vehicle to travel following the actual vehicle, and transmits the control data to the simulator unit. [Application example 2] The simulator system according to Application Example 1, the actual vehicle has a position acquisition unit that acquires actual vehicle position information indicating a current position of the actual vehicle; the position acquisition unit transmits the acquired actual vehicle position information to the vehicle control unit; The simulator unit a virtual sensor that acquires information related to the virtual vehicle, the information including virtual vehicle position information indicating a current position of the virtual vehicle; transmitting sensor information acquired by the virtual sensor to the vehicle control unit; The vehicle control unit generates the control data using the actual vehicle position information and the sensor information. [Application example 3] The simulator system according to Application Example 1 or Application Example 2, The vehicle control unit setting a target position in the virtual world, the target position being a position in the virtual world corresponding to the actual vehicle position information; a simulator system that automatically generates the control data for causing the virtual vehicle to travel from a current position of the virtual vehicle identified by the virtual vehicle position information of the sensor information to the target position; [Application example 4] The simulator system according to any one of Application Examples 1 to 3, The sensor information further includes an image of a front camera of the virtual vehicle; The vehicle control unit a display unit that allows a driver to view an image; and an operation unit that accepts an operation by the driver with respect to the virtual vehicle, displaying, on the display unit, an image in which a mark indicating a target position in the virtual world, which is a position in the virtual world corresponding to the actual vehicle position information, is superimposed on the image of the front camera of the sensor information; A simulator system that generates the control data according to the content of the operation acquired from the operation unit. [Application example 5] The simulator system according to any one of Application Examples 1 to 4, the real vehicle further includes a recognition unit that recognizes objects around the real vehicle in the real world; The simulator unit transmits to the vehicle control unit, instead of the image from the front camera, an image in which an object that reproduces the object recognized by the recognition unit is superimposed on the image from the front camera. [Application Example 6] The simulator system according to any one of Application Examples 1 to 5, The communication unit a first communication unit that realizes communication between the simulator unit and the vehicle control unit using a first communication means; a second communication unit that realizes communication between the simulator unit and the vehicle control unit using a second communication means different from the first communication unit; a communication control unit that controls multiplexing of communication using the first communication unit and the second communication unit; a simulator system. [Application Example 7] The simulator system according to any one of Application Examples 1 to 6, The output unit is an evaluation unit that compares the movement trajectory of the actual vehicle with the movement trajectory of the virtual vehicle and evaluates the degree of discrepancy between the two movement trajectories. [Application Example 8] A simulation method using a real vehicle that is an actual vehicle located in the real world, a step in which a first information processing device mounted on the real vehicle realizes a virtual world that reproduces the real world and a virtual vehicle that is a virtual vehicle placed in the virtual world; a step in which a second information processing device located away from the actual vehicle generates control data for running the virtual vehicle; a step of communicating between the first information processing device and the second information processing device; a step in which one of the first information processing device and the second information processing device outputs a movement trajectory of the actual vehicle and a movement trajectory of the virtual vehicle; Equipped with The second information processing device generates the control data for causing the virtual vehicle to travel following the actual vehicle, and transmits the control data to the first information processing device. [Explanation of symbols]

[0081] 10,10A~10E…In-vehicle device 11...Position acquisition section 12...Recognition part 20, 20A, 20B, 20E...Server 110, 110A, 110C...Simulator section 111,111A...Virtual sensor 112...Virtual actuator 113...3D city environment model 120, 120B1 to 120B3...Vehicle side communication unit 130...Evaluation Department 130D...Output section 150...Vehicle-side communication control unit 210, 210A...Vehicle control unit 213...Display device 214...Control device 220, 220B1 to 220B3...Server side communication section 230...Evaluation Department 250...Server-side communication control unit 1000, 1000A~1000E...Simulator system OB1~OB3...Objects C, CA~CE...actual car P...Driver

Claims

1. 1. A simulator system, comprising: Real vehicles are real vehicles placed in the real world, a simulator unit that realizes a virtual world that reproduces the real world and a virtual vehicle that is a virtual vehicle placed in the virtual world; a vehicle control unit that generates control data for running the virtual vehicle; a communication unit that realizes communication between the simulator unit and the vehicle control unit; an output unit that outputs a movement trajectory of the actual vehicle and a movement trajectory of the virtual vehicle; Equipped with the simulator unit is mounted on the actual vehicle, The vehicle control unit generates the control data for causing the virtual vehicle to travel following the actual vehicle, and transmits the control data to the simulator unit.

2. 2. The simulator system according to claim 1, the actual vehicle has a position acquisition unit that acquires actual vehicle position information indicating a current position of the actual vehicle; the position acquisition unit transmits the acquired actual vehicle position information to the vehicle control unit; The simulator unit a virtual sensor that acquires information related to the virtual vehicle, the information including virtual vehicle position information indicating a current position of the virtual vehicle; transmitting sensor information acquired by the virtual sensor to the vehicle control unit; The vehicle control unit generates the control data using the actual vehicle position information and the sensor information.

3. 3. The simulator system according to claim 2, The vehicle control unit setting a target position in the virtual world, the target position being a position in the virtual world corresponding to the actual vehicle position information; a simulator system that automatically generates the control data for causing the virtual vehicle to travel from a current position of the virtual vehicle identified by the virtual vehicle position information of the sensor information to the target position;

4. 3. The simulator system according to claim 2, The sensor information further includes an image of a front camera of the virtual vehicle; The vehicle control unit a display unit that allows a driver to view an image; and an operation unit that accepts an operation by the driver with respect to the virtual vehicle, displaying, on the display unit, an image in which a mark indicating a target position in the virtual world, which is a position in the virtual world corresponding to the actual vehicle position information, is superimposed on the image of the front camera of the sensor information; A simulator system that generates the control data according to the content of the operation acquired from the operation unit.

5. 5. The simulator system according to claim 4, the real vehicle further includes a recognition unit that recognizes objects around the real vehicle in the real world; The simulator unit transmits to the vehicle control unit, instead of the image from the front camera, an image in which an object that reproduces the object recognized by the recognition unit is superimposed on the image from the front camera.

6. The simulator system according to any one of claims 1 to 5, The communication unit a first communication unit that realizes communication between the simulator unit and the vehicle control unit using a first communication means; a second communication unit that realizes communication between the simulator unit and the vehicle control unit using a second communication means different from the first communication unit; a communication control unit that controls multiplexing of communication using the first communication unit and the second communication unit; a simulator system.

7. The simulator system according to any one of claims 1 to 5, The output unit is an evaluation unit that compares the movement trajectory of the actual vehicle with the movement trajectory of the virtual vehicle and evaluates the degree of discrepancy between the two movement trajectories.

8. A simulation method using a real vehicle that is an actual vehicle located in the real world, a step in which a first information processing device mounted on the real vehicle realizes a virtual world that reproduces the real world and a virtual vehicle that is a virtual vehicle placed in the virtual world; a step in which a second information processing device disposed away from the actual vehicle generates control data for causing the virtual vehicle to run; a step of communicating between the first information processing device and the second information processing device; a step in which one of the first information processing device and the second information processing device outputs a movement trajectory of the actual vehicle and a movement trajectory of the virtual vehicle; Equipped with The second information processing device generates the control data for causing the virtual vehicle to travel following the actual vehicle, and transmits the control data to the first information processing device.

Images