Vehicle operation simulator system and vehicle operation simulator method

Abstract

Provided are a vehicle operation simulator system and a vehicle operation simulator method that enable an occupant or an observer to visually recognize, in real time, a real field in which a virtual object is placed. Specifically, a virtual object is placed in a real field, and a vehicle 2, which includes virtual sensors 22A – 22L for detecting the virtual object, is allowed to travel in the real field on the basis of the detection results of the virtual sensors 22A – 22L. While the travel results of the vehicle are collected, the real field in which an image 7 representing the virtual object is superimposed on a position at which the virtual object is placed is caused to be visually recognized.

Description

Vehicle motion simulator system and vehicle motion simulator method

[0001] The present invention relates to a vehicle motion simulator system and a vehicle motion simulator method for simulating the motion of a vehicle.

[0002] In recent years, in addition to manual driving, in which a vehicle is driven based on the user's driving operations, new proposals have been made for automated driving assistance systems that assist the user in driving the vehicle by having the vehicle perform some or all of the user's driving operations.In automated driving assistance systems, detection sensors such as ultrasonic sensors, millimeter-wave radar sensors, and LiDAR sensors are installed in the vehicle to detect surrounding objects (e.g., people, bicycles, other vehicles, walls, etc.), and vehicle control such as steering, drive source, and braking is automatically performed based on the detection results of the detection sensors.

[0003] Here, in the development process of the above-mentioned automated driving assistance system, a process is required to test whether the prototype automated driving assistance system operates correctly in a real driving environment. Conventionally, the test was performed by installing the prototype automated driving assistance system in a vehicle and driving the vehicle through a field where the above-mentioned objects and the like were actually placed. However, the test must be performed in various driving environments, and changing the environment requires changing the type of objects or rearranging them, which requires a very long preparation period before starting the test. Therefore, it has been proposed to realize a virtual driving environment by placing virtual objects (hereinafter referred to as virtual objects) in the real field, and to perform the test by installing a virtual sensor on the vehicle to detect the virtual objects (e.g., JP 2021-507247 A).

[0004] Special Publication No. 2021-507247 (Paragraph 0010-0016)

[0005] Here, in Patent Document 1, the virtual objects set in the driving environment do not actually exist, and therefore, even if the vehicle recognizes them, they cannot be directly seen by the occupants in the vehicle or by test observers outside the vehicle. Therefore, the occupants and observers observe the vehicle's behavior, such as stopping in empty spaces or driving while avoiding empty spaces. This makes it unclear the positional relationship between the vehicle and the virtual objects, making it difficult to evaluate whether the vehicle's behavior is appropriate. Furthermore, while the above describes the issues using an example of testing an automated driving assistance system, similar issues have arisen in systems other than automated driving assistance systems. For example, similar issues have arisen when testing a virtual driving environment to confirm whether driving assistance, such as deceleration control and warnings for obstacles around the vehicle, is performed appropriately during manual driving.

[0006] The present invention has been made to solve the above-mentioned problems in the prior art, and has as its object to provide a vehicle movement simulator system and a vehicle movement simulator method that, in a vehicle driving test in which virtual objects are placed in a real field, enable occupants and observers to visually recognize the real field in which virtual objects are placed in real time.

[0007] To achieve the above object, the vehicle operation simulator system of the present invention includes a virtual object placement means for placing a virtual object in a real field, a driving result collection means for collecting driving results of a vehicle equipped with a virtual sensor for detecting the virtual object in the real field, and a field viewing means for visualizing the real field in which an image representing the virtual object is superimposed at the position where the virtual object is placed. The term "sensor" also includes a camera that performs detection using image recognition. The term "virtual object" is not limited to three-dimensional objects but may be any object that can be detected by image recognition using a distance sensor or a camera, such as lane markings and road markings. The term "virtual object" is not limited to stationary objects but may also be moving objects. In the case of a moving object, a schedule for movement is set in advance. The term "vehicle" may refer to a vehicle capable of not only manual driving based on user driving operations, but also automated driving assistance, in which the vehicle automatically drives without user driving operations. In this case, the vehicle may be driven in a real field using automated driving assistance based on detection information from the virtual sensor, and the driving results may be collected. On the other hand, the vehicle is not necessarily limited to a vehicle capable of assisted driving by the above-mentioned automatic driving assistance, but may be a vehicle capable of only manual driving. In that case, it is possible to drive the vehicle manually in an actual field and collect the driving results.

[0008] Furthermore, the vehicle operation simulator method according to the present invention includes a step in which a virtual object placement means places a virtual object in a real field, a step in which a driving result collection means collects driving results of a vehicle in the real field, the vehicle being equipped with a virtual sensor for detecting the virtual object, and a step in which a field viewing means makes the real field visible, with an image representing the virtual object superimposed at the position where the virtual object is placed.

[0009] The vehicle motion simulator system and vehicle motion simulator method according to the present invention, having the above-described configuration, enable a vehicle occupant or observer to visually confirm in real time the real field on which the virtual objects are placed during a vehicle running test in which virtual objects are placed. As a result, the positional relationship between the vehicle and the virtual objects becomes clear, and it becomes possible to evaluate from a human perspective whether the vehicle's behavior is appropriate.

[0010] 1 is a schematic configuration diagram showing a vehicle operation simulator system according to a first embodiment. FIG. 1 is a diagram showing an example of displaying a real scene video of a real field on which an image showing a virtual object is superimposed on an in-vehicle display. FIG. 2 is a diagram showing an example of displaying a real scene video of a real field on which an image showing a virtual object is superimposed on a VR head-mounted display. FIG. 3 is a diagram showing an example of displaying a real scene video of a real field on which an image showing a virtual object is superimposed on a rearview mirror. FIG. 4 is a diagram showing an example of displaying a real scene video of a real field on which an image showing a virtual object is superimposed from a plurality of viewpoints. FIG. 5 is a schematic configuration diagram of a vehicle according to the first embodiment. FIG. 6 is an external view of a virtual sensor provided in the simulator device. FIG. 7 is a diagram showing the internal structure of the virtual sensor. FIG. 8 is a diagram explaining a method of switching sensor information using a sensor selector switch according to the first embodiment. FIG. 9 is a block diagram showing the configuration of the simulator device according to the first embodiment. FIG. 10 is a flowchart of an operation evaluation program according to the first embodiment. FIG. 11 is a diagram explaining a method of switching sensor information using a sensor selector switch according to a second embodiment.

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment and a second embodiment of a vehicle operation simulator system according to the present invention will be described in detail below with reference to the drawings.

[0012] First Embodiment First, the schematic configuration of a vehicle operation simulator system 1 according to a first embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic configuration diagram showing the vehicle operation simulator system 1 according to this embodiment.

[0013] As shown in FIG. 1 , the vehicle operation simulator system 1 according to this embodiment basically includes a vehicle 2 that is the subject of a simulated driving test, and a simulator device 3 that provides the vehicle 2 with a virtual driving environment that serves as the driving environment for the driving test, and acquires, analyzes, evaluates, and outputs the driving test operation results of the vehicle 2. In the first embodiment described below, the subject of the quality evaluation in the driving test is an automated driving assistance system that assists the occupant in driving the vehicle by performing some or all of the occupant's driving operations on the vehicle side, and the vehicle 2 is a vehicle equipped with the automated driving assistance system to be evaluated. However, as described below, the vehicle 2 is not necessarily limited to a vehicle equipped with the automated driving assistance system, but may also be a vehicle that can only be driven manually. In this case, the driving test is intended to evaluate whether driving assistance, such as deceleration control and warnings regarding obstacles around the vehicle, is appropriately performed during manual driving, for example.

[0014] The driving test is one of the processes that manufacturers use to discover and correct defects in the automated driving assistance system before shipping, and involves simulating the operation of the automated driving assistance system under various expected driving environments, and acquiring and analyzing the operation results to confirm whether or not a defect will occur. The driving test is performed by actually driving the vehicle 2 in a real field (for example, a wide, flat road with nothing around), and by further adding a virtual driving environment using the simulator device 3, it becomes possible to perform driving tests under various driving environments even in the same real field.

[0015] The simulator device 3 is mounted on the vehicle 2 and is connected to the vehicle 2 via an in-vehicle network such as a CAN to enable two-way communication. However, the simulator device 3 does not necessarily have to be mounted on the vehicle 2; a part or all of the simulator device 3 may be installed outside the vehicle 2. In this case, the simulator device 3 is connected to the vehicle 2 via, for example, wireless communication to enable two-way communication. The simulator device 3 is also equipped with an operating means such as a keyboard and a display as a display means, and is configured to allow input and output of various information through operations by an evaluator (observer / occupant) 4 who evaluates the automated driving assistance system. In particular, the evaluator 4 inputs to the simulator device 3 the positions and types of virtual objects to be placed as a driving environment on the real field on which the vehicle 2 travels. The simulator device 3 then creates a virtual driving environment in which a driving test of the vehicle 2 is performed according to the input driving environment.

[0016] The simulator device 3 also includes a virtual sensor that enables the vehicle 2 to detect virtual objects constructed as the driving environment, and the detection results of the vehicle 2 detecting the virtual objects with the virtual sensor while driving on the real field are output to a control device that controls automatic driving assistance on the vehicle 2 side. This allows the vehicle side to recognize the virtual objects as objects that actually exist (e.g., people, bicycles, other vehicles, walls, etc.), making it possible to reproduce a driving environment that is closer to reality.

[0017] Furthermore, the simulator device 3 is configured to be able to perform either "constant analysis" or "post-analysis" for analyzing the driving results of the vehicle 2. "Constant analysis" monitors the vehicle 2 during the driving test, and if any malfunction occurs, automatically detects the malfunction and outputs the results. On the other hand, "post-analysis" evaluates and outputs the control details and driving trajectory of the vehicle 2 stored during the driving test after the driving test of the vehicle 2 is completed.

[0018] Therefore, when conducting a driving test of the vehicle 2, the evaluator 4 first inputs the type and position of virtual objects to be placed in the real field into the simulator device 3 as the driving environment for the driving test of the vehicle 2. This generates the driving environment for the driving test. Note that "virtual objects" are not limited to three-dimensional objects but also include, for example, lane markings and road markings. Furthermore, they may be stationary or moving objects. In the case of moving objects, a movement schedule (speed and direction) is also set in advance. For example, when conducting a driving test in a parking lot, parking lane marks painted on the road surface of the parking lot, other vehicles parked in parking spaces, pedestrians moving within the parking lot, and other vehicles traveling through the aisles within the parking lot are input as virtual objects. After that, when the driving test begins, virtual sensors installed in the vehicle detect the virtual objects generated as the driving environment, thereby simulating the vehicle's autonomous driving assistance in the virtual driving environment. The driving results are then analyzed, evaluated, and output through "on-the-fly analysis" or "post-analysis," making it possible to identify any malfunctions that may occur.

[0019] However, because the virtual objects placed in the real field by the simulator device 3 do not actually exist, they cannot be directly viewed by the evaluator 4 even if they are recognized on the vehicle side by virtual sensors. Therefore, particularly when performing "constant analysis," the positional relationship between the vehicle 2 and the virtual objects is unclear, making it difficult for the evaluator 4 to evaluate whether the vehicle's behavior is appropriate from his or her line of sight. Therefore, the vehicle operation simulator system 1 of the first embodiment is provided with the following means for allowing the evaluator 4 to visually view the real field in which an image representing a virtual object is superimposed at the position where the virtual object is placed. Note that the evaluator 4 may be riding in the vehicle 2 as a passenger or observing from outside the vehicle, and the following description will assume both cases.

[0020] First, a case where the evaluator 4, who is a passenger in the vehicle 2, is allowed to view the image will be described. In the first example, as shown in FIG. 2 , a real-time real-view image 6 of the real field captured by an external camera is displayed on an in-vehicle display 5 that is mounted on the vehicle 2 and visible to the passengers of the vehicle. Then, an image 7 showing a virtual object is superimposed on the real-view image 6 of the real field displayed on the in-vehicle display 5, thereby allowing the evaluator 4 to view the real field in which the image 7 showing the virtual object is superimposed at the position where the virtual object is placed. The simulator device 3 converts the placement coordinates (absolute coordinates) of the virtual object set as the driving environment into the coordinate system (relative coordinates) of the external camera, thereby enabling the position of the virtual object in the real-view image 6 of the real field displayed on the in-vehicle display 5 to be identified.

[0021] Next, in a second example, VR (Virtual Reality) technology is used to have the evaluator 4, who is a vehicle occupant, wear a VR head-mounted display 8. As shown in FIG. 3 , the VR head-mounted display 8 displays a pre-recorded, real-time image 6 of the real field captured by an external camera on the outside of the window, in addition to a pre-recorded, real-time image 6 of the real field. Then, by combining an image 7 of a virtual object with the displayed real-time image 6 of the real field, the evaluator 4 can visually recognize the real field with the image 7 of the virtual object superimposed at the position where the virtual object is located. The simulator device 3 converts the placement coordinates (absolute coordinates) of the virtual object set as the driving environment into a VR (i.e., user line-of-sight) coordinate system (relative coordinates), thereby enabling the position of the virtual object in the real-time image 6 of the real field displayed on the VR head-mounted display 8 to be identified.

[0022] Furthermore, when the evaluator 4, who is a passenger in the vehicle 2, is made to view the image 7 showing the virtual object as in the first and second examples, the image 7 showing the virtual object may also be superimposed on the real field reflected or displayed on one or both of the rearview mirror and the side mirror arranged in the vehicle. That is, the image 7 showing the virtual object is superimposed on the location where the virtual object is placed in the real field reflected or displayed on one or both of the rearview mirror and the side mirror. For example, if the rearview mirror and the side mirror are monitor-type mirrors that display images captured by an external camera, as shown in FIG. 4 , the image 7 showing the virtual object can be synthesized with the real scene image 6 of the real field displayed in the image display area of ​​the rearview mirror 9, thereby allowing the evaluator 4 to view the real field in which the image 7 showing the virtual object is superimposed at the location where the virtual object is placed. On the other hand, if the rearview mirror and the side mirrors are mirror-type displays with a transparent display on the mirror surface, the transparent display can superimpose an image 7 showing a virtual object on the actual scene of the real field reflected in the mirror, allowing the evaluator 4 to view the real field with the image 7 showing the virtual object superimposed at the position where the virtual object is placed.

[0023] In addition to the above examples, a HUD (head-up display) or a see-through VR head-mounted display can also be used as a means for allowing the evaluator 4, who is a passenger in the vehicle 2, to visually recognize a virtual object placed in the real field. These are technologies for visually recognizing an image by superimposing it on the real view outside the vehicle that is actually viewed through the front window or side window, and therefore display control is performed so that an image representing the virtual object is superimposed on the position of the virtual object within the field of view of the passenger.

[0024] Next, a case where the image is visually recognized by the evaluator 4 observing outside the vehicle will be described. For the evaluator 4 observing outside the vehicle, a real-time real-view video 6 of the real field captured by an outside-vehicle camera is displayed on the display of a tablet-type external terminal 11 communicably connected to the simulator device 3. The details are the same as when it is displayed on the in-vehicle display 5, and by combining the real-view video 6 of the real field displayed on the external terminal 11 with an image 7 showing a virtual object, the evaluator 4 can visually recognize the real field with the image 7 showing the virtual object superimposed at the position where the virtual object is placed.

[0025] However, particularly when displaying the real field on the external terminal 11, it is not necessary to display the real field included in the field of view from the viewpoint of the vehicle 2; it is also possible to display the real field included in the field of view from a position different from that of the vehicle 2. For example, if the external terminal 11 is equipped with a camera, a real-time real-view image 6 of the real field captured by the camera of the external terminal 11 is displayed on the display of the external terminal 11, as shown in FIG. 5 . The simulator device 3 then acquires optical axis information from the camera of the external terminal 11 and converts the placement coordinates (absolute coordinates) of the virtual objects set as the driving environment into the coordinate system (relative coordinates) of the camera of the external terminal 11, thereby making it possible to identify the positions of the virtual objects on the real field captured by the camera of the external terminal 11. Then, by superimposing an image 7 representing the virtual object on the real-view image 6 of the real field displayed on the display of the external terminal 11, it is possible to allow the evaluator 4 to visually recognize the real field in which the image 7 representing the virtual object is superimposed at the position where the virtual object is placed.

[0026] Furthermore, the evaluator 4 may operate the external terminal 11 to switch between displaying the real field from the viewpoint of the position of the vehicle 2 and displaying the real field from the viewpoint of the position of the evaluator 4. This makes it possible to visually recognize in real time the real field in which virtual objects are placed from various viewpoints on the real field, rather than from a fixed viewpoint. Furthermore, in addition to displaying the real field from the viewpoint of the position of the vehicle 2 and the position of the evaluator 4, if imaging devices are installed on the real field, it is also possible to display the real field from the viewpoint of those imaging devices.

[0027] Next, a description will be given below of a vehicle 2 equipped with an automatic driving assistance system that is the subject of evaluation by the vehicle operation simulator system 1. Fig. 6 is a schematic configuration diagram of the vehicle 2 according to this embodiment.

[0028] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine automobile) that uses an internal combustion engine (engine, etc.) as a drive source, an automobile (electric automobile, fuel cell automobile, etc.) that uses an electric motor (motor, etc.) as a drive source, or an automobile that uses both of these as drive sources (hybrid automobile). Furthermore, the vehicle type is not limited, and may be a standard car, a large commercial truck, a bus, construction machinery, or the like. Furthermore, although the following description will be of a four-wheeled automobile, it may also be a two-wheeled or three-wheeled vehicle.

[0029] However, vehicle 2 is a vehicle capable of manual driving, in which the vehicle drives based on the user's driving operation, as well as assisted driving using automatic driving assistance, in which the vehicle drives automatically without the user's driving operation.

[0030] Furthermore, the autonomous driving assistance may be performed only under specific circumstances, such as when parking or leaving a parking lot, or may be performed for all road sections, or may be performed only while the vehicle is traveling on a specific road section (for example, a highway with a gate (manned or unmanned, toll or free) at the boundary). In the following description, the autonomous driving section in which the autonomous driving assistance of the vehicle is performed includes all road sections, including general roads and highways, as well as parking lots. Furthermore, the autonomous driving assistance is performed only when the user selects to perform the autonomous driving assistance (for example, by turning on the autonomous driving start button) and it is determined that autonomous driving assistance is possible. On the other hand, the vehicle 2 may be a vehicle that is only capable of assisted driving with autonomous driving assistance. Alternatively, the autonomous driving assistance may be performed only when the vehicle is traveling to a parking space when parking (i.e., parking assistance).

[0031] In the vehicle control in the automated driving assistance of this embodiment, for example, the current position of the vehicle, the lane the vehicle is traveling on, and the positions of surrounding obstacles are detected as needed, and vehicle control of the steering, drive source, brakes, etc. is automatically performed so that the vehicle travels along the generated travel trajectory at a speed according to the generated speed plan. In particular, when performing parking assistance, the detection results of sensors and cameras are used to check the parking space where the vehicle will park and the surrounding conditions, and the parking trajectory to the parking space is calculated, and the vehicle is automatically controlled to enter the parking space along the calculated parking trajectory and complete parking.

[0032] As shown in FIG. 6, the vehicle 2 also has an operation unit 13 that accepts operations from the occupant, an in-vehicle display 5 (or a VR head-mounted display 8) that displays to the occupant the above-mentioned real field real-view image 6 (FIG. 2) and other output information from the simulator device 3, a speaker 15 that outputs audio guidance from the simulator device 3, a front camera 16, a rear camera 17, and side cameras 18A and 18B for capturing images of the area around the vehicle, ultrasonic sensors 19A to 19L that are real sensors that detect obstacles around the vehicle, a driving assistance ECU (electronic control unit) 20 that is a control unit that performs various calculations related to automatic driving assistance based on input information, the above-mentioned simulator device 3, and a sensor selector switch 21 that switches the sensor information to be input to the driving assistance ECU 20 as described below.

[0033] The components of the vehicle 2 will be described below. First, the operation unit 13 is disposed, for example, on the front of the steering wheel (also referred to as the steering wheel), and includes an operation button operated to start automatic driving assistance and an operation button operated to switch the sensor information of the sensor changeover switch 21. By operating the operation unit 13, the user can switch between manual driving, in which the vehicle travels based on the user's driving operation, and automatic driving assistance, in which the vehicle travels automatically without the user's driving operation. It is also possible to switch between real sensors and virtual sensors for the sensor information input to the driving assistance ECU 20 (described later). The operation unit 13 may include a touch panel provided on the front of the in-vehicle display. It may also include a microphone and a voice recognition device.

[0034] The in-vehicle display 5 is mounted on the instrument panel of the vehicle 2 and displays a real scene video 6 ( FIG. 2 ) of the real field on which an image 7 representing a virtual object is superimposed at the position where the virtual object is placed, as described above. The in-vehicle display 5 may be replaced with the VR head-mounted display 8 described above. Furthermore, as described above, a display device ( FIG. 4 ) may be provided on the rearview mirror and side mirrors to display the real scene video 6 of the real field on which an image 7 representing a virtual object is superimposed. The real scene video 6 of the real field displayed on the in-vehicle display 5 or the VR head-mounted display 8 is a video obtained by combining the image 7 representing the virtual object described above with real-time video captured by the front camera 16, rear camera 17, and side cameras 18A and 18B.

[0035] The speaker 15 is also provided on the instrument panel of the vehicle 2 and outputs voice guidance, warning sounds, etc. from the simulator device 3.

[0036] The forward camera 16 is an imaging device having a camera using a solid-state imaging element such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with its optical axis facing forward in the direction of travel of the vehicle.

[0037] The rear camera 17 is an imaging device that has a camera that also uses a solid-state imaging element such as a CCD, and is installed, for example, near the upper center of the license plate attached to the rear of the vehicle 2, with its optical axis facing toward the rear of the vehicle.

[0038] Furthermore, the side cameras 18A, 18B are imaging devices having cameras that similarly use solid-state imaging elements such as CCDs, and are installed, for example, on the left and right side mirrors of the vehicle 2 with their optical axes facing to the sides of the vehicle.

[0039] The simulator device 3 reads information about the type and installation position of the virtual object in the driving environment previously set by the evaluator 4, and synthesizes an image 7 showing the virtual object at the position where the virtual object is placed with the captured images taken by the front camera 16, rear camera 17, and side cameras 18A and 18B, thereby generating a real scene video 6 of the real field to be displayed on the in-vehicle display 5 or the VR head-mounted display 8. The real scene video onto which the image 7 showing the virtual object is synthesized may be the real scene video itself captured by the front camera 16, rear camera 17, and side cameras 18A and 18B, or may be, for example, a composite of images from multiple cameras or an image obtained by viewpoint conversion (for example, a bird's-eye view image or a bird's-eye view image).

[0040] On the other hand, ultrasonic sensors 19A-19L correspond to real sensors that detect real objects around the vehicle. Note that, as counterparts to ultrasonic sensors 19A-19L, there are virtual sensors 22A-22L (described below) that cannot detect real objects but can detect virtual objects, and these sensors are also installed on vehicle 2. Ultrasonic sensors 19A-19L are arranged at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as search waves around vehicle 2 and detect objects that reflect the search waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measurement sensor that can detect the distance (distance measurement value) to an object that reflects the search wave by measuring the time from transmission to reception. Furthermore, ultrasonic sensors 19A-19L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. The objects to be detected by the ultrasonic sensors 19A to 19L include obstacles that the vehicle 2 must avoid when traveling, such as people, bicycles, other vehicles, and walls, or obstacles that form parking spaces. A millimeter wave sensor or a radar sensor may be used as a distance measuring sensor instead of an ultrasonic sensor.

[0041] The installation position and installation direction of each ultrasonic sensor 19A-19L can be set as appropriate. In this embodiment, to detect objects in all directions (forward, backward, left, and right) of the vehicle 2, for example, ultrasonic sensors 19A-19D are installed on the front of the vehicle 2 facing the direction of travel so that the transmission direction of their search waves is forward of the vehicle's direction of travel. Ultrasonic sensors 19E and 19F are installed on the left side of the vehicle 2 facing leftward so that the transmission direction of their search waves is to the left of the vehicle's direction of travel. Ultrasonic sensors 19G and 19H are installed on the right side of the vehicle 2 facing rightward so that the transmission direction of their search waves is to the right of the vehicle's direction of travel. Ultrasonic sensors 19I-19L are installed on the rear of the vehicle 2 facing in the opposite direction to the vehicle's direction of travel so that the transmission direction of their search waves is to the rear of the vehicle. Each ultrasonic sensor 19A-19L is approximately the same height from the ground surface.

[0042] In this embodiment, among the ultrasonic sensors 19A to 19L, the ultrasonic sensors 19A to 19D on the front of the vehicle 2 and the ultrasonic sensors 19I to 19L on the rear of the vehicle 2 are particularly installed in positions where they can receive reflected waves from adjacent sensors as indirect waves, so that by receiving direct and indirect waves as received waves, it is possible to determine not only the distance to an object but also the specific position of the object (its position relative to the vehicle) using triangulation. The ultrasonic sensors 19E to 19H on the sides are installed at a distance from each other and therefore cannot receive indirect waves, but as the vehicle moves, it is possible to determine the specific position of the object (its position relative to the vehicle) by triangulation using the measured distance at the previous position, the measured distance at the current position, and the distance traveled between them.

[0043] Meanwhile, the driving assistance ECU 20 is an electronic control unit that performs various processes related to autonomous driving assistance. The driving assistance ECU 20 is connected to various sensors for detecting vehicle behavior, such as a vehicle speed sensor, a wheel speed sensor, an acceleration sensor, a gyro sensor, a steering sensor, and a shift position sensor, as well as various drive units of the vehicle, such as the steering, brake, accelerator, and transmission. Based on the detection results of these sensors, the driving assistance ECU 20 detects the current vehicle behavior, calculates control information (control variables) for controlling each drive unit, and controls each drive unit to perform autonomous driving assistance for the vehicle 2. Specific examples of autonomous driving assistance include, for example, constantly detecting the current vehicle position, the lane the vehicle is traveling on, and the positions of surrounding obstacles, and controlling the vehicle's steering, drive source, brakes, and other functions so that the vehicle travels along a generated travel trajectory at a speed according to a generated speed plan. In particular, when performing parking assistance, the driving assistance ECU 20 checks the parking space where the vehicle is to be parked and its surrounding conditions, calculates a parking trajectory to the parking space, and controls the vehicle to enter the parking space along the calculated parking trajectory and complete parking.

[0044] Here, the means by which the driving assistance ECU 20 acquires information about surrounding obstacles and parking spaces necessary for performing the above-mentioned autonomous driving assistance include means for acquiring detection information from the ultrasonic sensors 19A to 19L and means for acquiring detection information from the virtual sensors 22A to 22L. The ultrasonic sensors 19A to 19L are sensors for detecting real objects around the vehicle, while the virtual sensors 22A to 22L are sensors for detecting virtual objects in a driving environment virtually generated by the simulator device 3. The virtual sensors 22A to 22L are part of the simulator device 3 and are mounted on the vehicle 2.

[0045] 7 is an external view of the virtual sensors 22A to 22L provided in the simulator device 3, and FIG. 8 is a view showing the internal structure of the virtual sensors 22A to 22L. The virtual sensors 22A to 22L basically have the same structure, and the following description will be given taking the virtual sensor 22A as an example.

[0046] As shown in Figures 7 and 8, the virtual sensor 22A has a rectangular box shape, with a first ultrasonic sensor 25 at one end of the box and a second ultrasonic sensor 26 at the other end. The first ultrasonic sensor 25 and the second ultrasonic sensor 26 have essentially the same structure as the ultrasonic sensors 19A to 19L described above and are arranged facing each other so that their detection axes X are coaxial. The detection axis X is a virtual line extending from the first ultrasonic sensor 25 and the second ultrasonic sensor 26 along the transmission and reception direction of the search wave. The first ultrasonic sensor 25 and the second ultrasonic sensor 26 each have a piezoelectric vibrator (e.g., a ceramic plate) for transmitting and receiving the search wave. The piezoelectric vibrator is formed in a planar shape with the detection axis X as its normal. The first ultrasonic sensor 25 and the second ultrasonic sensor 26 are configured to transmit the search wave along the detection axis X by ultrasonically vibrating the piezoelectric vibrator based on a drive signal applied to the electromechanical transducer. In addition, when receiving waves from the outside, the first ultrasonic sensor 25 and the second ultrasonic sensor 26 are configured to generate a receiving signal, which is an electrical signal corresponding to the excitation state of the piezoelectric vibrator by the received ultrasonic waves, using an electro-mechanical conversion element.

[0047] The virtual sensor 22A detects that the first ultrasonic sensor 25 has received a reflected wave when the electromotive force generated by the vibration of the piezoelectric vibrator due to reception of the search wave exceeds a threshold, and calculates the distance to the target based on the time between when the first ultrasonic sensor 25 transmits the search wave and when it receives the search wave. However, the target here is a virtual object, and the first ultrasonic sensor 25 does not actually transmit a search wave; instead, it receives the search wave transmitted from the second ultrasonic sensor 26, recognizing it as a reflected wave. In this case, the distance measurement value detected by the virtual sensor 22A can be freely controlled by the timing of transmitting the search wave from the second ultrasonic sensor 26.

[0048] That is, the simulator device 3 identifies the current position and orientation of the vehicle based on information from sensors installed in the vehicle, such as a vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, and identifies the relative position of the virtual object with respect to the vehicle 2 based on the identified current position and orientation of the vehicle and the position of the virtual object placed in a driving environment previously set by the evaluator 4. Then, based on the identified relative position of the virtual object with respect to the vehicle 2 (more specifically, the ultrasonic sensors 19A to 19L equipped on the vehicle), the timing of transmitting search waves from the second ultrasonic sensor 26 is controlled, and the virtual sensors 22A to 22L thereby detect the non-existent virtual object.

[0049] In the vehicle operation simulator system 1 of the first embodiment, while a driving test is being conducted (while driving results are being collected), the driving assist ECU 20 detects surrounding obstacles and parking spaces based on the detection information from the virtual sensors 22A to 22L, and performs vehicle control such as steering, drive source, and braking. Meanwhile, during the preparation stage before the driving test (the preparation stage before collecting driving results) and after the driving test (after collecting driving results), the vehicle needs to be driven while avoiding real walls and obstacles. Therefore, the driving assist ECU 20 detects surrounding obstacles and parking spaces based on the detection information from the ultrasonic sensors 19A to 19L, which are real sensors, and performs vehicle control such as steering, drive source, and braking. The sensor information input to the driving assist ECU 20 is switched by a sensor selector switch 21.

[0050] The vehicle is equipped with the same number of virtual sensors 22A-22L as the ultrasonic sensors 19A-19L, which are real sensors, and virtual sensor 22A corresponds to ultrasonic sensor 19A. That is, during the driving test, the driving assistance ECU 20 recognizes virtual sensor 22A as ultrasonic sensor 19A and detects an object (a virtual object that does not actually exist) located to the left front of the vehicle. For example, if the distance from ultrasonic sensor 19A to the virtual object is 1 meter, the timing of transmitting a search wave from second ultrasonic sensor 26 is controlled so that the distance measurement value detected by virtual sensor 22A is 1 meter. This results in the detection of an object (actually a virtual object) located 1 meter from ultrasonic sensor 19A. Similarly, virtual sensor 22B corresponds to ultrasonic sensor 19B, virtual sensor 22C corresponds to ultrasonic sensor 19C, and virtual sensor 22D corresponds to ultrasonic sensor 19D. The same applies to the remaining virtual sensors 22E-22L.

[0051] In addition, vehicle 2 has basic components as vehicle 2 in addition to the components shown in Figure 6, but we will only explain the configuration related to the control of automatic driving assistance and the control related to that configuration.

[0052] Next, a description will be given of a method for switching the sensor information to be input to the driving assistance ECU 20 using the sensor changeover switch 21. As shown in Fig. 9, the vehicle operation simulator system 1 of the first embodiment is provided with a first input path for inputting detection information from the virtual sensors 22A to 22L to the driving assistance ECU 20, and a second input path for inputting detection information from the ultrasonic sensors 19A to 19L, which are real sensors, to the driving assistance ECU 20.

[0053] The sensor selector switch 21 selectively disconnects either the first input path or the second input path, thereby enabling the information used for autonomous driving assistance in the driving assistance ECU 20 to be switched between the detection information of the virtual sensors 22A-22L and the detection information of the ultrasonic sensors 19A-19L. The switching control of the sensor selector switch 21 is performed by the simulator device 3. While the autonomous driving assistance system is collecting driving results (i.e., during a driving test), the information used for autonomous driving assistance by the driving assistance ECU 20 is switched to the detection information of the virtual sensors 22A-22L, and after the collection of driving results has ended or during the preparation stage before collecting driving results, the information used for autonomous driving assistance in the driving assistance ECU 20 is switched to the detection information of the ultrasonic sensors 19A-19L. However, this switching may be performed automatically or may be performed based on, for example, the operation of the operation unit 13 or the external terminal 11 by the evaluator 4. In particular, when the test is performed based on the operation of the external terminal 11, for example, when start of test is selected on the external terminal 11, a signal is transmitted from the external terminal 11 to the simulator device 3, and the simulator device 3 controls the sensor changeover switch 21 based on the received signal to switch the information used for autonomous driving assistance in the driving assistance ECU 20 from the detection information of the ultrasonic sensors 19A to 19L to the detection information of the virtual sensors 22A to 22L. On the other hand, when end of test is selected on the external terminal 11, a signal is transmitted from the external terminal 11 to the simulator device 3, and the simulator device 3 controls the sensor changeover switch 21 based on the received signal to switch the information used for autonomous driving assistance in the driving assistance ECU 20 from the detection information of the virtual sensors 22A to 22L to the detection information of the ultrasonic sensors 19A to 19L.

[0054] Next, a detailed description will be given of the simulator device 3 provided in the vehicle operation simulator system 1. Fig. 10 is a block diagram showing the configuration of the simulator device 3 according to this embodiment.

[0055] As shown in FIG. 10, the simulator device 3 has a virtual control ECU (electronic control unit) 40, which is a control unit that performs various calculation processes such as generating a driving environment when conducting a driving test, controlling the virtual sensors 22A to 22L, and analyzing, evaluating, and outputting the operation results of the driving test of the vehicle 2, and the virtual sensors 22A to 22L described above.

[0056] The virtual control ECU 40 is an electronic control unit that controls the entire simulator device 3. It includes internal storage devices such as a CPU 41 as a calculation device and a control device, a RAM 42 used as a working memory when the CPU 41 performs various calculation processes, a ROM 43 in which control programs as well as a performance evaluation program (see FIG. 11 ) described below are stored, and a flash memory 44 for storing programs read from the ROM 43. The virtual control ECU 40, together with the driving assistance ECU 20, includes various means serving as processing algorithms. For example, the virtual object placement means places a virtual object in a real field. The driving result collection means collects driving results of the vehicle 2 equipped with virtual sensors 22A-22L for detecting the virtual object. The field visualization means allows the user to visualize a real field in which an image representing the virtual object is superimposed at the position where the virtual object is placed.

[0057] In addition, the virtual control ECU 40 is connected via an in-vehicle network such as a CAN to the aforementioned operation unit 13, in-vehicle display 5 (or VR head-mounted display 8), speaker 15, front camera 16, rear camera 17, side cameras 18A and 18B, ultrasonic sensors 19A to 19L, driving assistance ECU 20, and sensor selector switch 21, as well as various sensors 47 for detecting vehicle behavior, such as a vehicle speed sensor, a wheel speed sensor, an acceleration sensor, a gyro sensor, a steering sensor, and a shift position sensor, and is able to detect the current position, direction, and current behavior of the vehicle based on the detection results of these sensors 47. The virtual control ECU 40 provides the vehicle 2 with a virtual driving environment that serves as the driving environment for the driving test, while also acquiring, analyzing, evaluating, and outputting the operation results of the automated driving assistance system from the vehicle behavior during the driving test. Furthermore, based on the behavior of the vehicle during the driving test, a real-field scene image 6 (Figures 2 to 4) is generated and displayed on the aforementioned in-vehicle display 5, VR head-mounted display 8, and external terminal 11, and the image is output to these display devices.

[0058] The flash memory 44 also contains driving environment information 45 and a driving result DB 46. The driving environment information 45 stores information about the driving environment set in advance by the evaluator 4 when a driving test of the vehicle 2 is conducted. Specifically, the information includes the types and positions of virtual objects to be placed on the real field, and these are input in advance by the evaluator 4. It is also possible to store multiple patterns of driving environments in advance.

[0059] On the other hand, the driving result DB 46 is a storage means for cumulatively storing the vehicle control details, the vehicle's driving trajectory, and the history of the vehicle's behavior during the driving test. After the driving test operation of the vehicle 2 is completed, the simulator device 3 evaluates and outputs the control details, driving trajectory, etc. of the vehicle 2 stored in the driving result DB 46.

[0060] Next, an operation evaluation program executed in the simulator device 3 constituting the vehicle operation simulator system 1 having the above configuration will be described with reference to FIG. 11 . FIG. 11 is a flowchart of the operation evaluation program according to this embodiment. Here, the operation evaluation program is executed when the simulator device 3 receives a corresponding program startup operation, constructs a virtual driving environment, and evaluates the quality of the automated driving assistance system provided in the vehicle based on the results of a driving test. Note that the program startup operation may be triggered, for example, by an evaluator 4 riding as a vehicle occupant operating the operation unit 13, or may be initiated by an evaluator 4 outside the vehicle performing a predetermined operation on the external terminal 11. The program shown in the flowchart in FIG. 11 below is stored in the RAM 42, ROM 43, etc. provided in the simulator device 3, and executed by the CPU 41.

[0061] First, in step (hereinafter abbreviated as S) 1, the CPU 41 of the simulator device 3 performs various initial setting processes before executing a driving test, such as checking the operation of the virtual sensors 22A to 22L.

[0062] Next, in S2, the CPU 41 reads from the flash memory 44 setting information for the driving environment for carrying out a driving test of the vehicle 2. The driving environment includes, for example, the types and positions of virtual objects to be placed on the real field.

[0063] Here, the driving environment is basically set and stored in advance in the simulator device 3 by input operations by the evaluator 4. It is also possible to set multiple driving environments, in which case the driving environment to be used for this driving test can be selected arbitrarily. The driving environment can be set to, for example, driving on a public road or driving in a parking lot. Furthermore, the "virtual object" is not limited to a three-dimensional object but also includes, for example, a parking space line and road markings. For example, when a driving test is performed in a parking lot, parking space lines drawn on the road surface of the parking lot, other vehicles parked in parking spaces, pedestrians moving in the parking lot, other vehicles moving through the aisles of the parking lot, etc. are set as virtual objects. Furthermore, for moving virtual objects, a movement schedule (at what speed and in what way they move) is also set.

[0064] Next, in S3, the CPU 41 constructs a virtual driving environment for the real field where the vehicle is currently located based on the setting information read in S2. That is, the CPU 41 places virtual objects of the set type at set positions in the real field. Furthermore, moving virtual objects are subsequently moved according to a set schedule.

[0065] Thereafter, in S4, the CPU 41 operates the sensor changeover switch 21 to switch the information used for the autonomous driving assistance by the driving assistance ECU 20 to the information detected by the virtual sensors 22A to 22L. As described above, the sensor changeover switch 21 selectively disconnects one of the first input path and the second input path, thereby switching the information used for the autonomous driving assistance by the driving assistance ECU 20 between the information detected by the virtual sensors 22A to 22L and the information detected by the ultrasonic sensors 19A to 19L (FIG. 9). Here, the sensor changeover switch 21 may automatically switch the sensor information when the driving test starts, or may switch the sensor information when a predetermined operation is received from the evaluator 4. For example, when the start of the test is selected on the external terminal 11, a signal may be transmitted from the external terminal 11 to the simulator device 3, and the simulator device 3 may operate the sensor changeover switch 21 based on the received signal.

[0066] Note that until the switching is performed in S4, the information used for autonomous driving assistance in the driving assistance ECU 20 is the detection information of the ultrasonic sensors 19A to 19L, which are real sensors. Therefore, in the preparation stage before the start of the driving test, the vehicle 2 is capable of autonomous driving assistance driving based on the detection information of the ultrasonic sensors 19A to 19L, which are real sensors, and it becomes possible to move the vehicle 2 by autonomous driving while avoiding obstacles such as real walls and other vehicles when moving to the start position of the driving test, for example.

[0067] Thereafter, in S5, the CPU 41 starts a driving test of the vehicle 2. When the driving test starts, vehicle control by automated driving assistance is started in the vehicle 2, and, for example, the current position of the vehicle, the lane on which the vehicle is traveling, and the positions of surrounding obstacles are detected at any time, and vehicle control such as steering, drive source, and brakes is automatically performed so that the vehicle travels on the actual field along the generated driving trajectory at a speed according to the generated speed plan. In particular, when evaluating parking assistance using a parking lot as the driving environment, the detection information from the sensors is used to confirm the parking space in which the vehicle will park and the surrounding conditions, and the parking trajectory to the parking space is calculated, and the vehicle is automatically controlled to enter the parking space along the calculated parking trajectory and complete parking.

[0068] While the vehicle 2 is traveling with the above-described automatic driving assistance, the CPU 41 of the simulator device 3 outputs the detection results of the virtual objects detected by the virtual sensors 22A-22L by the vehicle 2 traveling in the real field to the driving assistance ECU 20, which controls the automatic driving assistance on the vehicle 2 side. Specifically, the CPU 41 determines the current position and orientation of the vehicle based on information from each sensor 47 installed on the vehicle, such as a vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, and determines the relative position of the virtual object with respect to the vehicle 2 based on the determined current position and orientation of the vehicle and the position of the virtual object placed in the traveling environment established in S3. Then, based on the determined relative position of the virtual object with respect to the vehicle 2 (more specifically, the ultrasonic sensors 19A-19L equipped on the vehicle), the CPU 41 controls the timing of transmitting search waves from the second ultrasonic sensors 26 equipped in the virtual sensors 22A-22L, thereby causing the virtual sensors 22A-22L to detect the non-existent virtual object.

[0069] Next, in S6, the CPU 41 acquires a real-time real-view image of the real field to be displayed on the in-vehicle display 5, the VR head-mounted display 8, or the external terminal 11, based on the captured images captured by the front camera 16, the rear camera 17, and the side cameras 18A and 18B. The real-view image may be the real-view image captured by the front camera 16, the rear camera 17, and the side cameras 18A and 18B, or may be an image obtained by combining images from multiple cameras or by converting viewpoints (for example, an overhead image or a bird's-eye view image).

[0070] The real-time image of the real field to be displayed on the external terminal 11 may be an image captured by a camera provided in the external terminal 11, instead of an image captured by a camera on the vehicle 2, or an image captured by a camera placed on the real field. The following processing is performed for each frame of the image acquired in S6.

[0071] Next, in S7, the CPU 41 converts the placement coordinates (absolute coordinates) of the virtual object in the driving environment constructed in S3 into the coordinate system (relative coordinates) of the real scene image of the real field acquired in S6, thereby identifying the position of the virtual object in the real scene image of the real field acquired in S6. For example, when the real scene image of the real field captured by the front camera 16 is displayed on the in-vehicle display 5, the placement coordinates (absolute coordinates) of the virtual object are converted into the coordinate system (relative coordinates) of the front camera 16, thereby identifying the position of the virtual object in the real scene image of the real field displayed on the in-vehicle display 5. On the other hand, when VR display is performed using the VR head-mounted display 8, the placement coordinates (absolute coordinates) of the virtual object are converted into the coordinate system (relative coordinates) of VR (i.e., the user's line of sight), thereby identifying the position of the virtual object in the real scene image of the real field displayed on the VR head-mounted display 8.

[0072] Then, in S8, the CPU 41 composites an image 7 representing a virtual object at the position identified in S7 onto the real-world image of the real field acquired in S6. As the image 7 representing a virtual object, 3D model images for each type of virtual object, such as a vehicle or a person, are pre-stored in the flash memory 44. The CPU 41 then reads the corresponding type of model image from the flash memory 44, rotates and scales it to match the virtual object placed in the real field, and composites it onto the real-world image. For virtual objects that have a certain orientation, such as vehicles, the orientation is also composited according to the constructed driving environment. Texture mapping is also performed to apply a texture image to the surface of the model image. It is desirable to draw the effect of the texture image so that it does not look unnatural when the model image is placed in the real field, taking into account the orientation of the placement, the ambient brightness, the position of light sources (e.g., sunlight, street lights), etc. The model image to be composited may be an opaque image with a transmittance of 0% or a semi-transparent image.

[0073] Next, in S9, the CPU 41 outputs a real-time image of the real field onto which the image 7 representing the virtual object has been superimposed to the in-vehicle display 5, the VR head-mounted display 8, or the external terminal 11, and displays it on each display device. As a result, the evaluator 4 can visually recognize the real field onto which the image 7 representing the virtual object has been superimposed at the position where the virtual object is placed (FIGS. 2 and 3). Furthermore, as described above, the real-field image onto which the image 7 representing the virtual object has been superimposed may be displayed on the rearview mirror and the side mirror (FIG. 4).

[0074] Next, in S10, the CPU 41 determines whether the driving test has been completed. For example, when evaluating parking assistance in a parking lot as the driving environment, the driving test is determined to be completed when the vehicle is parked in a parking space and the shift position is in P. On the other hand, the driving test may be determined to be completed when a predetermined distance or time has been driven since the start of the driving test.

[0075] If it is determined that the driving test has been completed (S10: YES), the process proceeds to S11. On the other hand, if it is determined that the driving test has not been completed (S10: NO), the process returns to S5, and the driving test of the vehicle 2 continues.

[0076] In S11, the CPU 41 operates the sensor selector switch 21 to switch the information used for autonomous driving assistance by the driving assistance ECU 20 to the information detected by the ultrasonic sensors 19A-19L, which are real sensors. As described above, the sensor selector switch 21 selectively disconnects one of the first input path and the second input path, thereby switching the information used for autonomous driving assistance by the driving assistance ECU 20 between the information detected by the virtual sensors 22A-22L and the information detected by the ultrasonic sensors 19A-19L (FIG. 9). The sensor selector switch 21 may switch the sensor information automatically upon completion of the driving test or upon receipt of a predetermined operation from the evaluator 4. For example, when the end of the test is selected on the external terminal 11, a signal may be transmitted from the external terminal 11 to the simulator device 3, and the simulator device 3 may operate the sensor selector switch 21 based on the received signal.

[0077] As a result, after the driving test is completed, the vehicle 2 is capable of autonomous driving assistance driving based on the detection information of the ultrasonic sensors 19A to 19L, which are real sensors, and for example, when moving the vehicle 2 to the vehicle storage position after the driving test is completed, it becomes possible to move the vehicle 2 by autonomous driving while avoiding obstacles such as real walls and other vehicles. After that, the operation evaluation program is terminated.

[0078] The analysis of the driving results of the vehicle 2 when a vehicle driving test is performed by the above-mentioned operation evaluation program can be performed in either "on-the-fly analysis" or "post-analysis." "On-the-fly analysis" monitors the vehicle 2 during the driving test, and if any malfunction occurs, automatically detects the malfunction and outputs the results. On the other hand, "post-analysis" evaluates and outputs the control details and driving trajectory of the vehicle 2 stored during the driving test, taking into account the detection results from the virtual sensors 22A to 22L, after the driving test operation of the vehicle 2 is completed.

[0079] The CPU 41 may also output an evaluation window containing details of any malfunctions that occurred after the driving test to an external display. The evaluation window displays the details of the malfunctions, along with the time at which the malfunctions occurred. As a result, the evaluator 4 can understand when and what kind of malfunction occurred in the automated driving assistance system. Furthermore, the CPU 41 may store the real-field scene video (including the virtual object image 7) output in S9 as an image log, extract from the image log an image output when it is determined that a malfunction occurred in the operation of the automated driving assistance system, and make the image available for viewing by the evaluator 4. The evaluator 4 can then make program modifications to the automated driving assistance system based on the output driving test results.

[0080] As described above in detail, the vehicle motion simulator system 1, the simulation method using the vehicle motion simulator system 1, and the computer program executed by the vehicle motion simulator system 1 according to this embodiment place a virtual object in a real field (S3), cause a vehicle 2 equipped with virtual sensors 22A-22L for detecting the virtual object to travel in the real field with autonomous driving assistance based on detection information from the virtual sensors 22A-22L (S5), collect the vehicle's travel results while allowing the vehicle to visually confirm the real field with an image 7 representing the virtual object superimposed at the position where the virtual object was placed (S9). Therefore, in a driving test of autonomous driving assistance in which a virtual object is placed in the real field, the real field with the virtual object placed can be visually confirmed in real time by a passenger or observer. As a result, the positional relationship between the vehicle and the virtual object becomes clear, and it becomes possible to evaluate whether the vehicle's behavior is appropriate from a human perspective. The system also has at least one of an in-vehicle display 5 or a VR head-mounted display 8 that is mounted on the vehicle 2 and visible to the occupants of the vehicle 2, and an external terminal 11 that is outside the vehicle 2 and visible to an observer outside the vehicle, and the real field is made visible to the occupants of the vehicle by displaying on the in-vehicle display 5 or the VR head-mounted display 8 a captured image of the real field in which an image 7 representing a virtual object has been superimposed at the position where the virtual object is placed, and the real field is made visible to the observer by displaying on the external terminal 11 a captured image of the real field in which an image 7 representing a virtual object has been superimposed at the position where the virtual object is placed, so that the real field in which the virtual object is placed can be made visible in real time to both the occupants in the vehicle and an observer monitoring from outside without being in the vehicle. In addition, it is possible to switch between a first viewing mode, which allows the user to view the real field included in the field of view from the viewpoint of the vehicle, and a second viewing mode, which allows the user to view the real field included in the field of view from a viewpoint different from the vehicle, so that the real field on which virtual objects are placed can be viewed in real time from various viewpoints in the real field rather than from a fixed viewpoint.In addition, an image 7 representing a virtual object is displayed superimposed on at least one of the rearview mirror and side mirrors arranged on the vehicle at a location where the virtual object is placed in the real field reflected or displayed in the rearview mirror and side mirror, allowing the virtual object placed in the real field to be viewed in a manner closer to reality. Furthermore, a virtual object is placed in the real field (S3), and a vehicle 2 equipped with virtual sensors 22A-22L for detecting the virtual object is caused to travel in the real field with autonomous driving assistance based on detection information from the virtual sensors 22A-22L (S5). While collecting the vehicle's travel results, the system is also equipped with a sensor selector switch 21 that switches the information used for the autonomous driving assistance between detection information from the virtual sensors 22A-22L and detection information from the real sensors. This makes it possible to perform preparation work and cleanup work after testing more efficiently than in the past. Furthermore, while collecting driving results to evaluate the automated driving assistance system, the information used for the automated driving assistance is switched to the detection information of the virtual sensors 22A to 22L, and after the collection of driving results is completed or during the preparation stage before collecting the driving results, the information used for the automated driving assistance is switched to the detection information of the ultrasonic sensors 19A to 19L, which are real sensors (S4, S11). This makes it possible to move the vehicle by automated driving while avoiding obstacles such as real walls and other vehicles when moving the vehicle to the start position of the driving test or to the vehicle storage position after the driving test is completed. Furthermore, since the information used for the automated driving assistance is switched between the detection information of the virtual sensors 22A to 22L and the detection information of the ultrasonic sensors 19A to 19L based on a signal input from outside the vehicle, it is possible to move the vehicle to the start position of the driving test and start the driving test by remote control from outside.The vehicle also has a driving assistance ECU 20 that calculates control information for driving with automatic driving assistance, a first input path that inputs detection information from virtual sensors 22A to 22L to the driving assistance ECU 20, and a second input path that inputs detection information from ultrasonic sensors 19A to 19L to the driving assistance ECU 20. By selectively disconnecting one of the first input path and the second input path, the information used for automatic driving assistance in the driving assistance ECU 20 can be switched between the detection information from virtual sensors 22A to 22L and the detection information from ultrasonic sensors 19A to 19L. This makes it possible to switch the sensor information used for automatic driving assistance by electrical switch control, without the need for manual work such as changing wiring.

[0081] [Second Embodiment] Next, a vehicle operation simulator system according to a second embodiment will be described with reference to Fig. 12. In the following description, the same reference numerals as those in the vehicle operation simulator system 1 according to the first embodiment shown in Figs. 1 to 11 indicate the same or corresponding parts as those in the vehicle operation simulator system 1 according to the first embodiment.

[0082] The schematic configuration of the vehicle operation simulator system according to the second embodiment is almost the same as that of the vehicle operation simulator system 1 according to the first embodiment. In addition, the various control processes are also almost the same as those of the vehicle operation simulator system 1 according to the first embodiment. However, while the vehicle operation simulator system 1 according to the first embodiment selectively disconnects one of the first and second input paths for the sensor selector switch 21 as shown in FIG. 9 to switch the information used for automatic driving assistance between the detection information of the virtual sensors 22A to 22L and the detection information of the ultrasonic sensors 19A to 19L, the vehicle operation simulator system according to the second embodiment is different in that it includes a first driving assistance ECU that inputs detection information from the virtual sensors to calculate control information for driving with automatic driving assistance, and a second driving assistance ECU that inputs detection information from the ultrasonic sensors 19A to 19L to calculate control information for driving with automatic driving assistance, and switches between acquiring control information from either the first driving assistance ECU or the second driving assistance ECU to switch the information used for automatic driving assistance between the detection information of the virtual sensors 22A to 22L and the detection information of the ultrasonic sensors 19A to 19L.

[0083] The following describes a method for switching the sensor information input to the driving assistance ECU using the sensor changeover switch 21 for the vehicle operation simulator system according to the second embodiment. As shown in Fig. 12, the vehicle operation simulator system according to the second embodiment includes a first driving assistance ECU 101 and a second driving assistance ECU 102 as control units that calculate control information (control information for each driving unit of the vehicle, such as the steering, brake, accelerator, and transmission) for driving with automatic driving assistance. The system also includes an input path for inputting detection information from virtual sensors 22A to 22L to the first driving assistance ECU 101, and an input path for inputting detection information from ultrasonic sensors 19A to 19L, which are real sensors, to the second driving assistance ECU 102. The first driving assistance ECU 101 calculates control information for driving with automatic driving assistance from the detection information of the virtual sensors 22A to 22L, and the second driving assistance ECU 102 calculates control information for driving with automatic driving assistance from the detection information of the ultrasonic sensors 19A to 19L, which are real sensors.

[0084] The sensor selector switch 21 switches the connection of the CAN of the vehicle 2, thereby enabling the information used for autonomous driving assistance to be switched between the detection information of the virtual sensors 22A-22L and the detection information of the ultrasonic sensors 19A-19L when autonomous driving assistance is performed. That is, when the first driving assistance ECU 101 is connected, autonomous driving assistance is performed based on the detection information of the virtual sensors 22A-22L, and when the second driving assistance ECU 102 is connected, autonomous driving assistance is performed based on the detection information of the ultrasonic sensors 19A-19L. Note that switching control of the sensor selector switch 21 is performed by the simulator device 3, but as in the first embodiment, while the autonomous driving assistance system is being evaluated (i.e., during a driving test), the switch is switched so that autonomous driving assistance is performed based on the detection information of the virtual sensors 22A-22L, and after the evaluation is completed or in the preparation stage before the evaluation, the switch is switched so that autonomous driving assistance is performed based on the detection information of the ultrasonic sensors 19A-19L.

[0085] It should be noted that the present invention is not limited to the above-described embodiments, and various improvements and modifications are possible without departing from the spirit and scope of the present invention. For example, in the first and second embodiments, virtual sensors 22A to 22L are used as sensors capable of detecting virtual objects instead of ultrasonic sensors 19A to 19L. However, instead of distance measuring sensors such as ultrasonic sensors, virtual sensors that replace cameras that detect targets through image recognition may also be used. In this case, the virtual sensors can detect virtual objects by using, for example, an image in which a virtual object is synthesized with an image to be recognized.

[0086] In addition, in the first and second embodiments, the information used for autonomous driving assistance is switched to detection information from a virtual sensor when the driving test begins, and the information used for autonomous driving assistance is switched to detection information from a real sensor when the driving test ends, but this may be done only at either the start or end of the driving test.

[0087] 11 is executed by the simulator device 3 mounted on the vehicle in the first and second embodiments, but the simulator device 3 may be provided outside the vehicle. In that case, the simulator device 3 and the vehicle 2 are capable of communicating with each other via wireless communication.

[0088] In the first and second embodiments, the subject of the quality evaluation in the driving test is an automated driving assistance system that assists the occupant in driving the vehicle by performing some or all of the occupant's driving operations on the vehicle side, and the vehicle 2 is a vehicle equipped with the automated driving assistance system to be evaluated. However, the vehicle 2 is not necessarily limited to a vehicle equipped with the automated driving assistance system and may be a vehicle that can only be driven manually. For such a vehicle that can only be driven manually, a driving test is performed to evaluate whether driving assistance, such as deceleration control and warnings for obstacles around the vehicle, is appropriately performed during manual driving. In this case, for example, vehicle control is not performed in S5, and the driver manually drives the vehicle 2 and drives it on the real field. Then, during driving, the CPU 41 of the simulator device 3 outputs detection results of virtual objects detected by the virtual sensors 22A to 22L of the vehicle 2 traveling on the real field to the ECU 20 on the vehicle 2, and vehicle driving assistance (e.g., deceleration control and warnings for obstacles) is performed. In addition, in S4 and S11, the information used for the driving assistance of the vehicle performed during manual driving is switched between the detection information of the virtual sensor and the detection information of the real sensor.

[0089] [Summary of this embodiment] This embodiment has at least the following configuration: virtual object placement means (20) that places a virtual object in a real field, driving result collection means (20) that collects driving results in the real field of a vehicle (2) that is equipped with virtual sensors (22A to 22L) for detecting the virtual object, and field viewing means (20) that allows the real field to be viewed with an image (7) representing the virtual object superimposed at the position where the virtual object is placed.

[0090] With this configuration, in a driving test in which a virtual object is placed in a real field, the real field on which the virtual object is placed can be visually recognized by passengers and observers in real time. As a result, the positional relationship between the vehicle and the virtual object becomes clear, and it becomes possible to evaluate from a human perspective whether the vehicle's behavior is appropriate.

[0091] Furthermore, this embodiment has at least one of an on-board display device (5, 8) mounted on the vehicle (2) and viewable by the occupants of the vehicle, and an exterior display device (11) located outside the vehicle and viewable by an observer outside the vehicle, and the field viewing means (20) preferably allows the occupants of the vehicle to view the real field by displaying, on the on-board display device, a captured image (6) of the real field on which an image (7) representing the virtual object has been superimposed at the position where the virtual object is placed, and allows the observer to view the real field by displaying, on the exterior display device, a captured image of the real field on which an image representing the virtual object has been superimposed at the position where the virtual object is placed.

[0092] With this configuration, it is possible for both the occupants in the vehicle and an observer monitoring from outside the vehicle to view the real field in which the virtual objects are placed in real time.

[0093] In addition, in this embodiment, it is preferable that the field viewing means (20) be switchable between a first viewing mode in which the real field is viewed within the field of view from the viewpoint of the vehicle (2), and a second viewing mode in which the real field is viewed within the field of view from a viewpoint different from that of the vehicle.

[0094] According to this configuration, it is possible to view the real field in which the virtual objects are arranged in real time from various viewpoints in the real field, rather than from a fixed viewpoint.

[0095] In addition, in this embodiment, it is preferable that the field viewing means (20) displays, on at least one of a rearview mirror (9) and a side mirror arranged on the vehicle (2), an image (7) showing the virtual object superimposed on the location where the virtual object is arranged in the real field reflected or displayed on the rearview mirror and the side mirror.

[0096] This configuration makes it possible to visually recognize virtual objects placed on the real field in a manner closer to reality.

[0097] 1...vehicle operation simulator system, 2...vehicle, 3...simulator device, 4...evaluator (observer, occupant), 5...in-vehicle display (in-vehicle display device), 6...real scene image of real field, 7...image of virtual object, 8...VR head-mounted display (in-vehicle display device), 9...room mirror, 11...external terminal (external display device), 19A to 19L...ultrasonic sensor (real sensor), 20...driving assistance ECU, 21...sensor selector switch, 22A to 22L...virtual sensor, 40...virtual control ECU, 41...CPU

Claims

1. A vehicle operation simulator system having: a virtual object placement means for placing a virtual object in a real field; a driving result collection means for collecting driving results in the real field of a vehicle equipped with a virtual sensor for detecting the virtual object; and a field visualization means for visualizing the real field with an image representing the virtual object superimposed at the position where the virtual object is placed.

2. A vehicle operation simulator system according to claim 1, comprising at least one of an on-board display device mounted on the vehicle and viewable by an occupant of the vehicle, and an exterior display device located outside the vehicle and viewable by an observer outside the vehicle, wherein the field viewing means allows the occupant of the vehicle to view the real field by displaying, on the on-board display device, a captured image of the real field onto which an image representing the virtual object has been superimposed at a position where the virtual object has been placed, and allows the observer to view the real field by displaying, on the exterior display device, a captured image of the real field onto which an image representing the virtual object has been superimposed at a position where the virtual object has been placed.

3. A vehicle operation simulator system as described in claim 1 or claim 2, wherein the field viewing means is switchable between a first viewing mode in which the real field is viewed within a field of view from the viewpoint of the vehicle, and a second viewing mode in which the real field is viewed within a field of view from a viewpoint different from that of the vehicle.

4. A vehicle operation simulator system according to claim 1 or claim 2, wherein the field viewing means displays, in at least one of a rearview mirror and a side mirror arranged on the vehicle, an image showing the virtual object superimposed on the location where the virtual object is arranged in the real field reflected or displayed on the rearview mirror and the side mirror.

5. A vehicle operation simulator method comprising: a step in which a virtual object placement means places a virtual object in a real field; a step in which a driving result collection means collects driving results in the real field of a vehicle equipped with a virtual sensor for detecting the virtual object; and a step in which a field viewing means makes the real field visible, with an image representing the virtual object superimposed at the position where the virtual object is placed.

Images