Vehicle operation simulator system

Abstract

This system provides a vehicle operation simulator that allows occupants and observers to visually confirm the detection range and status of virtual objects detected by virtual sensors in real time, overlaid on the actual field. [Solution] A virtual object is placed in a real field, and the results of a vehicle 2 equipped with virtual sensors 22A to 22L for detecting the virtual object are collected when the vehicle travels in the real field. At the same time, the virtual sensors 22A to 22L detect the virtual object by simulating the process of transmitting probe waves from their installation position to the area around the vehicle and receiving reflected waves that are reflected by virtual objects around the vehicle. The system is configured to allow the user to view a real field overlaid with a probe wave image 10 that shows the probe waves transmitted from the virtual sensors 22A to 22L traveling in the direction of installation from their installation position.

Description

Technical Field

[0001] The present invention relates to a vehicle operation simulator system for simulating the operation of a vehicle.

Background Art

[0002] In recent years, as a driving mode of a vehicle, in addition to manual driving that travels based on a user's driving operation, an automatic driving support system that assists the user in driving a vehicle by executing part or all of the user's driving operations on the vehicle side has been newly proposed. In the automatic driving support system, detection sensors for detecting surrounding objects (for example, people, bicycles, other vehicles, walls, etc.) such as ultrasonic sensors, millimeter-wave radar sensors, LiDAR sensors, etc. are arranged on the vehicle, and vehicle control such as steering, drive source, brake, etc. is automatically performed based on the detection results of the detection sensors.

[0003] Here, in the process of developing a vehicle control system such as the above automatic driving support system, a process of testing whether the prototype vehicle control system operates correctly in an actual driving environment is necessary. Conventionally, the above test has been performed by mounting a prototype vehicle control system on a vehicle and actually driving on a field where the above objects are arranged. However, although the above test needs to be performed in various driving environments, in order to change the environment, it is necessary to change the types of objects or rearrange them, and a very long preparation period is required until the test starts. Therefore, it has been proposed to perform the above test by realizing a virtual driving environment by arranging virtual objects (hereinafter referred to as virtual objects) that do not exist in the actual field and mounting a virtual sensor for detecting the virtual objects on the vehicle (for example, Korean Registered Patent No. 10-1357596).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] In the above-mentioned Patent Document 1, the virtual object set in the driving environment is not actually present, and the virtual sensor used to detect it is not actually installed on the exterior wall of the vehicle, nor does it transmit probe waves around the vehicle. Therefore, while it is possible to confirm how the virtual sensor transmits probe waves and receives reflected waves to detect the virtual object by analyzing the results after the test, it is difficult for the vehicle occupants or test observers outside the vehicle to grasp this in real time. Consequently, there was a problem in evaluating whether the vehicle's behavior in response to the detection result of the virtual object was appropriate.

[0006] The present invention was made to solve the aforementioned problems of the conventional method, and aims to provide a vehicle operation simulator system that enables the detection range and detection status of virtual objects by virtual sensors to be superimposed on the real field and viewed by occupants or observers in real time during vehicle driving tests in which virtual objects are placed in a real field. [Means for solving the problem]

[0007] To achieve the above objective, the vehicle operation simulator system according to the present invention includes: virtual object placement means for placing virtual objects in a real field; virtual object detection means for detecting the virtual objects by using a virtual sensor assumed to be installed at a predetermined installation position in a predetermined direction on the vehicle, transmitting a probe wave from the installation position to the area around the vehicle in accordance with the installation direction, and receiving reflected waves reflected by the virtual objects around the vehicle; and image viewing means for viewing the real field on which at least one of a probe wave image showing the propagation of the probe wave transmitted from the virtual sensor and a reflected wave image showing the propagation of the reflected wave when the probe wave is reflected by the virtual object is superimposed. Furthermore, a "virtual object" can be any object detectable by the probe waves, and is not limited to stationary objects; it can also be a moving object. If it is a moving object, a schedule for its movement will be set in advance. [Effects of the Invention]

[0008] According to the vehicle operation simulator system of the present invention having the above configuration, in a vehicle driving test conducted by placing a virtual object in a real field, it becomes possible to superimpose the detection range and detection status of the virtual object by the virtual sensor onto the real field and allow the occupants or observers to see it in real time. As a result, it becomes clear how the virtual sensor transmits search waves and receives reflected waves to detect the virtual object, and it becomes possible to evaluate from a human perspective whether the vehicle's behavior in response to the detection result of the virtual object is appropriate. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing the vehicle operation simulator system according to this embodiment. [Figure 2] This diagram shows actual sensors installed on a vehicle. [Figure 3] This diagram illustrates how an object is detected using a real sensor positioned in front of the vehicle. [Figure 4]This diagram illustrates how to detect an object using actual sensors positioned on the side of a vehicle. [Figure 5] This figure shows an example of displaying a real-world field image, overlaid with a probe image, on an in-vehicle display. [Figure 6] This diagram shows an example of displaying a real-world field image (overhead view) with a probe image superimposed on it on an in-vehicle display. [Figure 7] This is a diagram showing the probe wave image. [Figure 8] This figure shows the reflected wave image. [Figure 9] This figure shows a modified version of the probe wave image. [Figure 10] This figure illustrates an example of displaying a real-world field image, overlaid with a probe image, on a VR head-mounted display. [Figure 11] This diagram shows an example of displaying a real-world field image, superimposed with a probe image, on a rearview mirror. [Figure 12] This diagram illustrates an example of displaying real-world footage of a real field, overlaid with images representing virtual objects, from multiple viewpoints. [Figure 13] This is a schematic diagram of the vehicle according to this embodiment. [Figure 14] This is an external view of the virtual sensor installed in the simulator device. [Figure 15] This is a diagram showing the internal structure of a virtual sensor. [Figure 16] This is a block diagram showing the configuration of the simulator device according to this embodiment. [Figure 17] This is a flowchart of the operation evaluation program according to this embodiment. [Figure 18] This diagram shows the direction (directivity) and range (detection area) of the probe wave transmitted from the virtual sensor. [Modes for carrying out the invention]

[0010] Hereinafter, a detailed description will be given of an embodiment in which the vehicle operation simulator system according to the present invention is embodied, with reference to the drawings. First, the schematic configuration of the vehicle operation simulator system 1 according to the present embodiment will be described with reference to FIG. 1. FIG. 1 is a schematic configuration diagram showing the vehicle operation simulator system 1 according to the present embodiment.

[0011] As shown in FIG. 1, the vehicle operation simulator system 1 according to the present embodiment basically includes a vehicle 2 that is the object of a driving test by simulation, and a simulator device 3 that gives a virtual driving environment that is the driving environment for the driving test to the vehicle 2, and acquires, analyzes, evaluates, and outputs the operation results of the driving test of the vehicle 2. In the following embodiments, the object of quality evaluation in the above driving test is an automatic driving support system that assists the driving of the vehicle by the driver by the vehicle side executing part or all of the driving operations of the driver. The vehicle 2 is a vehicle equipped with the automatic driving support system to be evaluated.

[0012] Note that the driving test is one of the processes for the manufacturer to discover and correct defects in the automatic driving support system before shipment. By simulating the operations of the automatic driving support system in various assumed driving environments and acquiring and analyzing the operation results, it is confirmed whether defects occur. The driving test is performed by actually driving the vehicle 2 in a real field (for example, a wide flat road surface with nothing around), and by further giving a virtual driving environment by the simulator device 3, it becomes possible to perform driving tests in various driving environments even in the same real field.

[0013] Furthermore, the simulator device 3 is mounted on the vehicle 2 and is connected to the vehicle 2 via an in-vehicle network such as CAN, enabling bidirectional communication. However, the simulator device 3 does not necessarily have to be mounted on the vehicle 2; it may be partially or entirely installed outside the vehicle 2. In that case, it can be connected to the vehicle 2 via, for example, wireless communication, enabling bidirectional communication. The simulator device 3 is also equipped with operating means such as a keyboard and display means such as a display, and is configured to allow input and output of various information through operation by the evaluator (observer / occupant) 4 who evaluates the automated driving support system. In particular, the evaluator 4 inputs the position and type of virtual objects to be placed as the driving environment on the actual field in which the vehicle 2 travels to the simulator device 3. The simulator device 3 then constructs a virtual driving environment for conducting driving tests of the vehicle 2 according to the input driving environment.

[0014] Furthermore, the simulator device 3 is equipped with virtual sensors for vehicle 2 to detect virtual objects constructed as the driving environment described above. The detection results obtained when vehicle 2, driving in the real field, detects virtual objects with the virtual sensors are output to a control device that controls the automatic driving assistance on vehicle 2. As a result, the vehicle recognizes the virtual objects as actual objects (for example, people, bicycles, other vehicles, walls, etc.), making it possible to reproduce a driving environment that is closer to reality.

[0015] Furthermore, evaluator 4 inputs and sets the assumed installation position and orientation of the virtual sensor by operating the simulator device 3 in advance. Here, the installation position and orientation of the virtual sensor follow the installation position and orientation of the sensor actually mounted on the vehicle 2 being tested. For example, Figure 2 shows the sensors 9A to 9L that are actually installed on vehicle 2 (hereinafter referred to as actual sensors in contrast to the virtual sensor). In the following explanation, the actual sensors 9A to 9L will be described as ultrasonic sensors, but millimeter-wave sensors or radar sensors may be used instead of ultrasonic sensors.

[0016] As shown in Figure 2, the actual sensors 9A to 9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as probe waves around the vehicle 2 and detect objects that reflected the probe waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measuring sensor that can detect the distance (measured distance value) to the object that reflected the probe wave by measuring the time from transmission to reception. The actual sensors 9A to 9L are also configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the vehicle's control unit. However, at least during testing, virtual sensors are used instead of the actual sensors 9A to 9L, so a switch, for example, is provided to allow switching between the output signals from the actual sensors 9A to 9L and the output signals from the virtual sensors when inputting signals to the control unit.

[0017] Furthermore, while the installation position and direction of each sensor 9A to 9L can be set as appropriate, in this embodiment, in order to make the detection range of objects in all directions—forward, backward, and to the left and right sides of the vehicle 2's direction of travel—for example, sensors 9A to 9D are installed on the front of the vehicle 2 facing the direction of travel so that the direction of transmission of the probe wave is forward of the vehicle's direction of travel. Sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the direction of transmission of the probe wave is to the left of the vehicle's direction of travel. Sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the direction of transmission of the probe wave is to the right of the vehicle's direction of travel. Sensors 9I to 9L are installed on the rear of the vehicle 2 facing the opposite direction of travel so that the direction of transmission of the probe wave is to the rear of the vehicle. The height of each sensor 9A to 9L from the ground surface is approximately the same.

[0018] To explain using the actual sensors 9A to 9D as an example, it is desirable that the actual sensors 9A to 9D are installed at different positions on the front bumper or around the front grille above it on the front of the vehicle 2, as shown in Figure 3, with even spacing between them without bias in the left-right direction, so that they can transmit detection waves to a wider area in front of the vehicle (i.e., to widen the range in which objects can be detected).

[0019] Specifically, as shown in Figure 3, the actual sensor 9A is installed near the left front corner of the vehicle 2, with the direction of its wave transmission slightly tilted to the left of the vehicle's direction of travel, so as to emit a search wave towards the left front of the vehicle 2. The actual sensor 9B is installed slightly to the left of the vehicle 2's centerline, with the direction of its wave transmission facing the vehicle's direction of travel, so as to emit a search wave mainly on the left front of the vehicle 2. The actual sensor 9C is installed slightly to the right of the vehicle 2's centerline, with the direction of its wave transmission facing the vehicle's direction of travel, so as to emit a search wave mainly on the right front of the vehicle 2. The actual sensor 9D is installed near the right front corner of the vehicle 2, with the direction of its wave transmission slightly tilted to the right of the vehicle's direction of travel, so as to emit a search wave towards the right front of the vehicle 2. Furthermore, the actual sensors 9A and 9D, and 9B and 9C are each arranged symmetrically across the vehicle's centerline in a plan view. Although not shown in the diagram, the actual sensors 9I to 9L on the rear of vehicle 2 are also arranged similarly, symmetrically vertically. Furthermore, although not shown in the diagram, the transmission direction of the probe waves relative to the height (vertical direction) is to be horizontal in all cases.

[0020] On the other hand, as shown in Figure 4, the lateral sensors 9E and 9F are each installed to transmit search waves in a direction that intersects the direction of travel of the vehicle 2 at a 90-degree angle. Although not shown in the illustration, the same symmetry applies to the sensors 9G and 9H on the right side of the vehicle 2. Furthermore, although not shown in the illustration, the direction of transmission of search waves in the height direction (up and down direction) is to be horizontal in all cases.

[0021] Furthermore, the installation position and direction of the virtual sensors set in this embodiment follow the installation position and direction of the actual sensors 9A to 9L described above. That is, it is assumed that the virtual sensors (12 in total) are installed in the same installation position and direction as the actual sensors 9A to 9L, and this is a prerequisite for conducting the test.

[0022] Furthermore, the simulator device 3 is configured to allow for both "continuous analysis" and "post-analysis" of the vehicle 2's driving results. "Continuous analysis" monitors vehicle 2 during the driving test and automatically detects and outputs any malfunctions that occur. On the other hand, "post-analysis" evaluates and outputs the control content and driving trajectory of vehicle 2 that were stored during the driving test after the vehicle 2's driving test is completed.

[0023] Therefore, when conducting a driving test of vehicle 2, evaluator 4 first inputs the types and positions of virtual objects to be placed in the actual field, in order to create the driving environment for the vehicle 2 driving test, into the simulator device 3. This generates the driving environment for the driving test. Note that "virtual objects" are not limited to three-dimensional objects, but also include, for example, lane markings and road markings. Furthermore, they are not limited to stationary objects but may also be moving objects, and if they are moving objects, a schedule for their movement (at what speed and how they will move) is set in advance. For example, when conducting a driving test in a parking lot, parking lane markings drawn on the road surface in the parking lot, other vehicles parked in parking spaces, pedestrians moving within the parking lot, and other vehicles driving on the pathways within the parking lot are input as virtual objects. After that, when the driving test is started, virtual sensors mounted on the vehicle detect the virtual objects generated as the driving environment, and a simulation is performed in which the vehicle is driven with autonomous driving assistance in the virtual driving environment. The driving results are then analyzed, evaluated, and output by "continuous analysis" or "post-analysis," making it possible to identify any malfunctions that occur.

[0024] However, since the virtual objects placed in the real field by the simulator device 3 do not actually exist, even if the vehicle recognizes them using the virtual sensors, evaluator 4 cannot directly see them. Furthermore, since the virtual sensors are also pseudo-sensors, they are not actually installed on the exterior walls of the vehicle, nor do they transmit probe waves around the vehicle. Therefore, while it is possible to confirm how the virtual sensors transmit probe waves and detect the virtual objects by analyzing the results after the test, it is difficult for evaluator 4 to grasp this in real time. Consequently, especially when performing "continuous analysis," there is a lack of clarity regarding how the virtual sensors transmit probe waves and detect the virtual objects, making it difficult for evaluator 4 to evaluate whether the vehicle's behavior is appropriate. Therefore, the vehicle operation simulator system 1 of this embodiment includes the following means to allow the evaluator 4 to visually observe a real field in which an image representing a virtual object is superimposed at the position where the virtual object is placed, and at least one of the following is superimposed: a probe wave image showing how the probe wave of a virtual sensor transmitted from a hypothetical installation position (see Figures 3 and 4) travels (including cases where it spreads out as it travels and cases where it narrows as it travels, the same applies hereinafter), and a reflected wave image showing how the reflected wave travels when the probe wave is reflected by the virtual object. Note that the evaluator 4 may be riding in the vehicle 2 as a passenger or observing from outside the vehicle, and the explanation will be based on each case.

[0025] First, we will explain using the example of allowing evaluator 4, who is riding in vehicle 2 as a passenger, to view the real field with the above-mentioned probe wave image superimposed. In this embodiment, it is possible to selectively view one of the following: a real field with only the probe wave image superimposed, a real field with only the reflected wave image superimposed, or a real field with both the probe wave image and the reflected wave image superimposed. For example, evaluator 4 can switch between these by operating the operation unit 14, which will be described later.

[0026] In the first example, as shown in Figure 5, a real-time video of the actual field captured by an external camera is displayed on an in-vehicle display 5 mounted on the vehicle 2 and visible to the vehicle's occupants. Then, by combining an image 7 representing a virtual object and an image 10 showing the progress of a probe wave transmitted from the position of a set virtual sensor (corresponding to the position of the actual sensor) with the real-time video 6 of the actual field displayed on the in-vehicle display 5, it becomes possible for the evaluator 4 to see a real field in which the image 7 representing the virtual object is superimposed at the position where the virtual object is placed, and the probe wave image 10 showing the progress of the probe wave transmitted from the virtual sensor is also superimposed. Furthermore, the simulator device 3 can identify the position of the virtual object in the real-time video 6 of the actual field displayed on the in-vehicle display 5 by converting the placement coordinates (absolute coordinates) of the virtual object set as the driving environment to the coordinate system (relative coordinates) of the external camera. Furthermore, the simulator device 3 can determine the direction (directivity) and range (detection area) of the probe wave based on the pre-set installation position and direction of the virtual sensor, the performance of the virtual sensor, and the output strength of the probe wave (e.g., burst length, drive current, base gain). The performance of the virtual sensor follows that of the actual sensors 9A to 9L. In the first example, a real-time image of the actual field captured by an external camera 6 (i.e., an image in the optical axis direction with the camera's position as the viewpoint) is displayed, but a bird's-eye view or overhead view image generated by synthesizing images from multiple cameras installed in front of, behind, and to the sides of the vehicle may also be displayed. In addition, except when displaying an overhead view image, it is desirable that the probe wave image 10 be a three-dimensional image to represent the movement of the probe wave in three dimensions.

[0027] For example, Figure 6 shows a second example in which an overhead view image is displayed as a real-time field video 6. Based on real-time images captured by multiple cameras installed in front of, behind, and to the sides of the vehicle, an overhead view image is generated that looks down vertically from above at the area around the vehicle, and the generated overhead view image is displayed on the in-vehicle display 5. The generation of the overhead view image is publicly known, so the details are omitted, but the overhead view image is generated by performing a viewpoint transformation on each image captured by the multiple cameras, then combining (stitching) them together, and further inserting an illustrative image schematically representing the vehicle in the center. Similarly, by combining an image 7 showing a virtual object and a probe wave image 10 showing the progress of probe waves transmitted from the set virtual sensor position (corresponding to the position of the actual sensor) with the real field field video 6 displayed on the in-vehicle display 5, it becomes possible for the evaluator 4 to visually perceive a real field in which the image 7 showing the virtual object is superimposed at the position where the virtual object is placed, and the probe wave image 10 showing the progress of probe waves transmitted from the virtual sensor is also superimposed. Furthermore, the overhead view shown in Figure 6 makes it possible to visualize areas that are blind spots for the crew.

[0028] Next, to explain in more detail the probe wave image 10, which shows the progress of the probe wave transmitted from the virtual sensor, as shown in Figure 7, the probe wave image 10 is an image that allows visualization of the progress of the probe wave in the direction of installation from the installation position X in the detection area Y where a virtual object can be detected, which is set based on the installation position X of the virtual sensor. Furthermore, as shown in Figure 7, it is desirable that the probe wave image 10 be a video in which arcs representing the probe wave appear and move continuously in the transmission direction. As a result, it becomes possible to understand the direction of propagation of the probe wave.

[0029] Furthermore, in the examples described so far, we have shown an example in which evaluator 4 is shown a real field with only the probe wave image 10 superimposed. However, especially when a virtual object is detected by a virtual sensor, it is also possible to show a real field with only the reflected wave image 21 (hereinafter referred to as the reflected wave image) superimposed, or a real field with both the probe wave image 10 and the reflected wave image 21 superimposed. For example, Figure 8(A) shows a real field (overhead view) with both the probe wave image 10 and the reflected wave image 21 superimposed, and Figure 8(B) shows a real field (overhead view) with only the reflected wave image 21 superimposed.

[0030] As shown in Figures 8(A) and 8(B), when a virtual object is detected, the reflected wave image 21, which shows the reflected wave reflected by the virtual object, can also be made visible, allowing the evaluator 4 to see the detection status of the virtual object in real time, superimposed on the actual field. When both the search wave image 10 and the reflected wave image 21 are made visible, it is desirable to make them distinguishable, for example, by using different display colors for each image. Furthermore, when a virtual object is detected, the distance to the detected virtual object (for example, in the example shown in Figure 8, it indicates that a virtual object has been detected 1m away) may also be displayed.

[0031] Furthermore, as another variation, the exploration wave image 10 may be configured to display the waveform 13 of the exploration wave, which travels along the detection axis 12 in the direction of the exploration wave transmission, along with the detection axis 12 extending from the installation position X of the virtual sensor, as shown in Figure 9.

[0032] Next, in the third example, VR (Virtual Reality) technology is used, and a VR head-mounted display 8 is fitted to the evaluator 4, who is an occupant of the vehicle. As shown in Figure 10, the VR head-mounted display 8 displays not only pre-recorded live footage of the vehicle interior, but also real-time live footage 6 of the actual field captured by an external camera on the outside of the window. Then, by combining the displayed live footage 6 of the actual field with an image 7 representing a virtual object and a probe wave image 10 showing the progress of probe waves transmitted from the set virtual sensor's position (corresponding to the actual sensor's position), the evaluator 4 can see a real field where the image 7 representing the virtual object is superimposed at the location where the virtual object is placed, and the probe wave image 10 showing the progress of probe waves transmitted from the virtual sensor is also superimposed. Furthermore, the simulator device 3 can identify the position of the virtual object in the live footage 6 of the actual field displayed on the VR head-mounted display 8 by converting the placement coordinates (absolute coordinates) of the virtual object set as the driving environment to the VR (i.e., user's viewpoint) coordinate system (relative coordinates). Furthermore, the simulator device 3 can determine the direction (directivity) and range (detection area) of the probe wave based on the pre-set installation position and orientation of the virtual sensor, as well as the performance of the virtual sensor and the output strength of the probe wave (e.g., burst length, drive current, base gain). The performance of the virtual sensor follows that of the actual sensors 9A to 9L. When a virtual object is detected, the VR head-mounted display 8 can also display the reflected wave image 21, similar to the example shown in Figure 8.

[0033] Furthermore, in cases where the evaluator 4, who is riding in the vehicle 2 as a passenger, is shown images 7, probe images 10, and reflected images 21 representing the virtual object, as in the first to third examples above, the images 7, probe images 10, and reflected images 21 representing the virtual object may also be superimposed and displayed on the real field reflected or displayed in one or both of the rearview mirror and side mirrors of the vehicle. That is, the image 7 representing the virtual object is superimposed and displayed at the location where the virtual object is located in the real field reflected or displayed in one or both of the rearview mirror and side mirrors. In addition, if the real field reflected or displayed in one or both of the rearview mirror and side mirrors includes the detection area of ​​the virtual sensor, the probe images 10 and reflected images 21 are displayed. For example, if the rearview mirror and side mirrors are monitor-type displays showing images captured by external cameras, as shown in Figure 11, by combining an image 7 representing a virtual object, a search wave image 10, and a reflected wave image 21 with the actual field image 6 displayed in the image display area of ​​the rearview mirror 19, it becomes possible to make the evaluator 4 visually perceive an actual field where the image 7 representing the virtual object is superimposed at the location where the virtual object is placed, and the search wave image 10 showing the propagation of the search wave transmitted from the virtual sensor and the reflected wave image 21 showing the propagation of the reflected wave reflected by the virtual object are superimposed. On the other hand, if the rearview mirror and side mirrors are mirror-type displays equipped with a transparent display on the mirror surface, the transparent display can superimpose images 7, probe wave images 10, and reflected wave images 21 representing virtual objects onto the actual field scene reflected in the mirror. This allows the evaluator 4 to see an actual field where image 7 representing the virtual object is superimposed at the location where the virtual object is placed, and where probe wave images 10 showing the propagation of probe waves transmitted from the virtual sensor and reflected wave images 21 showing the propagation of reflected waves reflected by the virtual object are superimposed.

[0034] Furthermore, in addition to the above example, other means of allowing evaluators 4, who are passengers in vehicle 2, to visually perceive virtual objects placed in the real field and probe waves transmitted from virtual sensors can also be used, such as a HUD (Dead-Up Display) or a see-through type VR head-mounted display. These technologies superimpose images onto the actual scenery outside the vehicle that is visible through the front windshield or side windows. Therefore, images representing virtual objects are superimposed on the locations where virtual objects are placed within the real field included in the passenger's field of view, and display control is performed to display probe wave images 10 and reflected wave images 21 in the detection area of ​​the virtual sensor.

[0035] Next, we will explain the case where the system is made visible to evaluator 4 observing from outside the vehicle. For evaluator 4 observing from outside the vehicle, the real-time field video 6 captured by the external camera is displayed on the display of a tablet-type external terminal 11 that is communicatively connected to the simulator device 3. The details are the same as when it is displayed on the in-vehicle display 5. By combining images 7 showing virtual objects, exploration wave images 10, and reflected wave images 21 with the real-time field video 6 displayed on the external terminal 11, images 7 showing virtual objects are superimposed at the positions where the virtual objects are located, and exploration wave images 10 showing the propagation of exploration waves transmitted from the virtual sensor and reflected wave images 21 showing the propagation of reflected waves reflected by the virtual objects are also superimposed, making it possible for evaluator 4 to visually perceive the real field.

[0036] However, especially when displaying the actual field on the external terminal 11, it is not necessary to display the actual field included in the field of view from the perspective of vehicle 2; it is also possible to display the actual field included in the field of view from a different position than vehicle 2. For example, if the external terminal 11 has a camera, as shown in Figure 12, the real-time actual field image 6 captured by the camera of the external terminal 11 is displayed on the display of the external terminal 11. The simulator device 3 then acquires the optical axis information of the camera of the external terminal 11 and converts the placement coordinates (absolute coordinates) of the virtual objects set as the driving environment to the coordinate system (relative coordinates) of the camera of the external terminal 11, thereby making it possible to identify the position where the virtual objects are placed in the actual field captured by the camera of the external terminal 11. Subsequently, by compositing an image 7 representing the virtual objects onto the actual field image 6 displayed on the display of the external terminal 11, it becomes possible for the evaluator 4 to visually perceive the actual field with the image 7 representing the virtual objects superimposed on the position where the virtual objects are placed. Furthermore, the simulator device 3 can determine the direction (directivity) and range (detection area) of the probe wave based on the pre-set installation position and orientation of the virtual sensor, as well as the performance of the virtual sensor and the output strength of the probe wave (e.g., burst length, drive current, base gain). Subsequently, by compositing the probe wave image 10 and the reflected wave image 21 onto the identified detection area in the actual field video 6 displayed on the external terminal 11's display, the evaluator 4 can visually perceive an actual field where the probe wave image 10 showing the propagation of the probe wave transmitted from the virtual sensor and the reflected wave image 21 showing the propagation of the reflected wave reflected by a virtual object are superimposed.

[0037] Furthermore, the evaluator 4 may, by operating their external terminal 11, switch between displaying the real field from the perspective of vehicle 2 and displaying the real field from the perspective of evaluator 4. This makes it possible to view the real field with virtual objects placed in it in real time from various perspectives, rather than from a fixed viewpoint. In addition to displaying the real field from the perspective of vehicle 2 and evaluator 4, if imaging devices are installed on the real field, it is also possible to display the real field from the perspective of those imaging devices.

[0038] Furthermore, in the embodiment described above, the image 7 representing a virtual object, the probe wave image 10, and the reflected wave image 21 are superimposed on the real-world field image 6 displayed on the in-vehicle display 5, the VR head-mounted display 8, and the external terminal 11. However, the image 7 representing the virtual object may be omitted, and only the probe wave image 10 or the reflected wave image 21 may be superimposed.

[0039] Next, we will describe the vehicle 2 equipped with the automated driving support system that will be evaluated by the vehicle operation simulator system 1. Figure 13 is a schematic diagram of the vehicle 2 according to this embodiment.

[0040] Here, Vehicle 2 may be, for example, an automobile powered by an internal combustion engine (internal combustion engine vehicle), an automobile powered by an electric motor (electric vehicle, fuel cell vehicle, etc.), or an automobile powered by both (hybrid vehicle). Furthermore, there is no restriction on the type of vehicle; it may be a regular passenger car, a large commercial truck, a bus, construction machinery, etc. Also, although the following explanation will refer to it as a four-wheeled vehicle, it may also be a two-wheeled or three-wheeled vehicle.

[0041] However, Vehicle 2 shall be a vehicle capable of not only manual driving based on the user's driving operations, but also assisted driving through automated driving assistance, in which the vehicle drives automatically without user operation.

[0042] Furthermore, autonomous driving assistance may be performed only under specific circumstances, such as when parking or exiting a parking space, or it may be performed on all road sections, or it may be configured to be performed only while the vehicle is traveling on a specific road section (for example, a highway with a gate (regardless of whether it is manned or unmanned, tolled or free) at the boundary). In the following explanation, the autonomous driving sections in which the vehicle's autonomous driving assistance is performed will include all road sections, including general roads and highways, as well as parking lots, and will only be performed when the user has selected to perform autonomous driving assistance (for example, by turning on the autonomous driving start button) and it has been determined that it is possible to perform driving with autonomous driving assistance. On the other hand, vehicle 2 may be a vehicle that is only capable of driving with autonomous driving assistance. Alternatively, autonomous driving assistance may be performed only when the vehicle is driving to a parking space (i.e., parking assistance).

[0043] In the vehicle control for the automated driving assistance of this embodiment, for example, the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles are detected in real time, and the steering, drivetrain, brakes, and other vehicle controls are automatically performed so that the vehicle travels along the generated driving trajectory at a speed according to the generated speed plan. In particular, when providing parking assistance, the system uses the detection results from sensors and cameras to check the parking space to which the vehicle is to park and the surrounding conditions, calculates a parking trajectory to the parking space, and automatically performs vehicle control to guide the vehicle into the parking space along the calculated parking trajectory and complete the parking.

[0044] Furthermore, as shown in Figure 13, the vehicle 2 includes an operating unit 14 that receives input from the occupant, an in-vehicle display 5 (or VR head-mounted display 8) that displays the aforementioned real-world field video 6 (Figure 5) and other output information from the simulator device 3 to the occupant, a speaker 15 that outputs voice guidance from the simulator device 3, a front camera 16, a rear camera 17, and side cameras 18A, 18B for imaging the area around the vehicle, real sensors 9A to 9L for detecting obstacles around the vehicle, a driver assistance ECU (electronic control unit) 20 which is a control unit that performs various calculation processing related to automatic driving assistance based on the input information, and the aforementioned simulator device 3.

[0045] The following describes the various components of vehicle 2. First, the control unit 14 is located, for example, in front of the steering wheel and includes control buttons that are operated when starting the automated driving assistance system. By operating the control unit 14, the user can switch between manual driving, where the vehicle drives based on the user's driving input, and automated driving assistance, where the vehicle drives automatically without user input. The control unit 14 may also have a touch panel located in front of the in-vehicle display. It may also have a microphone and a voice recognition device.

[0046] The in-vehicle display 5 is a type of display device mounted on the instrument panel of the vehicle 2. As described above, it displays a real-field image 6 (Figure 5) in which an image 7 representing a virtual object is superimposed at the position where the virtual object is placed, and a probe wave image 10 showing the propagation of probe waves transmitted from the virtual sensor, or a reflected wave image 21 showing the propagation of reflected waves when the probe wave reflects off the virtual object, is superimposed. Alternatively, the VR head-mounted display 8, also described above, may be provided instead of the in-vehicle display 5. Furthermore, as described above, the rearview mirror and side mirrors may be equipped with display devices (Figure 11) that also display a real-field image 6 in which an image 7 representing a virtual object, a probe wave image 10, and a reflected wave image 21 are superimposed. The real-field image 6 displayed on the in-vehicle display 5 and the VR head-mounted display 8 is an image created by combining the aforementioned image 7 representing a virtual object, a probe wave image 10, and a reflected wave image 21 with real-time images captured by the front camera 16, rear camera 17, and side cameras 18A and 18B.

[0047] Additionally, speaker 15 is mounted on the instrument panel of vehicle 2 and outputs guidance voices, warning sounds, etc., from simulator device 3.

[0048] Furthermore, the front camera 16 is an imaging device that has a camera using a solid-state image sensor such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with the optical axis facing forward in the direction of travel of the vehicle.

[0049] The rear camera 17 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0050] Furthermore, the side cameras 18A and 18B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of vehicle 2, with the optical axis facing the side of the vehicle.

[0051] Furthermore, the simulator device 3 reads information regarding the type and placement of virtual objects in the driving environment, which has been pre-set by the evaluator 4. By compositing images 7 showing the virtual objects at their positions onto the images captured by the front camera 16, rear camera 17, and side cameras 18A and 18B, the simulator device 3 generates real-world field footage 6 that will be displayed on the in-vehicle display 5 and VR head-mounted display 8. In addition, the simulator device 3 determines the direction (directivity) and range (detection area) of the probe wave based on the placement and orientation of the virtual sensor, which have been pre-set by the evaluator 4, as well as the performance of the virtual sensor and the output strength of the probe wave (e.g., burst length, drive current, base gain). Similarly, it composites probe wave images 10 and reflected wave images 21 onto the detection area of ​​the virtual sensor onto the images captured by the front camera 16, rear camera 17, and side cameras 18A and 18B. Furthermore, the real-world footage on which images 7 representing virtual objects and probe images 10 are synthesized may be the real-world footage itself captured by the forward camera 16, rear camera 17, and side cameras 18A and 18B, or it may be footage synthesized from images of multiple cameras or footage with a transformed viewpoint (for example, an overhead view or a bird's-eye view).

[0052] On the other hand, the real sensors 9A to 9L are distance measuring sensors for detecting real objects around the vehicle, and are, for example, ultrasonic sensors. In contrast to the real sensors 9A to 9L, there are virtual sensors 22A to 22L, which cannot detect real objects but can detect virtual objects, and these are also mounted on the vehicle 2. As mentioned above, the real sensors 9A to 9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle, respectively, and transmit ultrasonic waves as probe waves to the area around the vehicle 2. They also detect objects that reflected the probe waves by receiving reflected waves that were reflected by objects around the vehicle (Figures 3 and 4).

[0053] On the other hand, the driver assistance ECU 20 is an electronic control unit that performs various processes related to automated driving assistance. The driver assistance ECU 20 is connected to various sensors for detecting the vehicle's behavior, such as a vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as to various drive units of the vehicle, such as the steering, brakes, accelerator, and transmission. Based on the detection results of these sensors, it detects the vehicle's current behavior, calculates control information (control amounts) to control each drive unit, and performs automated driving assistance for vehicle 2 by controlling each drive unit. Specifically, the automated driving assistance includes, for example, continuously detecting the vehicle's current position, the lane the vehicle is traveling in, and the position of surrounding obstacles, and controlling the vehicle, such as the steering, drive source, and brakes, so that the vehicle travels along the generated driving trajectory at a speed according to the generated speed plan. In particular, when performing parking assistance, it checks the parking space to which the vehicle is to park and the surrounding conditions, calculates a parking trajectory to the parking space, and controls the vehicle to enter the parking space along the calculated parking trajectory and complete the parking.

[0054] Here, the driver assistance ECU 20 has two means of acquiring information about surrounding obstacles and parking spaces necessary for performing the above-mentioned automated driving assistance: a means of acquiring detection information from real sensors 9A to 9L and a means of acquiring detection information from virtual sensors 22A to 22L. Real sensors 9A to 9L are sensors for detecting objects that actually exist around the vehicle, while virtual sensors 22A to 22L are sensors for detecting virtual objects in a driving environment virtually generated by the simulator device 3. Virtual sensors 22A to 22L are part of the simulator device 3 and are mounted on the vehicle 2.

[0055] Here, Figure 14 shows the external view of the virtual sensors 22A to 22L provided by the simulator device 3, and Figure 15 shows the internal structure of the virtual sensors 22A to 22L. Note that the virtual sensors 22A to 22L have basically the same structure, and the following explanation will use virtual sensor 22A as an example.

[0056] As shown in Figures 14 and 15, the virtual sensor 22A has a rectangular box shape, with a first ultrasonic sensor 25 at one end of the box and a second ultrasonic sensor 26 at the other end. The first ultrasonic sensor 25 and the second ultrasonic sensor 26 have basically the same structure as the actual sensors 9A to 9L described above, and are arranged facing each other so that the detection axis X is coaxial. The detection axis X is a virtual straight line extending from the first ultrasonic sensor 25 and the second ultrasonic sensor 26 along the direction of transmission and reception of the probe wave. The first ultrasonic sensor 25 and the second ultrasonic sensor 26 each have a piezoelectric vibrator (e.g., a ceramic plate) for transmitting and receiving the probe wave. The piezoelectric vibrator is formed in a planar shape with the detection axis X as its normal. The first ultrasonic sensor 25 and the second ultrasonic sensor 26 are configured to emit a probe wave along the detection axis X by causing the piezoelectric vibrator to vibrate ultrasonically based on a drive signal applied to an electromechanical conversion element. Furthermore, the first ultrasonic sensor 25 and the second ultrasonic sensor 26 are configured to generate a received signal, which is an electrical signal corresponding to the excitation state of the piezoelectric vibrator caused by the received ultrasonic waves, using an electromechanical conversion element when receiving a wave from an external source.

[0057] The virtual sensor 22A detects that the first ultrasonic sensor 25 has received a reflected wave when the electromotive force generated by the vibration of the piezoelectric transducer accompanying the reception of the probe wave exceeds a threshold, and calculates the distance to the target object based on the time from when the first ultrasonic sensor 25 transmits the probe wave until it receives the probe wave. However, the target object referred to here is a virtual object, and the first ultrasonic sensor 25 does not actually transmit a probe wave. Instead, it assumes that it has transmitted a probe wave and recognizes and receives the probe wave transmitted from the second ultrasonic sensor 26 as a reflected wave. In this case, the distance measurement value detected by the virtual sensor 22A can be freely controlled by the timing of when the second ultrasonic sensor 26 transmits the probe wave.

[0058] Specifically, the simulator device 3 determines the current position and orientation of the vehicle based on information from various sensors installed on the vehicle, such as the vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor. Based on the determined current position and orientation of the vehicle and the positions of virtual objects placed in a driving environment set in advance by the evaluator 4, the simulator device 3 determines the relative position of the virtual objects to the vehicle 2. Then, based on the determined relative position of the virtual objects to the vehicle 2 (more specifically, the actual sensors 9A to 9L installed on the vehicle), the simulator device 3 controls the timing of transmitting probe waves from the second ultrasonic sensor 26, thereby enabling the virtual sensors 22A to 22L to detect the non-existent virtual objects. In other words, the virtual sensors 22A to 22L can detect virtual objects by simulating the process of transmitting probe waves from their installation position to the area around the vehicle 2 and receiving reflected waves that have been reflected by virtual objects around the vehicle 2.

[0059] In the vehicle operation simulator system 1 of this embodiment, during a driving test (while collecting driving results), the driver assistance ECU 20 detects surrounding obstacles and parking spaces based on the detection information of virtual sensors 22A to 22L and controls the vehicle, such as steering, drive source, and brakes. On the other hand, during the preparation stage before the driving test (preparation stage before collecting driving results) and after the driving test (after the collection of driving results is completed), it is necessary to drive the vehicle while avoiding real walls and obstacles, so the driver assistance ECU 20 detects surrounding obstacles and parking spaces based on the detection information of real sensors 9A to 9L and controls the vehicle, such as steering, drive source, and brakes. The switching of sensor information input to the driver assistance ECU 20 is performed by a switch or the like.

[0060] Furthermore, the virtual sensors 22A to 22L are provided in the same number as the actual sensors 9A to 9L, with virtual sensor 22A corresponding to actual sensor 9A. That is, during driving tests, the driver assistance ECU 20 recognizes virtual sensor 22A as actual sensor 9A and detects an object (a virtual object that does not actually exist) located to the left front of the vehicle. For example, if the distance from actual sensor 9A to the virtual object is 1m, the timing of transmitting a probe wave from the second ultrasonic sensor 26 is controlled so that the distance measured by virtual sensor 22A becomes 1m. As a result, it is detected that there is an object (a virtual object) at a position 1m from actual sensor 9A. Similarly, virtual sensor 22B corresponds to actual sensor 9B, virtual sensor 22C corresponds to actual sensor 9C, and virtual sensor 22D corresponds to actual sensor 9D. The same applies to the remaining virtual sensors 22E to 22L.

[0061] In addition to the components shown in Figure 13, Vehicle 2 also has other basic components as Vehicle 2, but only the configuration related to the control of the automated driving assistance system and the control related to said configuration will be explained.

[0062] Next, we will describe the details of the simulator device 3 included in the vehicle operation simulator system 1. Figure 16 is a block diagram showing the configuration of the simulator device 3 according to this embodiment.

[0063] As shown in Figure 16, the simulator device 3 includes a virtual control ECU (electronic control unit) 40, which is a control unit that performs various calculation processing such as generating a driving environment when conducting a driving test, controlling virtual sensors 22A to 22L, and analyzing, evaluating, and outputting the operational results of the vehicle 2's driving test, as well as the aforementioned virtual sensors 22A to 22L.

[0064] The virtual control ECU 40 is an electronic control unit that controls the entire simulator device 3, and includes a CPU 41 as an arithmetic unit and control device, RAM 42 used as working memory when the CPU 41 performs various arithmetic processing, ROM 43 which stores control programs and operation evaluation programs (see Figure 17) described later, and internal storage devices such as flash memory 44 which stores programs read from ROM 43. The virtual control ECU 40, together with the aforementioned driving support ECU 20, has various means as processing algorithms. For example, the virtual object placement means places virtual objects in the real field. The virtual object detection means detects virtual objects by using virtual sensors 22A to 22L, which are assumed to be installed at predetermined installation positions and in predetermined installation directions on the vehicle, to transmit probe waves from the installation position to the surroundings of the vehicle in accordance with the installation direction, and to simulate the reception of reflected waves reflected by virtual objects around the vehicle. The image viewing means allows the user to view a real field in which at least one of a probe wave image 10 showing the propagation of probe waves transmitted from virtual sensors 22A to 22L and a reflected wave image 21 showing the propagation of reflected waves when the probe waves are reflected by a virtual object is superimposed. In other words, the virtual control ECU 40 is an example of a virtual object placement means, a virtual object detection means, and an image viewing means.

[0065] Furthermore, the virtual control ECU 40 is connected via an in-vehicle network such as CAN to the aforementioned operating unit 14, in-vehicle display 5 (or VR head-mounted display 8), speaker 15, front camera 16, rear camera 17, side cameras 18A, 18B, real sensors 9A~9L, and driver assistance ECU 20, as well as to various sensors 48 for detecting the vehicle's behavior, such as a vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor. Based on the detection results of these sensors 48, it is possible to detect the vehicle's current position, direction, and current behavior. The virtual control ECU 40 then provides the vehicle 2 with a virtual driving environment that serves as the driving environment for the driving test, while simultaneously acquiring, analyzing, evaluating, and outputting the operation results of the automated driving assistance system from the vehicle's behavior during the driving test. In addition, it generates real-world field footage 6 (Figures 5, 6, 10, and 11) displayed on the aforementioned in-vehicle display 5, VR head-mounted display 8, and external terminal 11 from the vehicle's behavior during the driving test, and outputs the footage to these display devices.

[0066] Furthermore, the flash memory 44 includes driving environment information 45, sensor information 46, and a driving results DB 47. The driving environment information 45 stores the driving environment for the vehicle 2 driving test, which is set in advance by the evaluator 4. Specifically, it includes the types and positions of virtual objects to be placed in the real field, which are entered in advance by the evaluator 4. It is also possible to store multiple driving environment patterns in advance.

[0067] On the other hand, sensor information 46 stores various information about the actual sensors 9A to 9L installed on vehicle 2, such as installation location, installation direction, and sensor performance. The installation location, installation direction, and sensor performance of the virtual sensors follow those of the actual sensors 9A to 9L, and as described later, the simulator device 3 generates a probe wave image 10 to be superimposed on the actual field based on sensor information 46.

[0068] The driving results DB47 is a storage means for cumulatively storing the control details of the vehicle, the vehicle's driving trajectory, and the history of its behavior during the driving test. After the vehicle 2's driving test operation is completed, the simulator device 3 evaluates and outputs the control details and driving trajectory of the vehicle 2 stored in the driving results DB47.

[0069] Next, the operation evaluation program executed in the simulator device 3, which constitutes the vehicle operation simulator system 1 having the above configuration, will be described with reference to Figure 17. Figure 17 is a flowchart of the operation evaluation program according to this embodiment. Here, the operation evaluation program is executed when the simulator device 3 receives a startup operation for the corresponding program, and is a program that constructs a virtual driving environment and evaluates the quality of the automatic driving support system equipped in the vehicle based on the results of the driving test. Note that the program startup operation may be triggered, for example, by an evaluator 4 riding as a passenger in the vehicle operating the operation unit 14, or it may be started by an evaluator 4 outside the vehicle performing a predetermined operation on an external terminal 11. The program shown in the flowchart in Figure 17 below is stored in the RAM 42, ROM 43, etc., of the simulator device 3 and is executed by the CPU 41.

[0070] First, in step 1 (hereinafter abbreviated as S), the CPU 41 of the simulator device 3 performs various initial setup processes before executing the driving test. For example, it checks the operation of virtual sensors 22A to 22L.

[0071] Next, in S2, the CPU 41 acquires the sensor performance (specifications) of the actual sensors 9A to 9L. The sensor performance is pre-stored in the flash memory 44. The sensor performance of the actual sensors 9A to 9L also corresponds to the sensor performance of the virtual sensors.

[0072] Next, in S3, CPU41 sets the output intensity of the probe wave in the virtual sensor.

[0073] Subsequently, in S4, the CPU 41 performs a sound wave analysis simulation of the exploration wave output from the virtual sensor, based on the sensor performance of the actual sensors 9A to 9L acquired in S2 and the output intensity of the exploration wave set in S3. That is, it simulates how the exploration wave would actually propagate if a virtual sensor having performance equivalent to the actual sensors 9A to 9L outputs an exploration wave at the set output intensity.

[0074] Next, in S5, the CPU 41 determines, based on the sound wave analysis simulation results of S4, the direction (directivity) and how the probe wave propagates (detection area) relative to the installation position and direction of the virtual sensor. The installation position and direction of the virtual sensor follow those of the actual sensors 9A to 9L that are actually mounted on the vehicle 2. As a result, as shown in Figure 18, it becomes possible to determine the direction (directivity) and range (detection area) of the probe wave transmitted from a virtual sensor assumed to be installed at the same installation position and direction as the actual sensors 9A to 9L.

[0075] Next, in S6, the CPU 41 reads setting information for the driving environment for conducting a driving test of vehicle 2 from the flash memory 44. The driving environment includes, for example, the types and positions of virtual objects to be placed in the real field.

[0076] Here, the driving environment is basically pre-set and stored in the simulator device 3 through the input operations of evaluator 4. It is also possible to set multiple driving environments, in which case the driving environment to be used for the current driving test can be arbitrarily selected. For example, it is possible to set the driving environment to be driving on a public road or driving in a parking lot. Furthermore, "virtual objects" are not limited to three-dimensional objects, but also include, for example, lane markings and road markings. For example, when conducting a driving test in a parking lot, parking lane markings drawn on the road surface of the parking lot, other vehicles parked in parking spaces, pedestrians moving in the parking lot, and other vehicles driving in the parking lot's pathways are set as virtual objects. In addition, for moving virtual objects, the movement schedule (at what speed and how they move) is also set.

[0077] Next, in S7, the CPU 41 constructs a virtual driving environment for the real field where the vehicle is currently located, based on the configuration information read in S6. That is, it places virtual objects of the specified type at the specified locations in the real field. Furthermore, it moves any virtual objects that are to move according to the set schedule.

[0078] Subsequently, in S8, the CPU 41 initiates a driving test of vehicle 2. Once the driving test begins, vehicle control using automated driving assistance starts in vehicle 2. For example, the current position of the vehicle, the lane the vehicle is traveling in, and the positions of surrounding obstacles are detected in real time, and vehicle control such as steering, drivetrain, and brakes is automatically performed so that the vehicle travels on the real field along the generated driving trajectory at a speed according to the generated speed plan. In particular, when evaluating parking assistance using a parking lot as the driving environment, the system uses sensor detection information to confirm the parking space to which the vehicle will park and the surrounding conditions, calculates a parking trajectory to the parking space, and automatically performs vehicle control to enter the parking space along the calculated parking trajectory and complete the parking.

[0079] While the above-described automated driving assistance is being performed, the CPU 41 of the simulator device 3 outputs the detection results obtained when the vehicle 2, which is driving in the real field, detects a virtual object with the virtual sensors 22A to 22L, to the driving assistance ECU 20 that controls the automated driving assistance on the vehicle 2 side. Specifically, the CPU 41 determines the current position and orientation of the vehicle based on information from various sensors 48 installed on the vehicle, such as the vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor. Based on the determined current position and orientation of the vehicle and the position of the virtual object placed in the driving environment constructed in S7, the CPU 41 determines the relative position of the virtual object to the vehicle 2. Then, based on the determined relative position of the virtual object to the vehicle 2 (more specifically, the real sensors 9A to 9L that the vehicle is equipped with), the CPU 41 controls the timing of transmitting probe waves from the second ultrasonic sensors 26 equipped with the virtual sensors 22A to 22L, thereby causing the virtual sensors 22A to 22L to detect a virtual object that does not actually exist.

[0080] Next, in S9, the CPU 41 acquires real-time footage of the actual field to be displayed on the in-vehicle display 5, VR head-mounted display 8, or external terminal 11, based on the images captured by the front camera 16, rear camera 17, and side cameras 18A and 18B. The footage may be the actual footage captured by the front camera 16, rear camera 17, and side cameras 18A and 18B themselves, or it may be footage synthesized from images from multiple cameras or footage with a transformed viewpoint (for example, an overhead view or a bird's-eye view).

[0081] Furthermore, the real-time video footage of the actual field to be displayed on the external terminal 11 may be video footage captured by a camera on the external terminal 11, rather than video footage captured by a camera on the vehicle 2, or video footage captured by a camera placed on the actual field. The following processing is performed on a frame-by-frame basis of the video acquired in S9.

[0082] Next, in S10, the CPU 41 converts the placement coordinates (absolute coordinates) of the virtual object in the driving environment constructed in S7 to the coordinate system (relative coordinates) of the real-world field image acquired in S9, thereby identifying the position of the virtual object in the real-world field image acquired in S9. For example, when displaying the real-world field image captured by the front camera 16 on the in-vehicle display 5, the position of the virtual object in the real-world field image displayed on the in-vehicle display 5 can be identified by converting the placement coordinates (absolute coordinates) of the virtual object to the coordinate system (relative coordinates) of the front camera 16. On the other hand, when performing VR display using the VR head-mounted display 8, the position of the virtual object in the real-world field image displayed on the VR head-mounted display 8 can be identified by converting the placement coordinates (absolute coordinates) of the virtual object to the coordinate system (relative coordinates) of VR (i.e., the user's viewpoint).

[0083] Subsequently, in S11, the CPU 41 synthesizes an image 7 representing a virtual object at the location identified in S10 onto the real-world image of the real field acquired in S9. For the image 7 representing the virtual object, 3D model images for each type of virtual object, such as a vehicle or a person, are pre-stored in the flash memory 44. The CPU 41 then reads the corresponding model image from the flash memory 44, rotates and scales it to match the virtual object placed in the real field, and synthesizes it onto the real-world image. For virtual objects with a defined orientation, such as a vehicle, the orientation is also synthesized to match the constructed driving environment. Furthermore, texture mapping is performed by applying a texture image to the surface of the model image. When placing the model image in the real field, it is desirable to consider the orientation of placement, the surrounding brightness, and the position of light sources (e.g., sunlight, streetlights) to ensure that the effect of the texture image is rendered in a way that does not appear unnatural. The model image to be synthesized may be an opaque image with 0% transparency, or a semi-transparent image.

[0084] Subsequently, in S12, the CPU 41 synthesizes a probe wave image 10 onto the real field image acquired in S9, based on the direction (directivity) and range (detection area) of the probe wave identified in S5, showing how the probe wave transmitted from the virtual sensor's position (corresponding to the real sensor's position). The probe wave image 10, as described above, is an image that allows the viewer to see how the probe wave travels in the direction following the installation direction from the installation position within the detection area where a virtual object can be detected based on the virtual sensor's installation position (Figure 7). Furthermore, as described above, in this embodiment, it is possible to selectively view one of three real fields: one with only the probe wave image 10 superimposed, one with only the reflected wave image 21 superimposed, or one with both the probe wave image 10 and the reflected wave image 21 superimposed. For example, this can be switched by the evaluator 4 operating the operation unit 14.

[0085] Furthermore, when the reflected wave image 21 is superimposed on the actual field image in S12, first, based on the direction (directivity) and range (detection area) of the probe wave identified in S5 and the position of the virtual object in the actual field, the range and direction (directivity) in which the probe wave (reflected wave) travels after being reflected by the virtual object is identified. Then, the reflected wave image 21 showing how the reflected wave travels within the identified range is superimposed. This makes it possible to visualize the actual field with the reflected wave image 21 superimposed.

[0086] Next, in S13, the CPU 41 outputs a real-time video of the actual field, which is a composite of an image 7 showing a virtual object and a probe wave image 10 or a reflected wave image 21, to the in-vehicle display 5, VR head-mounted display 8, or external terminal 11, and displays it on each display device. As a result, the evaluator 4 can see the actual field, in which the image 7 showing the virtual object is superimposed on the position where the virtual object is placed, and the probe wave image 10 showing the propagation of the probe wave transmitted from the virtual sensor, or the reflected wave image 21 showing the propagation of the reflected wave, is superimposed (Figures 5, 6, 8, and 10). Furthermore, as mentioned above, the real-time video of the actual field, with the same image 7 showing the virtual object, probe wave image 10, or reflected wave image 21 superimposed, may also be displayed on the rearview mirror and side mirrors (Figure 11).

[0087] Next, in S14, the CPU 41 determines whether the driving test has been completed. For example, when evaluating parking assistance using a parking lot as the driving environment, the driving test is determined to be completed when the vehicle is parked in a parking space and the shift position is set to P. Alternatively, the driving test may be determined to be completed when a predetermined distance has been driven or when a predetermined time has been spent driving since the start of the driving test.

[0088] If it is determined that the driving test has been completed (S14: YES), the operation evaluation program is terminated. On the other hand, if it is determined that the driving test has not been completed (S14: NO), the program returns to S8 and continues the driving test of vehicle 2.

[0089] Furthermore, the analysis of the vehicle's driving results when the vehicle is tested using the above-mentioned operation evaluation program can be performed using either "continuous analysis" or "post-analysis." "Continuous analysis" monitors vehicle 2 during the driving test and automatically detects and outputs any malfunctions that occur. On the other hand, "post-analysis" evaluates and outputs the control content and driving trajectory of vehicle 2 stored during the driving test, based on the detections made by virtual sensors 22A to 22L, after the vehicle's driving test operation is completed.

[0090] Furthermore, the CPU 41 may output an evaluation window to an external display that shows the details of any malfunctions that occurred after the driving test. The evaluation window will display the details of the malfunction along with the time at which it occurred. As a result, evaluator 4 will be able to understand when and what kind of malfunction occurred in the automated driving support system. In addition, the CPU 41 may save the actual field video (including the virtual object image 7 and the probe wave image 10) output in S13 as an image log, and extract the images output when it is determined that a malfunction occurred in the operation of the automated driving support system from the image log and make them available for evaluator 4 to view. Evaluator 4 will then be able to modify the program of the automated driving support system based on the outputted driving test results.

[0091] As described in detail above, in the vehicle operation simulator system 1 and the computer program executed by the vehicle operation simulator system 1 according to this embodiment, a virtual object is placed in a real field (S7), and the results of the vehicle 2, which is equipped with virtual sensors 22A to 22L for detecting the virtual object, are collected when the vehicle runs in the real field. Meanwhile, it is assumed that the virtual sensors 22A to 22L are installed in predetermined locations and directions on the vehicle according to the real sensors 9A to 9L, and the virtual object is detected by simulating the transmission of probe waves from the installation locations to the area around the vehicle and the reception of reflected waves that are reflected by virtual objects around the vehicle. At least one of a probe wave image 10 showing the propagation of the probe waves transmitted from the virtual sensors 22A to 22L and a reflected wave image 21 showing the propagation of the reflected waves when the probe waves are reflected by the virtual objects are superimposed on the real field (S13). Therefore, in a vehicle running test in which a virtual object is placed in a real field, the detection range and detection status of the virtual object by the virtual sensors can be superimposed on the real field and viewed by the occupants or observers in real time. As a result, it becomes clear how the virtual sensor transmits search waves and receives reflected waves to detect the virtual object, and it also becomes possible to evaluate from a human perspective whether the vehicle's behavior in response to the detection of the virtual object is appropriate. Furthermore, since it is possible to selectively visualize one of three real fields: one with only the exploration wave image 10 superimposed, one with only the reflected wave image 21 superimposed, or one with both the exploration wave image 10 and the reflected wave image 21 superimposed, it is possible to distinguish and visualize the exploration wave and the reflected wave, making it possible to visualize the detection status of virtual objects by the virtual sensor more clearly. Furthermore, in another example of the probe wave image 10 (Figure 9), the waveform of the probe wave traveling in the direction of transmission along the detection axis is displayed along with the detection axis of the virtual sensor. This makes it possible to superimpose the detection range and detection status of the virtual object by the virtual sensor onto the actual field and allow the crew or observer to see it in real time. Furthermore, the virtual sensor has a detection area set based on its installation location where it can detect virtual objects. The probe wave image 10 shows the probe wave moving within the detection area in the direction from the installation location to the installation direction. This makes it possible to superimpose the detection range and detection status of virtual objects by the virtual sensor onto the actual field and allow the crew or observers to see it in real time.

[0092] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, both the image 7 representing a virtual object and the probe wave image 10 are superimposed on the real-world field video 6 displayed on the in-vehicle display 5, VR head-mounted display 8, or external terminal 11. However, it is also possible to superimpose only the probe wave image 10 without displaying the image representing the virtual object.

[0093] Furthermore, in this embodiment, the real-world field video 6 (including the virtual object image 7, the probe wave image 10, and the reflected wave image 21) displayed on the in-vehicle display 5 and the external terminal 11 is a real-time field video, but it may also be a past field video captured by a camera. In that case, the evaluator 4 can confirm how the virtual sensor transmitted probe waves and detected the virtual object after the vehicle 2 has completed its driving test. In addition, the real-world field video 6 may be displayed on the in-vehicle display 5 and the external terminal 11 not only while the vehicle is in motion, but also when the vehicle is stopped.

[0094] Furthermore, in this embodiment, the subject of quality evaluation in the driving test is the automated driving support system, which assists the occupant in driving the vehicle by having the vehicle perform some or all of the occupant's driving operations. However, it is also possible to evaluate various other vehicle control systems besides the automated driving support system. For example, an automatic braking system that detects approaching obstacles can be evaluated.

[0095] Furthermore, the exploration wave image 10 in this embodiment is an image showing the direction (directivity) and range (detection area) of the exploration wave transmitted from the virtual sensor, but it may also be an image showing only one of these. The same applies to the reflected wave image 21.

[0096] Furthermore, in this embodiment, the main entity executing the operation evaluation program shown in Figure 17 was the simulator device 3 mounted on the vehicle, but the simulator device 3 may be located outside the vehicle. In that case, the simulator device 3 and the vehicle 2 can communicate via wireless communication.

[0097] 1…Vehicle operation simulator system, 2…Vehicle, 3…Simulator device, 4…Evaluator (observer, occupant), 5…In-vehicle display, 6…Actual field footage, 8…VR head-mounted display, 9A~9L…Actual sensors, 10…Surveillance wave image, 11…External terminal, 20…Driving support ECU, 21…Reflected wave image, 22A~22L…Virtual sensors, 40…Virtual control ECU (example of virtual object placement means, virtual object detection means, image viewing means), 41…CPU

Claims

1. A virtual object placement means for placing virtual objects in a real field, A virtual object detection means for detecting a virtual object by using a virtual sensor assumed to be installed at a predetermined location and in a predetermined direction on a vehicle, transmitting a probe wave from the installation location to the area around the vehicle in the direction of the installation, and simulating the reception of reflected waves that have been reflected by the virtual object around the vehicle. A vehicle operation simulator system comprising: an image viewing means for viewing the real field on which at least one of a probe wave image showing the propagation of the probe wave transmitted from the virtual sensor and a reflected wave image showing the propagation of the reflected wave when the probe wave is reflected by the virtual object is superimposed.

2. The vehicle operation simulator system according to claim 1, wherein the image viewing means selectively views the real field in which only the exploration wave image is superimposed, the real field in which only the reflected wave image is superimposed, or the real field in which both the exploration wave image and the reflected wave image are superimposed.

3. The vehicle operation simulator system according to claim 1 or 2, wherein the image viewing means allows the waveform of the exploration wave to be viewed as the exploration wave image, along with the detection axis of the virtual sensor, and the waveform of the exploration wave advancing in the transmission direction of the exploration wave along the detection axis.

4. The virtual sensor has a detection area set based on the installation position in which it can detect the virtual object. The vehicle operation simulator system according to claim 1 or 2, wherein the image viewing means causes the exploration wave to be viewed as an exploration wave image, showing how the exploration wave moves within the detection area from the installation position in a direction corresponding to the installation direction.

Images