Vehicle motion simulator system

Abstract

The invention provides a vehicle motion simulator system, which enables a detection range and a detection state of a virtual object by a virtual sensor to be overlapped with an actual scene, and enables passengers and observers to visually confirm the detection range and the detection state in real time. A virtual object is disposed in an actual scene, and travel results of a vehicle (2) provided with virtual sensors (22A-22L) for detecting the virtual object are collected when the vehicle (2) travels in the actual scene. The virtual sensors (22A-22L) detect a virtual object by simulating and reproducing a probe wave transmitted from the installation position to the surroundings of the vehicle and receiving a reflected wave obtained by reflecting the probe wave by a virtual object located around the vehicle. And visually confirms an actual scene in which a probe wave image (10) indicating a situation in which a probe wave transmitted from the virtual sensors (22A-22L) propagates from the installation position in a direction along the installation direction is superimposed.

Description

Technical Field

[0001] This invention relates to a vehicle motion simulator system that simulates the movements of a vehicle. Background Technology

[0002] In recent years, in addition to manual driving based on user operation, a new mode of vehicle operation has emerged: automated driving assistance systems. These systems assist users in driving by executing some or all of the user's driving operations on the vehicle side. Automated driving assistance systems are equipped with sensors such as ultrasonic sensors, millimeter-wave radar sensors, and LiDAR sensors to detect surrounding objects (e.g., people, bicycles, other vehicles, walls, etc.). Based on the detection results from these sensors, the system automatically controls the vehicle, including steering, drive system, and brakes.

[0003] Here, in the development of vehicle control systems such as the aforementioned automated driving assistance systems, it is necessary to test the prototype vehicle control system to ensure it can perform correct actions in actual driving environments. Previously, this testing was conducted by mounting the prototype vehicle control system onto a vehicle and actually driving it on a site containing the aforementioned objects. However, this testing needs to be performed under various driving environments, but changing the environment requires changing or reconfiguring the types of objects, and necessitates a very long preparation period before starting the test. Therefore, a technique has been proposed that uses virtual objects (hereinafter referred to as virtual objects) that do not actually exist in the actual scene to create a virtual driving environment, and then mounts virtual sensors for detecting virtual objects onto the vehicle to perform the aforementioned tests (e.g., Korean Patent Registration No. 10-1357596).

[0004] Patent Document 1: Korean Patent Registration No. 10-1357596 (paragraphs 0041-0045)

[0005] In Patent Document 1 mentioned above, the virtual object set in the driving environment is an object that does not actually exist. The virtual sensor used to detect it is not actually installed on the vehicle's exterior wall and does not send detection waves around the vehicle. Therefore, while it can be confirmed after testing and analysis of the results how the virtual sensor sends detection waves and receives reflected waves to detect the virtual object, it is difficult for the vehicle's passengers and observers outside the vehicle to monitor this in real time. Therefore, it is difficult to evaluate whether the vehicle's behavior regarding the detection of the virtual object is appropriate. Summary of the Invention

[0006] This invention was made to solve the aforementioned problems and aims to provide a vehicle motion simulator system that, in vehicle driving tests in which virtual objects are configured in a real-world scenario, can overlap the detection range and detection status of virtual objects detected by virtual sensors with the real-world scenario and provide real-time visual confirmation for passengers and observers.

[0007] To achieve the above objectives, the vehicle motion simulator system of the present invention includes: a virtual object configuration mechanism that configures virtual objects in an actual scene; a virtual object detection mechanism that uses a virtual sensor, assumed to be set at a predetermined position on the vehicle in a predetermined setting direction, to detect the virtual object by simulating the transmission of a probe wave from the predetermined position along the predetermined setting direction to the vicinity of the vehicle and receiving the reflected wave after the probe wave is reflected by the virtual object located around the vehicle; and an image visual confirmation mechanism that visually confirms the actual scene by superimposing at least one of a probe wave image representing the propagation of the probe wave transmitted from the virtual sensor and a reflected wave image representing the propagation of the reflected wave when the probe wave is reflected to the virtual object.

[0008] Furthermore, a "virtual object" can be any object that can be detected by a probe wave; it is not limited to stationary objects and can also be moving objects. In the case of a moving object, a schedule for its movement is pre-set.

[0009] According to the vehicle motion simulator system of the present invention having the above-described structure, in vehicle driving tests where virtual objects are configured in a real-world scenario, the detection range and detection status of the virtual objects detected by the virtual sensors can be superimposed on the real-world scenario, allowing passengers and observers to visually confirm in real time. As a result, the method by which the virtual sensors send probe waves and receive reflected waves to detect the virtual objects becomes clear, and the appropriateness of the vehicle's behavior relative to the detection results of the virtual objects can be evaluated through human line of sight. Attached Figure Description

[0010] Figure 1 This is a simplified structural diagram illustrating the vehicle motion simulator system of this embodiment.

[0011] Figure 2 This is a diagram showing the actual sensors installed in the vehicle.

[0012] Figure 3 This diagram illustrates a method for detecting objects using actual sensors positioned at the front of a vehicle.

[0013] Figure 4 This diagram illustrates a method for detecting objects using actual sensors positioned on the side of a vehicle.

[0014] Figure 5 This diagram illustrates an example of displaying a real-world image of a scene superimposed with a probe wave image on an in-vehicle display.

[0015] Figure 6 This diagram illustrates an example of displaying a real-world image (overhead view) of the actual scene superimposed with a probe wave image on an in-vehicle display.

[0016] Figure 7 It is a diagram representing a probe wave image.

[0017] Figure 8 It is a diagram representing the image of the reflected wave.

[0018] Figure 9 This is a diagram showing a deformed example of a probe wave image.

[0019] Figure 10 This diagram illustrates an example of displaying a real-world image of a scene overlaid with a probe wave image on a VR headset.

[0020] Figure 11 This diagram illustrates an example of displaying a real-world image of the actual scene superimposed with a probe wave image in a car's rearview mirror.

[0021] Figure 12 This is an example of a real-world imagery that shows overlapping images of a real scene representing virtual objects from multiple viewpoints.

[0022] Figure 13 This is a simplified structural diagram of the vehicle according to this embodiment.

[0023] Figure 14 This is an image showing the appearance of the virtual sensors on the simulator device.

[0024] Figure 15 This is a diagram showing the internal structure of a virtual sensor.

[0025] Figure 16 This is a block diagram illustrating the structure of the simulator device according to this embodiment.

[0026] Figure 17 This is a flowchart of the action evaluation procedure in this embodiment.

[0027] Figure 18 It is a diagram that determines the propagation direction (directivity) and propagation range (detection area) of the probe waves sent from the virtual sensor.

[0028] Explanation of reference numerals in the attached figures

[0029] 1…Vehicle motion simulator system, 2…Vehicle, 3…Simulator device, 4…Evaluator (observer, passenger), 5…In-vehicle display, 6…Real-world image of actual scene, 8…VR head-mounted display, 9A~9L…Actual sensor, 10…Probe wave image, 11…External terminal, 20…Driver assistance ECU, 21…Reflected wave image, 22A~22L…Virtual sensor, 40…Virtual control ECU (an example of a virtual object configuration mechanism, virtual object detection mechanism, or image visual verification mechanism), 41…CPU. Detailed Implementation

[0030] Hereinafter, an embodiment of the vehicle motion simulator system of the present invention will be described in detail with reference to the accompanying drawings. First, using... Figure 1 The general structure of the vehicle motion simulator system 1 in this embodiment will be explained. Figure 1 This is a simplified structural diagram of the vehicle motion simulator system 1 according to this embodiment.

[0031] like Figure 1 As shown, the vehicle motion simulator system 1 of this embodiment basically includes a vehicle 2 that serves as the object of a simulated driving test; and a simulator device 3 that provides a virtual driving environment for the vehicle 2 to serve as the driving environment for the driving test, and obtains, analyzes, evaluates, and outputs the motion results of the vehicle 2's driving test. Furthermore, in this embodiment below, the evaluation object that becomes the quality evaluation object through the aforementioned driving test is an automated driving assistance system that assists passengers in driving the vehicle by performing part or all of the passenger's driving operations on the vehicle side; the vehicle 2 is a vehicle equipped with the automated driving assistance system that becomes the evaluation object.

[0032] Furthermore, driving tests are one of the processes conducted by the manufacturer before the vehicle leaves the factory to identify and correct malfunctions in the automated driving assistance system. This involves simulating the actions of the automated driving assistance system under various hypothetical driving environments and obtaining and analyzing the results to confirm whether any malfunctions have occurred. Moreover, driving tests are conducted by having the vehicle 2 actually drive in a real-world scenario (e.g., on a wide, flat road with nothing around it), and by using a simulator device 3 to provide a virtual driving environment, driving tests can be performed under various driving conditions even in the same real-world scenario.

[0033] Furthermore, the simulator device 3 is mounted on the vehicle 2 and connects to the vehicle 2 via a vehicle network such as CAN for bidirectional communication. The simulator device 3 does not necessarily need to be mounted on the vehicle 2; it can be partially or entirely located outside the vehicle 2. In this case, it can connect to the vehicle 2 via wireless communication for bidirectional communication. The simulator device 3 is configured with operating mechanisms such as a keyboard and a display, allowing for the input and output of various information through the operation of the evaluator (observer / passenger) 4 who is evaluating the autonomous driving assistance system. Specifically, the evaluator 4 inputs the positions and types of virtual objects configured in the actual driving scenario of the vehicle 2 as the driving environment into the simulator device 3. Furthermore, the simulator device 3 constructs a virtual driving environment for testing the vehicle 2 based on the input driving environment.

[0034] Furthermore, the simulator device 3 is equipped with virtual sensors for vehicle 2 to detect virtual objects constructed as the aforementioned driving environment. When vehicle 2 is driving in the actual scene, the detection results of the virtual objects detected by the virtual sensors are output to the control device for autonomous driving assistance on the vehicle 2 side. As a result, the vehicle side can recognize the virtual objects as actual objects (e.g., people, bicycles, other vehicles, walls, etc.) and reproduce a driving environment that is closer to reality.

[0035] Furthermore, the evaluator 4 inputs the assumed setup position and orientation of the virtual sensor by pre-operating the simulator device 3, and then sets it. Here, the setup position and orientation of the virtual sensor are based on the actual setup position and orientation of the sensors mounted on the vehicle 2 of the test subject. For example, Figure 2 This diagram shows the actual sensors (hereinafter referred to as actual sensors, compared to virtual sensors) 9A to 9L that are actually installed in vehicle 2. Although actual sensors 9A to 9L are described as ultrasonic sensors in the following explanation, millimeter-wave sensors or radar sensors can also be used instead of ultrasonic sensors.

[0036] like Figure 2As shown, actual sensors 9A to 9L are arranged at predetermined intervals at the front, rear, and sides of the vehicle. They detect objects that reflect the probe waves by transmitting ultrasonic waves as probe waves around the vehicle 2 and receiving reflected waves after the transmitted probe waves are reflected by objects located around the vehicle. Specifically, a ranging sensor exists that can detect the distance (range value) of the object reflecting the probe waves by measuring the time from transmission to reception. Furthermore, the actual sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the vehicle's control unit. Since virtual sensors are used instead of actual sensors 9A to 9L at least during testing, the signal input to the control unit can be switched between the output signals from actual sensors 9A to 9L and the output signals from virtual sensors by setting a switching switch, for example.

[0037] Furthermore, although the placement position and orientation of each actual sensor 9A to 9L can be appropriately set, in this embodiment, in order to define all directions in front of, behind, and to the left and right sides of the vehicle 2 as the detection range of the object, for example, actual sensors 9A to 9D are placed in front of the vehicle 2 facing the propagation direction, with the direction of the probe wave transmission being in front of the vehicle's propagation direction. Additionally, actual sensors 9E and 9F are placed on the left side of the vehicle 2 facing the left side, with the direction of the probe wave transmission being to the left of the vehicle's propagation direction. Furthermore, actual sensors 9G and 9H are placed on the right side of the vehicle 2 facing the right side, with the direction of the probe wave transmission being to the right of the vehicle's propagation direction. Finally, actual sensors 9I to 9L are placed behind the vehicle 2 facing the opposite direction to the propagation direction, with the direction of the probe wave transmission being behind the vehicle. The height of each actual sensor 9A to 9L above the ground surface is set to the same level.

[0038] Especially when illustrating actual sensors 9A to 9D, such as Figure 3 As shown, the actual sensors 9A to 9D are not offset to the left or right and are evenly spaced apart and are set at different positions around the front bumper or the front grille above the vehicle 2. Preferably, they can send detection waves to a wider range in front of the vehicle (i.e., set the detectable range of the object to a wider range).

[0039] Specifically, such as Figure 3As shown, actual sensor 9A is positioned near the left front corner of vehicle 2, oriented so that the direction of the detection wave is slightly tilted to the left compared to the propagation direction of vehicle 2, in order to send detection waves towards the left front of vehicle 2. Actual sensor 9B is positioned slightly to the left of the vehicle's centerline, oriented so that the detection waves are sent towards the front of vehicle 2, particularly centered on the left side, in order that the direction of the detection waves is towards the propagation direction of vehicle 2. Actual sensor 9C is positioned slightly to the right of the vehicle's centerline, oriented so that the detection waves are sent towards the front of vehicle 2, particularly centered on the right side, in order that the direction of the detection waves is towards the propagation direction of vehicle 2. Actual sensor 9D is positioned near the right front corner of vehicle 2, oriented so that the direction of the detection waves is slightly tilted to the right compared to the propagation direction of vehicle 2, in order to send detection waves towards the right front of vehicle 2. Furthermore, actual sensors 9A and 9D, and actual sensors 9B and 9C, are symmetrically arranged along the vehicle's centerline in the top view. Furthermore, although the illustration is omitted, the actual sensors 9I to 9L at the rear of vehicle 2 are also arranged vertically symmetrically. Additionally, although the transmission direction of the detection waves in the height direction (vertical direction) is omitted from the illustration, they are all set to be horizontal as well.

[0040] On the other hand, such as Figure 4 As shown, the actual sensors 9E and 9F on the sides are each configured to send detection waves in a direction that intersects the propagation direction of vehicle 2 at a 90-degree angle. Additionally, although not shown in the diagram, the actual sensors 9G and 9H on the right side of vehicle 2 are also configured to be symmetrical. Furthermore, although the direction of transmission of detection waves in the height direction (vertical direction) is not shown in the diagram, it is configured to be horizontal.

[0041] Furthermore, the virtual sensor placement positions and orientations set in this embodiment are based on the actual sensor placement positions and orientations of the actual sensors 9A to 9L. That is, the premise for conducting the test is that virtual sensors (a total of 12) are placed in the same placement positions and orientations as the actual sensors 9A to 9L.

[0042] Furthermore, the analysis of the driving results of vehicle 2 in simulator device 3 is configured to perform either "real-time analysis" or "post-event analysis." "Real-time analysis" monitors vehicle 2 during the driving test and automatically detects and outputs any adverse situations that occur. On the other hand, "post-event analysis" evaluates and outputs the control data, driving trajectory, etc., of vehicle 2 stored during the driving test after the test is completed.

[0043] Therefore, when conducting a driving test on vehicle 2, the evaluator 4 first inputs the types and positions of virtual objects configured relative to the actual scene into the simulator device 3, specifically as the driving environment for the driving test of vehicle 2. This generates the driving environment for the driving test. Furthermore, "virtual objects" are not limited to three-dimensional objects; for example, they include dividing lines, road markings, etc. Moreover, they are not limited to stationary objects; they can also be moving objects. If they are moving objects, a schedule for their movement (at what speed and how) is pre-set. For example, when conducting a driving test in a parking lot, the virtual objects input include parking space markings depicted on the road surface within the parking lot, other vehicles parked in parking spaces, pedestrians moving within the parking lot, and other vehicles traveling on the passageways within the parking lot. Then, when the driving test begins, the virtual objects generated as the driving environment are detected by virtual sensors mounted on the vehicle, simulating the vehicle's driving in a virtual driving environment using automated driving assistance. Furthermore, the driving results are analyzed, evaluated, and output through "real-time analysis" or "post-event analysis," and in the event of an adverse situation, the resulting adverse situation can be identified.

[0044] In this system, the virtual objects configured in the actual scene via simulator device 3 are not actually existing objects. Therefore, even if they are identified by virtual sensors on the vehicle side, the evaluator 4 cannot directly visually confirm them. Furthermore, the virtual sensors are also pseudo-sensors; they are not actually installed on the vehicle's exterior wall and do not emit detection waves around the vehicle. Therefore, while the method by which the virtual sensors emit detection waves and detect the virtual objects can be confirmed through post-test analysis, the evaluator 4 has difficulty grasping it in real time. Therefore, especially in the case of "real-time analysis," the method by which the virtual sensors emit detection waves and detect the virtual objects is opaque, making it difficult for the evaluator 4 to assess whether the vehicle's behavior is appropriate from their perspective. Therefore, in the vehicle motion simulator system 1 of this embodiment, to enable the evaluator 4 to visually confirm the actual scene, the following mechanism is provided: an image representing the virtual object at the location where the virtual object is configured is overlaid in the actual scene, and an image representing the location from the assumed setting position (see reference 1) is overlaid. Figure 3 , Figure 4 The method includes at least one of the following: a probe wave image showing the propagation of the probe wave from the virtual sensor (including both diffuse and narrow propagation, hereinafter the same), and a reflected wave image showing the propagation of the reflected wave when the probe wave is reflected onto the virtual object. Furthermore, the method is described assuming various scenarios, such as an evaluator 4 riding in vehicle 2 as a passenger, and an observation from outside the vehicle.

[0045] First, let's illustrate an example of a scenario where an evaluator 4, acting as a passenger in vehicle 2, visually confirms an actual scene where the aforementioned probe wave image overlaps. Here, in this embodiment, it is possible to selectively visually confirm any one of the following: an actual scene where only the probe wave image overlaps, an actual scene where only the reflected wave image overlaps, or an actual scene where both the probe wave image and the reflected wave image overlap. For example, this can be switched by the evaluator 4 operating the operation unit 14, described later.

[0046] In the first case, such as Figure 5 As shown, the in-vehicle display 5, mounted on vehicle 2 and visually accessible to the vehicle's passengers, displays a real-time scene image 6 captured by an external camera. Furthermore, by combining an image 7 representing a virtual object with an image 10 representing the propagation of a probe wave emitted from a set virtual sensor location (which is equivalent to the location of the actual sensor) displayed on the in-vehicle display 5, the evaluator 4 can visually confirm the actual scene, which includes an image 7 representing a virtual object at the location where the virtual object is positioned and an image 10 representing the propagation of a probe wave emitted from a virtual sensor. Additionally, the simulator device 3 can determine the position of the virtual object in the real-scene image 6 displayed on the in-vehicle display 5 by converting the virtual object's configuration coordinates (absolute coordinates) set as a driving environment into the coordinate system of the external camera (relative coordinates). Furthermore, the simulator device 3 can determine the propagation direction (directivity) and propagation range (detection area) of the probe wave based on the preset virtual sensor's location and orientation, the virtual sensor's performance, and the probe wave's output intensity (e.g., pulse length, drive current, base gain). The virtual sensor's performance is based on the actual sensors 9A to 9L. In the first example, while a real-time scene image 6 of the actual scene captured by an external camera (i.e., an image along the optical axis with the camera's position as the viewpoint) is displayed, a bird's-eye view or a top-down view generated by synthesizing images from multiple cameras positioned in front, behind, and to the side of the vehicle can also be displayed. In addition to displaying a top-down view, it is preferable to display the probe wave image 10 as a three-dimensional image to represent the movement of the probe wave.

[0047] For example, Figure 6As shown in the second example of displaying a top-down image, the real-time scene image 6 generates a top-down image of the vehicle's surroundings from above, based on real-time images captured by multiple cameras positioned at the front, rear, and sides of the vehicle. This generated top-down image is then displayed on the vehicle display 5. The generation of the top-down image is well-known, so details are omitted. However, the images captured by the multiple cameras are composited (stitched into a single image) after viewpoint conversion, and a schematic illustration of the vehicle is inserted in the center to generate the top-down image. Furthermore, by similarly combining the image 7 representing a virtual object and the probe wave image 10 representing the propagation of probe waves emitted from the location of a set virtual sensor (which is also equivalent to the location of the actual sensor) onto the scene image 6 displayed on the vehicle display 5, the evaluator 4 can visually confirm the actual scene where the image 7 representing the virtual object at the location where the virtual object is located is superimposed, and the probe wave image 10 representing the propagation of probe waves emitted from the virtual sensor is superimposed. Additionally, in Figure 6 In the overhead image shown, passengers can visually identify areas that are considered blind spots.

[0048] Next, the probe wave image 10, which illustrates the propagation of the probe wave transmitted from the virtual sensor, is described in more detail, such as... Figure 7 As shown, the probe wave image 10 is an image visually confirming the propagation of the probe wave from the setting position X in a detection area Y capable of detecting virtual objects, with the setting position X of the virtual sensor as a reference. Additionally, as... Figure 7 As shown, the preferred probe wave image 10 is a moving image representing the continuous appearance and movement of the probe wave's arc in the transmission direction. As a result, the propagation direction of the probe wave can be determined.

[0049] Furthermore, while the previously described example showed an actual scene where only the probe wave image 10 was superimposed, it is also possible to visually confirm, particularly in the case of detecting a virtual object using a virtual sensor, an actual scene where only the reflected wave (reflected probe wave) 21 representing the probe wave reflected onto the virtual object is superimposed, and an actual scene where both the probe wave image 10 and the reflected wave image 21 are superimposed. For example, Figure 8 (A) represents the actual scene (overhead view) where the probe wave image 10 and the reflected wave image 21 are superimposed. Figure 8 (B) represents the actual scene (overhead view) with only the reflected wave image 21 superimposed.

[0050] like Figure 8 (A) Figure 8As shown in (B), when detecting virtual objects, the reflected wave image 21, which represents the reflected wave after being reflected by the virtual object, can be visually confirmed by visually verifying it. This allows the detection status of the virtual object to be superimposed on the actual scene and visually confirmed in real time by the evaluator 4. Furthermore, when visually confirming both the probe wave image 10 and the reflected wave image 21, it is preferable to use a method that allows identification of both the probe wave image 10 and the reflected wave image 21, for example, by setting the display color of each image to a different color. Moreover, when detecting virtual objects, the distance to the detected virtual object can also be displayed (e.g., in...). Figure 8 The example shown represents the case where a virtual object is detected 1m in front of the target.

[0051] Moreover, such as Figure 9 As shown, as another variation, the probe wave image 10 can also be configured to display the waveform 13 of the probe wave propagating along the detection axis 12 in the direction of probe wave transmission, together with the detection axis 12 extending from the setting position X of the virtual sensor.

[0052] Next, in the third example, VR (Virtual Reality) technology is used, with the vehicle's passengers, i.e., the evaluators 4, wearing VR headsets 8. For example... Figure 10 As shown, the VR headset 8 displays not only pre-recorded live-action images of the vehicle interior, but also real-time live-action images 6 of the actual scene captured by an external camera outside the vehicle window. Furthermore, by combining the image 7 representing a virtual object with the probe wave image 10 representing the propagation of probe waves emitted from the location of the set virtual sensor (the actual sensor's location) into the displayed live-action image 6 of the actual scene, the evaluator 4 can visually confirm the actual scene superimposed with the image 7 representing the virtual object at the location where the virtual object is positioned, and the probe wave image 10 representing the propagation of probe waves emitted from the virtual sensor. Additionally, the simulator device 3 can determine the position of the virtual object in the live-action image 6 of the actual scene displayed by the VR headset 8 by converting the configuration coordinates (absolute coordinates) of the virtual object set as a driving environment into the VR (i.e., user's line-of-sight) coordinate system (relative coordinates). Furthermore, the simulator device 3 can determine the propagation direction (directivity) and propagation range (detection area) of the probe wave based on the preset setting position and orientation of the virtual sensor, the performance of the virtual sensor, and the output intensity of the probe wave (e.g., pulse length, drive current, fundamental gain). The performance of the virtual sensor is based on the performance of the actual sensors 9A to 9L. Additionally, in the state of detecting virtual objects, it is compatible with... Figure 8 The same example shown can also display the reflected wave image 21 in the VR head-mounted display 8.

[0053] Furthermore, as in the first to third examples described above, when the evaluator 4, as a passenger in vehicle 2, visually confirms the image 7, probe wave image 10, and reflected wave image 21 representing a virtual object, it is also possible to overlay the image 7, probe wave image 10, and reflected wave image 21 onto the actual scene reflected or displayed in one or both of the vehicle's interior and exterior rearview mirrors. That is, in the actual scene reflected or displayed in one or both of the vehicle's interior and exterior rearview mirrors, the image 7 representing the virtual object is overlaid at the location where the virtual object is located. Additionally, if the actual scene reflected or displayed in one or both of the vehicle's interior and exterior rearview mirrors includes the detection area of ​​a virtual sensor, the probe wave image 10 and reflected wave image 21 are displayed. For example, if the interior and exterior rearview mirrors are monitor types displaying images captured by an exterior camera, such as... Figure 11 As shown, by combining the image 7 representing a virtual object, the probe wave image 10, and the reflected wave image 21 onto the image display area 6 of the rearview mirror 19, the evaluator 4 can visually confirm the actual scene displayed, which includes the image 7 representing the virtual object at the location where the virtual object is located, the probe wave image 10 representing the propagation of the probe wave sent from the virtual sensor, and the reflected wave image 21 representing the propagation of the reflected wave after being reflected by the virtual object. On the other hand, when the rearview mirror and the exterior rearview mirror are mirror-type displays with a perspective display on the mirror surface, the image 7 representing the virtual object, the probe wave image 10, and the reflected wave image 21 are superimposed on the actual scene reflected in the mirror using the perspective display. This allows the evaluator 4 to visually confirm the actual scene displayed, which includes the image 7 representing the virtual object at the location where the virtual object is located, the probe wave image 10 representing the propagation of the probe wave sent from the virtual sensor, and the reflected wave image 21 representing the propagation of the reflected wave after being reflected by the virtual object.

[0054] In addition to the examples mentioned above, other mechanisms that enable the evaluator 4, as a passenger in vehicle 2, to visually confirm virtual objects positioned in the actual scene and the detection waves emitted from the virtual sensor can also be, for example, a HUD (Head-Up Display) or a see-through VR headset. These are essentially technologies that overlay images onto the real-world scene outside the vehicle, which can be visually confirmed through the windshield and side windows, and then visually confirm it. Therefore, the display control is performed by overlaying images representing virtual objects at their positions within the actual scene contained in the passenger's field of vision, and displaying the detection wave image 10 and the reflected wave image 21 in the detection area of ​​the virtual sensor.

[0055] Next, the situation where the evaluator 4 observes from outside the vehicle will be described. For the evaluator 4 observing from outside the vehicle, a real-time scene image 6 of the actual scene captured by an external camera is displayed on the display of a tablet-type external terminal 11 that is connected to the simulator device 3 in a communicative manner. The details are the same as those displayed on the in-vehicle display 5. By combining the image 7 representing the virtual object, the probe wave image 10, and the reflected wave image 21 of the actual scene displayed on the external terminal 11, the evaluator 4 can visually confirm the actual scene, which is superimposed with the image 7 representing the virtual object at the location where the virtual object is located, and superimposed with the probe wave image 10 representing the propagation of the probe wave sent from the virtual sensor, and the reflected wave image 21 representing the propagation of the reflected wave after being reflected by the virtual object.

[0056] In particular, when the external terminal 11 displays the actual scene, it is not necessary to display the actual scene contained in the viewpoint of vehicle 2, but rather to display the actual scene contained in the viewpoint of a position different from vehicle 2. For example, if the external terminal 11 is equipped with a camera, then... Figure 12 As shown, the display of the external terminal 11 shows a real-time scene image 6 of the actual scene captured by the camera of the external terminal 11. Furthermore, the simulator device 3 obtains the optical axis information of the camera of the external terminal 11 and converts the configuration coordinates (absolute coordinates) of the virtual objects set as the driving environment into the coordinate system (relative coordinates) of the camera of the external terminal 11, thereby determining the position of the virtual objects in the actual scene captured by the camera of the external terminal 11. Then, by combining the image 7 representing the virtual objects with the scene image 6 of the actual scene displayed on the display of the external terminal 11, the evaluator 4 can visually confirm the actual scene overlaid with the image 7 representing the virtual objects at the positions where the virtual objects are located. Additionally, the simulator device 3 can determine the propagation direction (directivity) and propagation range (detection area) of the probe wave based on the preset setting position and orientation of the virtual sensor, as well as the performance of the virtual sensor and the output intensity of the probe wave (e.g., pulse length, drive current, basic gain). Then, by synthesizing the probe wave image 10 and the reflected wave image 21 in the determined detection area within the real scene image 6 displayed on the monitor of the external terminal 11, the evaluator 4 can visually confirm the real scene that is superimposed with the probe wave image 10, which represents the propagation of the probe wave sent from the virtual sensor, and the reflected wave image 21, which represents the propagation of the reflected wave after being reflected by the virtual object.

[0057] Furthermore, the display can be switched appropriately between showing the actual scene from the perspective of vehicle 2 and from the perspective of evaluator 4 via the operation of the external terminal 11 of evaluator 4. This allows for real-time visual confirmation of the actual scene containing virtual objects from various viewpoints, rather than from a fixed viewpoint. In addition to displaying the actual scene from the perspective of vehicle 2 and from the perspective of evaluator 4, the display can also show the actual scene from the perspective of any camera device present in the actual scene.

[0058] In addition, in the above embodiment, although the image 7 representing the virtual object, the probe wave image 10, and the reflected wave image 21 are superimposed on the real scene image 6 of the actual scene displayed on the vehicle display 5, the VR head-mounted display 8, and the external terminal 11, it is also possible not to display the image 7 representing the virtual object and only superimpose the probe wave image 10 or the reflected wave image 21.

[0059] Next, the following describes a vehicle 2 equipped with an autonomous driving assistance system that is evaluated by the vehicle motion simulator system 1. Figure 13 This is a simplified structural diagram of vehicle 2 in this embodiment.

[0060] Here, vehicle 2 can be, for example, a car powered by an internal combustion engine (engine, etc.) (internal combustion engine car), a car powered by an electric motor (motor, etc.) (electric car, fuel cell car, etc.), or a car powered by both (hybrid car). Furthermore, regardless of the vehicle type, it can be a regular car, or a large commercial truck, bus, construction machinery, etc. Although described below as a four-wheeled vehicle, it can also be a two-wheeled or three-wheeled vehicle.

[0061] Among them, vehicle 2 is a vehicle that is assisted in driving by autonomous driving assistance, which can drive automatically regardless of whether the user is driving or not, except for manual driving based on the user's driving operation.

[0062] Furthermore, autonomous driving assistance can be implemented only in specific situations such as parking or exiting a parking space, or it can be implemented across all road sections, or it can be configured to only be implemented while the vehicle is traveling on a specific road section (e.g., on a highway where the boundary is a toll gate (regardless of whether there are people or not, or whether it is toll-free)). In the following description, the autonomous driving area for which autonomous driving assistance is implemented includes all road sections in addition to general roads and highways, as well as parking lots, and it is implemented only when the user selects to implement autonomous driving assistance (e.g., turns on the autonomous driving start button) and it is determined that autonomous driving assistance-based driving is possible. On the other hand, vehicle 2 can also be configured to be a vehicle that can only perform assisted driving based on autonomous driving assistance. Alternatively, the vehicle can also perform assisted driving based on autonomous driving assistance only for driving to a parking space when parking (i.e., parking assistance).

[0063] Furthermore, in the vehicle control of the automated driving assistance system in this embodiment, for example, the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles are detected at any time. The vehicle is then automatically controlled by adjusting the steering, drive system, and brakes, moving along a generated driving trajectory at a speed planned based on the same speed. Particularly in the case of parking assistance, automatic vehicle control is performed. In this vehicle control, the detection results of sensors and cameras are used to confirm the parking space to which the vehicle will park and the surrounding conditions. A parking trajectory to the parking space is calculated, and the vehicle is driven into the parking space along the calculated parking trajectory to complete the parking.

[0064] In addition, such as Figure 13 As shown, vehicle 2 includes: an operation unit 14 that receives operations from passengers; and a real-scene image 6 that displays the actual scene described above to passengers. Figure 5 The vehicle-mounted display 5 (or VR head-mounted display 8) receives output information from other simulator devices 3; the speaker 15 outputs sound guidance from simulator devices 3; the front camera 16, rear camera 17, and side cameras 18A and 18B are used to capture images of the vehicle's surroundings; the actual sensors 9A to 9L detect obstacles around the vehicle; the control unit, i.e., the driver assistance ECU (electronic control unit) 20, performs various calculations related to autonomous driving assistance based on the input information; and the simulator device 3.

[0065] The following describes the various structural components of vehicle 2. First, the operation unit 14 is located, for example, in front of the steering wheel (also called the steering wheel), and includes operation buttons that are activated when autonomous driving assistance is initiated. By operating the operation unit 14, the user can switch between manual driving based on the user's driving actions and assisted driving performed by autonomous driving assistance, where the vehicle drives automatically regardless of the user's driving actions. Furthermore, the operation unit 14 may also have a touch panel located in front of the vehicle's display. Additionally, it may include a microphone and a voice recognition device.

[0066] The in-vehicle display 5 is a type of display device installed on the dashboard of the vehicle 2. As described above, it displays a real-scene image 6 of the actual scene, which is superimposed with an image 7 representing the virtual object at the location where the virtual object is located, and superimposed with a probe wave image 10 representing the propagation of the probe wave sent from the virtual sensor, or a reflected wave image 21 representing the propagation of the reflected wave after the probe wave is reflected by the virtual object. Figure 5 Alternatively, the same VR head-mounted display 8 can be used instead of the in-vehicle display 5. Furthermore, as described above, a display device can also be provided that displays a real-scene image 6 of the actual scene, which is superimposed on both the interior and exterior rearview mirrors, showing an image 7 representing a virtual object, a probe wave image 10, and a reflected wave image 21. Figure 11 In addition, the real-world image 6 of the actual scene displayed in the vehicle display 5 and the VR head-mounted display 8 is a composite image formed by combining the image 7 representing the virtual object, the probe wave image 10, and the reflected wave image 21 with the real-time images captured by the front camera 16, the rear camera 17, and the side cameras 18A and 18B.

[0067] In addition, a speaker 15 is installed on the dashboard of the vehicle 2 to output guidance sounds, warning sounds, etc. from the simulator device 3.

[0068] In addition, the front camera 16 is, for example, a camera with a solid-state imaging element such as a CCD, and is provided above the front bumper of the vehicle 2 or inside the rearview mirror, with the optical axis pointing in the direction of the vehicle's propagation.

[0069] The rear camera 17 is a shooting device that uses a camera with the same solid-state shooting element such as a CCD. For example, it is installed near the upper center of the license plate number installed at the rear of the vehicle 2, and the optical axis is oriented towards the rear of the vehicle.

[0070] Furthermore, the side cameras 18A and 18B are imaging devices that use the same solid-state imaging element such as a CCD, for example, mounted on the left and right exterior rearview mirrors of the vehicle 2, and set with the optical axis direction facing the side of the vehicle.

[0071] Furthermore, the simulator device 3 reads information about the types and locations of virtual objects in the driving environment pre-set by the evaluator 4, and synthesizes the image 7 representing the virtual objects at the locations where the virtual objects are located onto the images captured by the front camera 16, rear camera 17, and side cameras 18A and 18B, thereby generating a real-world image 6 of the actual scene displayed on the vehicle display 5 and the VR headset display 8. Additionally, the simulator device 3 determines the propagation direction (directivity) and propagation range (detection area) of the probe wave based on the location and orientation of the virtual sensor pre-set by the evaluator 4, as well as the performance of the virtual sensor and the output intensity of the probe wave (e.g., pulse length, drive current, basic gain). Similarly, relative to the images captured by the front camera 16, rear camera 17, and side cameras 18A and 18B, it synthesizes the probe wave image 10 and the reflected wave image 21 onto the detection area of ​​the virtual sensor. In addition, the real-world image that becomes the object of the synthesis of the image 7 representing the virtual object and the probe wave image 10 can also be the real-world image itself captured by the front camera 16, the rear camera 17, and the side cameras 18A and 18B. For example, it can also be the image synthesized from multiple cameras or the image after viewpoint transformation (e.g., overhead image, bird's-eye view image).

[0072] On the other hand, the actual sensors 9A to 9L are ranging sensors used to detect objects that actually exist around the vehicle, for example, ultrasonic sensors. Additionally, corresponding to the actual sensors 9A to 9L, there are also virtual sensors 22A to 22L (described later), which, while unable to detect actual objects, can detect virtual objects; these are also mounted on the vehicle 2. As described above, the actual sensors 9A to 9L are respectively arranged at predetermined intervals at the front, rear, and sides of the vehicle. They detect objects that reflect the detection waves by sending ultrasonic waves as detection waves around the vehicle 2 and by receiving the reflected waves after the transmitted detection waves are reflected by objects located around the vehicle. Figure 3 , Figure 4 ).

[0073] On the other hand, the driver assistance ECU 20 is an electronic control unit that performs various processes related to autonomous driving assistance. The driver assistance ECU 20 is also connected to various sensors used to detect vehicle behavior, such as vehicle speed sensors, wheel speed sensors, acceleration sensors, gyroscope sensors, steering sensors, and shift position sensors, as well as various drive units of the vehicle, such as the steering gear, brakes, accelerator, and transmission. Based on the detection results of these sensors, it detects the current behavior of the vehicle and calculates control information (control quantities) for controlling each drive unit, thereby controlling each drive unit and implementing autonomous driving assistance for vehicle 2. Specific aspects of autonomous driving assistance include vehicle control, in which the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles are constantly detected, and vehicle control, such as steering, drive source, and brakes, is performed to ensure the vehicle travels along a generated driving trajectory at a speed planned based on the same generated speed. Especially in the case of parking assistance, the parking space to which the vehicle is to be parked and its surrounding conditions are confirmed, and parking control is performed by calculating a parking trajectory to the parking space and guiding the vehicle along the calculated parking trajectory to complete the parking.

[0074] Here, the driver assistance ECU 20, as a mechanism for acquiring information related to surrounding obstacles and parking spaces necessary for performing the aforementioned autonomous driving assistance, includes mechanisms for acquiring detection information from actual sensors 9A-9L and mechanisms for acquiring detection information from virtual sensors 22A-22L. The actual sensors 9A-9L are sensors used to detect objects actually existing around the vehicle, while the virtual sensors 22A-22L are sensors used to detect virtual objects in a driving environment virtually generated by the simulator device 3. The virtual sensors 22A-22L are part of the simulator device 3 and are mounted on the vehicle 2.

[0075] here, Figure 14 This is an external view of the virtual sensors 22A to 22L equipped in simulator device 3. Figure 15 This diagram shows the internal structure of virtual sensors 22A to 22L. Furthermore, virtual sensors 22A to 22L have essentially the same structure; the following explanation will use virtual sensor 22A as an example.

[0076] like Figure 14 as well as Figure 15As shown, the virtual sensor 22A has a cubic box shape, with a first ultrasonic sensor 25 at one end and a second ultrasonic sensor 26 at the other end inside the box. Furthermore, the first ultrasonic sensor 25 and the second ultrasonic sensor 26 have essentially the same structure as the actual sensors 9A-9L described above, and are arranged opposite each other coaxially with the detection axis X. The detection axis X is a virtual straight line extending from the first ultrasonic sensor 25 and the second ultrasonic sensor 26 along the transmission and reception direction of the probe wave. The first ultrasonic sensor 25 and the second ultrasonic sensor 26 each have a piezoelectric vibrator (e.g., a ceramic plate) for transmitting and receiving probe waves. The piezoelectric vibrator is formed as a plane with the detection axis X as its normal. Moreover, the first ultrasonic sensor 25 and the second ultrasonic sensor 26 are configured to transmit probe waves along the detection axis X by causing the piezoelectric vibrator to vibrate ultrasonically based on a drive signal applied to an electromechanical conversion element. Furthermore, the first ultrasonic sensor 25 and the second ultrasonic sensor 26 are configured to, when receiving a received wave from the outside, cause the electromechanical conversion element to generate an electrical signal, i.e., a received signal, corresponding to the excitation state of the piezoelectric oscillator caused by the received ultrasonic wave.

[0077] Furthermore, the virtual sensor 22A detects the arrival of a reflected wave by the electromotive force generated by the vibration of the piezoelectric vibrator of the first ultrasonic sensor 25 as it receives the probe wave, which exceeds a threshold. It then calculates the distance to the object by the time elapsed between the first ultrasonic sensor 25 sending the probe wave and receiving it. Here, the object is a virtual object; the first ultrasonic sensor 25 does not actually send a probe wave, but assumes it does. The probe wave sent from the second ultrasonic sensor 26 is identified as a reflected wave and received. In this case, the distance value detected by the virtual sensor 22A can be freely controlled by adjusting the timing of the probe wave transmission from the second ultrasonic sensor 26.

[0078] That is, the simulator device 3 determines the current position and orientation of the vehicle based on information from various sensors installed on the vehicle, such as the vehicle speed sensor, wheel speed sensor, acceleration sensor, gyroscope sensor, steering sensor, and shift position sensor. Based on the determined current position and orientation of the vehicle and the position of virtual objects configured in a driving environment pre-set by the evaluator 4, it determines the relative position of the virtual objects relative to the vehicle 2. Furthermore, based on the relative position of the virtual objects relative to the determined vehicle 2 (more specifically, the actual sensors 9A-9L on the vehicle), it controls the timing of sending probe waves from the second ultrasonic sensor 26, thereby enabling the virtual sensors 22A-22L to detect virtual objects that do not actually exist. In other words, the virtual sensors 22A-22L can detect virtual objects by simulating and reproducing the sending of probe waves from their setting position to the vicinity of the vehicle 2 and receiving the reflected waves after the probe waves are reflected by virtual objects located around the vehicle 2.

[0079] Furthermore, in the vehicle motion simulator system 1 of this embodiment, during the driving test (during the collection of driving results), the driver assistance ECU 20 detects surrounding obstacles and parking spaces based on the detection information from virtual sensors 22A to 22L, and performs vehicle control such as steering, drive, and braking. On the other hand, during the preparation phase before the driving test (the preparation phase before collecting driving results) and after the driving test (after the collection of driving results is completed), it is necessary to avoid actual walls and obstacles to drive the vehicle. Therefore, the driver assistance ECU 20 detects surrounding obstacles and parking spaces based on the detection information from actual sensors 9A to 9L, and performs vehicle control such as steering, drive, and braking. In addition, the switching of sensor information input to the driver assistance ECU 20 is performed by switches or the like.

[0080] Furthermore, the number of virtual sensors 22A to 22L is the same as that of actual sensors 9A to 9L, with virtual sensor 22A corresponding to actual sensor 9A. That is, during driving tests, the driver assistance ECU 20 identifies virtual sensor 22A as actual sensor 9A and detects objects (virtual objects that do not actually exist) located to the left front of the vehicle. For example, if the distance from actual sensor 9A to the virtual object is 1m, the timing of sending a detection wave from the second ultrasonic sensor 26 is controlled so that the distance detected in virtual sensor 22A is 1m. Thus, an object (actually a virtual object) is detected at a position 1m from actual sensor 9A. Similarly, virtual sensor 22B corresponds to actual sensor 9B, virtual sensor 22C corresponds to actual sensor 9C, and virtual sensor 22D corresponds to actual sensor 9D. The same applies to the remaining virtual sensors 22E to 22L.

[0081] In addition, vehicle 2 Figure 13 In addition to the structural components shown, there are also basic structural components for vehicle 2, but only the structure related to the control of autonomous driving assistance and the control related to the structure are described.

[0082] Next, the details of the simulator device 3 included in the vehicle motion simulator system 1 will be explained. Figure 16 This is a block diagram showing the structure of the simulator device 3 in this embodiment.

[0083] like Figure 16 As shown, the simulator device 3 includes: a control unit, namely a virtual control ECU (electronic control unit) 40, which performs various calculations and processing such as generating the driving environment during driving tests, controlling the virtual sensors 22A to 22L, and analyzing, evaluating, and outputting the action results of the driving tests of the vehicle 2; and the aforementioned virtual sensors 22A to 22L.

[0084] The virtual control ECU 40 is an electronic control unit that performs overall control of the simulator device 3. In addition to a CPU 41 serving as both a computing and control unit, RAM 42 used as working memory for the CPU 41 during various calculations, and a control program, it also contains a motion evaluation program (described later). Figure 17 The virtual control ECU 40 includes an internal storage device such as a ROM 43 and a flash memory 44 storing programs read from the ROM 43. Furthermore, the virtual control ECU 40 and the aforementioned driving assistance ECU 20 share various mechanisms as processing algorithms. For example, a virtual object configuration mechanism configures virtual objects in a real-world scenario. A virtual object detection mechanism detects virtual objects by using virtual sensors 22A-22L, assumed to be positioned at a predetermined location on the vehicle in a predetermined orientation, and by simulating and reproducing the transmission of probe waves from the location along the predetermined orientation towards the vehicle's surroundings, and receiving reflected waves after the probe waves are reflected by virtual objects located around the vehicle. An image visual verification mechanism visually verifies a real-world scenario superimposed with at least one of a probe wave image 10 showing the propagation of probe waves transmitted from virtual sensors 22A-22L, and a reflected wave image 21 showing the propagation of reflected waves when the probe waves are reflected onto virtual objects. In other words, the virtual control ECU 40 is an example of a virtual object configuration mechanism, a virtual object detection mechanism, and an image visual verification mechanism.

[0085] In addition, the virtual control ECU 40 connects to the aforementioned operating unit 14, vehicle display 5 (or VR head-mounted display 8), speaker 15, front camera 16, rear camera 17, side cameras 18A, 18B, actual sensors 9A-9L, and driver assistance ECU 20 via a vehicle network such as CAN. It also connects to various sensors 48 used to detect vehicle behavior, such as vehicle speed sensor, wheel speed sensor, acceleration sensor, gyroscope sensor, steering sensor, and shift position sensor. Based on the detection results of these sensors 48, it can detect the vehicle's current position, orientation, and current behavior. Furthermore, the virtual control ECU 40 can provide the vehicle 2 with a virtual driving environment for driving tests. On the other hand, it can obtain, analyze, evaluate, and output the action results of the autonomous driving assistance system from the vehicle's behavior during driving tests. Moreover, based on the same vehicle behavior during driving tests, it generates a real-scene image 6 of the actual scene displayed on the aforementioned vehicle display 5, VR head-mounted display 8, and external terminal 11. Figure 5 , Figure 6 , Figure 10 , Figure 11 It outputs images to the aforementioned display device.

[0086] Additionally, the flash memory 44 contains driving environment information 45, sensor information 46, and driving results DB 47. The driving environment information 45 stores the driving environment pre-set by the evaluator 4 for the driving test of vehicle 2. Specifically, this refers to the types and locations of virtual objects configured in the actual scene, which are pre-input by the evaluator 4. Furthermore, multiple driving environment modes can also be pre-stored.

[0087] On the other hand, sensor information 46 stores various information related to the actual sensors 9A-9L configured in vehicle 2, such as setting position, setting direction, and sensor performance. The setting position, setting direction, and sensor performance of the virtual sensors are based on the actual sensors 9A-9L. As described later, simulator device 3 generates a probe wave image 10 superimposed on the actual scene based on sensor information 46.

[0088] The driving result DB47 is a storage mechanism that accumulates and stores the vehicle's control content, driving trajectory, and behavioral history during the driving test. Furthermore, after the driving test of vehicle 2 is completed, simulator device 3 evaluates and outputs the control content, driving trajectory, etc. of vehicle 2 stored in driving result DB47.

[0089] Next, based on Figure 17 This section explains the motion evaluation procedure executed in the simulator device 3 that constitutes the vehicle motion simulator system 1 having the above-described structure. Figure 17This is a flowchart of the motion evaluation procedure in this embodiment. Here, the motion evaluation procedure is executed when the corresponding program is started in the simulator device 3, constructing a virtual driving environment and evaluating the quality of the vehicle's automated driving assistance system based on the results of driving tests. Furthermore, the program can be started, for example, by an evaluator 4 operating the operation unit 14 while riding in the vehicle as a passenger, or by an evaluator 4 located outside the vehicle performing a prescribed operation using an external terminal 11. In the following... Figure 17 The program illustrated in the flowchart is stored in RAM 42, ROM 43, etc. of the simulator device 3 and is executed by CPU 41.

[0090] First, in step (hereinafter referred to as S)1, the CPU 41 of the simulator device 3 performs various initial setting processes before executing the driving test. For example, it confirms the operation of the virtual sensors 22A to 22L.

[0091] Next, in S2, CPU 41 obtains the sensor performance (specifications) of the actual sensors 9A to 9L. Furthermore, the sensor performance is pre-stored in flash memory 44. Additionally, the sensor performance of the actual sensors 9A to 9L is equivalent to the sensor performance of the virtual sensor.

[0092] Next, in S3, CPU41 sets the output intensity of the probe wave from the virtual sensor.

[0093] Then, in S4, CPU41 performs acoustic wave analysis simulation of the probe wave output from the virtual sensor based on the sensor performance of the actual sensors 9A to 9L obtained in S2 above and the output intensity of the probe wave set in S3 above. That is, it simulates how the actual probe wave propagates when the probe wave is output with the output intensity of a virtual sensor that has the same performance as the actual sensors 9A to 9L.

[0094] Next, in S5, CPU41, based on the acoustic wave analysis simulation results from S4, determines the placement position and direction (directivity) of the probe wave relative to the virtual sensor, and how it propagates (detection area). Furthermore, the placement position and direction of the virtual sensor are based on the placement position and direction of the actual sensors 9A to 9L actually mounted in vehicle 2. The result is as follows: Figure 18 As shown, the propagation direction (directivity) and propagation range (detection area) of a probe wave sent from a virtual sensor assumed to be set at the same location and orientation as the actual sensors 9A to 9L can be determined.

[0095] Next, in S6, CPU 41 reads from flash memory 44 the setting information of the driving environment for conducting driving tests on vehicle 2. The driving environment includes, for example, the types and positions of virtual objects configured relative to the actual scene.

[0096] Here, the driving environment is basically preset and stored in the simulator device 3 through the input operation of the evaluator 4. Multiple driving environments can be set, allowing any driving environment to be selected for this driving test. Furthermore, the driving environment can be set to either driving on a highway or driving in a parking lot. Additionally, the "virtual objects" are not limited to three-dimensional objects; for example, they include lane markings, road markings, etc. For example, in the case of a parking lot driving test, virtual objects include parking space markings depicted on the road surface within the parking lot, other vehicles parked in parking spaces, pedestrians moving within the parking lot, and other vehicles driving in the passageways within the parking lot. Moreover, a movement schedule (at what speed and how) is set for moving virtual objects.

[0097] Next, in S7, CPU41 constructs a virtual driving environment relative to the actual scene where the vehicle is currently located, based on the setting information read in S6 above. That is, virtual objects of the set type are configured at the set locations in the actual scene. In addition, the moving virtual objects move according to the set timetable.

[0098] Then, in S8, CPU41 begins driving tests of vehicle 2. If a driving test begins, vehicle control executed by autonomous driving assistance is initiated in vehicle 2. This includes continuously detecting the vehicle's current position, the lane it is traveling in, and the positions of surrounding obstacles, and automatically controlling the steering, drive system, and brakes while driving along a generated driving trajectory at a speed planned based on the same generated speed in the actual scenario. Especially when evaluating parking assistance using a parking lot as the driving environment, automatic vehicle control is performed. In this control, sensor detection information is used to confirm the parking space and its surrounding conditions, calculate the parking trajectory for the parking space, and guide the vehicle along the calculated trajectory to enter the parking space, thus completing the parking maneuver.

[0099] During driving performed by the aforementioned autonomous driving assistance, the CPU 41 of the simulator device 3 outputs the detection results of virtual objects detected by the virtual sensors 22A-22L in the actual driving scenario to the driving assistance ECU 20 controlling the autonomous driving assistance on the vehicle 2 side. Specifically, the CPU 41 determines the current position and orientation of the vehicle based on information from various sensors 48 installed in the vehicle, such as the vehicle speed sensor, wheel speed sensor, acceleration sensor, gyroscope sensor, steering sensor, and shift position sensor. Based on the determined current position and orientation of the vehicle and the position of the virtual objects configured in the driving environment constructed in S7, it determines the relative position of the virtual objects relative to the vehicle 2. Furthermore, based on the determined relative position of the virtual objects relative to the vehicle 2 (more specifically, the actual sensors 9A-9L provided by the vehicle), it controls the timing of sending detection waves from the second ultrasonic sensors 26 provided by the virtual sensors 22A-22L, thereby enabling the virtual sensors 22A-22L to detect virtual objects that do not actually exist.

[0100] Next, in S9, the CPU 41 obtains a real-time scene image of the actual scene of the object to be displayed on the in-vehicle display 5, the VR head-mounted display 8, or the external terminal 11, based on the images captured by the front camera 16, the rear camera 17, and the side cameras 18A and 18B. Alternatively, the scene image can be the scene image itself captured by the front camera 16, the rear camera 17, and the side cameras 18A and 18B, or it can be a composite image of multiple cameras or an image transformed by viewpoint (e.g., a bird's-eye view).

[0101] Furthermore, the real-time scene image of the actual object displayed on the external terminal 11 may not be an image captured by the camera of the vehicle 2, but rather an image captured by the camera of the external terminal 11. Alternatively, it may be an image captured by a camera located in the actual scene. The following processing is performed frame by frame of the image acquired in S9 described above.

[0102] Next, in S10, the CPU 41 determines the position of the virtual object in the real-world image of the actual scene obtained in S9 by converting the configuration coordinates (absolute coordinates) of the virtual object in the driving environment constructed in S7 into the coordinate system (relative coordinates) of the real-world image of the actual scene obtained in S9. For example, when displaying the real-world image of the actual scene captured by the front camera 16 on the vehicle display 5, the position of the virtual object in the real-world image of the actual scene displayed on the vehicle display 5 can be determined by converting the configuration coordinates (absolute coordinates) of the virtual object into the coordinate system (relative coordinates) of the front camera 16. On the other hand, when performing VR display through the VR headset 8, the position of the virtual object in the real-world image of the actual scene displayed on the VR headset 8 can be determined by converting the configuration coordinates (absolute coordinates) of the virtual object into the coordinate system (relative coordinates) of VR (i.e., the user's line of sight).

[0103] Then, in S11, the CPU 41 composites the image 7 representing the virtual object at the position determined in S10 with the real-world image of the actual scene obtained in S9. The image 7 representing the virtual object is stored in the flash memory 44 in advance as a type of virtual object, such as a vehicle or a person. Furthermore, the CPU 41 reads the corresponding type of model image from the flash memory 44, matches it with the virtual object positioned in the actual scene, rotates and scales it, and composites it with the real-world image. For virtual objects with orientation, such as vehicles, the orientation is also matched to the constructed driving environment. Additionally, it is preferable to perform texture mapping on the surface of the model image, and the texture image effect is depicted in a way that does not appear unnatural when the model image is positioned in the actual scene, taking into account the orientation of the placement and the surrounding brightness, the position of light sources (e.g., sunlight, streetlights), etc. The composite model image can be set as an opaque image with 0% transmittance or a semi-transparent image.

[0104] Then, in S12, the CPU41, based on the propagation direction (directivity) and propagation range (detection area) of the probe wave determined in S5 above, synthesizes the probe wave image 10, representing the propagation of the probe wave sent from the virtual sensor's position (the actual sensor's position), into the real-scene image of the actual scene obtained in S9 above. Furthermore, as described above, the probe wave image 10 becomes an image showing the propagation of the probe wave from the setting position in a direction along the setting direction within a detection area capable of detecting a virtual object set based on the virtual sensor's setting position. Figure 7Furthermore, as described above, in this embodiment, it is possible to selectively visually confirm any one of the following: a real scene that only overlaps with the probe wave image 10, a real scene that only overlaps with the reflected wave image 21, and a real scene that overlaps with both the probe wave image 10 and the reflected wave image 21. For example, switching can be performed by the evaluator 4 operating the operation unit 14.

[0105] Furthermore, in S12 above, when the reflected wave image 21 is composited onto a real-world image of the actual scene, firstly, based on the propagation direction (directivity) and propagation range (detection area) of the probe wave determined in S5 above, and the position of the virtual object in the actual scene, it is determined within what range and in what direction (directivity) the probe wave (reflected wave) propagates after being reflected in the virtual object. Then, the reflected wave image 21, representing the propagation of the reflected wave, is composited onto the specified range. Thus, the actual scene with the reflected wave image 21 superimposed can be visually confirmed.

[0106] Next, in S13, the CPU 41 synthesizes a real-time scene image of the actual scene, which includes the image 7 representing the virtual object and the probe wave image 10 or the reflected wave image 21, and outputs it to the vehicle display 5, the VR headset display 8, or the external terminal 11, and displays it on each display device. As a result, the evaluator 4 can visually confirm the actual scene, which includes the image 7 representing the virtual object at the location where the virtual object is positioned, and the probe wave image 10 representing the propagation of the probe wave sent from the virtual sensor, or the reflected wave image 21 representing the propagation of the reflected wave. Figure 5 , Figure 6 , Figure 8 , Figure 10 Furthermore, as described above, it is also possible to display a real-world image of the actual scene with an overlay of an image 7 representing the same virtual object, a probe wave image 10, and a reflected wave image 21 on both the interior and exterior rearview mirrors. Figure 11 ).

[0107] Next, in S14, CPU41 determines whether the driving test has been completed. For example, when evaluating parking assistance in a parking lot environment, the driving test is considered complete when the vehicle parks in a parking space and the gear shift is P. Alternatively, the driving test can be completed by using the condition of traveling a specified distance or a specified time since the start of the driving test.

[0108] Furthermore, if the driving test is deemed completed (S14: Yes), the action evaluation procedure ends. Conversely, if the driving test is deemed incomplete (S14: No), the process returns to S8 to continue the driving test of vehicle 2.

[0109] Furthermore, the analysis of the driving results of vehicle 2 during the driving test, performed through the aforementioned action evaluation procedure, is configured to perform both "real-time analysis" and "post-test analysis." "Real-time analysis" monitors vehicle 2 during the driving test and automatically detects and outputs any adverse situations that occur. On the other hand, "post-test analysis" evaluates and outputs the control data and driving trajectory of vehicle 2 stored during the driving test, based on the detection data from virtual sensors 22A-22L, after the driving test of vehicle 2 has concluded.

[0110] Furthermore, CPU41 can also output an evaluation window containing the information about any adverse events that occurred after the driving test to an external display. Additionally, the time at which the adverse event occurred, along with the details of the adverse event, is displayed in the evaluation window. As a result, evaluator 4 can understand at what time and what kind of adverse event occurred in the autonomous driving assistance system. Moreover, CPU41 can be configured to save the real-world image of the actual scene output in S13 (including the image 7 of the virtual object and the probe wave image 10) as an image log, and extract the image output when an adverse event was determined to have occurred during the autonomous driving assistance system's operation from the image log, allowing evaluator 4 to view it. Based on the output driving test results, evaluator 4 can perform program corrections for the autonomous driving assistance system, etc.

[0111] As explained in detail above, in the vehicle motion simulator system 1 and the computer program executed by the vehicle motion simulator system 1 in this embodiment, virtual objects are configured in the actual scene (S7), and the driving results of the vehicle 2 equipped with virtual sensors 22A to 22L for detecting virtual objects are collected when it drives in the actual scene. On the other hand, the virtual sensors 22A to 22L are assumed to be set in a predetermined position on the vehicle according to the actual sensors 9A to 9L in a predetermined setting direction. Virtual objects are detected by simulating the transmission of probe waves from the setting position to the surroundings of the vehicle and receiving the reflected waves after the probe waves are reflected by virtual objects located around the vehicle. The actual scene is visually confirmed by superimposing at least one of the probe wave image 10, which shows the propagation of the probe waves transmitted from the virtual sensors 22A to 22L, and the reflected wave image 21, which shows the propagation of the reflected waves when the probe waves are reflected to the virtual objects (S13). Therefore, in the vehicle driving test with virtual objects configured in the actual scene, the detection range and detection status of virtual objects detected by virtual sensors can be superimposed on the actual scene and can be visually confirmed by passengers and observers in real time. As a result, it will become clear how virtual sensors send probe waves and receive reflected waves to detect the aforementioned virtual objects, and it will also be possible to evaluate whether the vehicle's behavior relative to the detection results of virtual objects is appropriate through human line of sight.

[0112] In addition, it is possible to selectively visually confirm any one of the following: the actual scene with only the probe wave image 10 superimposed, the actual scene with only the reflected wave image 21 superimposed, and the actual scene with both the probe wave image 10 and the reflected wave image 21 superimposed. Therefore, it is possible to identify and visually confirm the probe wave and the reflected wave, and to more clearly visually confirm the detection status of the virtual object by the virtual sensor.

[0113] In addition, in other examples of probe wave image 10 ( Figure 9 In this system, the detection axis of the virtual sensor and the waveform of the detection wave propagating along the detection axis in the direction of the detection wave transmission are visually confirmed. Therefore, the detection range and detection status of the virtual object by the virtual sensor can be superimposed on the actual scene through the displayed detection axis and waveform, enabling passengers and observers to visually confirm in real time.

[0114] In addition, the virtual sensor is set to detect a detection area that can detect virtual objects based on the set position. In the probe wave image 10, the visual confirmation is made of the probe wave propagating from the set position in the detection area along the set direction. Therefore, the detection range and detection status of virtual objects by the virtual sensor can be superimposed on the actual scene and visually confirmed by passengers and observers in real time.

[0115] Furthermore, the present invention is not limited to the above-described embodiments, and various improvements and modifications can be made without departing from the spirit of the present invention.

[0116] For example, in this embodiment, although the image 7 representing the virtual object and the probe wave image 10 are overlaid relative to the real scene image 6 of the actual scene displayed by the vehicle display 5, the VR head-mounted display 8, and the external terminal 11, it is also possible to not display the image representing the virtual object and only overlay the probe wave image 10.

[0117] Furthermore, in this embodiment, although the real-scene image 6 (including the image 7 of the virtual object, the probe wave image 10, and the reflected wave image 21) displayed on the vehicle display 5 and the external terminal 11 is set as a real-time real-scene image, it can also be set as a past real-scene image captured by a camera. In this case, the evaluator 4 can confirm how the virtual sensor sends probe waves to detect the virtual object after the vehicle 2's driving test is completed. In addition, it is not limited to when the vehicle is moving; even when the vehicle is stationary, the real-scene image 6 can be displayed on the vehicle display 5 and the external terminal 11.

[0118] Furthermore, in this embodiment, although the evaluation object for quality evaluation in driving tests is set as an automated driving assistance system that assists passengers in driving the vehicle by performing part or all of the passenger's driving operations on the vehicle side, various vehicle control systems can also be set as evaluation objects in addition to automated driving assistance systems. For example, an automatic braking system for approaching obstacles can be set as an evaluation object.

[0119] Furthermore, although the probe wave image 10 in this embodiment represents the propagation direction (directivity) and propagation range (detection area) of the probe wave transmitted from the virtual sensor, it could also be an image representing only one of them. The same applies to the reflected wave image 21.

[0120] Furthermore, in this embodiment, Figure 17 Although the simulator device 3, mounted on the vehicle, is the main executor of the motion evaluation program shown, the simulator device 3 can also be located outside the vehicle. In this case, the simulator device 3 and the vehicle 2 can communicate wirelessly.

Claims

1. A vehicle motion simulator system, characterized in that, have: A virtual object configuration mechanism that configures virtual objects in a real-world scenario; A virtual object detection mechanism uses a virtual sensor, assumed to be positioned at a predetermined location on a vehicle with a predetermined orientation, to detect virtual objects by simulating the transmission of probe waves from the predetermined location along the predetermined orientation towards the vicinity of the vehicle and receiving reflected waves from virtual objects located around the vehicle. An image visual verification mechanism visually verifies the actual scene by superimposing at least one of a probe wave image showing the propagation of the probe wave transmitted from the virtual sensor and a reflected wave image showing the propagation of the reflected wave when the probe wave is reflected onto the virtual object.

2. The vehicle motion simulator system according to claim 1, characterized in that, The aforementioned image visual verification mechanism enables selective visual verification of any one of the following: the actual scene with only the aforementioned probe wave image superimposed, the actual scene with only the aforementioned reflected wave image superimposed, and the actual scene with both the aforementioned probe wave image and the aforementioned reflected wave image superimposed.

3. The vehicle motion simulator system according to claim 1 or 2, characterized in that, The aforementioned image visual verification mechanism, as the aforementioned probe wave image, enables the detection axis of the aforementioned virtual sensor to be visually verified together with the waveform of the aforementioned probe wave propagating along the aforementioned detection axis in the transmission direction of the aforementioned probe wave.

4. The vehicle motion simulator system according to claim 1 or 2, characterized in that, The aforementioned virtual sensor is configured to detect the aforementioned virtual object within a detection area based on the aforementioned configured position. The aforementioned image visual confirmation mechanism, as the aforementioned probe wave image, visually confirms the situation in which the aforementioned probe wave propagates from the aforementioned setting position in the aforementioned detection area along the aforementioned setting direction.

Images