Reproduction system, reproduction method, and reproduction program

Abstract

To provide a reproduction system, a reproduction method and a reproduction program with which it is possible to reproduce in an easily understandable manner a state at a specific traffic point where a host vehicle and other traffic participants interfere.SOLUTION: The reproduction system comprises: an external detection unit that acquires external information of a host vehicle; a vehicle sensor that acquires vehicle information of the host vehicle; a reproduction video generation unit that generates a video which are reproduced together the movement history of the host vehicle at a specific traffic point and the movement history of other traffic participants that have their courses crossing the host vehicle at the specific traffic point, on the basis of external time-series data that is generated based on the external information and indicates the external state of the host vehicle in time series, and vehicle time-series data that is generated based on the vehicle information and indicates the driving state of the host vehicle; and a display control unit that causes a display device to display the video.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a reproduction system, a reproduction method, and a reproduction program. [Background technology]

[0002] In recent years, efforts to provide sustainable transportation-related technologies that take into consideration vulnerable traffic participants have become more active. To achieve this, we are focusing on research and development into driver assistance technologies to further improve traffic safety and convenience.

[0003] Patent Document 1 discloses an image display system that enables efficient investigation of traffic accident scenes by displaying images taken from various points along the vehicle's travel route at the scene of the traffic accident in association with a floor plan. This system reproduces the changes in the field of view image that the driver would have seen as a video. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-109986 Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional technology, it was sometimes impossible to clearly reproduce the situation at a specific traffic location where the vehicle interfered with another traffic participant.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide a reproduction system, a reproduction method, and a reproduction program that can clearly reproduce the situation at a specific traffic location where one's own vehicle interferes with another traffic participant. [Means for solving the problem]

[0007] (1): In order to solve the above problem, a reproduction system according to one embodiment of the present invention includes an external detection unit that acquires external information about the host vehicle, a vehicle sensor that acquires vehicle information about the host vehicle, external time series data that is generated based on the external information and shows the external situation of the host vehicle in time series, and vehicle time series data that is generated based on the vehicle information and shows the driving situation of the host vehicle in time series, based on the reproduction video generation unit that generates a video that reproduces the movement history of the host vehicle at a specific traffic location and the movement history of other traffic participants whose paths intersect with the host vehicle at the specific traffic location, and a display control unit that displays the video on a display device.

[0008] (2): Furthermore, a reproduction system according to one aspect of the present invention includes an external detection unit that acquires external information about the host vehicle, a vehicle sensor that acquires vehicle information about the host vehicle, and a reproduction video generation unit that generates a video that reproduces the movement history of the host vehicle at a specific traffic location from a perspective outside the host vehicle, based on external time series data that indicates the external situation of the host vehicle in time series, generated based on the external information, and vehicle time series data that indicates the driving situation of the host vehicle in time series, generated based on the vehicle information.

[0009] (3): In the above aspect (1) or (2), the external detection unit may include an image sensor that captures images ahead of the vehicle, the external time series data may include the time-series positions of stop lines and lane markers on the road on which the vehicle has traveled, and the vehicle time series data may be input to the reproduction video generation unit via an in-vehicle communication network of the vehicle.

[0010] (4): In any one of the above aspects (1) to (3), the specific traffic location is an intersection, and the reproduction video generation unit may display the intersection in the video using information about a reference line marked on the road before the vehicle changes course at the intersection and information about a second reference line marked on the road after the vehicle changes course at the intersection.

[0011] (5): In any one of the above aspects (1) to (4), the reproduction video generation unit may use information about the other traffic participants contained in the external time series data to generate abstracted image data corresponding to the other traffic participants, and display the image data as the other traffic participants in the video.

[0012] (6): In one aspect of the present invention, a reproduction method is provided in which a computer of a reproduction system, which includes an external detection unit that acquires external information about the host vehicle and a vehicle sensor that acquires vehicle information about the host vehicle, generates a video that reproduces both the movement history of the host vehicle at a specific traffic location and the movement history of other traffic participants whose paths intersect with the host vehicle at the specific traffic location, based on external time series data that shows the external situation of the host vehicle in time series, generated based on the external information, and vehicle time series data that shows the driving situation of the host vehicle in time series, generated based on the vehicle information, and displays the video on a display device.

[0013] (7): A reproduction program according to one embodiment of the present invention causes a computer of a reproduction system, which includes an external detection unit that acquires external information about the host vehicle and a vehicle sensor that acquires vehicle information about the host vehicle, to generate a video that reproduces both the movement history of the host vehicle at a specific traffic location and the movement history of other traffic participants whose paths intersected with the host vehicle at the specific traffic location, based on external time series data that shows the external situation of the host vehicle in time series, generated based on the external information, and vehicle time series data that shows the driving situation of the host vehicle in time series, generated based on the vehicle information, and then displays the video on a display device.

[0014] (8): In one aspect of the present invention, a reproduction method is provided in which a computer of a reproduction system including an external detection unit that acquires external information about the vehicle and a vehicle sensor that acquires vehicle information about the vehicle generates a video that reproduces the movement history of the vehicle at a specific traffic location from the perspective of another traffic participant whose path intersects with the vehicle at the specific traffic location, based on external time series data that shows the external situation of the vehicle in time series, generated based on the external information, and vehicle time series data that shows the driving situation of the vehicle in time series, generated based on the vehicle information, and displays the video on a display device.

[0015] (9) A reproduction program according to one aspect of the present invention causes a computer of a reproduction system, which includes an external detection unit that acquires external information about the host vehicle and a vehicle sensor that acquires vehicle information about the host vehicle, to generate a video that reproduces the movement history of the host vehicle at a specific traffic location from the perspective of another traffic participant whose path intersects with that of the host vehicle at the specific traffic location, based on external time series data that shows the external situation of the host vehicle in time series, generated based on the external information, and vehicle time series data that shows the driving situation of the host vehicle in time series, generated based on the vehicle information, and to display the video on a display device. [Effects of the Invention]

[0016] According to the above aspects of the present invention, it is possible to provide a reproduction system, a reproduction method, and a reproduction program that can easily reproduce a situation at a specific traffic location where one's own vehicle interferes with another traffic participant. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a reproduction system according to an embodiment of the present invention. [Figure 2] 10A and 10B are diagrams illustrating an example of a scene from a bird's-eye view reproduction video generated by the reproduction system of this embodiment. [Figure 3] 10 is a diagram illustrating an example of a scene from a replay video from the viewpoint of the vehicle, generated by the replay system of the present embodiment. FIG. [Figure 4] 10 is a diagram illustrating an example of a scene from a replay video from the other person's point of view, generated by the replay system of this embodiment. FIG. [Figure 5] 10 is a diagram illustrating an example of a scene from a replay video from the other person's point of view, generated by the replay system of this embodiment. FIG. [Figure 6] 10 is a flowchart illustrating an example of processing executed by the reproduction system of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] The reenactment system, reenactment method, and reenactment program of this embodiment will be described below with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a reenactment system 1 according to this embodiment. The reenactment system 1 is configured to generate, by computer graphics, a video (hereinafter also referred to as a reenactment video) that reenacts the driving history of a vehicle at a location where multiple traffic participants are involved (hereinafter referred to as a specific traffic location). The reenactment system 1 is also configured to display the generated reenactment video.

[0019] The reproduction system 1 includes, for example, an external detection unit 10, a vehicle sensor 20, a display device 30, and a display control device 100. The external detection unit 10, the vehicle sensor 20, the display device 30, and the display control device 100 may be mounted on the vehicle M. These devices and equipment may be connected to each other via a multiplex communication line such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like. Note that the configuration shown in FIG. 1 is merely an example, and part of the configuration may be omitted, or other configuration may be added.

[0020] The host vehicle M may have, for example, two, three, or four wheels, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a secondary battery or a fuel cell.

[0021] The external detection unit 10 is a device for acquiring external information about the host vehicle M. The "external information" includes, for example, information about the shapes and positions (relative positions as seen from the host vehicle M) of other traffic participants, road dividing lines (hereinafter referred to as dividing lines) drawn or affixed to the road surface, road markings such as crosswalks, bicycle crossings, and stop lines, road signs, traffic signals, railroad crossings, curbs, medians, guardrails, fences, etc. The "other traffic participants" include, for example, vehicles (two-wheeled vehicles, four-wheeled vehicles, etc.), pedestrians, etc. In the video example shown in FIG. 2 etc., an oncoming vehicle X is displayed as an example of an "other traffic participant." In addition, in the video example shown in FIG. 2 etc., the host vehicle M is turning right at an intersection, and the oncoming vehicle X is traveling straight. Therefore, the paths of the host vehicle M and the oncoming vehicle X cross each other at the intersection.

[0022] The external detection unit 10 includes, for example, a camera, a LIDAR (Light Detection and Ranging), a radar device, a sensor fusion device, etc. As the camera, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), a stereo camera, etc. The camera as the external detection unit 10 captures an image of the area ahead of the vehicle M and outputs the image data.

[0023] The vehicle sensors 20 include a vehicle speed sensor that detects the speed of the host vehicle M, a gyro sensor that detects angular velocity, an accelerator pedal sensor that detects the amount of accelerator pedal operation or whether or not the accelerator pedal is operated, a brake pedal sensor that detects the amount of brake pedal operation or whether or not the brake pedal is operated, an acceleration sensor that detects acceleration, and a direction sensor that detects the orientation of the host vehicle M. The gyro sensor includes, for example, a yaw rate sensor that detects angular velocity around a vertical axis.

[0024] The display control device 100 includes, for example, a display control unit 110, a recognition unit 120, a reproduction moving image generation unit 130, and a storage unit 140. The functions of the display control unit 110, the recognition unit 120, and the reproduction moving image generation unit 130 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (not shown), or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM, and installed in the storage device by inserting the storage medium into a drive device. Furthermore, some or all of the functions of the display control unit 110, the recognition unit 120, and the reproduced moving image generation unit 130 may be realized by the same hardware.

[0025] The recognition unit 120 recognizes the type, position, and other conditions of objects around the vehicle M based on the external information output by the external detection unit 10. The "objects" include, for example, some or all of moving objects such as vehicles, bicycles, and pedestrians, road boundaries such as road dividing lines, steps, guardrails, road shoulders, and median strips, structures installed on the road such as road signs and billboards, and fallen objects present (fallen) on the road.

[0026] For example, when image data captured by a camera of the external detection unit 10 is input to the recognition unit 120, the recognition unit 120 may acquire the external situation by inputting the image data into a trained model that has been trained to output the results of image recognition. The type of object can also be estimated based on the size of the object in the image data or the intensity of the reflected wave received by the radar device of the external detection unit 10. The recognition unit 120 may acquire the moving speed of other traffic participants detected by the radar device using Doppler shift, for example.

[0027] The position of an object is recognized as a position on a relative coordinate system (i.e., a relative position relative to the vehicle M), with a representative point (such as the center of gravity or the center of the drive shaft of the vehicle M) as the origin. The position of an object includes, for example, the relative positions of other traffic participants, the left road dividing line, the right road dividing line, the left curb, the right curb, a stop line, a traffic signal, etc., as seen from the vehicle M.

[0028] The display control unit 110 generates external time series data that indicates the external situation of the vehicle M in time series based on the recognition result output by the recognition unit 120. Furthermore, the display control unit 110 generates vehicle time series data that indicates the driving situation of the vehicle M in time series based on the vehicle information output by the vehicle sensor 20.

[0029] The external time series data includes, for example, the distance from the vehicle M to the left road dividing line, the distance from the vehicle M to the right road dividing line, the distance from the vehicle M to the left curb, the distance from the vehicle M to the right curb, the distance from the vehicle M to the stop line, the distance from the vehicle M to the traffic light, the type of other traffic participants and their relative positions as seen from the vehicle M, etc., as time series.

[0030] The vehicle time series data includes, for example, yaw rate, vehicle speed, accelerator operation information, brake operation information, light operation information, blinker operation information, wiper operation information, hazard operation information, and the like, in time series.

[0031] The display control unit 110 inputs the external time-series data and the vehicle time-series data to the reproduction moving image generating unit 130. The display control unit 110 also determines whether to cause the reproduction moving image generating unit 130 to generate a reproduction moving image, whether to cause the reproduction moving image to be displayed on the display device 30, and the like. The display control unit 110 also executes control based on the determination result. For example, when the display control unit 110 determines that the reproduction moving image should be displayed on the display device 30, it performs processing to cause the reproduction moving image to be displayed on the display device 30.

[0032] The display device 30 displays the reconstructed video generated by the reconstructed video generating unit 130. When the display device 30 is mounted on the host vehicle M, the display device 30 may display, in addition to the reconstructed video, information images that represent various information to be presented to the driver of the host vehicle M. The display device 30 is, for example, an LCD (Liquid Crystal Display) incorporated in the dashboard or instrument panel of the host vehicle M. It should be noted that the display device 30 does not have to be mounted on the host vehicle M.

[0033] The storage unit 140 stores abstracted image data (hereinafter referred to as abstract data) corresponding to the type of object. For example, when the type of object is a vehicle, abstract data may be prepared for each type of vehicle (two-wheeled vehicle, four-wheeled vehicle, large vehicle, etc.). The storage unit 140 may also store abstract data corresponding to objects other than traffic participants (for example, traffic lights, signs, buildings, etc.).

[0034] The reproduction video generation unit 130 may use the size of the object recognized by the recognition unit 120 as a parameter, adjust the shape of the abstract data using the parameter, and then use the adjusted shape to generate the reproduction video. The reproduction video generation unit 130 may also correct the shape of the object to be displayed in the reproduction video based on the shape of a straight reference line on the road (e.g., a stop line, etc.). Because the reference line is a straight line, for example, by correcting the shape of each object so that the reference line becomes a straight line in a bird's-eye view reproduction video, a reproduction video closer to reality can be generated.

[0035] The reproduction moving image generating unit 130 generates a reproduction moving image at a specific traffic location based on the external time-series data and the vehicle time-series data. The reproduction moving image generated by the reproduction moving image generating unit 130 will be described below.

[0036] The reproduction video generating unit 130 generates a background image of the specific traffic location where the vehicle M has traveled, based on external time-series data. The background image may include, for example, roads and surrounding structures (buildings, guardrails, three-dimensional signs, etc.). Lines marked on the road, such as road dividing lines, may be rendered (drawn) in the video based on geometric data recognized by the recognition unit 120, rather than based on abstract data stored in the memory unit 140. If the specific traffic location is an intersection, the reproduction video generating unit 130 may calculate the intersection angle at which multiple roads intersect, and draw the roads included in the background image based on the calculation result.

[0037] The intersection angle can be calculated, for example, as follows. First, a trajectory of the host vehicle M is calculated, starting from a specific first time point and ending at a second time point after the first time point, based on the yaw rate and vehicle speed of the host vehicle M included in the vehicle time-series data. Furthermore, a relative position of the host vehicle M with respect to the reference line at the first time point is calculated based on information such as a reference line (e.g., a stop line on a road before changing the direction of travel) included in the external time-series data. Furthermore, a relative position of the host vehicle M with respect to the second reference line at the second time point is calculated based on information such as a second reference line (e.g., a stop line on a road after changing the direction of travel) included in the external time-series data. Then, the angle between the reference line and the second reference line can be calculated by using the angle of change in the direction of travel of the host vehicle M from the first time point to the second time point, the angle of the reference line with respect to the host vehicle M at the first time point, and the angle of the second reference line with respect to the host vehicle M at the second time point. In this manner, the intersection angle can be calculated.

[0038] Instead of calculating the intersection angle, the intersection angle may be obtained by referring to pre-stored map information. In this case, the position of the intersection in the map information is identified using location information acquired by a GPS (Global Positioning System) device mounted on the vehicle M, and the intersection angle at the intersection is extracted from the map information. However, calculating the intersection angle may reduce the amount of processing and processing time required for generating a video, compared to using location information acquired by a GPS device.

[0039] The reproduced video generation unit 130 reads out, from the storage unit 140, abstract data corresponding to the type of object recognized by the recognition unit 120. For example, when the recognition unit 120 recognizes the presence of an oncoming vehicle ahead of the host vehicle M, the reproduced video generation unit 130 reads out abstract data corresponding to the type of the oncoming vehicle (two-wheeled vehicle, four-wheeled vehicle, etc.).

[0040] Furthermore, the reproduction video generation unit 130 adjusts the shape of the abstract data displayed in the reproduction video based on the size of the object recognized by the recognition unit 120. For example, if the recognition unit 120 recognizes the width dimension of an oncoming vehicle, the width dimension is used to adjust the width of the abstract data representing the oncoming vehicle. In the scenes of the reproduction video shown in Figures 2, 3, and 5, oncoming vehicle X is displayed using abstract data whose shape has been adjusted.

[0041] Furthermore, the reproduced video generating unit 130 dynamically displays the movement history of the vehicle M at a specific traffic location in a reproduced video based on information such as the yaw rate and vehicle speed included in the vehicle time-series data.

[0042] Furthermore, the reenactment video generation unit 130 dynamically displays the movement history of other traffic participants at the specific traffic location in the reenactment video based on external time-series data. Note that if the other traffic participants are stationary when the host vehicle M passes through the specific traffic location, the positions of the other traffic participants may be statically displayed in the reenactment video. If the reenactment video is from the perspective of the other traffic participants, the appearance of the host vehicle M in the reenactment video changes depending on the positions of the other traffic participants. Therefore, in this case, when displaying the movement history of the host vehicle M, position information of the other traffic participants in the external time-series data is also used. Furthermore, the reenactment video generation unit 130 may display the trajectory L of the host vehicle M (see FIG. 2, etc.) in the reenactment video.

[0043] 2 to 5 are examples of reproduced videos generated by the reproduced video generating unit 130 and displayed on the display device 30. FIG. 2 shows a bird's-eye view video, FIG. 3 shows a video from the view of the own vehicle, and FIGS. 4 and 5 show videos from the view of the other party. A bird's-eye view is a view looking down on a specific traffic point from above. A view of the other party is a view of the own vehicle M from the side of another traffic participant. In this specification, a bird's-eye view and a view of the other party are collectively referred to as "a view from outside the own vehicle M." The "view of the other party" may be a view from the other traffic participant (for example, a view from the driver's seat of the oncoming vehicle X as shown in FIG. 4) or a view from above the other traffic participant (for example, a view from above the oncoming vehicle X as shown in FIG. 5).

[0044] As shown in FIGS. 2 to 5, the reproduced video includes a background image showing a specific traffic location (an intersection in the illustrated example), the movement history of the vehicle M, and the movement history of other traffic participants. The background image may be displayed statically or dynamically. For example, in the bird's-eye view video shown in FIG. 2, the background image is displayed statically. In the video from the vehicle's perspective or the other vehicle's perspective shown in FIGS. 2 to 5, the background image is displayed dynamically in accordance with the movement of the line of sight (i.e., the movement of the vehicle M or the oncoming vehicle X).

[0045] As shown in FIGS. 2 to 5, the image displayed by display device 30 may include a switch display area A1 and a traffic information display area A2. Switch display area A1 includes icons IC1 to IC3 for changing the viewpoint of the video. Icon IC1 is a GUI (Graphical User Interface) switch for selecting a bird's-eye view video. Icon IC2 is a GUI switch for selecting a video from the viewpoint of the vehicle itself. Icon IC3 is a GUI switch for selecting a video from the viewpoint of the other party. Furthermore, the image displayed by display device 30 may include icon IC4, which is a GUI switch for replaying the video. Icon IC4 may be included in switch display area A1.

[0046] When the user operates icons IC1 to IC4, the display control unit 110 executes the corresponding control. For example, as shown in Fig. 2, when a bird's-eye view video is displayed and icon IC2 (own vehicle's eye view) is selected, the display control unit 110 transmits a video output command to the reproduction video generation unit 130 to output a video from the own vehicle's eye view. Receiving this video output command, the reproduction video generation unit 130 displays the bird's-eye view video on the display device 30.

[0047] The reproduced moving image generating unit 130 may generate a moving image every time it receives a moving image output command, and display the generated moving image on the display device 30. Alternatively, the reproduced moving image generating unit 130 may have a storage area for storing the generated moving image, and upon receiving a moving image output command, may call up the moving image stored in the storage area and output it to the display device 30.

[0048] Furthermore, the display control unit 110 may determine whether to display the reenactment video on the display device 30 each time the host vehicle M stops. In this case, the display control unit 110 may display the reenactment video on the display device 30 only when a predetermined condition is met. The "predetermined condition" may be a specific condition that indicates that there is room for improvement in terms of safety, based on the driving situation of the host vehicle M. Specific examples of the "predetermined condition" are listed below. A first specific example is when the vehicle speed exceeds a threshold when the host vehicle M changes direction at an intersection. A second specific example is when the closest distance between the host vehicle M and another traffic participant falls below a threshold. A third specific example is when the distance between the host vehicle M and another traffic participant falls below a threshold, and the relative speed with respect to the other traffic participant exceeds a threshold.

[0049] The traffic information display area A2 displays information about the traffic of the subject vehicle M and other traffic participants (for example, an oncoming vehicle X) at a specific traffic location (for example, an intersection). In the examples of FIGS. 2 to 5, the traffic information display area A2 displays the oncoming vehicle speed (the speed of the oncoming vehicle X), the subject vehicle speed, and the distance between the oncoming vehicle and the subject vehicle M, corresponding to the time series of the reproduced video. Note that the traffic information display area A2 does not have to be displayed in the video. It may be possible to switch whether or not to display the traffic information display area A2.

[0050] 6 is a flowchart showing an example of processing performed by the reproduction system 1. Note that this flowchart is just an example, and processing may be performed according to another flow.

[0051] In the example of Fig. 6, first, the display control unit 110 determines whether the host vehicle M has passed through an intersection (specific traffic location). If it is determined that the host vehicle M has passed through the intersection (step S1: Yes), the process proceeds to step S2. In step S2, the display control unit 110 generates external time series data and vehicle time series data before and after the time when the host vehicle M passed through the intersection. Note that the vehicle time series data and external time series data may be generated continuously while the host vehicle M is traveling.

[0052] In step S3, the display control device 100 (for example, the display control unit 110 or the reproduction moving image generation unit 130) integrates information necessary for generating a moving image. In step S4, the reproduction moving image generation unit 130 generates a reproduction moving image. In step S5, the display control unit 110 determines whether the playback timing has arrived. If it is determined that the playback timing has arrived, the process proceeds to step S6, where the display control unit 110 causes the display device 30 to display the reproduction moving image.

[0053] As described above, the reproduction system 1 of this embodiment includes an external detection unit 10 that acquires external information, a vehicle sensor 20 that acquires vehicle information, a reproduction video generation unit 130 that generates a video that reproduces the movement history of the host vehicle M at a specific traffic location and the movement history of another traffic participant (e.g., oncoming vehicle X) whose path intersects with that of the host vehicle M at the specific traffic location, and a display control unit 110 that displays the video on the display device 30. The reproduction video generation unit 130 generates a video based on external time-series data that indicates the external situation of the host vehicle M in time series and that is generated based on external information, and vehicle time-series data that indicates the driving situation of the host vehicle M in time series and that is generated based on vehicle information.

[0054] In addition, the reproduction video generation unit 130 generates a video that reproduces the movement history of the vehicle M at a specific traffic location from an external perspective of the vehicle M (for example, a bird's eye view or the perspective of another traffic participant) based on the external time series data and the vehicle time series data.

[0055] According to this reproduction system 1, the movement history of the vehicle M at a specific traffic location can be reproduced as a video generated by computer graphics. This makes it possible to reproduce in an easy-to-understand manner the situation at a specific traffic location where the vehicle M interferes with another traffic participant. Furthermore, compared to a video obtained by video recording, for example, the flexibility of displaying the movement history of the vehicle M can be improved, and for example, a video from an objective perspective can be shown to the driver. This makes it possible to effectively encourage safe driving.

[0056] The external detection unit 10 may also include an image sensor that captures images ahead of the vehicle M, and the external time-series data may include the time-series positions of stop lines and lane markings on the road on which the vehicle M has traveled, and the vehicle time-series data may be input to the reproduction video generation unit 130 via an in-vehicle communication network (CAN) of the vehicle M. With this configuration, a reproduction video can be generated using existing hardware resources installed in the vehicle.

[0057] The specific traffic location may also be an intersection. The reproduction video generation unit 130 may display the intersection in the reproduction video using information about a reference line marked on the road before the host vehicle M changes course at the intersection and information about a second reference line marked on the road after the host vehicle M changes course at the intersection. In this case, the intersection can be displayed in the reproduction video without using GPS, map information, etc., and the processing load when generating the reproduction video can be reduced. Furthermore, by reducing the processing load, the reproduction video can be generated in a short time, and the reproduction video can be displayed on the display device 30 in a timely manner, for example, when the host vehicle M stops.

[0058] Furthermore, the reproduction video generator 130 may use information about other traffic participants included in the external time-series data to generate abstracted image data corresponding to the other traffic participants, and display the image data as the other traffic participants in the reproduction video. By using abstracted image data in this way, the processing load can be reduced compared to, for example, displaying live-action data of the other traffic participants in the reproduction video.

[0059] The replay video may be displayed from a bird's-eye view, from the perspective of another traffic participant, or from the perspective of the vehicle itself. Furthermore, the view of the replay video may be selectable. This allows for greater flexibility in video display.

[0060] The above-described embodiment can be expressed as follows. an external detection unit that acquires external information about the vehicle; a vehicle sensor for acquiring vehicle information of the host vehicle; a storage medium storing computer readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: generating external time-series data indicating a time-series state of an external situation of the vehicle based on the external information; generating vehicle time-series data indicating a driving situation of the vehicle in a time series based on the vehicle information; generating a video that reproduces both the movement history of the vehicle at the specific traffic location and the movement history of other traffic participants whose paths intersect with the vehicle at the specific traffic location based on the external time-series data and the vehicle time-series data; A reproduction system that displays the video on a display device.

[0061] The above-described embodiment can be expressed as follows. an external detection unit that acquires external information about the vehicle; a vehicle sensor for acquiring vehicle information of the host vehicle; a storage medium storing computer readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: generating external time-series data indicating a time-series state of an external situation of the vehicle based on the external information; generating vehicle time-series data indicating a driving situation of the vehicle in a time series based on the vehicle information; generating a video that reproduces the movement history of the vehicle at a specific traffic location from the perspective of another traffic participant whose path intersects with the vehicle at the specific traffic location based on the external time-series data and the vehicle time-series data; A reproduction system that displays the video on a display device.

[0062] The technical scope of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

[0063] For example, a part of the reproduction system 1 (e.g., the display device 30, the display control device 100, etc.) may not be mounted on the vehicle M, but may be mounted on a device (a server, a personal computer, a smartphone, a tablet terminal, etc.) that communicates wirelessly with the vehicle M. In other words, the generation and playback of the video may be performed by a device located away from the vehicle M.

[0064] As a specific example, the video generated by the reenactment video generating unit 130 may be stored in a storage unit provided in a server or the like. Then, based on an operation on a device connected to the server, the reenactment video may be displayed on a display device provided in the device. In this case, the driver can watch the video at home or the like and reflect on their own driving operation. Alternatively, the reenactment video may be used for driving lessons at a driving school or the like.

[0065] In the above embodiment, an intersection is given as an example of a specific traffic location. However, the specific traffic location does not have to be an intersection, and may be, for example, a location in a parking lot. Alternatively, the specific traffic location may be a connecting point between a parking lot or the like and a road. In these cases, too, by objectively showing the movement history of the vehicle M to the driver, safer driving can be encouraged.

[0066] In addition, it is possible to replace the components in the above-described embodiments with well-known components as appropriate, and the above-described embodiments and variations may be combined as appropriate, without departing from the spirit of the present invention. [Explanation of symbols]

[0067] 1 Reproduction System 10 External detection unit 20 Vehicle Sensors 30 Display device 110 Display control unit 130 Reproduced video generation unit M Vehicle

Claims

1. an external detection unit that acquires external information about the vehicle; a vehicle sensor for acquiring vehicle information of the host vehicle; a replay video generation unit that generates a video that replays the movement history of the host vehicle at a specific traffic location from the perspective of another traffic participant whose path intersects with that of the host vehicle at the specific traffic location, based on external time-series data that indicates a time-series state of the outside of the host vehicle and that is generated based on the external information, and vehicle time-series data that indicates a time-series state of the driving of the host vehicle and that is generated based on the vehicle information; a display control unit that causes a display device to display the video; A reproduction system comprising:

2. 2. The reenactment system according to claim 1, wherein the reenactment video generation unit generates a video that reenacts the movement history of the other traffic participants together with the movement history of the vehicle at the specific traffic location based on the external time series data and the vehicle time series data.

3. the external detection unit includes an image sensor that captures an image of the area ahead of the host vehicle; the external time-series data includes time-series positions of stop lines and lane markings on a road on which the host vehicle has traveled, The reproduction system according to claim 1 or 2, wherein the vehicle time-series data is input to the reproduction video generating unit through an in-vehicle communication network of the host vehicle.

4. The specific traffic location is an intersection, 3. The reenactment system according to claim 1, wherein the reenactment video generation unit calculates an intersection angle between a road before the vehicle changes course and a road after the vehicle changes course, and uses the result of the calculation to display the intersection in the video.

5. 3. The reenactment system according to claim 1, wherein the reenactment video generation unit generates abstracted image data corresponding to the other traffic participants using information about the other traffic participants contained in the external time-series data, and displays the image data as the other traffic participants in the video.

6. A computer of a reproduction system including an external detection unit that acquires external information of a host vehicle and a vehicle sensor that acquires vehicle information of the host vehicle, generating a video that reproduces the movement history of the vehicle at a specific traffic location from the perspective of another traffic participant whose path intersects with that of the vehicle at the specific traffic location, based on external time-series data that indicates a time-series external situation of the vehicle, the external time-series data being generated based on the external information, and vehicle time-series data that indicates a time-series driving situation of the vehicle, the vehicle information being generated based on the external time-series data; Displaying the video on a display device. How to reproduce.

7. The reproduction method according to claim 6, further comprising generating a video that reproduces the movement history of the other traffic participants together with the movement history of the vehicle at the specific traffic location based on the external time series data and the vehicle time series data.

8. A computer of a reproduction system including an external detection unit that acquires external information of a host vehicle and a vehicle sensor that acquires vehicle information of the host vehicle, generating a video that reproduces the movement history of the vehicle at a specific traffic location from the perspective of another traffic participant whose path intersects with that of the vehicle at the specific traffic location, based on external time-series data that indicates a time-series external situation of the vehicle, the external time-series data being generated based on the external information, and vehicle time-series data that indicates a time-series driving situation of the vehicle, the vehicle information being generated based on the external time-series data; displaying the moving image on a display device; Reproduction program.

9. 9. The reproduction program according to claim 8, which causes a computer of the reproduction system to generate a video that reproduces the movement history of the other traffic participants together with the movement history of the vehicle at the specific traffic location based on the external time series data and the vehicle time series data.

Images