Reproduction system, reproduction method, and storage medium

Abstract

A reproduction system, a reproduction method, and a storage medium capable of easily reproducing a situation at a specific traffic site are provided. The reproduction system includes an external sensing unit that acquires external information of a host vehicle; a vehicle sensor that acquires vehicle information of the host vehicle; a reproduction moving picture generation unit that generates a moving picture that reproduces a movement history of the host vehicle at the specific traffic site, based on external time-series data that represents a situation outside the host vehicle in time series, which is generated based on the external information, and vehicle time-series data that represents a driving situation of the host vehicle in time series, which is generated based on the vehicle information; and a display control unit that causes a display device to display the moving picture.

Description

Technical Field

[0001] This invention relates to a reproduction system, a reproduction method, and a storage medium. Background Technology

[0002] In recent years, efforts to provide sustainable transportation-related technologies that also take into account vulnerable groups among transportation participants have become increasingly active. To achieve this, research and development related to driver assistance technologies are being undertaken to further improve the safety and convenience of transportation.

[0003] Japanese Patent Application Publication No. 2001-109986 discloses an image display system that enables efficient on-site verification of traffic accidents by displaying images taken at various points along the vehicle's path at the scene of the accident in conjunction with a schematic diagram. In this system, changes in the field of vision inferred from the driver's perspective are reproduced as dynamic images. Summary of the Invention

[0004] In the past, it was sometimes impossible to easily reproduce the situation at a specific point in traffic where the vehicle interfered with other traffic participants.

[0005] In addition, with previous technologies, it was difficult for drivers to objectively review their driving history when they stopped their vehicle according to traffic signals, stop signs, or other indicators.

[0006] This invention was developed in consideration of such circumstances, and its purpose is to provide a reproduction system, reproduction method, and storage medium capable of easily reproducing the conditions at a specific traffic location.

[0007] (1) One aspect of the present invention relates to a reproduction system, wherein the reproduction system comprises: an external detection unit that acquires external information of the vehicle; a vehicle sensor that acquires vehicle information of the vehicle; a reproduction dynamic image generation unit that generates a dynamic image reproducing the movement history of the vehicle at a specific traffic location based on external time series data representing the external conditions of the vehicle in a time series generated based on the external information and vehicle time series data representing the driving conditions of the vehicle in a time series generated based on the vehicle information; and a display control unit that causes a display device to display the dynamic image.

[0008] (2): Based on the above (1) scheme, the dynamic image generation unit generates a dynamic image that reproduces the movement history of other traffic participants whose forward road intersects with the forward road of the vehicle at the specific traffic location, together with the movement history of the vehicle at the specific traffic location, based on the external time series data and the vehicle time series data.

[0009] (3): Based on the above (1) scheme, the dynamic image generation unit generates a dynamic image that reproduces the movement history of the vehicle at the specific traffic location from the perspective of other traffic participants whose road path at the specific traffic location intersects with the road path of the vehicle at the specific traffic location, based on the external time series data and the vehicle time series data.

[0010] (4): Based on the above scheme (2) or (3), the external detection unit includes an image sensor that captures images of the front of the vehicle, and the external time series data includes the positions of stop lines and vehicle traffic separation lines in the road through which the vehicle travels in a time series. The vehicle time series data is input to the dynamic image generation unit via the vehicle's in-vehicle communication network.

[0011] (5): Based on any of the above schemes (2) to (4), the specific traffic location is an intersection. The dynamic image generation unit uses information related to the baseline marked on the road before the vehicle changes its forward path at the intersection, and information related to the second baseline marked on the road after the vehicle changes its forward path at the intersection, to make the dynamic image display the intersection.

[0012] (6): Based on any of the above schemes (2) to (5), the dynamic image generation unit uses the information related to the other traffic participants contained in the external time series data to generate abstract image data corresponding to the other traffic participants, and displays the image data as the other traffic participants in the dynamic image.

[0013] (7): Based on the above (1) scheme, the specific traffic location includes a stop indicator set on the road on which the vehicle is traveling to indicate the stopping of the vehicle. The dynamic image generation unit generates a dynamic image that reproduces the movement history of the vehicle from a position closer to the target stopping position to the target stopping position at the specific traffic location based on the external time series data and the vehicle time series data.

[0014] (8): Based on the above (7) scheme, the target stopping position is the position of the stop line in the road in which the vehicle is traveling. The external detection unit includes an image sensor that takes pictures of the front of the vehicle. The external time series data includes the position of the stop line in the time series and the position of the road dividing line in the road in the road in the time series. The vehicle time series data is input to the dynamic display generation unit via the vehicle's in-vehicle communication network. The dynamic display generation unit uses the information related to the stop line and the road dividing line contained in the external time series data to make the dynamic display show the stop line and the road dividing line.

[0015] (9): Based on the above scheme (7) or (8), the stop indicator includes a traffic signal, and the dynamic image generation unit generates a dynamic image that reproduces the movement history of the vehicle when the traffic signal is red.

[0016] (10): Based on any of the above schemes (7) to (9), the reproduction system further includes a chart image generation unit, which generates a chart image representing the change in the vehicle speed at the specific traffic location based on the vehicle time series data, and the display control unit causes the display device to display the chart image.

[0017] (11): Based on any of the above schemes (7) to (10), the dynamic image is a dynamic image of the vehicle when viewed from a direction that intersects with the driving direction of the vehicle.

[0018] (12): Based on any of the above schemes (1) to (11), the reproduction system further includes an internal detection unit that obtains the internal information of the vehicle, and the reproduction dynamic image generation unit generates a dynamic image that reproduces the driver's actions together with the vehicle's movement history at the specific traffic location based on the driver time series data generated based on the internal information, which represents the driver's status of the vehicle in a time series.

[0019] (13) One aspect of the present invention relates to a reproduction method, wherein the reproduction system comprises: an external detection unit that acquires external information of the vehicle; and a vehicle sensor that acquires vehicle information of the vehicle, the reproduction method causing a computer of the reproduction system to perform the following processing: generating a dynamic image reproducing the movement history of the vehicle at a specific traffic location based on external time-series data representing the external conditions of the vehicle in a time series generated based on the external information, and vehicle time-series data representing the driving conditions of the vehicle in a time series generated based on the vehicle information; and displaying the dynamic image on a display device.

[0020] (14): Based on the above (13) scheme, a dynamic picture is generated based on the external time series data and the vehicle time series data, which reproduces the movement history of other traffic participants whose forward road intersects with the forward road of the vehicle at the specific traffic location, together with the movement history of the vehicle at the specific traffic location.

[0021] (15): Based on the above (13) scheme, a dynamic picture is generated based on the external time series data and the vehicle time series data to reproduce the movement history of the vehicle at the specific traffic location in the line of sight of other traffic participants whose forward road intersects with the forward road of the vehicle at the specific traffic location.

[0022] (16): Based on the above (13) scheme, the specific traffic location includes a stop indicator set on the road on which the vehicle travels to indicate the stopping of the vehicle. Based on the external time series data and the vehicle time series data, a dynamic picture is generated to reproduce the movement history of the vehicle from a position closer to the target stopping position to the target stopping position at the specific traffic location.

[0023] (17) One aspect of the present invention relates to a storage medium storing a playback program, wherein the playback system comprises: an external detection unit that acquires external information of the vehicle; and a vehicle sensor that acquires vehicle information of the vehicle, the playback program causing the computer of the playback system to perform the following processing: generating a dynamic image that reproduces the movement history of the vehicle at a specific traffic location based on external time-series data representing the external conditions of the vehicle in a time series generated based on the external information, and vehicle time-series data representing the driving conditions of the vehicle in a time series generated based on the vehicle information; and displaying the dynamic image on a display device.

[0024] (18): Based on the above (17) scheme, the reproduction program causes the computer of the reproduction system to perform the following processing: based on the external time series data and the vehicle time series data, to generate a dynamic picture that reproduces the movement history of other traffic participants whose forward road intersects with the forward road of the vehicle at the specific traffic location, together with the movement history of the vehicle at the specific traffic location.

[0025] (19): Based on the above (17) scheme, the reproduction program causes the computer of the reproduction system to perform the following processing: based on the external time series data and the vehicle time series data, to generate a dynamic picture that reproduces the movement history of the vehicle at the specific traffic location in the line of sight of other traffic participants whose forward road intersects with the forward road of the vehicle at the specific traffic location.

[0026] (20): Based on the above (17) scheme, the specific traffic location includes a stop indicator set on the road on which the vehicle travels to indicate the stopping of the vehicle. The reproduction program causes the computer of the reproduction system to perform the following processing: based on the external time series data and the vehicle time series data, to generate a dynamic picture that reproduces the movement history of the vehicle from a position closer to the target stopping position of the vehicle to the target stopping position at the specific traffic location.

[0027] Invention Effects

[0028] According to the above-described solution of the present invention, a reproduction system, reproduction method, and storage medium capable of easily reproducing the conditions at a specific traffic location can be provided. Attached Figure Description

[0029] Figure 1 This is a block diagram illustrating a structural example of the reproduction system according to the first embodiment.

[0030] Figure 2 This is a diagram illustrating a scene of a bird's-eye view reproduced by the reproduction system of the first embodiment.

[0031] Figure 3 This is a diagram illustrating a scene of a dynamic image of the vehicle's line of sight generated by the reproduction system of the first embodiment.

[0032] Figure 4 This is a diagram illustrating a scene of a dynamic image of the other party's gaze generated by the reproduction system of the first embodiment.

[0033] Figure 5 This is a diagram illustrating a scene of a dynamic image of the other party's gaze generated by the reproduction system of the first embodiment.

[0034] Figure 6 This is a flowchart illustrating an example of the processing performed by the reproduction system of the first embodiment.

[0035] Figure 7 This is a block diagram illustrating the structure of a reproduction system that represents a variation of the first embodiment.

[0036] Figure 8 This is a block diagram illustrating a structural example of the reproduction system according to the second embodiment.

[0037] Figure 9 This is a diagram illustrating a scene of a horizontally reproducing dynamic image generated by the reproduction system of the second embodiment.

[0038] Figure 10 This is a diagram illustrating a scene of a horizontally reproducing dynamic image generated by the reproduction system of the second embodiment.

[0039] Figure 11 This is a diagram illustrating a scene of a recreated dynamic image from a diagonal rear direction generated by the recreated system of the second embodiment.

[0040] Figure 12 This is a graph illustrating the changes in the vehicle speed generated by the reproduction system of the second embodiment.

[0041] Figure 13 This is a flowchart illustrating an example of the processing performed by the reproduction system of the second embodiment. Detailed Implementation

[0042] <First Implementation Method>

[0043] Hereinafter, the reproduction system, reproduction method and storage medium storing the reproduction program of the first embodiment will be described based on the accompanying drawings. Figure 1 This is a block diagram illustrating a structural example of the reproduction system 1 according to this embodiment. The reproduction system 1 is configured to generate, using computer graphics, a dynamic image (hereinafter also referred to as a reproduced dynamic image) that reproduces the driving history of vehicles at a location (hereinafter referred to as a specific traffic location) in which multiple traffic participants participate. Furthermore, the reproduction system 1 is configured to display the generated reproduced dynamic image.

[0044] The reproduction system 1, for example, includes an external detection unit 10, a vehicle sensor 20, a display device 30, and a display control device 100. The external detection unit 10, vehicle sensor 20, display device 30, and display control device 100 can also be mounted on the vehicle M. These devices and equipment can also be interconnected via multiplexed communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. It should be noted that... Figure 1The structure shown is just one example; you can omit part of the structure or add other structures.

[0045] The vehicle M is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, whose power source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or by discharge electricity from a secondary battery or fuel cell.

[0046] The external detection unit 10 comprises various devices used to acquire external information about the vehicle M. "External information" includes, for example, information regarding the shape and position (relative position as observed from the vehicle M) of other traffic participants, road markings (hereinafter referred to as markings) painted or affixed to the road surface, pedestrian crossings, bicycle crossings, temporary stop lines, and other road markings, traffic signals, railway crossings, curbs, median strips, guardrails, fences, etc. "Other traffic participants" include, for example, vehicles (two-wheeled vehicles, four-wheeled vehicles, etc.) and pedestrians. Figure 2 In the example of the dynamic image shown, an oncoming vehicle X is displayed as an example of "other traffic participants." That is, "external information" includes information related to oncoming vehicle X. Additionally, in Figure 2 In the example of the dynamic scene shown, vehicle M turns right at an intersection while oncoming vehicle X goes straight. Therefore, the path of vehicle M intersects with the path of oncoming vehicle X at the intersection.

[0047] The external detection unit 10 includes, for example, a camera, a LIDAR (Light Detection and Ranging) device, a radar device, and a sensor fusion device. As a camera, for example, a digital camera or stereo camera utilizing a solid-state imaging element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) can be used. The camera of the external detection unit 10 captures images of the front of the vehicle M and outputs them as image data.

[0048] The vehicle sensors 20 include a vehicle speed sensor for detecting the speed of the vehicle M, a gyroscope sensor for detecting angular velocity, a steering sensor for detecting the amount of steering (e.g., steering angle) of the vehicle M via the steering wheel, an accelerator pedal sensor for detecting the amount of operation or presence / absence of operation of the accelerator pedal, a brake pedal sensor for detecting the amount of operation or presence / absence of operation of the brake pedal, an acceleration sensor for detecting acceleration, and an orientation sensor for detecting the orientation of the vehicle M. The gyroscope sensors include, for example, a yaw rate sensor for detecting angular velocity about a vertical axis.

[0049] The display control device 100 includes, for example, a display control unit 110, a recognition unit 120, a motion picture generation unit 130, and a storage unit 140. The functions of the display control unit 110, the recognition unit 120, and the motion picture generation unit 130 are implemented, for example, by executing a program (software) using a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can be implemented by hardware (including the circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or through the coordinated operation of software and hardware. The program can be pre-saved in a storage device (not shown) or stored in a removable storage medium (non-temporary storage medium) such as a DVD or CD-ROM, and installed in the storage device by mounting the storage medium onto a drive device. Furthermore, some or all of the functions of the display control unit 110, the recognition unit 120, and the motion picture generation unit 130 can be implemented by the same hardware.

[0050] The identification unit 120 identifies the types and locations of objects surrounding the vehicle M based on external information output by the external detection unit 10. "Objects" include, for example, moving objects such as vehicles, bicycles, and pedestrians; road markings, steps, guardrails, shoulders, and median strips; road signs, notice boards, and other structures placed on the road; and some or all of objects that have fallen onto the road.

[0051] For example, when image data captured by the camera of the external detection unit 10 is input to the recognition unit 120, the recognition unit 120 can also input the image data into a learned model that has been trained to output image recognition results, thereby obtaining the external situation. The type of object can also be estimated based on the size of the object in the image data, the intensity of the reflected wave received by the radar device of the external detection unit 10, etc. The recognition unit 120 can also obtain, for example, the moving speed of other traffic participants detected by the radar device using Doppler frequency shift, etc.

[0052] The position of an object is identified, for example, as its position on a relative coordinate system with a representative point (the center of gravity of vehicle M, the center of the drive shaft, etc.) as the origin (i.e., its relative position relative to vehicle M). The position of an object includes, for example, the relative positions of other traffic participants, the left and right lane markings, the left and right curbs, the stop lines, traffic signals, etc., as viewed from vehicle M.

[0053] The display control unit 110 generates external time-series data representing the external conditions of the vehicle M in a time series based on the recognition result output by the recognition unit 120. Additionally, the display control unit 110 generates vehicle time-series data representing the driving conditions of the vehicle M in a time series based on the vehicle information output by the vehicle sensor 20.

[0054] The external time series data includes, as a time series, the following: the distance from vehicle M to the left side of the road dividing line, the distance from vehicle M to the right side of the road dividing line, the distance from vehicle M to the left side of the curb, the distance from vehicle M to the right side of the curb, the distance from vehicle M to the stop line, the distance from vehicle M to the traffic signal, the types of other traffic participants, and their relative positions as observed from vehicle M.

[0055] Vehicle time series data includes, as a time series, yaw rate, vehicle speed, steering wheel operation information, acceleration operation information, braking operation information, light operation information, direction indicator operation information, windshield wiper operation information, hazard warning operation information, etc.

[0056] The display control unit 110 inputs external time-series data and vehicle time-series data into the replay motion image generation unit 130. Furthermore, the display control unit 110 determines whether the replay motion image generation unit 130 should generate a replay motion image, and whether the display device 30 should display the replay motion image. Additionally, the display control unit 110 performs control based on the determination results. For example, if the display control unit 110 determines that it intends to display the replay motion image on the display device 30, it performs processing to cause the display device 30 to display the replay motion image.

[0057] Display device 30 displays the reproduced dynamic images generated by dynamic image generation unit 130. When display device 30 is mounted on the vehicle M, in addition to reproducing dynamic images, display device 30 can also display information images indicating various information prompts to the driver of vehicle M. Display device 30 may be, for example, mounted on the front bulkhead of vehicle M or an LCD (Liquid Crystal Display) in the instrument panel. It should be noted that display device 30 may also not be mounted on vehicle M.

[0058] The storage unit 140 stores abstracted image data (hereinafter referred to as abstract data) corresponding to the types of objects. For example, if the type of object is a vehicle, the abstract data can also be prepared according to the vehicle category (two-wheeled vehicle, four-wheeled vehicle, large vehicle, etc.). In addition, the storage unit 140 can also store abstract data corresponding to objects other than traffic participants (such as traffic lights, signs, buildings, etc.).

[0059] The motion picture generation unit 130 can also use the size of the object recognized by the recognition unit 120 as a parameter, and use this parameter to adjust the shape of the abstract data before generating the motion picture. Alternatively, the motion picture generation unit 130 can also correct the shape of the objects displayed in the motion picture based on the shape of a baseline (such as a stop line) on a road. Since the baseline is a straight line, in a motion picture from a bird's-eye view, for example, by correcting the shape of each object so that the baseline is a straight line, a more realistic motion picture can be generated.

[0060] The motion picture generation unit 130 generates motion pictures of specific traffic locations based on external time series data and vehicle time series data. The motion pictures generated by the motion picture generation unit 130 will be described below.

[0061] The dynamic image generation unit 130 generates a background image of the specific traffic location where the vehicle M is traveling, based on external time-series data. The background image may include, for example, roads and surrounding structures (buildings, guardrails, three-dimensional signs, etc.). Lines marked on the road, such as road markings, may be drawn in the dynamic image based on geometric data recognized by the recognition unit 120, rather than on abstract data stored in the storage unit 140. When the specific traffic location is an intersection, the dynamic image generation unit 130 may also calculate the intersection angles of multiple road intersections and draw the roads included in the background image based on the calculation results.

[0062] The calculation of the intersection angle can be performed as follows: First, based on the yaw rate and speed of the vehicle M contained in the vehicle time series data, the trajectory of the vehicle M is calculated, with a specific first moment as the starting point and a second moment after the first moment as the ending point. Then, based on information such as a baseline (e.g., a stop line on the road before a change of direction) contained in the external time series data, the relative position of the vehicle M at the first moment relative to the baseline is calculated. Finally, based on information such as a second baseline (e.g., a stop line on the road after a change of direction) contained in the external time series data, the relative position of the vehicle M at the second moment relative to the second baseline is calculated. Then, by using the angle of change of the vehicle M's direction of travel from the first moment to the second moment, the angle of the baseline relative to the vehicle M at the first moment, and the angle of the second baseline relative to the vehicle M at the second moment, the angle between the baseline and the second baseline can be calculated. In this way, the intersection angle can be calculated.

[0063] It should be noted that the intersection angle can also be obtained by referring to pre-stored map information instead of calculating it. In this case, the location of the intersection in the map information is determined using the location information obtained by the GPS (Global Positioning System) device installed in the vehicle M, and the intersection angle at the determined intersection is extracted from the map information. However, compared with the case of using the location information obtained by the GPS device, calculating the intersection angle can sometimes reduce the amount of processing and processing time involved in dynamic image generation.

[0064] The motion picture generation unit 130 reads abstract data corresponding to the type of object identified by the recognition unit 120 from the storage unit 140. For example, when the recognition unit 120 detects the presence of an oncoming vehicle in front of the vehicle M, it reads abstract data corresponding to the type of the oncoming vehicle (two-wheeled vehicle, four-wheeled vehicle, etc.).

[0065] Furthermore, the motion picture generation unit 130 adjusts the shape of the abstract data displayed on the motion picture based on the size of the object recognized by the recognition unit 120. For example, if the recognition unit 120 recognizes the lateral width of an oncoming vehicle, it uses that lateral width to adjust the lateral width of the abstract data representing the oncoming vehicle. Figure 2 , Figure 3 , Figure 5 In the scene depicting a recreated dynamic image, abstract data with adjusted shapes is used to display the oncoming car X.

[0066] In addition, the dynamic image generation unit 130 dynamically displays the movement history of the vehicle M at a specific traffic location in the dynamic image based on information such as yaw rate and vehicle speed contained in the vehicle time series data.

[0067] Furthermore, the motion picture generation unit 130 dynamically displays the movement history of other traffic participants at a specific traffic location in the reproduced motion picture based on external time series data. It should be noted that even if other traffic participants are stationary when the vehicle M passes through a specific traffic location, their positions can also be statically displayed in the reproduced motion picture. When the reproduced motion picture is a view from the perspective of other traffic participants, the way the vehicle M is displayed in the reproduced motion picture changes according to the positions of the other traffic participants. Therefore, in this case, the position information of other traffic participants from the external time series data is also used when displaying the movement history of the vehicle M. Additionally, the motion picture generation unit 130 can also display the trajectory L of the vehicle M in the reproduced motion picture (see reference). Figure 2 wait).

[0068] Figures 2-5 This is an example of a reproduced dynamic image generated by the reproduced dynamic image generation unit 130 and displayed on the display device 30. Figure 2 This represents a dynamic scene from a bird's-eye view. Figure 3 This indicates the dynamic view as seen from the perspective of this vehicle. Figure 4 , Figure 5 These refer to the dynamic view from the perspective of another party. Bird's-eye view refers to the view of a specific traffic location from above. The other party's view is the view of this vehicle M from the perspective of other traffic participants. In this specification, bird's-eye view and the other party's view are collectively referred to as "views from outside this vehicle M." "The other party's view" can be the view from other traffic participants (e.g., Figure 4 The line of sight from the driver's seat of the oncoming vehicle X (as shown) can also be a line of sight from above other road users (e.g., the line of sight from the driver's seat of the oncoming vehicle X), or a line of sight from above other road users (e.g., the line of sight from the driver's seat of the oncoming vehicle X). Figure 5 (The line of sight from above the oncoming vehicle X, as shown).

[0069] like Figures 2-5 As shown, the reproduced dynamic scene includes a background image representing a specific traffic location (an intersection in the example), the movement history of the vehicle M, and the movement history of other traffic participants. The background image can be displayed statically or dynamically. For example, in Figure 2 In the dynamic footage showing the bird's line of sight, the background is displayed statically. Figures 3-5 In the dynamic image showing the line of sight of this vehicle or the other vehicle, the background image is dynamically displayed in conjunction with the movement of the line of sight (i.e., the movement of this vehicle M or the oncoming vehicle X).

[0070] like Figures 2-5 As shown, the image displayed on the display device 30 may also include a switch display area A1 and a traffic information display area A2. The switch display area A1 includes icons IC1 to IC3 for changing the viewpoint of the dynamic image. Icon IC1 is a GUI (Graphical User Interface) switch for selecting the bird's-eye view dynamic image. Icon IC2 is a GUI switch for selecting the vehicle's own view dynamic image. Icon IC3 is a GUI switch for selecting the oncoming view dynamic image. Additionally, the image displayed on the display device 30 may also include an icon IC4, which serves as a GUI switch for playing dynamic images. Icon IC4 may also be included in the switch display area A1.

[0071] When the user performs an operation on icons IC1 to IC4, the display control unit 110 executes the control corresponding to each icon. For example, Figure 2As shown, when displaying a bird's-eye view animation, and icon IC2 (vehicle's view) is selected, the display control unit 110 sends an animation output command to the playback animation generation unit 130 to output the animation of the vehicle's view. After receiving the animation output command, the playback animation generation unit 130 causes the display device 30 to display the animation of the vehicle's view.

[0072] It should be noted that the motion picture generation unit 130 may also generate motion pictures whenever a motion picture output command is received, and display the generated motion picture on the display device 30. Alternatively, the motion picture generation unit 130 may have a storage area for storing the generated motion pictures, and when a motion picture output command is received, it may retrieve the motion picture stored in the storage area and output it to the display device 30.

[0073] Additionally, the display control unit 110 can also determine whether to display the dynamic image on the display device 30 whenever the vehicle M stops. In this case, the display control unit 110 can also display the dynamic image on the display device 30 only if predetermined conditions are met. The "prescribed conditions" can be specifically set based on the driving conditions of the vehicle M, considering conditions where there is room for improvement in terms of safety. Specific examples of the "prescribed conditions" are given below. The first specific example is when the vehicle M changes direction at an intersection and its speed exceeds a threshold. The second specific example is when the closest distance between the vehicle M and other traffic participants is less than a threshold. The third specific example is when the distance between the vehicle M and other traffic participants is less than a threshold, and the relative speed between the vehicle M and other traffic participants exceeds a threshold.

[0074] Traffic information related to vehicle M and other traffic participants (e.g., oncoming vehicle X) at a specific traffic location (e.g., an intersection) is displayed in traffic information display area A2. Figures 2-5 In the example, the traffic information display area A2 displays the oncoming vehicle speed (speed of oncoming vehicle X), the vehicle's speed, and the distance between the oncoming vehicle and the vehicle M, corresponding to the time sequence of the reproduced dynamic image. It should be noted that the traffic information display area A2 may not be displayed in the dynamic image. It is also possible to toggle whether the traffic information display area A2 is displayed.

[0075] Figure 6 This is a flowchart illustrating an example of the processing performed by Reproduction System 1. It should be noted that this flowchart is an example, and other processes can also be followed.

[0076] exist Figure 6In the example, firstly, the display control unit 110 determines whether the vehicle M has passed through an intersection (specific traffic location). If it is determined that the vehicle M has passed through the intersection (step S1: Yes), the process proceeds to step S2. In step S2, the display control unit 110 generates external time-series data and vehicle time-series data before and after the time of passing through the intersection. It should be noted that the vehicle time-series data and external time-series data can also be generated continuously while the vehicle M is in motion.

[0077] In step S3, the display control device 100 (e.g., display control unit 110 or motion picture generation unit 130) synthesizes the information required for motion picture generation. In step S4, the motion picture generation unit 130 generates the motion picture. In step S5, the display control unit 110 determines whether the playback timing has arrived. If the playback timing is determined to have arrived, the process proceeds to step S6, where the display control unit 110 causes the display device 30 to display the motion picture.

[0078] As explained above, the reproduction system 1 of this embodiment includes: an external detection unit 10 that acquires external information; a vehicle sensor 20 that acquires vehicle information of the vehicle M; a reproduction dynamic image generation unit 130 that generates a dynamic image that reproduces the movement history of the vehicle M at a specific traffic location; and a display control unit 110 that causes the display device 30 to display the dynamic image. The reproduction dynamic image generation unit 130 generates the dynamic image based on external time-series data representing the external conditions of the vehicle M in a time series, generated based on external information, and vehicle time-series data representing the driving conditions of the vehicle M in a time series, generated based on vehicle information.

[0079] According to this reproduction system 1, the movement history of the vehicle M at a specific traffic location can be reproduced using a computer-generated dynamic image. This allows for an easily understandable reproduction of the situation at that specific traffic location. Furthermore, compared to dynamic images obtained through video recording, the display flexibility of the vehicle M's movement history is improved, allowing the driver to view a dynamic image from, for example, an objective line of sight. This effectively promotes safe driving.

[0080] In addition, the dynamic image generation unit 130 generates a dynamic image based on external time series data and vehicle time series data, which reproduces the movement history of the vehicle M at a specific traffic location and the movement history of other traffic participants (e.g., oncoming vehicles X) whose roadway intersects with the roadway of the vehicle M at the specific traffic location.

[0081] In addition, the dynamic image generation unit 130 generates a dynamic image based on external time series data and vehicle time series data, which reproduces the movement history of the vehicle M at a specific traffic location from the external viewpoint of the vehicle M (e.g., bird's-eye view or viewpoint from other traffic participants).

[0082] Based on such a reproduction system 1, the situation at a specific traffic location where the vehicle M interferes with other traffic participants can be easily reproduced.

[0083] In addition, the external detection unit 10 includes an image sensor that captures images of the front of the vehicle M. The external time-series data includes the positions of stop lines and traffic lane markings along the road traveled by the vehicle M in a time-series manner. The vehicle time-series data can also be input to the motion picture generation unit 130 via the vehicle communication network (CAN) of the vehicle M. According to this structure, existing hardware resources mounted on the vehicle can be used to generate motion pictures.

[0084] Alternatively, a specific traffic location can be an intersection. Furthermore, the dynamic image generation unit 130 can use information related to the baseline marked on the road before the vehicle M changes direction at the intersection, and information related to the second baseline marked on the road after the vehicle M changes direction at the intersection, to display the intersection in the dynamic image. In this case, the intersection can be displayed in the dynamic image even without using GPS, map information, etc., reducing the processing load when generating the dynamic image. Furthermore, by reducing the processing load, the dynamic image can be generated in a short time, allowing the display device 30 to display the dynamic image in a timely manner, for example, when the vehicle M stops.

[0085] Alternatively, the motion picture generation unit 130 can use information related to other traffic participants contained in external time series data to generate abstracted image data corresponding to other traffic participants, and display the image data as other traffic participants in the motion picture. In this way, by using abstracted image data, the processing load can be reduced compared to, for example, displaying actual captured data of other traffic participants in the motion picture.

[0086] Furthermore, the displayed dynamic images can also be from the perspective of a bird, other road users, or the vehicle itself. Moreover, the perspective from which the dynamic images are displayed can be selected. This further enhances the flexibility of dynamic image display.

[0087] Figure 7 This is a block diagram illustrating a structural example of the reproduction system 1 according to a variation of the first embodiment. For example... Figure 7As shown, in this modified example, the reproduction system 1 further includes an internal detection unit 40 for acquiring internal information of the vehicle M. "Internal information" includes, for example, information related to the driver of the vehicle M. The internal detection unit 40 includes, for example, an in-vehicle camera. The in-vehicle camera is, for example, a digital camera utilizing a solid-state imaging element such as a CCD or CMOS. The in-vehicle camera is mounted at a position capable of photographing the driver of the vehicle M. The in-vehicle camera, serving as the internal detection unit 40, photographs the driver of the vehicle M and outputs the image data.

[0088] The recognition unit 120 identifies the driver's condition in the vehicle M based on the internal information output by the internal detection unit 40. For example, the recognition unit 120 analyzes the images captured by the in-vehicle camera of the internal detection unit 40 and identifies the driver's facial orientation and the direction of the driver's gaze based on the analysis results.

[0089] Based on the recognition result output by the recognition unit 120, the display control unit 110 generates driver time-series data that represents the driver's condition of the vehicle M in a time sequence. The display control unit 110 inputs the driver time-series data to the dynamic image generation unit 130.

[0090] The dynamic image generation unit 130 generates a dynamic image of a specific traffic location based on external time-series data, vehicle time-series data, and driver time-series data. Specifically, in addition to the background image of the specific traffic location, the dynamic image representing the movement history of the vehicle M, and the dynamic image representing the movement history of other traffic participants, the dynamic image generation unit 130 also generates a dynamic image representing the actions of the driver of the vehicle M based on the driver time-series data. That is, the dynamic image generation unit 130 dynamically displays the actions of the driver of the vehicle M in the dynamic image. For example, the dynamic image generation unit 130 generates a dynamic image representing the direction of the driver's head based on the orientation of the driver's face as identified by the recognition unit 120. In this case, it is easy to review the driver's driving operations. It should be noted that the dynamic image generation unit 130 can also dynamically display the driver's field of vision in the dynamic image based on the direction of the driver's gaze as identified by the recognition unit 120. The driver's field of vision can also be displayed as a semi-transparent cone or the like. In this situation, it is easy to review the driver's line of sight while driving.

[0091] Furthermore, in other variations of the first embodiment, the motion picture generation unit 130 may also generate motion pictures representing the actions of the driver of the vehicle M based on vehicle time-series data. For example, the motion picture generation unit 130 may dynamically display the driver's hand and foot movements in the motion picture based on steering wheel operation information, acceleration operation information, braking operation information, light operation information, and direction indicator operation information included in the vehicle time-series data. In this case, it is possible to easily review the driver's driving operations.

[0092] Furthermore, in other variations of the first embodiment, the motion picture generation unit 130 may also display the movement history of other traffic participants (e.g., oncoming vehicle X) in the motion picture by highlighting them in red or other bright spots. In this case, changes in the status of other traffic participants, such as the deceleration of oncoming vehicle X caused by braking, can be easily identified from the front and side of the vehicle M.

[0093] <Second Implementation Method>

[0094] Hereinafter, the reproduction system, reproduction method and storage medium storing the reproduction program of the second embodiment will be described based on the accompanying drawings. Figure 8 This is a block diagram illustrating a structural example of the reproduction system 1A according to the embodiment. The reproduction system 1A is configured to generate, using computer graphics, a dynamic image (hereinafter also called a reproduced dynamic image) showing the driving history of vehicle M at the periphery of its parking position (hereinafter referred to as a specific traffic area) when vehicle M stops according to indicators (hereinafter referred to as stop indicators) such as traffic signals and temporary stop signs. Furthermore, the reproduction system 1A is configured to display the generated reproduced dynamic image.

[0095] The reproduction system 1A, for example, includes an external detection unit 10, a vehicle sensor 20, a display device 30, and a display control device 200. The external detection unit 10, vehicle sensor 20, display device 30, and display control device 200 can also be mounted on the vehicle M. These devices and equipment can also be interconnected via multiplexed communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. It should be noted that... Figure 8 The structure shown is only one example; some parts of the structure may be omitted, and other structures may be added. It should be noted that the basic structure of the external detection unit 10, vehicle sensor 20, and display device 30 in this embodiment is the same as that in the first embodiment, so it is omitted here.

[0096] The external detection unit 10 comprises various devices used to acquire external information about the vehicle M. "External information" includes, for example, information regarding the shape and position (relative position as viewed from the vehicle M) of road markings such as road lines, pedestrian crossings, bicycle lanes, stop lines, traffic signals, temporary stop signs, railway crossings, curbs, median strips, guardrails, and fences. The external detection unit 10 may include, for example, cameras, LIDAR (Light Detection and Ranging) devices, radar devices, and sensor fusion devices.

[0097] The display control device 200 includes, for example, a display control unit 210, a recognition unit 220, a dynamic image generation unit 230, a chart image generation unit 231, and a storage unit 240. The functions of the display control unit 210, the recognition unit 220, the dynamic image generation unit 230, and the chart image generation unit 231 are implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components can be implemented by hardware (including a circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or through cooperation between software and hardware. The program can be pre-saved in a storage device (not shown) or stored in a removable storage medium (non-temporary storage medium) such as a DVD or CD-ROM, and installed in the storage device by mounting the storage medium onto a drive device. Furthermore, some or all of the functions of the display control unit 210, the recognition unit 220, the dynamic image generation unit 230, and the chart image generation unit 231 can also be implemented by the same hardware.

[0098] The identification unit 220 identifies the types and locations of objects surrounding the vehicle M based on external information output by the external detection unit 10. "Objects" include, for example, parts or all of road markings such as road lines, pedestrian crossings, bicycle lanes, stop lines, traffic signals, temporary stop signs, railway crossings, curbs, median strips, guardrails, and fences. The types of objects include the color and shape of road lines (solid lines, dashed lines) and the color of traffic signals (including arrow signals).

[0099] For example, when image data captured by the camera of the external detection unit 10 is input into the recognition unit 220, the recognition unit 220 can also input the image data into a learned model that has been learned in the manner of outputting image recognition results, thereby obtaining the external situation. The type of object can also be estimated based on the size of the object in the image data, the intensity of the reflected wave received by the radar device of the external detection unit 10, etc.

[0100] The position of an object is identified, for example, as its position on a relative coordinate system with a representative point (the center of gravity of vehicle M, the center of the drive shaft, etc.) as the origin (i.e., its relative position relative to vehicle M). The position of an object includes, for example, the relative positions of road markings on the left and right, curbs on the left and right, stop lines, traffic signals, temporary stop signs, etc., as viewed from vehicle M.

[0101] The display control unit 210 generates external time-series data representing the external conditions of the vehicle M in a time series based on the recognition result output by the recognition unit 220. Additionally, the display control unit 210 generates vehicle time-series data representing the driving conditions of the vehicle M in a time series based on the vehicle information output by the vehicle sensor 20.

[0102] The external time series data includes, as a time series, the following: distance from vehicle M to the left side of the road dividing line, distance from vehicle M to the right side of the road dividing line, distance from vehicle M to the left side of the curb, distance from vehicle M to the right side of the curb, distance from vehicle M to the stop line, distance from vehicle M to the traffic signal, distance from vehicle M to the temporary stop sign, color of the road dividing line, shape of the road dividing line, and color of the traffic signal (including arrow signals).

[0103] Vehicle time-series data includes, as a time series, information such as vehicle speed, steering wheel operation, acceleration operation, braking operation, light operation, direction indicator operation, windshield wiper operation, and hazard warning operation.

[0104] The display control unit 210 inputs external time-series data and vehicle time-series data to the replay motion image generation unit 230. Based on the external time-series data and vehicle time-series data, the display control unit 210 causes the replay motion image generation unit 230 to generate a replay motion image. The display control unit 210 then causes the display device 30 to display the replay motion image. Furthermore, the display control unit 210 determines whether to cause the replay motion image generation unit 230 to generate a replay motion image and whether to cause the display device 30 to display the replay motion image. Additionally, the display control unit 210 performs control based on the determination results.

[0105] The display control unit 210 inputs vehicle time-series data to the chart image generation unit 231. Based on the vehicle time-series data, the display control unit 210 causes the chart image generation unit 231 to generate a chart image. The display control unit 210 then causes the display device 30 to display the chart image. Furthermore, the display control unit 210 determines whether to cause the chart image generation unit 231 to generate a chart image and whether to cause the display device 30 to display the chart image. Additionally, the display control unit 210 performs control based on the determination results.

[0106] The display device 30 displays the reproduced dynamic images generated by the dynamic image generation unit 230 and the chart images generated by the chart image generation unit 231. When the display device 30 is mounted on the vehicle M, in addition to reproducing dynamic images and chart images, the display device 30 can also display information images that indicate various information prompts to the driver of the vehicle M.

[0107] The storage unit 240 stores abstracted image data (hereinafter referred to as abstract data) corresponding to the types of objects. For example, the storage unit 240 stores abstract data corresponding to traffic signals, temporary stop signs, and other road signs. In the case where the type of object is, for example, a traffic signal, the abstract data can also be prepared according to each type of traffic signal (shape, color, etc.).

[0108] The motion picture generation unit 230 can also use the size of the object recognized by the recognition unit 220 as a parameter, and adjust the shape of the abstract data using this parameter before generating the motion picture. Alternatively, the motion picture generation unit 230 can also correct the shape of the objects displayed in the motion picture based on the shape of a baseline (e.g., a stop line) on a road. Since the baseline is a straight line, the shape of each object can be corrected in a motion picture from a horizontal direction, for example, by making the baseline a straight line, thereby generating a motion picture that is closer to reality.

[0109] The motion picture generation unit 230 generates motion pictures of specific traffic locations based on external time series data and vehicle time series data. The motion pictures generated by the motion picture generation unit 230 will be described below.

[0110] It should be noted that specific traffic locations include stop indicators such as red traffic lights and temporary stop signs that instruct vehicle M to stop. Specific traffic locations include the area surrounding the parking position of vehicle M when it stops according to the stop indicator. The area surrounding the parking position of vehicle M includes the range from the position in front of it to the target stopping position, where "in front of it" refers to a position a predetermined distance (e.g., a few meters) ahead of the target stopping position of vehicle M. The area surrounding the parking position of vehicle M may include, for example, the range from a position where it is presumed the driver of vehicle M can recognize the stop indicator (hereinafter also referred to as the target recognition position) to the target stopping position of vehicle M. The target stopping position is, for example, the position of the stop line on the road in which vehicle M is traveling.

[0111] The dynamic image generation unit 230 generates background images of specific traffic locations traversed by the vehicle M based on external time-series data. These background images may include, for example, road markings such as road lines, pedestrian crossings, bicycle lanes, and stop lines; traffic signals; temporary stop signs; railway crossings; curbs; median strips; guardrails; and fences. Regarding road lines and other lines marked on the road, they may be drawn (drawn) in the dynamic image based on geometric data recognized by the recognition unit 220, rather than on abstract data stored in the storage unit 240.

[0112] The motion picture generation unit 230 reads abstract data corresponding to the type of object identified by the recognition unit 220 from the storage unit 240. For example, if the recognition unit 220 identifies a traffic signal, it reads abstract data corresponding to the type (shape, color, etc.) of that traffic signal. Furthermore, the motion picture generation unit 230 adjusts the shape of the abstract data displayed on the motion picture based on the size of the object identified by the recognition unit 220. For example, if the recognition unit 220 identifies the vertical dimension of a traffic signal, it uses that vertical dimension to adjust the vertical width of the abstract data representing the traffic signal.

[0113] In addition, the dynamic image generation unit 230 dynamically displays the movement history of the vehicle M at a specific traffic location in the dynamic image based on information such as vehicle speed contained in the vehicle time series data.

[0114] Figures 9-11 This is an example of a reproduced motion picture generated by the reproduced motion picture generation unit 230 and displayed on the display device 30. The reproduced motion picture is a motion picture obtained by reproducing the movement history of the vehicle M from the viewpoint outside the vehicle M. For example, the reproduced motion picture is a motion picture viewed from a direction intersecting the driving direction of the vehicle M. Figure 9 This indicates a lateral approach motion image from vehicle M. Figure 10This represents a lateral, remote dynamic view from vehicle M. Figure 11 This displays a dynamic image from the oblique rear direction of the vehicle M. It should be noted that the image displayed on the display device 30 may also include icons for switching dynamic images. Additionally, the image displayed on the display device 30 may include icons for playing dynamic images. When the user operates on the icons, the display control unit 210 executes the corresponding controls.

[0115] like Figures 9-11 As shown, the reproduced dynamic screen includes a background image representing a specific traffic location and the movement history of the vehicle M. The background image can be displayed statically or dynamically. For example, in Figure 9 In the approaching dynamic view shown laterally, the background image is dynamically displayed in conjunction with the movement of the vehicle M. Figure 10 The shown is a remote dynamic image from the horizontal direction, and Figure 11 In the dynamic image shown from a diagonal rear direction, the background image is displayed statically.

[0116] The chart image generation unit 231 generates a chart image representing the change in the vehicle speed of the vehicle M at a specific traffic location based on the speed of the vehicle M detected by the vehicle sensor 20, accelerator pedal operation information, brake pedal operation information, etc. Additionally, the chart image generation unit 231 generates a chart image representing the target speed of the vehicle M. The target speed, for example, refers to the speed of the vehicle M when the driver begins to operate the brake pedal at the target identification location, decelerates at a certain rate, and brings the vehicle M to a stop at the target stopping position.

[0117] Figure 12 This is an example of a chart image generated by the chart image generation unit 231 and displayed on the display device 30. For example... Figure 12 As shown, the graph displays the vehicle speed (solid line) and the target speed (dashed line) of vehicle M. Figure 12 As shown, the graph can also display the time when the driver ends the operation of the accelerator pedal and the time when the driver begins the operation of the brake pedal. The graph can also display the deceleration of the vehicle M and the deceleration at the target speed.

[0118] It should be noted that the motion picture generation unit 230 and the chart image generation unit 231 may also generate the motion picture and the chart image respectively each time a motion picture output command is received, and then display the generated motion picture and chart image on the display device 30. Alternatively, the motion picture generation unit 230 and the chart image generation unit 231 may also have a storage area for storing the generated motion picture and chart image, and when a motion picture output command is received, the motion picture and chart image stored in the storage area are retrieved and output to the display device 30.

[0119] Furthermore, the display control unit 210 can also determine whether to display the dynamic images and charts on the display device 30 whenever the vehicle M stops. In this case, the display control unit 210 can also display the dynamic images and charts on the display device 30 only if predetermined conditions are met. The "prescribed conditions" can be specifically set based on the driving conditions of the vehicle M, considering conditions where there is room for improvement in terms of safety. For example, the "prescribed conditions" refer to situations where the deceleration of the vehicle M exceeds a threshold.

[0120] Figure 13 This is a flowchart illustrating an example of the processing performed by the reproduction system 1A. It should be noted that this flowchart is an example, and other processes can also be followed.

[0121] exist Figure 13 In the example, firstly, the display control unit 210 determines whether there is a stop indicator in front of the vehicle M (step S101). If it is determined that there is a stop indicator in front of the vehicle M (step S101: Yes), the process proceeds to step S102. If it is not determined that there is a stop indicator in front of the vehicle M (step S101: No), the display control unit 210 executes the processing of step S101 again.

[0122] In step S102, the display control unit 210 generates external time-series data and vehicle time-series data within the interval from the target identification position to the target stopping position. It should be noted that the vehicle time-series data and external time-series data can also be generated continuously while the vehicle M is in motion.

[0123] In step S103, the display control device 200 (e.g., display control unit 210 or motion picture generation unit 230) synthesizes the information required to generate the motion picture. In step S104, the motion picture generation unit 230 generates the motion picture. In step S105, the chart image generation unit 231 generates a chart image. In step S106, the display control unit 210 determines whether the playback timing has arrived. If the playback timing is determined to have arrived (step S106: Yes), the process proceeds to step S107, where the display control unit 210 causes the display device 30 to display the motion picture and the chart image. The processing of this flowchart ends here.

[0124] As explained above, the reproduction system 1A of this embodiment includes: an external detection unit 10 that acquires external information; a vehicle sensor 20 that acquires vehicle information of the vehicle M; a reproduction dynamic image generation unit 230 that generates a dynamic image that reproduces the movement history of the vehicle M at a specific traffic location; and a display control unit 210 that causes the display device 30 to display the dynamic image. The reproduction dynamic image generation unit 230 generates the dynamic image based on external time-series data representing the external conditions of the vehicle M in a time series, generated based on external information, and vehicle time-series data representing the driving conditions of the vehicle M in a time series, generated based on vehicle information.

[0125] According to this reproduction system 1A, the movement history of the vehicle M at a specific traffic location can be reproduced through dynamic images generated by computer graphics. Therefore, the situation at a specific traffic location can be easily reproduced.

[0126] In addition, the dynamic image generation unit 230 generates a dynamic image that reproduces the movement history of the vehicle M from the position in front to the target stopping position at a specific traffic location including the stop indicator. The position in front refers to the position in front of the target stopping position of the vehicle M.

[0127] According to this reproduction system 1A, the movement history of the vehicle M at a specific traffic location, including stop indicators such as traffic signals and temporary stop signs, can be reproduced through dynamic images generated by computer graphics. Therefore, the driver can objectively review the driving history of the vehicle M when it stopped according to stop indicators such as traffic signals and temporary stop signs.

[0128] Alternatively, the target stopping position can be the position of the stop line on the road in which the vehicle M is traveling. The external detection unit 10 includes an image sensor that captures images of the front of the vehicle M. External time-series data includes the positions of the stop line and road markings in a time-series format. The vehicle time-series data is input to the motion picture generation unit 230 via the vehicle's in-vehicle communication network (CAN). The motion picture generation unit 230 uses the information related to the stop line and road markings contained in the external time-series data to display the stop line and road markings in a motion picture. According to this structure, motion pictures can be generated using existing hardware resources mounted on the vehicle. Furthermore, even without using GPS, map information, etc., the stop line and road markings can be displayed in the motion picture, reducing the processing load when generating motion pictures. In addition, by reducing the processing load, motion pictures can be generated in a short time, and the display device 30 can display the motion picture in a timely manner, for example, when the vehicle M stops.

[0129] Alternatively, the stopping indicator could include a traffic signal, and the dynamic image generation unit 230 could generate a dynamic image reproducing the movement history of the vehicle M when the traffic signal is red. According to this structure, the driver can objectively review the driving history of the vehicle M when it stops at a red light.

[0130] Alternatively, the reproduction system 1A may also include a graph image generation unit 231 that generates a graph image representing the speed change of the vehicle M at a specific traffic location based on vehicle time-series data, and a display control unit 210 that causes the display device 30 to display the graph image. According to this configuration, by enabling the driver to recognize the speed changes of the vehicle M, the driving history of the vehicle M when it stops due to a stop indicator can be reviewed more objectively.

[0131] In addition, the dynamic image can also be a dynamic image viewed from a direction intersecting with the driving direction of vehicle M. This allows the driver to more objectively review the driving history of vehicle M.

[0132] It should be noted that, similar to the first embodiment, the reproduction system 1A also includes an internal detection unit 40 that acquires internal information of the vehicle M. In this case, the recognition unit 220 identifies the driver's condition of the vehicle M based on the internal information output by the internal detection unit 40. The display control unit 210 generates driver time-series data that represents the driver's condition of the vehicle M in a time sequence based on the recognition result output by the recognition unit 220. The reproduction dynamic image generation unit 230 generates a reproduction dynamic image of a specific traffic location based on the external time-series data, vehicle time-series data, and driver time-series data. Specifically, in addition to the background image of the specific traffic location and the dynamic image showing the movement history of the vehicle M, the reproduction dynamic image generation unit 230 also generates a dynamic image showing the driver's actions based on the driver time-series data. For example, the reproduction dynamic image generation unit 230 generates a dynamic image showing the direction of the driver's head based on the orientation of the driver's face as identified by the recognition unit 220. The dynamic image generation unit 230 can also dynamically display the driver's field of vision in the dynamic image based on the direction of the driver's gaze recognized by the recognition unit 220.

[0133] Furthermore, the dynamic image generation unit 230 can also generate dynamic images representing the actions of the driver of the vehicle M based on vehicle time-series data. For example, the dynamic image generation unit 230 can also dynamically display the actions of the driver's hands and feet in the dynamic image based on steering wheel operation information, acceleration operation information, braking operation information, light operation information, and direction indicator operation information contained in the vehicle time-series data.

[0134] The implementation methods described above can be performed as follows.

[0135] A reproduction system comprising:

[0136] External detection unit, which obtains external information about this vehicle;

[0137] Vehicle sensors acquire vehicle information of the vehicle.

[0138] Storage media that hold commands that can be read by a computer; and

[0139] The processor, which is connected to the storage medium,

[0140] The processor performs the following processing by executing commands that can be read by the computer:

[0141] Based on the external information, external time series data representing the external conditions of the vehicle in a time series format is generated;

[0142] Based on the vehicle information, generate vehicle time series data that represents the driving status of the vehicle in a time series format;

[0143] Based on the external time-series data and the vehicle time-series data, a dynamic image is generated that reproduces the movement history of the vehicle at a specific traffic location, as well as the movement histories of other traffic participants whose paths intersect with the vehicle's path at that specific traffic location; and

[0144] The display device displays the dynamic image.

[0145] Furthermore, the embodiments described above can be implemented as follows.

[0146] A reproduction system comprising:

[0147] External detection unit, which obtains external information about this vehicle;

[0148] Vehicle sensors acquire vehicle information of the vehicle.

[0149] Storage media that hold commands that can be read by a computer; and

[0150] The processor, which is connected to the storage medium,

[0151] The processor performs the following processing by executing commands that can be read by the computer:

[0152] Based on the external information, external time series data representing the external conditions of the vehicle in a time series format is generated;

[0153] Based on the vehicle information, generate vehicle time series data that represents the driving status of the vehicle in a time series format;

[0154] Based on the external time-series data and the vehicle time-series data, a dynamic image is generated that reproduces the vehicle's movement history at a specific traffic location using the following line of sight: the line of sight refers to the line of sight of other traffic participants whose paths intersect with the vehicle's path at the specific traffic location; and

[0155] The display device displays the dynamic image.

[0156] Furthermore, the embodiments described above can be implemented as follows.

[0157] A reproduction system comprising:

[0158] External detection unit, which obtains external information about this vehicle;

[0159] Vehicle sensors acquire vehicle information of the vehicle.

[0160] Storage media that hold commands that can be read by a computer; and

[0161] The processor, which is connected to the storage medium,

[0162] The processor performs the following processing by executing commands that can be read into the computer:

[0163] Based on the external information, external time series data representing the external conditions of the vehicle in a time series format is generated;

[0164] Based on the vehicle information, generate vehicle time series data that represents the driving status of the vehicle in a time series format;

[0165] Based on the external time-series data and the vehicle time-series data, a dynamic image is generated that reproduces the vehicle's movement history at a specific traffic location, including stop indicators set on the road where the vehicle is traveling, from a position closer to the target stop position to the target stop position; and

[0166] The display device displays the dynamic image.

[0167] It should be noted that the technical scope of the present invention is not limited to the described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0168] For example, a part of the playback system 1, 1A (e.g., display device 30, display control device 100, 200, etc.) may not be mounted on the vehicle M, but on a device (server, personal computer, smartphone, tablet terminal, etc.) that communicates wirelessly with the vehicle M. That is, the generation and playback of dynamic images can also be performed by a device separated from the vehicle M.

[0169] As a specific example, the animated images generated by the motion image generation units 130 and 230 can also be stored in a storage unit provided by a server or the like. Furthermore, based on operations performed on a device connected to the server, the animated images can be displayed on a display device provided by that device. In this case, the driver can view the animated images at home or elsewhere to review their driving operations. Alternatively, the use of animated images can be used for instruction in motor vehicle training facilities or the like.

[0170] Furthermore, in the first embodiment, an intersection is cited as an example of a specific traffic location. However, a specific traffic location may not be an intersection; for example, it could be a location within a parking lot. Alternatively, a specific traffic location could be a connection point between a parking lot and a road. In these cases, the driver can objectively observe the movement history of the vehicle M, thereby promoting safer driving.

[0171] In addition, without departing from the spirit of the present invention, the constituent elements in the above embodiments can be appropriately replaced with well-known constituent elements. Furthermore, the above embodiments and variations can be appropriately combined.

Claims

1. A reproduction system, wherein, The reproduction system includes: External detection unit, which obtains external information about this vehicle; Vehicle sensors acquire vehicle information of the vehicle. The dynamic image generation unit generates a dynamic image that reproduces the movement history of the vehicle at a specific traffic location, based on external time-series data representing the external conditions of the vehicle in a time series, generated based on the external information, and vehicle time-series data representing the driving conditions of the vehicle in a time series, generated based on the vehicle information; and The display control unit causes the display device to display the dynamic image. The specific traffic location is an intersection. The dynamic image generation unit uses information related to the baseline marked on the road at a first moment before the vehicle changes its course at the intersection, and information related to the second baseline marked on the road at a second moment after the vehicle changes its course at the intersection, to display the intersection in the dynamic image. The dynamic image generation unit uses the change angle of the vehicle's direction of travel from the first moment to the second moment, the angle of the baseline relative to the vehicle at the first moment, and the angle of the second baseline relative to the vehicle at the second moment to calculate the angle between the baseline and the second baseline, and draws the road included in the background image based on the result of the calculation. The display control unit determines whether to display the dynamic image on the display device each time the vehicle stops, and if the predetermined conditions are met, it displays the dynamic image on the display device. As a condition of the aforementioned regulations, conditions are set based on the driving conditions of the vehicle, where there is room for improvement in terms of safety.

2. The reproduction system according to claim 1, wherein, The dynamic image generation unit generates a dynamic image based on the external time series data and the vehicle time series data, which reproduces the movement history of other traffic participants whose forward roads intersect with the forward road of the vehicle at the specific traffic location, together with the movement history of the vehicle at the specific traffic location.

3. The reproduction system according to claim 1, wherein, The dynamic image generation unit generates a dynamic image based on the external time series data and the vehicle time series data, which reproduces the movement history of the vehicle at the specific traffic location from the perspective of other traffic participants whose road path intersects with the vehicle's road path at the specific traffic location.

4. The reproduction system according to claim 2 or 3, wherein, The external detection unit includes an image sensor that captures images of the front of the vehicle. The external time series data includes the time series positions of stop lines and vehicle traffic separation lines on the roads traversed by the vehicle. The vehicle time-series data is input to the motion picture generation unit via the vehicle's in-vehicle communication network.

5. The reproduction system according to claim 2 or 3, wherein, The dynamic image generation unit uses the information related to the other traffic participants contained in the external time series data to generate abstracted image data corresponding to the other traffic participants, and displays the image data as the other traffic participants in the dynamic image.

6. The reproduction system according to claim 1, wherein, The specific traffic location includes a stop indicator placed on the road on which the vehicle is traveling, indicating that the vehicle should stop. The dynamic image generation unit generates a dynamic image based on the external time series data and the vehicle time series data, which reproduces the movement history of the vehicle at the specific traffic location from a position closer to the target stopping position to the target stopping position.

7. The reproduction system according to claim 6, wherein, The target stopping position is the location of the stop line on the road in which the vehicle is traveling. The external detection unit includes an image sensor that captures images of the front of the vehicle. The external time series data includes the position of the stop line in the time series and the position of the road markings in the road where the vehicle is traveling in the time series. The vehicle time-series data is input to the motion picture generation unit via the vehicle's in-vehicle communication network. The dynamic image generation unit uses the information related to the stop line and the road dividing line contained in the external time series data to make the dynamic image display the stop line and the road dividing line.

8. The reproduction system according to claim 6 or 7, wherein, The stop indicators include traffic signals. The dynamic image generation unit generates a dynamic image that reproduces the vehicle's movement history when the traffic signal is red.

9. The reproduction system according to claim 6 or 7, wherein, The reproduction system also includes a chart image generation unit, which generates a chart image representing the change in the vehicle's speed at the specific traffic location based on the vehicle time series data. The display control unit causes the display device to display the chart image.

10. The reproduction system according to claim 6 or 7, wherein, The dynamic image is a dynamic image viewed from a direction intersecting the vehicle's direction of travel.

11. The reproduction system according to claim 1, wherein, The reproduction system also includes an internal detection unit that acquires internal information of the vehicle. The dynamic image generation unit generates a dynamic image that reproduces the driver's actions along with the vehicle's movement history at the specific traffic location, based on driver time series data generated from the internal information, which represents the driver's status in a time series.

12. A reproduction method, wherein, The reproduction system includes: an external detection unit that acquires external information about the vehicle; and a vehicle sensor that acquires vehicle information about the vehicle. The reproduction method causes the computer of the reproduction system to perform the following processing: Based on external time-series data representing the external conditions of the vehicle in a time series generated based on the external information, and vehicle time-series data representing the driving conditions of the vehicle in a time series generated based on the vehicle information, a dynamic picture that reproduces the movement history of the vehicle at a specific traffic location is generated. The display device displays the dynamic image; The specific traffic location is an intersection. The dynamic display of the intersection is made using information related to the baseline marked on the road at a first moment before the vehicle changes its course at the intersection, and information related to the second baseline marked on the road at a second moment after the vehicle changes its course at the intersection. Using the change angle of the vehicle's direction of travel from the first time point to the second time point, the angle of the baseline relative to the vehicle at the first time point, and the angle of the second baseline relative to the vehicle at the second time point, the angle between the baseline and the second baseline is calculated, and based on the result of the calculation, a road included in the background image is drawn. Whenever the vehicle stops, a determination is made as to whether the display device should display the dynamic image, and if the determination is that the predetermined conditions are met, the display device should display the dynamic image. As a condition of the aforementioned regulations, conditions are set based on the driving conditions of the vehicle, where there is room for improvement in terms of safety.

13. The reproduction method according to claim 12, wherein, Based on the external time series data and the vehicle time series data, a dynamic picture is generated that reproduces the movement history of other traffic participants whose forward roads intersect with the forward road of the vehicle at the specific traffic location, together with the movement history of the vehicle at the specific traffic location.

14. The reproduction method according to claim 12, wherein, Based on the external time series data and the vehicle time series data, a dynamic picture is generated that reproduces the movement history of the vehicle at the specific traffic location in the line of sight of other traffic participants whose roadway intersects with the vehicle's roadway at the specific traffic location.

15. The reproduction method according to claim 12, wherein, The specific traffic location includes a stop indicator placed on the road on which the vehicle is traveling, indicating that the vehicle should stop. Based on the external time series data and the vehicle time series data, a dynamic image is generated that reproduces the movement history of the vehicle from a position closer to the target stopping position at the specific traffic location to the target stopping position.

16. A storage medium storing a playback program, wherein, The reproduction system includes: an external detection unit that acquires external information about the vehicle; and a vehicle sensor that acquires vehicle information about the vehicle. The reproduction program causes the computer of the reproduction system to perform the following processing: Based on external time-series data representing the external conditions of the vehicle in a time series generated based on the external information, and vehicle time-series data representing the driving conditions of the vehicle in a time series generated based on the vehicle information, a dynamic picture that reproduces the movement history of the vehicle at a specific traffic location is generated. The display device displays the dynamic image; The specific traffic location is an intersection. The dynamic display of the intersection is made using information related to the baseline marked on the road at a first moment before the vehicle changes its course at the intersection, and information related to the second baseline marked on the road at a second moment after the vehicle changes its course at the intersection. Using the change angle of the vehicle's direction of travel from the first time point to the second time point, the angle of the baseline relative to the vehicle at the first time point, and the angle of the second baseline relative to the vehicle at the second time point, the angle between the baseline and the second baseline is calculated, and based on the result of the calculation, a road included in the background image is drawn. Whenever the vehicle stops, a determination is made as to whether the display device should display the dynamic image, and if the determination is that the predetermined conditions are met, the display device should display the dynamic image. As a condition of the aforementioned regulations, conditions are set based on the driving conditions of the vehicle, where there is room for improvement in terms of safety.

17. The storage medium according to claim 16, wherein, The reproduction program causes the computer of the reproduction system to perform the following processing: Based on the external time series data and the vehicle time series data, a dynamic picture is generated that reproduces the movement history of other traffic participants whose forward roads intersect with the forward road of the vehicle at the specific traffic location, together with the movement history of the vehicle at the specific traffic location.

18. The storage medium according to claim 16, wherein, The reproduction program causes the computer of the reproduction system to perform the following processing: Based on the external time series data and the vehicle time series data, a dynamic picture is generated that reproduces the movement history of the vehicle at the specific traffic location in the line of sight of other traffic participants whose roadway intersects with the vehicle's roadway at the specific traffic location.

19. The storage medium according to claim 16, wherein, The specific traffic location includes a stop indicator placed on the road on which the vehicle is traveling, indicating that the vehicle should stop. The reproduction program causes the computer of the reproduction system to perform the following processing: Based on the external time series data and the vehicle time series data, a dynamic image is generated that reproduces the movement history of the vehicle from a position closer to the target stopping position at the specific traffic location to the target stopping position.

Images