Information provision device and information provision method

Abstract

To make it easy to analyze a situation using an actual gaze direction of a driver.SOLUTION: An information processing device according to an embodiment comprises a controller configured to acquire information obtained from images inside and outside a vehicle from an in-vehicle device, and draw a gaze direction together with an object on a map screen on the basis of the acquired information.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The disclosed embodiments relate to an information providing device and an information providing method. [Background technology]

[0002] In recent years, insurance companies have been using video footage from drive recorders at the time of traffic accidents to analyze the circumstances at the time of the accident and using the analysis results as a basis for determining, for example, the degree of fault.

[0003] Since reviewing video footage is time-consuming, image recognition technology is often used to automatically extract objects captured in the footage using so-called AI (Artificial Intelligence) and visualize the situation at the time of the accident. When using this type of image recognition technology, for example, vehicle trajectories and speeds, traffic light colors, intersection shapes, etc. are visualized.

[0004] By the way, when determining the degree of negligence mentioned above, important factors include whether the driver was looking away from the road and whether they were able to see other vehicles and pedestrians. However, when using conventional image recognition technology, it is not possible to grasp such points.

[0005] Regarding visualization of the field of view of a driver while driving, a technology has been proposed for a drive recorder in which the field of view of the driver according to the vehicle speed is superimposed on the camera image of the drive recorder (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2009-089022 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional techniques have room for further improvement in terms of easily analyzing a situation using the actual line of sight of the driver.

[0008] For example, when the technology disclosed in Patent Document 1 is used, it is easy to check the driver's visual field according to the vehicle speed, but it is difficult to grasp the actual line of sight of the driver included in the visual field. In other words, when the above-mentioned conventional technology is used, it is difficult to easily check whether the driver was looking away from the road or whether he or she was able to see other vehicles or pedestrians.

[0009] One aspect of the embodiment has been made in consideration of the above, and aims to provide an information providing device and an information providing method that can easily analyze a situation using the driver's actual line of sight. [Means for solving the problem]

[0010] An information providing device according to one aspect of an embodiment includes a controller that acquires information obtained from an outside image and an inside image of the vehicle from an in-vehicle device, and draws a line of sight direction together with an object on a map screen based on the acquired information. Effect of the Invention

[0011] According to one aspect of the embodiment, the driver's line of sight obtained from in-vehicle footage is drawn and visualized together with objects obtained from outside the vehicle footage on a map screen corresponding to a situation such as an accident, making it easy to analyze the situation using the driver's actual line of sight. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an outline of an information providing method according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a situation analysis system according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of the drive recorder according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a server device according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of the operator terminal according to the embodiment. [Figure 6] FIG. 6 is a diagram showing a processing sequence executed by the situation analysis system according to the embodiment. [Figure 7] FIG. 7 is a diagram (part 1) showing a display example of the analysis information. [Figure 8] FIG. 8 is a diagram (part 2) showing a display example of the analysis information. [Figure 9] FIG. 9 is a diagram (part 3) showing a display example of the analysis information. [Figure 10] FIG. 10 is a diagram (part 4) showing a display example of the analysis information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of an information providing device and an information providing method disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiment.

[0014] In the following, the in-vehicle device according to the embodiment is described as a drive recorder 10 (see FIG. 1 onwards). In the following, the information providing system according to the embodiment is a situation analysis system 1 (see FIG. 1 onwards) including the drive recorder 10.

[0015] The situation analysis system 1 is a system that visualizes the situation at the time when a specified event occurs based on the outside and inside images contained in the vehicle data when a specified event is detected by the drive recorder 10, and the image recognition results of these images.

[0016] The visualized analysis information is displayed on, for example, an operator terminal used by an operator at a data center. The operator can grasp the situation at the time of the event occurrence based on the analysis information displayed on the operator terminal, and can also use this analysis information, for example, to determine the degree of fault at the time of an accident at an insurance company.

[0017] In the following description, the information providing device according to the embodiment is a server device 100 (see FIG. 1 and subsequent figures) included in the situation analysis system 1. The information providing method according to the embodiment is an information providing method executed by a controller 103 (see FIG. 4) of the server device 100. The drive recorder 10 is of a communication type that is capable of communicating with the server device 100.

[0018] In addition, the expressions "specific," "predetermined," and "constant" in the following description may be read as "predetermined."

[0019] First, an overview of an information providing method according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating an overview of an information providing method according to an embodiment. As shown in Fig. 1, a situation analysis system 1 includes a drive recorder 10, a server device 100, and an operator terminal 200.

[0020] The drive recorder 10 is a video recording device mounted on a vehicle. The drive recorder 10 according to the embodiment has an exterior camera 13a and an interior camera 13b. The exterior camera 13a is provided so as to be able to capture at least an exterior image in front of the vehicle. The interior camera 13b is provided so as to be able to capture an interior image of the vehicle.

[0021] During the startup of the vehicle, the drive recorder 10 executes a recording process for recording vehicle data including the vehicle exterior video and vehicle interior video captured by the vehicle exterior camera 13a and vehicle interior camera 13b in a ring buffer memory in an overwritable manner for a certain period of time. The certain period is, for example, 24 hours. As shown in FIG. 1, the vehicle data includes various data indicating the vehicle status, such as date and time information, position information, G (acceleration) value, and vehicle speed, in addition to the vehicle exterior video and vehicle interior video.

[0022] In addition to performing the recording process, the drive recorder 10 also performs image recognition processing on the vehicle exterior image and the vehicle interior image using an AI model for image recognition. The AI ​​model is, for example, a DNN (Deep Neural Network) model trained using a machine learning algorithm. This AI model is trained in advance so as to be able to recognize the type, position, speed, color, etc. of each object shown in the vehicle exterior image and the vehicle interior image.

[0023] The AI ​​model is configured to be able to recognize, for example, other vehicles other than the vehicle equipped with the drive recorder 10, motorcycles, bicycles, pedestrians, traffic lights, etc. from the outside-vehicle video. In addition, when the AI ​​model recognizes a traffic light, it is configured to be able to recognize the color of the light (hereinafter, appropriately referred to as the "color of the traffic light").

[0024] The AI ​​model is also set up to be able to recognize, for example, the driver's face from the in-car image. In the image recognition process, when the driver's face is recognized, the position and posture of the face are estimated, for example, from the arrangement of the feature points on the face. Then, in the image recognition process, the driver's line of sight direction is estimated based on the estimated position and posture of the face.

[0025] The image recognition results from the image recognition process include the type of object, its position, its speed, the color of the traffic light, and the driver's line of sight, as shown in Figure 1. These image recognition results are recorded together with the corresponding vehicle data in the recording process described above.

[0026] The drive recorder 10 is also configured to be capable of detecting specific events such as the occurrence of an accident, the occurrence of a near miss, the arrival of a predetermined position, etc. The drive recorder 10 detects a specific event when, for example, a change in vehicle speed, a change in G value, a change in position information, etc., satisfy a predetermined event condition corresponding to the occurrence of a pre-set accident, the occurrence of a near miss, the arrival of a predetermined position, etc.

[0027] The drive recorder 10 detects, for example, the detection of acceleration equal to or greater than a threshold as a specific event such as the occurrence of an accident or a near-miss. When this embodiment is used, the situation when an accident or a near-miss occurs can be visualized, including the driver's line of sight. A specific example will be described later with reference to Figs. 7 and 8.

[0028] When the drive recorder 10 detects a specific event, the drive recorder 10 sets the vehicle data and the image recognition result for a certain period of time before and after the detection time to be overwritten. Alternatively, the drive recorder 10 records the vehicle data and the image recognition result for a certain period of time before and after the detection time on a separate recording medium. Note that this overwriting prevention process and the recording process on a separate recording medium may be performed by instructions from the server device 100.

[0029] Furthermore, when the drive recorder 10 detects a specific event, the drive recorder 10 transmits to the server device 100 the vehicle data to be set as overwrite-prohibited and the image recognition result.

[0030] The server device 100 is provided so as to be capable of generating analysis information that visualizes the vehicle situation when a specific event is detected in the drive recorder 10, based on the vehicle data and the image recognition result acquired from the drive recorder 10. The server device 100 is also provided so as to be capable of providing the analysis information to the operator terminal 200.

[0031] In the information providing method according to the embodiment, the server device 100 generates the analysis information so that the situation can be easily analyzed using the actual line of sight of the driver.

[0032] That is, in the information providing method according to the embodiment, the server device 100 acquires an image recognition result including the type and position of an object obtained from an outside-vehicle image and the driver's line-of-sight direction obtained from an inside-vehicle image from the drive recorder 10. Furthermore, based on the image recognition result, the server device 100 draws the line-of-sight direction together with the object on a map screen corresponding to the situation to be analyzed.

[0033] 1, the drive recorder 10 first executes the above-mentioned recording process and image recognition process while the vehicle is running (step S1). Then, when the drive recorder 10 detects a specific event based on a preset event condition (step S2), it transmits vehicle data and image recognition results for a certain period of time before and after the event detection time to the server device 100 (step S3).

[0034] The server device 100 then acquires map information corresponding to the situation to be analyzed based on the position information included in the vehicle data, and generates a map screen that visualizes the map information.The server device 100 then generates analysis information that depicts the driver's line of sight direction together with each recognized object on the map screen based on the image recognition result acquired from the drive recorder 10 (step S4).

[0035] Then, the server device 100 provides the generated analysis information to the operator terminal 200 (step S5). This allows the operator to easily grasp the situation at the time of an accident, etc., by using the analysis information in which the driver's line of sight direction obtained from the in-vehicle image is visualized on the map screen together with the object obtained from the outside-vehicle image.

[0036] Specific display examples of the analysis information provided by the server device 100 will be described later with reference to FIGS.

[0037] A configuration example of the situation analysis system 1 including the server device 100 to which the information providing method according to the embodiment described above is applied will be described in more detail below.

[0038] Fig. 2 is a diagram showing a configuration example of a situation analysis system 1 according to an embodiment. As shown in Fig. 2, the situation analysis system 1 includes drive recorders 10-1, 10-2, ... 10-m (m is a natural number equal to or greater than 1), a server device 100, and operator terminals 200-1, 200-2, ... 200-n (n is a natural number equal to or greater than 1). The server device 100 and each operator terminal 200 form a data center.

[0039] Each drive recorder 10 and the server device 100 are connected to each other so as to be able to communicate with each other via a network N1, which may be the Internet, a mobile phone network, a C-V2X (Cellular Vehicle to Everything) communication network, or the like.

[0040] The server device 100 and each operator terminal 200 are connected to each other so as to be able to communicate with each other via a network N2, which is a private network managed by a business operator (e.g., an insurance company) that operates a data center. Also, each operator terminal 200 can communicate with each drive recorder 10 connected to the network N1 via the server device 100.

[0041] As described above, the drive recorder 10 executes the recording process and the image recognition process, and records the vehicle data and the image recognition results in a ring buffer memory in a manner that allows the data to be overwritten for a certain period of time.

[0042] As described above, when the drive recorder 10 detects a specific event while the vehicle is running, the drive recorder 10 sets the vehicle data and the image recognition results for a certain period of time before and after the detection time to be overwritten. When the drive recorder 10 detects a specific event, the drive recorder 10 transmits the vehicle data and the image recognition results to be overwritten to the server device 100.

[0043] The server device 100 is realized as, for example, a private cloud server. The server device 100 is managed by, for example, a business operator (such as an insurance company) that operates a data center. The server device 100 collects vehicle data and image recognition results transmitted from each drive recorder 10.

[0044] In addition, the server device 100 generates analysis information that depicts the driver's line of sight direction together with each recognized object on the map screen to be analyzed based on the collected vehicle data and the image recognition result. In addition, the server device 100 provides the generated analysis information to the operator terminal 200.

[0045] As described above, the operator terminal 200 is a terminal device used by an operator of a data center, and is realized by a PC (Personal Computer), a smartphone, etc. The operator terminal 200 may also be realized by, for example, a tablet terminal, a wearable device, etc.

[0046] Next, a configuration example of the drive recorder 10 will be described. Fig. 3 is a diagram showing a configuration example of the drive recorder 10 according to the embodiment. As shown in Fig. 3, the drive recorder 10 has a communication unit 11, an HMI (Human Machine Interface) unit 12, a sensor unit 13, a storage unit 14, and a controller 15.

[0047] The communication unit 11 is realized by a network adapter etc. The communication unit 11 is wirelessly connected to the network N1, and transmits and receives information to and from the server device 100 via the network N1.

[0048] The HMI unit 12 is a component that provides interface components related to input and output to a driver or the like who uses the drive recorder 10. The HMI unit 12 includes an input interface that accepts input operations from the driver or the like. The HMI unit 12 also includes an output interface that presents visual information and audio information to the driver or the like.

[0049] 3, for example, the HMI unit 12 includes a touch panel display 12a, a microphone 12b, and a speaker 12c. The touch panel display 12a corresponds to the input interface and the output interface described above. The touch panel display 12a displays, for example, operation components for operating the drive recorder 10. Note that these operation components may be provided in the HMI unit 12 as hardware components, rather than being displayed on the touch panel display 12a as software components.

[0050] The microphone 12b corresponds to an input interface that collects sounds from inside the vehicle, and the speaker 12c corresponds to an output interface that outputs guidance sounds from the drive recorder 10 and the like.

[0051] The sensor unit 13 is a group of various sensors mounted on the drive recorder 10. The sensor unit 13 includes, for example, an exterior camera 13a, an interior camera 13b, a GPS (Global Positioning System) sensor 13c, and a G sensor 13d.

[0052] As described above, the exterior camera 13a is provided so as to be able to capture at least an image of the exterior of the vehicle in front of the vehicle. The exterior camera 13a is attached near the windshield, near the dashboard, etc. The exterior camera 13a may be attached near the rear window, etc., so as to be able to capture an image of the rear of the vehicle.

[0053] As described above, the in-vehicle camera 13b is provided so as to be able to capture an image of the interior of the vehicle. The in-vehicle camera 13b is attached near the windshield, the dashboard, or the like so that at least the driver's face is included in the imaging range.

[0054] It should be noted that the exterior camera 13a and the interior camera 13b do not necessarily need to be separate, and may be integrated into one camera, for example a 360-degree camera.

[0055] The GPS sensor 13c measures the GPS position of the vehicle. The G sensor 13d measures the acceleration applied to the drive recorder 10. The sensor unit 13 may include various sensors other than the exterior camera 13a, the interior camera 13b, the GPS sensor 13c, and the G sensor 13d.

[0056] In addition to the sensor unit 13, the drive recorder 10 is connected to an in-vehicle sensor 5, which is a group of various sensors mounted on a vehicle. The in-vehicle sensor 5 includes, for example, a vehicle speed sensor, an accelerator sensor, a brake sensor, etc. The in-vehicle sensor 5 is connected to the drive recorder 10 via an in-vehicle network such as a CAN (Controller Area Network).

[0057] The storage unit 14 is realized by a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc. In the example of FIG. 3, the storage unit 14 stores user information 14a, an image recognition AI 14b, and vehicle record information 14c.

[0058] The user information 14a is information that links information about a user who uses the drive recorder 10 with identification information of the drive recorder 10. The information about the user is the identification information of the user, the identification information of the vehicle in which the drive recorder 10 is installed (for example, the vehicle registration number), etc.

[0059] The image recognition AI 14b corresponds to the AI ​​model described above. The image recognition AI 14b is read into the controller 15 as an AI model, and is provided so that when each frame of an outside image or an inside image of the vehicle is input to the controller 15, the image recognition AI 14b can detect various objects and the like included in each frame.

[0060] The image recognition AI 14b is provided so as to be able to recognize the type, position, speed, and color of the traffic light of each of the above-mentioned objects shown in each frame when each frame of the vehicle exterior video is input. Also, the image recognition AI 14b is provided so as to be able to recognize the face of the driver shown in each frame when each frame of the vehicle interior video is input.

[0061] The controller 15 estimates the driver's line of sight direction based on the face recognized by the image recognition AI 14b, for example, as follows: For example, the controller 15 extracts a group of facial feature points from the detection frame of the recognized face.

[0062] Then, the controller 15 estimates the three-dimensional position and posture of the driver's face as the face direction based on the arrangement of the extracted feature points. The position and posture of the face are specifically the position and posture of a specific part of the face (for example, the area between the eyebrows).

[0063] Based on the arrangement of the group of feature points, the controller 15 estimates the roll angle, pitch angle, and yaw angle for three orthogonal axes with the center of the specific part on the driver's median plane as the origin as the position and posture of the face. The controller 15 then estimates the driver's line of sight as a three-dimensional vector for the estimated position and posture of the face. This allows the controller 15 to accurately estimate the line of sight based on the position and posture of the face.

[0064] This method of estimating the gaze direction is merely an example, and other methods may be used to estimate the gaze direction of the driver. For example, the image recognition AI 14b may be provided so as to directly recognize the gaze direction of the driver based on the face of the driver shown in each frame when each frame of the in-vehicle video is input.

[0065] The vehicle record information 14c is a database of vehicle data and image recognition results recorded by the drive recorder 10.

[0066] The controller 15 corresponds to a so-called processor. The controller 15 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or the like. The controller 15 executes a program according to an embodiment (not shown) stored in the storage unit 14, using the RAM as a working area. The controller 15 can also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0067] The controller 15 executes information processing according to the processing sequence shown in Fig. 6. The explanation using Fig. 6 will be given later.

[0068] Next, a configuration example of the server device 100 will be described. Fig. 4 is a diagram showing a configuration example of the server device 100 according to an embodiment. As shown in Fig. 4, the server device 100 includes a communication unit 101, a storage unit 102, and a controller 103.

[0069] The communication unit 101 is realized by a network adapter, etc. The communication unit 101 is connected to a network N1 by wire or wirelessly, and transmits and receives information to and from the drive recorder 10 via the network N1. The communication unit 101 is also connected to a network N2 by wire or wirelessly, and transmits and receives information to and from the operator terminal 200 via the network N2.

[0070] The storage unit 102 is realized by a storage device such as a ROM, a RAM, a flash memory, a HDD (Hard Disk Drive), etc. In the example of Fig. 4, the storage unit 102 stores a user information DB (Database) 102a, a map information DB 102b, a collected information DB 102c, and a screen generation information DB 102d.

[0071] The user information DB 102a is a database of the user information 14a of each drive recorder 10. The user information 14a of each drive recorder 10 is transmitted from the drive recorder 10, for example, at the time of initial setting of the drive recorder 10, and is stored in the user information DB 102a.

[0072] The map information DB 102b is a database in which various map information for generating a map screen in the analysis information to be provided to the operator terminal 200 is stored.

[0073] The collected information DB 102c is a database that stores vehicle data and image recognition results collected from each drive recorder 10. The vehicle data and image recognition results collected from each drive recorder 10 may be associated with each user information 14a in the user information DB 102a and stored in the collected information DB 102c.

[0074] The screen generation information DB 102d is a database that stores various parameters, a GUI (Graphical User Interface), and the like used when the controller 103 generates analysis information. The various parameters include, for example, parameters related to the screen layout in the analysis information.

[0075] The controller 103 corresponds to a so-called processor. The controller 103 is realized by a CPU, an MPU, a GPU, or the like. The controller 103 executes a program according to an embodiment (not shown) stored in the storage unit 102, using a RAM as a working area. The controller 103 can also be realized by an integrated circuit such as an ASIC or an FPGA.

[0076] The controller 103, like the above-mentioned controller 15, executes information processing according to the processing sequence shown in Fig. 6. The explanation using Fig. 6 will be given later.

[0077] Next, a configuration example of the operator terminal 200 will be described. Fig. 5 is a diagram showing a configuration example of the operator terminal 200 according to the embodiment. As shown in Fig. 5, the operator terminal 200 has a communication unit 201, an HMI unit 202, a storage unit 203, and a controller 204.

[0078] The communication unit 201 is realized by a network adapter etc. The communication unit 201 is connected to the network N2 by wire or wirelessly, and transmits and receives information to and from the server device 100 via the network N2.

[0079] The HMI unit 202 is a component that provides interface components related to input and output to the operator who uses the operator terminal 200. The HMI unit 202 includes an input interface that accepts input operations from the operator. The HMI unit 202 also includes an output interface that presents visual information and audio information to the operator.

[0080] 5, the HMI unit 202 includes, for example, a display 202a, a speaker 202b, and a keyboard 202c. The display 202a corresponds to the output interface described above. The display 202a displays, for example, analysis information provided by the server device 100.

[0081] The speaker 202b corresponds to an output interface when the analysis information includes voice. The keyboard 202c corresponds to an input interface that accepts an operation of the operator terminal 200 by the operator, for example.

[0082] The operator inputs, via the keyboard 202c, for example, identification information corresponding to the analysis information that the operator wishes to receive from the server device 100. When this identification information is input, the operator terminal 200 transmits a display request for the corresponding analysis information to the server device 100.

[0083] The input interface may also be realized by a mouse, a pen tablet, etc. The input interface may also be realized by a software component.

[0084] Furthermore, the HMI unit 202 may provide the operator with an input interface and an output interface in one body, for example, by using a touch panel display.

[0085] The storage unit 203 is realized by a storage device such as a ROM, a RAM, a flash memory, a HDD, etc. In the example of Fig. 5, the storage unit 203 stores operator application information 203a.

[0086] The operator application information 203a is information related to an operator application executed in the operator terminal 200, and includes a program of the operator application. The operator application information 203a may be realized as a Web application with the server device 100 serving as a Web server.

[0087] The controller 204 corresponds to a so-called processor. The controller 204 is realized by a CPU, an MPU, a GPU, or the like. The controller 204 executes the above-mentioned operator application program stored in the storage unit 203, using the RAM as a working area. The controller 204 can also be realized by an integrated circuit such as an ASIC or an FPGA.

[0088] The controller 204, like the above-mentioned controller 15 and controller 103, executes information processing according to the processing sequence shown in FIG.

[0089] Next, information processing according to this processing sequence will be described. Fig. 6 is a diagram showing a processing sequence executed by the situation analysis system 1 according to the embodiment.

[0090] The controller 15 of the drive recorder 10 executes the above-mentioned recording process and image recognition process while the vehicle is running (step S101). In addition, the controller 15 determines whether or not the above-mentioned specific event is detected while the vehicle is running (step S102).

[0091] If an event is detected (step S102, Yes), the controller 15 sets the vehicle data and image recognition results for a certain period of time before and after the detection time to be overwritten prohibited, and transmits the vehicle data and image recognition results to the server device 100 (step S103). This makes it possible to visualize the situation for a certain period of time before and after the event is detected. If no event is detected (step S102, No), the controller 15 repeats the process from step S101.

[0092] The controller 103 of the server device 100 acquires the vehicle data and the image recognition result in step S103, and stores them in the collected information DB 102c (step S104).

[0093] Furthermore, when the controller 204 of the operator terminal 200 receives an input of identification information of desired analysis information from the operator at any timing, the controller 204 transmits a display request for the analysis information corresponding to the identification information to the server device 100 (step S105).

[0094] When the controller 103 of the server device 100 receives the display request in step S105, the controller 103 extracts the vehicle data and the image recognition result corresponding to the display request from the collected information DB 102c (step S106).

[0095] Then, the controller 103 draws each object recognized in the extracted image recognition result on a map screen corresponding to the position information included in the extracted vehicle data (step S107). The controller 103 also draws the driver's line of sight direction on the same map screen (step S108).

[0096] Then, the controller 103 transmits the analysis information generated in steps S106 to S108 to the operator terminal 200 (step S109). Upon acquiring the analysis information in step S109, the controller 204 of the operator terminal 200 displays the acquired analysis information on the display 202a (step S110).

[0097] Next, specific display examples of this analysis information will be described with reference to Figs. 7 to 10. Fig. 7 is a diagram (part 1) showing a display example of the analysis information. Fig. 8 is a diagram (part 2) showing a display example of the analysis information. Fig. 9 is a diagram (part 3) showing a display example of the analysis information. Fig. 10 is a diagram (part 4) showing a display example of the analysis information.

[0098] As shown in Fig. 7, the analysis information is displayed as visual information depicting the host vehicle and each object included in the image recognition result on a map screen M corresponding to the situation to be analyzed. Fig. 7 shows an example in which other vehicles, motorcycles, bicycles, pedestrians, and traffic lights 500-1, 500-2, and 500-3 are depicted as objects together with the host vehicle. By depicting the type of each object, it is possible to easily grasp how each type of object is related to, for example, the situation in which an accident occurred.

[0099] 7 also shows an example in which traffic lights 500-1 and 500-2 are red and traffic light 500-3 is green. The host vehicle can be drawn from position information included in the vehicle data.

[0100] The analysis information is also displayed as visual information depicting the driver's line of sight on the map screen M. Figure 7 shows an example in which the line of sight is depicted as an arrow extending from the vehicle toward the pedestrian W-1.

[0101] The analysis information is thus visual information in which the driver's line of sight direction is depicted together with each object on the map screen M, making it easy to analyze the situation using the driver's actual line of sight direction. Although not shown in FIG. 7, the speed of each object may also be displayed.

[0102] The analysis information may have an image display area M1 in which an actual vehicle exterior image and vehicle interior image corresponding to the map screen M are displayed. Although Fig. 7 shows an example in which the map screen M is displayed in 2D (two-dimensional), it may be displayed in 3D (three-dimensional). In this case, a display switching button M2 may be provided, and the display may be switched to 3D by operating the display switching button M2.

[0103] The analysis information may be displayed as visual information depicting changes in the line of sight and changes in the object's position over a certain period of time according to the vehicle data and image analysis results for the certain period of time before and after the event detection time. This makes it possible to easily grasp the changes in the line of sight and the changes in the object's position.

[0104] Specifically, as shown in FIG. 8, the analysis information may be displayed, for example, as an animation, showing the changes in the positions of the host vehicle Vs and the other vehicle V-o1 and the changes in the line of sight D1 from scene SC-1 to scene SC-3.

[0105] This makes it easy to understand the situation in the example of Figure 8, where the driver of vehicle Vs is distracted and ignores the red light of traffic light 500-1 in the direction of travel, resulting in a collision in scene SC-3 with another vehicle V-o1, which is traveling at a green light of traffic light 500-2.

[0106] In addition, in the examples of 2D display shown in Figures 7 and 8, the line of sight direction is represented by an arrow extending from the host vehicle Vs, but in the case of 3D display, the line of sight direction may be represented in a manner appropriate to the 3D display.

[0107] In the case of this 3D display, the line of sight direction may be displayed to indicate a point in 3D space instead of an arrow, as shown in Fig. 9. Fig. 9 shows an example in which the line of sight direction is displayed as a star indicating a point in 3D space. In the case of 3D display, the change in line of sight direction can be shown, for example, by moving the position of the star. In this way, by showing the line of sight direction according to the 3D display, it is possible to judge whether or not the driver has performed a safety confirmation action, such as looking away from the vehicle, including the height direction in 3D space.

[0108] For example, as shown in Figure 9, if it is determined that the driver's line of sight is directed toward the side wall of a building, if there is a signboard or digital signage on this side wall, it will be possible to grasp situations in which the driver has become distracted by these.

[0109] In addition, although examples have been given so far in which the driver's gaze direction is shown as an arrow or a single point, the gaze direction may be shown as a range according to the estimated accuracy of the gaze direction, as shown in Fig. 10. Fig. 10 shows an example of a gaze direction range R1 in the case of 2D display and an example of a gaze direction range R2 in the case of 3D display. By roughly showing the driver's gaze direction in this way, it becomes possible to easily analyze the situation using the driver's actual gaze direction.

[0110] As described above, the server device 100 (corresponding to an example of an "information providing device") according to the embodiment includes a controller 103 that acquires information obtained from outside-vehicle images and inside-vehicle images from a drive recorder 10 (corresponding to an example of an "in-vehicle device") and draws the line-of-sight direction together with an object on a map screen based on the acquired information.

[0111] Specifically, the controller 103 obtains image recognition results from the drive recorder 10, including the type and position of the object obtained from the outside-vehicle image and the driver's line of sight obtained from the inside-vehicle image, and draws the line of sight direction together with the object on the map screen corresponding to the situation to be analyzed based on the image recognition results.

[0112] Therefore, according to the server device 100 of the embodiment, the driver's line of sight obtained from the in-vehicle image is drawn and visualized together with objects obtained from the outside-vehicle image on a map screen corresponding to the situation such as the occurrence of an accident, so that the situation can be easily analyzed using the driver's actual line of sight.

[0113] In the above embodiment, for example, an example of the specific event being an accident is given using Fig. 8, but the event may be when the vehicle reaches a predetermined position as already described. That is, although this embodiment is also related to the example of Fig. 9, when this embodiment is used, it is possible to easily grasp the change in the driver's line of sight when the vehicle reaches a predetermined position where a signboard, digital signage, etc. is provided. That is, this embodiment can be used for marketing analysis of tourist spots, specific stores, facilities, etc., in addition to advertising media such as signboards and digital signage.

[0114] Further advantages and modifications may readily occur to those skilled in the art. Thus, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof. [Explanation of symbols]

[0115] 1. Situation Analysis System 5. Vehicle sensors 10. Drive recorder 11 Communications Department 12 HMI section 13 Sensor section 14 Storage section 15 Controller 100 Server device 101 Communications Department 102 Storage section 103 Controller 200 Operator terminal 201 Communications Department 202 HMI Department 203 Storage section 204 Controller

Claims

1. Acquire information obtained from the outside image and the inside image from an in-vehicle device; a controller for drawing a line-of-sight direction together with an object on a map screen based on the acquired information; An information providing device comprising:

2. The controller: acquiring from the in-vehicle device an image recognition result including a type and a position of the object obtained from the outside-vehicle image and the line-of-sight direction of the driver obtained from the inside-vehicle image; Based on the image recognition result, the line of sight direction is drawn together with the object on the map screen corresponding to the situation to be analyzed.

2. The information providing device according to claim 1.

3. The controller: acquiring the image recognition result from the in-vehicle device when the in-vehicle device detects a predetermined event; 3. The information providing device according to claim 2.

4. The controller: acquiring, from the in-vehicle device, the image recognition results for a predetermined period of time before and after the event is detected; based on the image recognition result, plotting the change in the line of sight direction for the time period on the map screen; 4. The information providing device according to claim 3.

5. The controller: based on the image recognition result, plotting a change in the position of the object over the time period on the map screen; 5. The information providing device according to claim 4.

6. The event is detection of acceleration equal to or greater than a threshold.

4. The information providing device according to claim 3.

7. The event is arrival at a predetermined position.

4. The information providing device according to claim 3.

8. The object includes a vehicle other than the vehicle in which the on-board device is mounted, a motorcycle, a bicycle, a pedestrian, and a traffic light.

3. The information providing device according to claim 2.

9. The image recognition result includes a color of the traffic light, The controller: Drawing the color of the traffic light on the map screen based on the image recognition result. The information providing device according to claim 8.

10. The controller: drawing the line of sight direction in accordance with the 2D display or 3D display of the map screen, respectively; 10. An information providing device according to claim 1.

11. An information providing method executed by a controller, comprising: acquiring information obtained from an outside image and an inside image from an in-vehicle device; Drawing a line of sight direction together with the object on a map screen based on the acquired information; Methods of providing information, including:

Images