Information providing device, information providing method, and program

Abstract

To provide information on a scene to be easily shared with others.SOLUTION: The information providing device is provided with: a detection unit for detecting the behavior of a vehicle; a determination unit for determining whether the behavior of the vehicle driven by a driver is favorable or unfavorable to a predetermined standard based on one or both of the behavior of the vehicle detected by the detection unit and an image captured by an imaging unit for capturing a situation around the vehicle; an extraction unit for extracting a first image taken between a first time when the favorable behavior is performed and a time before a first predetermined time or a second image taken between a second time when the unfavorable behavior is performed and a time before a second predetermined time from the images captured by the imaging unit; and a provision unit for providing a content corresponding to the type of behaviors using the extracted image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information providing device, an information providing method, and a program. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transportation systems that take into consideration vulnerable traffic participants have become more active. To achieve this, we are focusing on research and development into preventive safety technologies to further improve road safety and convenience. Traditionally, awareness-raising activities have been conducted to share undesirable and desirable behaviors when driving vehicles, etc., with others in order to reduce accidents in similar situations, and to encourage desirable behavior. For example, efforts have been made to reduce near-accident situations (near-miss situations). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7021899 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies have sometimes been unable to provide scene information that can be easily shared with others. For example, the information is scattered, making it difficult to identify and provide a scene in which a particular action or event (such as a near miss) occurred, at what location and at what time.

[0005] The present invention has been made in consideration of the above circumstances, and aims to provide an information providing device, an information providing method, and a program that can easily provide information about a situation that can be shared with others, thereby contributing to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The information providing device, the information providing method, and the program according to the present invention employ the following configuration. (1): An information providing device according to one embodiment of the present invention includes a detection unit that detects vehicle behavior; a determination unit that determines whether the driver's driving of the vehicle is desirable or undesirable relative to a predetermined standard based on one or both of the vehicle behavior detected by the detection unit and an image captured by an imaging unit that captures the situation around the vehicle; an extraction unit that extracts, from the images captured by the imaging unit, a first image from a first time when the desirable behavior occurred to a first predetermined time before, or a second image from a second time when the undesirable behavior occurred to a second predetermined time before, and a provision unit that provides content according to the type of behavior using the extracted images.

[0007] (2): In the above aspect (1), the content includes the first image or the second image, and presents a question corresponding to the type of behavior before the behavior is performed, without requiring an answer to the question.

[0008] (3): In the above aspect (1) or (2), the judgment unit judges whether the driving of the vehicle performed by the driver is desirable or undesirable based on a predetermined standard, based on the image captured by the imaging unit and the behavior of the vehicle, and inputs the image and the behavior of the vehicle into a trained model that is trained to output the type of behavior when the image and the behavior of the vehicle are input, and judges whether the driving of the vehicle performed by the driver is desirable or undesirable based on a predetermined standard.

[0009] (4): In the above aspect (1) or (2), a gaze detection unit is provided that detects the direction of the driver's gaze, and the judgment unit judges whether the driving of the vehicle performed by the driver is desirable or undesirable relative to a predetermined standard based on the image captured by the imaging unit, the behavior of the vehicle, and information indicating the gaze direction, and inputs the image, the behavior of the vehicle, and the information indicating the gaze direction into a trained model that has been trained to output the type of behavior when the image, the behavior of the vehicle, and the information indicating the gaze direction are input, and judges whether the driving of the vehicle performed by the driver is desirable or undesirable relative to a predetermined standard.

[0010] (5): In the above aspect (4), the trained model is a model trained from training data, the training data includes the image, the vehicle behavior, information indicating the gaze direction, and correct answer data, and the correct answer data is information indicating the type of behavior corresponding to the combination of the image, the vehicle behavior, and the information indicating the gaze direction.

[0011] (6): In the above aspect (5), the trained model outputs information indicating that the undesirable behavior has been performed if the behavior of the vehicle deviates by a predetermined value or more from a standard based on the situation in the image, or outputs information indicating that the undesirable behavior has been performed if the direction of the gaze deviates by a predetermined value or more from a standard direction based on the situation in the image.

[0012] (7): In the above aspect (1) or (2), the detection unit is mounted on the vehicle or on equipment installed in the vehicle, and the determination unit is provided in a device different from the vehicle.

[0013] (8): In another aspect of the information provision method of the present invention, a computer detects the behavior of a vehicle, and determines whether the driver's driving of the vehicle is desirable or undesirable relative to a predetermined standard based on one or both of the detected behavior of the vehicle and images captured by an imaging unit that captures the situation around the vehicle.The computer then extracts, from the images captured by the imaging unit, a first image from a first time when the desirable behavior occurred to a first predetermined time before, or a second image from a second time when the undesirable behavior occurred to a second predetermined time before, and provides content using the extracted images according to the type of behavior.

[0014] (9): Another aspect of the present invention provides a program that causes a computer to detect vehicle behavior, and, based on one or both of the detected vehicle behavior and images captured by an imaging unit that captures images of the vehicle's surroundings, determine whether the driver's driving of the vehicle is desirable or undesirable relative to a predetermined standard; extract, from the images captured by the imaging unit, a first image from a first time when the desirable behavior occurred to a first predetermined time before, or a second image from a second time when the undesirable behavior occurred to a second predetermined time before, and provide content according to the type of behavior using the extracted images. [Effects of the Invention]

[0015] According to the aspects (1) to (9), by providing content according to the type of driver's behavior, it is possible to provide scene information that can be easily shared with others. For example, if the driving behavior of the driver is negative, the content provided based on the type of driving behavior of the driver can encourage the driver to be conscious of changing their behavior to improve driving, and if the driving behavior is positive, it can be expected to foster an awareness of more considerate driving toward other traffic participants.

[0016] According to (2), driving behavior scenes are provided as videos or images starting from a position a predetermined time before the occurrence of the event, and related questions are output in time with the playback of the driving behavior scenes. However, since the system does not require the driver to answer the questions, it is possible to reduce the burden on the driver, and by not forcing the driver to answer correctly, it is expected to have a coaching effect that draws out spontaneous behavior from the driver. In the case of negative driving behavior, the questions encourage behavioral change, and in the case of positive driving behavior, the questions provide incentives.

[0017] According to (3), the acquired data can be used to understand what kind of driving behavior the driver exhibited (for example, whether it was risky or exemplary).

[0018] According to (4) or (5), the type of behavior can be determined more accurately by also using the driver's line of sight.

[0019] According to (6), a trained model is constructed to output an appropriate type of behavior depending on the vehicle's behavior or gaze direction, making it possible to determine the type of behavior more accurately.

[0020] According to (7), the detection unit is mounted on the vehicle or an equipment installed in the vehicle, and the judgment unit is provided in a device different from the vehicle, thereby reducing the processing load on the vehicle or the equipment installed in the vehicle. [Brief explanation of the drawings]

[0021] [Figure 1] 2 is a diagram illustrating an example of a functional configuration of the information providing device 1. FIG. [Figure 2] A diagram for explaining the information input to the trained model 194 and the information output by the trained model 194. [Figure 3] 10 is a flowchart showing an example of a flow of processing executed by the extraction device 100. [Figure 4]10 is a flowchart showing an example of the flow of processing executed by the provision server 200. [Figure 5] FIG. 2 is a diagram illustrating video content provided by a providing server 200. [Figure 6] FIG. 2 is a diagram illustrating video content provided by a providing server 200. [Figure 7] 10 is a diagram showing an example of the contents of extracted data 260 stored in a storage unit 250 of a provision server 200. FIG. [Figure 8] 10 is a flowchart showing an example of the flow of processing executed by the provision server 200. [Figure 9] FIG. 10 is a diagram illustrating an example of information provided to a user. [Figure 10] FIG. 10 is a diagram illustrating an example of learning data. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, an information providing apparatus, an information providing method, and a program according to embodiments of the present invention will be described with reference to the drawings.

[0023] FIG. 1 is a diagram illustrating an example of the functional configuration of an information providing device 1. The information providing device 1 includes, for example, an extraction device 100, a provision server 200, a learning device 300, and a user terminal device 400. Some or all of the functional configuration included in the extraction device 100, some or all of the functional configuration included in the provision server 200, or some or all of the functional configuration included in the learning device 300 may be included in other devices. For example, some or all of the functional configuration included in the extraction device 100 may be included in the functional configuration included in the provision server 200, or vice versa. The functional configuration of the information providing device may be distributed across multiple devices or may be included in a single device.

[0024] [Extraction device] The extraction device 100 is, for example, a device mounted on or installed in a vehicle. The vehicle may be, for example, a four-wheeled vehicle, a motorcycle, or a micromobile. The following description will be given for a four-wheeled vehicle. The extraction device 100 may also be a drive recorder or a user's smartphone installed in the vehicle.

[0025] The extraction device 100 includes, for example, an imaging unit 110, a state detection unit 120, a gaze detection unit 130, a determination unit 140, an extraction unit 150, a transmission control unit 160, and a storage unit 180. Of these functional components, the state detection unit 120, the gaze detection unit 130, the determination unit 140, the extraction unit 150, and the transmission control unit 160 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or an SOC (System On Chip), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

[0026] The storage unit 180 may be realized by the various storage devices described above, or a solid-state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), or a random-access memory (RAM). The storage unit 180 stores, for example, image data (video data) 192 and a trained model 194. The image data is image data (video) captured by the imaging unit 110. The trained model 194 will be described later.

[0027] The image capturing unit 110 is a camera that captures images of the surroundings of the vehicle. For example, the image capturing unit 110 captures images of the surroundings in any direction (for example, ahead) of the vehicle.

[0028] The state detection unit 120 detects the state of the vehicle. The state of the vehicle is, for example, the acceleration, deceleration, yaw rate, etc. of the vehicle. The state detection unit detects the above state based on the detection results of sensors of the vehicle or a terminal device provided in the vehicle. In other words, the state of the vehicle can be said to be information indicating the driving state of the driver. The state of the vehicle may be time-series data (similarly during learning).

[0029] The gaze detection unit 130 detects the direction of the driver's gaze. The gaze detection unit identifies the driver's line of sight from an image captured by a driver monitor camera installed in the vehicle, and detects the direction of the driver's gaze based on the identified result. The direction of the driver's gaze is detected by a known method.

[0030] The determination unit 140 determines whether the driver's driving of the vehicle is desirable or undesirable based on either or both of the vehicle behavior and the images captured by the imaging unit. The determination unit 140 may make the determination using the trained model 194, as described below, or may make the determination by analyzing either or both of the vehicle behavior and the images captured by the imaging unit. In the latter case, the determination unit 140 makes the above determination based on whether the vehicle behavior (degree of change in acceleration or yaw rate) deviates from the predetermined standard by a predetermined degree or more. The determination unit 140 also makes the above determination by analyzing the images using a known image analysis algorithm. The determination unit 140 combines these determinations to determine whether the behavior is desirable or undesirable. One or both of the determination unit 140 and the extraction unit 150, described below, may be provided in a device (e.g., the provider server 200) separate from the vehicle.

[0031] The extraction unit 150 extracts a first image (specifically, a video) from a first time when a desirable behavior occurred to a first predetermined time before, or a second image (specifically, a video) from a second time when an undesirable behavior occurred to a second predetermined time before, from the images captured by the imaging unit 110. Details of the processing by the determination unit 140 and the extraction unit 150 will be described later.

[0032] The transmission control unit 160 provides the video (image) extracted by the extraction unit 150 to the provision server 200. The transmission control unit 160 associates the above information, such as the weather, time, and season when the video was captured, with the video and transmits it to the provision server 200. The weather may be estimated by the extraction device 100 from the captured image, or may be obtained from another device.

[0033] [Providing server] The provision server 200 includes, for example, an information processing unit 210 and a storage unit 250. The information processing unit 210 is realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, GPU, or SOC, or may be realized by a combination of software and hardware.

[0034] The information processing unit 210 generates video content (or image content) using the video provided by the extraction device 100. The information processing unit 210 generates video content by, for example, adding a caption containing a predetermined text to the video. The information processing unit 210 provides the generated video content to the user's terminal device 400. Details of this process will be described later.

[0035] The storage unit 250 stores extracted data 260. The extracted data 260 is data extracted by the extraction device 100 (details will be described later).

[0036] The processing performed by the learning device 300 will be described later.

[0037] [Trained model] 2 is a diagram for explaining information input to trained model 194 and information output by trained model 194. When trained model 194 receives an image captured by imaging unit 110, the state of the vehicle according to the timing at which the image was captured, information indicating the direction of the driver's line of sight at the timing, and environmental information (such as the weather and date and time at which the image was captured), trained model 194 outputs information indicating the type of driver's behavior at the timing according to the input information.

[0038] The vehicle state refers to the vehicle's behavior, such as the vehicle's longitudinal acceleration, lateral acceleration, yaw rate, and the degree of change thereof. More specifically, the vehicle state refers to an index indicating the degree of deviation between an index indicating the vehicle's behavior acquired a predetermined time ago and an index indicating the vehicle's behavior acquired this time (an index indicating the degree of change in the index). The weather information may be estimated by the extraction device 100 from an image, or may be acquired from another device.

[0039] The information indicating the type of driver's behavior is, for example, information indicating whether the driver's behavior is desirable or undesirable. Specifically, it is a type indicating whether the driver's driving caused a near-miss event. The trained model 194 may output information indicating other types in addition to the above-mentioned types. The other types are types (e.g., other) that are different from the desirable and undesirable types of driver's behavior.

[0040] [Flowchart (1)] 3 is a flowchart showing an example of a flow of processing executed by the extraction device 100. For example, the processing of this flowchart is repeatedly executed at predetermined intervals.

[0041] First, the extraction device 100 acquires various information (images, vehicle states, and information indicating gaze directions) (step S100). Next, the extraction device 100 inputs the acquired information into the trained model 194 and acquires the results output by the trained model 194 (step S102).

[0042] Next, the extraction device 100 determines whether or not the driver is behaving undesirably based on the information acquired in step S102 (step S104). If the determination in step S104 is negative, the extraction device 100 determines whether or not the driver is behaving undesirably based on the information acquired in step S102 (step S106). If the determination in step S106 is negative, the processing of one routine of this flowchart ends.

[0043] If the determination in step S104 is positive, or if the processing in step S106 is positive, the extraction device 100 extracts videos captured at the timing at which it is determined that an undesirable behavior occurred or the timing at which it is determined that a desirable behavior occurred (step S108). For example, the extraction device 100 extracts videos from the above timing up to a predetermined time before. Next, the extraction device 100 transmits the extracted videos to the provision server 200. The extraction device 100 may acquire, along with the videos, information such as location information, weather information, and time at which the videos were captured. The extraction device 100 acquires location information from a location identification unit that identifies location information or from another device. The extraction device 100 may acquire weather information from another device, or may estimate the weather based on images captured by its own device.

[0044] For example, when a series of desirable behaviors are performed, extraction device 100 may extract a video from the timing of the last desirable behavior (for example, the timing when a vehicle stops at a crosswalk to allow a pedestrian to cross the crosswalk) to a predetermined time before. For example, when a series of undesirable behaviors are performed, extraction device 100 may extract a video from the timing of the last undesirable behavior (the timing of the last behavior in the series of behaviors within a predetermined time (image IM2 in FIG. 5, which will be described later)). This completes the processing of one routine of this flowchart.

[0045] In this way, the extraction device 100 can automatically extract videos in which desirable behaviors are performed or videos in which undesirable behaviors are performed, thereby helping to provide scene information that can be easily shared with others.

[0046] [Flowchart (2)] FIG. 4 is a flowchart showing an example of the flow of processing executed by the providing server 200. First, the providing server 200 selects a predetermined video from the extracted data 260 (step S200). Next, the providing server 200 assigns a question to the selected video (step S202). Next, the providing server 200 generates video content based on the video to which the question has been assigned, and provides the generated video content to the user's terminal device 400 (step S204). The above video content is content that presents a question corresponding to the type of behavior before the behavior is performed, and does not require an answer to the question (see FIGS. 5 and 6 described below). This ends the processing of one routine of this flowchart.

[0047] [Example of video content provided to users (1)] FIG. 5 is a diagram illustrating video content provided by the provider server 200. The video content in FIG. 5 is video content about undesirable behavior. A question appears in caption within the video (in IM1) a predetermined time after the video starts. The video may be stopped for a predetermined time when the caption appears. At this time, the user can think about the question. For example, a caption may appear saying, "You are about to turn right at an intersection. What should you pay attention to at this time?" After that, a video (IM2) is played in which an electric scooter approaches straight ahead just as the vehicle is about to turn right, and the vehicle suddenly brakes, and the video ends.

[0048] [Example of video content provided to users (2)] FIG. 6 is a diagram for explaining video content provided by the provision server 200. The video content in FIG. 6 is video content of desirable behavior. After the video starts, a question appears in caption within the video (in IM3) a predetermined time later. The video may be stopped for a predetermined time when the caption appears. At this time, the user can think about the question. For example, a caption may appear saying, "You are about to cross a pedestrian crossing. What should you pay attention to at this time?" After that, a video (IM4) of a vehicle stopping in front of a pedestrian crossing is played, and the video ends.

[0049] In this way, the provision server 200 can provide users with video content that is useful for drivers. For example, by referring to this video content, drivers can learn how to drive better and what they should pay attention to when driving.

[0050] Here, the information processing unit 210 of the provision server 200 may generate video content by assigning different questions for each type of video. For example, the information processing unit 210 may input a video (one or more images) into a trained model (not shown) and assign a question based on the type of video output by the trained model. The trained model is a trained model that has been trained to output a type corresponding to the input video when a video (more than one image) is input. The trained model is a model trained using training data. The training data is, for example, information in which the type of video and the video are associated with each other.

[0051] For example, correspondence information is prepared in advance. The correspondence information is information in which the type of video is associated with a question. The information processing unit 210 refers to the correspondence information and assigns a question to the video that corresponds to the type of video output by the trained model. For example, the question "Can you see the surrounding vehicles?" may be assigned to a video of another vehicle changing lanes ahead, or the question "Is there anything hiding in the blind spot of the oncoming vehicle?" may be assigned to a video of a motorcycle appearing from the blind spot of an oncoming vehicle.

[0052] The correspondence information may also be information in which the type of video, information on the environment (weather, date and time such as day or night, season, road surface condition, etc.), and questions are associated with each other. The road surface condition is, for example, information acquired by the providing server 200 by analyzing an image. In this case, the information processing unit 210 assigns different questions to videos for each environment, even if the video type (event) is the same. In this way, the providing server 200 can automatically assign questions according to the type of video (according to the event).

[0053] [Another example of the provider server's processing] In addition to providing the questions as described above, the provision server 200 may provide the user with video content that corresponds to the route that the user plans to travel. This process will be described below.

[0054] FIG. 7 is a diagram showing an example of the contents of the extracted data 260 stored in the storage unit 250 of the providing server 200. This is information that associates video content with weather, time zones, seasons, and locations. The weather, time zones, and seasons correspond to the video content associated with them. The weather, time zones, and seasons are information transmitted by the extraction device 100 together with the video content. The providing server 200 generates video content by adding questions to the extracted data 260 as described above or below, and provides the video content to the user.

[0055] [Flowchart (3)] 8 is a flowchart showing an example of the flow of processing executed by the providing server 200. First, the providing server 200 determines whether or not a route has been acquired from the user's terminal device 400 (step S300). If a route has been acquired from the user's terminal device 400, the providing server 200 searches the extracted data 260 for video content corresponding to a position along the route and a planned driving situation (step S302). For example, the providing server 200 searches for video content of a position along the route along which the user plans to drive, and that matches the expected weather, time zone, and season when the user will be driving.

[0056] Next, the providing server 200 determines whether or not video content exists as a result of the search (step S304). If video content does not exist, the processing of one routine of this flowchart ends. If video content exists, the providing server 200 provides the video content to the user's terminal device (step S306). This ends the processing of one routine of this flowchart.

[0057] [Video content provided to users] Through the above process, content according to the user's route and situation is provided to the user, as shown in Fig. 9. For example, when a user searches for a route search service to travel a specific route to a destination on a specific date and time, the providing server 200 provides the user's terminal device 400 with video content that matches the time zone, weather, and season of the specific date and time and is located at a position according to the route. At this time, the providing server 200 provides the user's terminal device 400 with information indicating the existence of video content associated with the location of the route.

[0058] In the example shown in the upper diagram of Fig. 9, for example, the user is heading to a destination during the day, so daytime video content is provided to the user. In the example shown in the lower diagram of Fig. 9, for example, the user is heading to a destination at night in the rain, so nighttime video content in the rain is provided to the user.

[0059] In this way, the providing server 200 can provide video content that is suited to the driving situation of the user.

[0060] [About the learning device] The learning device 300 generates a trained model 194 and provides the generated trained model 194 to the extraction device 100. The learning device 300 generates the trained model 194 by training a pre-training model (a machine learning model such as a neural network) with training data. FIG. 10 is a diagram showing an example of training data. The training data is information in which correct answer data (desirable or undesirable behavior) is associated with information combining an image of the vehicle's surroundings, the vehicle's state (degree of change in acceleration or yaw rate), information indicating the direction of gaze, and environmental information (weather, date, time, season). When the combined information is input to the model, the learning device 300 trains the model to output information indicating the correct answer data associated with the combined information, thereby generating the trained model 194.

[0061] The trained model 194 outputs information indicating that an undesirable behavior has been performed when the behavior of the vehicle deviates by a predetermined value or more from a standard based on the situation in the image. The trained model 194 outputs information indicating that an undesirable behavior has been performed when the direction of the gaze deviates by a predetermined value or more from a standard direction based on the situation in the image.

[0062] For example, if the driver's gaze is not directed at the electric scooter and the driver is not slowing down, a label indicating undesirable behavior is associated with the combined information of image IM1 in Figure 5, the vehicle state, information indicating the direction of gaze, and environmental information. When the combined information is input, the pre-training model is trained to output information indicating undesirable behavior.

[0063] For example, if the driver's gaze is directed at the electric scooter but the vehicle is suddenly decelerating, a label indicating undesirable behavior is associated with the combined information of image IM2 in Figure 5, information indicating the direction of gaze, and environmental information. When the combined information above is input, the pre-training model is trained to output information indicating undesirable behavior.

[0064] For example, if the driver's gaze is directed at a pedestrian and the driver is decelerating, a label indicating a desirable behavior is associated with the combined information of image IM3 in Figure 6, the vehicle state, information indicating the gaze direction, and environmental information. When the combined information is input, the pre-training model is trained to output information indicating that the behavior is desirable.

[0065] For example, in the combined information of image IM4 in Figure 6, the vehicle state, information indicating the gaze direction, and environmental information, if the driver's gaze is directed at a pedestrian, the speed is zero, and the vehicle is stopped, a label of desirable behavior is associated. When the combined information above is input, the pre-training model is trained to output information indicating desirable behavior.

[0066] In the above example, the combination information has been described as a combination of an image, a vehicle state, information indicating a gaze direction, and environmental information, but some of this information may be omitted. For example, the image or the gaze direction may be omitted. In this case, information in which the combination information that does not include the omitted information is associated with correct answer data is trained as training data, and a trained model 194 is generated. Then, the extraction device 100 executes processing using the trained model 194. At this time, the information input to the trained model 194 is combination information that does not include the omitted information.

[0067] As described above, the learning device 300 uses the learning data to generate the trained model 194. In this way, the learning device 300 can generate a trained model that more accurately outputs whether the driver has performed a desirable or undesirable behavior.

[0068] For example, although there are near-miss videos out there, the information is scattered, and the details of where the incident occurred are not known, meaning there is insufficient information to use in terms of accident prevention when actually passing through that location. Also, if the video is old, the road conditions may have changed and the video may not be usable (previous near-miss videos can still be used for education purposes for their universal aspects, but they are not suitable for use in messages such as "This accident occurred at this location and this time, so be careful when driving!").

[0069] By automatically extracting near misses from videos and generating quizzes such as "What happened next?" and storing them in the provision server 200, it is possible to continue providing real and hot video content. These videos can be accessed from the route search screen. The provision server 200 deletes old videos and updates them as needed, but may also leave videos with a high number of views.

[0070] The information providing device 1 can support people who are eager to learn (such as novice drivers or people who have had serious accidents in the past) (voluntary social contribution). The information providing device 1 may also cooperate with a navigation device to allow users to view videos of past near misses that have occurred along the route they are traveling (since it is not known where dangerous points are when driving on a road for the first time, this allows users to prepare in advance and feel at ease). For example, when searching for a route, the information providing device 1 displays videos of near misses that have occurred along that route in the past. For example, when searching for a route, the user can perform a predetermined operation to view videos of near misses that actually occurred at the target location, allowing them to learn about specific dangers before traveling.

[0071] According to the embodiment described above, the information providing device 1 provides video content according to the type of activity, thereby making it possible to easily provide information on scenes to be shared with others.

[0072] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Detects vehicle behavior, determining whether the driver's driving of the vehicle is a desirable or undesirable behavior relative to a predetermined standard based on one or both of the detected behavior of the vehicle and an image captured by an imaging unit that captures an image of the surroundings of the vehicle; extracting, from the images captured by the imaging unit, a first image taken from a first time when the desirable behavior was performed to a first predetermined time before, or a second image taken from a second time when the undesirable behavior was performed to a second predetermined time before, Execute a process of providing video content according to the type of the action using the extracted image. Device.

[0073] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0074] 1 Information provision device 100 extraction equipment 110 Imaging unit 120 Status detection unit 130 Gaze detection unit 140 Judgment section 150 Extraction part 160 Transmission control section 192 Image data (video data) 194 trained models 200 Provided Server 210 Information Processing Department 260 Extracted Data 300 Learning Device

Claims

1. a detection unit that detects the behavior of a vehicle; a determination unit that determines whether the driver's driving of the vehicle is a desirable or undesirable behavior with respect to a predetermined standard, based on one or both of the behavior of the vehicle detected by the detection unit and an image captured by an imaging unit that captures an image of the surroundings of the vehicle; and an extracting unit that extracts, from the images captured by the imaging unit, a first image taken from a first time when the desirable behavior was performed to a first predetermined time before, or a second image taken from a second time when the undesirable behavior was performed to a second predetermined time before, a providing unit that provides content according to the type of the action using the extracted image; An information providing device comprising:

2. the content includes the first image or the second image, and presents a question corresponding to the type of the behavior before the behavior is performed, without requiring an answer to the question; The information providing device according to claim 1 .

3. the determination unit determines whether the driving of the vehicle performed by the driver is a desirable behavior or an undesirable behavior with respect to a predetermined standard, based on the image captured by the imaging unit and the behavior of the vehicle; inputting the image and the vehicle behavior into a trained model that has been trained to output a type of behavior when the image and the vehicle behavior are input, and determining whether the driving of the vehicle performed by the driver is a desirable or undesirable behavior with respect to a predetermined standard; 3. The information providing device according to claim 1.

4. a gaze detection unit that detects the direction of the driver's gaze, the determination unit determines whether the driving of the vehicle performed by the driver is a desirable behavior or an undesirable behavior with respect to a predetermined standard based on the image captured by the imaging unit, the behavior of the vehicle, and information indicating the direction of the line of sight; inputting the image, the vehicle behavior, and the information indicating the gaze direction into a trained model that has been trained to output the type of behavior when the image, the vehicle behavior, and the information indicating the gaze direction are input, and determining whether the driving of the vehicle performed by the driver is a desirable behavior or an undesirable behavior with respect to a predetermined standard; 3. The information providing device according to claim 1.

5. The trained model is a model trained using training data, the learning data includes the image, the behavior of the vehicle, information indicating the line of sight direction, and correct answer data; The correct answer data is information indicating a type of behavior corresponding to a combination of the image, the behavior of the vehicle, and information indicating the line of sight.

5. The information providing device according to claim 4.

6. The trained model is If the behavior of the vehicle deviates from a standard based on the image situation by a predetermined value or more, outputting information indicating that the undesirable behavior has occurred; or outputting information indicating that the undesirable behavior has been performed when the direction of the line of sight deviates by a predetermined value or more from a reference direction based on the situation of the image; The information providing device according to claim 5 .

7. the detection unit is mounted on the vehicle or on equipment installed in the vehicle, The determination unit is provided in a device different from the vehicle.

3. The information providing device according to claim 1.

8. The computer Detects vehicle behavior, determining whether the driver's driving of the vehicle is a desirable or undesirable behavior relative to a predetermined standard based on one or both of the detected behavior of the vehicle and an image captured by an imaging unit that captures an image of the surroundings of the vehicle; extracting, from the images captured by the imaging unit, a first image taken from a first time when the desirable behavior was performed to a first predetermined time before, or a second image taken from a second time when the undesirable behavior was performed to a second predetermined time before, providing content according to the type of the action using the extracted image; Information processing methods.

9. On the computer, Detecting vehicle behavior, Based on one or both of the detected behavior of the vehicle and an image captured by an imaging unit that captures an image of the surroundings of the vehicle, the driving behavior of the vehicle performed by the driver is judged to be desirable or undesirable with respect to a predetermined standard; extracting, from the images captured by the imaging unit, a first image taken from a first time when the desirable behavior was performed to a first predetermined time before, or a second image taken from a second time when the undesirable behavior was performed to a second predetermined time before, providing content according to the type of the action using the extracted image; program.

Images