Driving assistance device, vehicle, driving assistance system, driving education system, and driving assistance method

Abstract

Provided is a driving assistance device for automatically generating advice information for assisting a driving operation of a driver on the basis of sensor data from the time of operation.　A driving assistance device for generating advice information for assisting a driving operation of a driver, the driving assistance device being characterized by having: a sensor data reception unit 1 for receiving sensor data from a sensor for measuring the surroundings of a host vehicle on which a driver rides; a target information generation unit 2 for generating target information pertaining to a target included in the sensor data; a map information acquisition unit 3 for acquiring map information in a travel environment of the host vehicle; a driving-operation-related information generation unit 5 for generating, by using the map information and the target information, driving-operation-related information having high relevance related to advice information to be reported to the driver; an advice information generation unit 6 for generating, on the basis of the driving-operation-related information, advice information to be reported to the driver; and a notification unit 7 for notifying the driver regarding the advice information generated by the advice information generation unit.

Description

Driving assistance device, vehicle, driving assistance system, driving education system, and driving assistance method

[0001] The present invention relates to a driving assistance device, a vehicle, a driving assistance system, a driving education system, and a driving assistance method that assist a driver in driving operations.

[0002] Regular safety training for drivers in the logistics and transportation industries includes on-board instruction and instruction using dashcams.

[0003] On-board instruction is an educational method in which a safety instructor, such as an experienced driver, rides in the vehicle driven by the driver (trainee) to evaluate the driver's driving and provide guidance and advice during or after the drive.

[0004] In addition, instruction using a drive recorder is an educational method in which a safety instructor reviews the footage from the drive recorder installed in the vehicle, evaluates the driver's driving using a driving simulator, etc., and provides guidance and advice.

[0005] However, in the logistics and transportation industries, one safety instructor is responsible for instructing and advising multiple drivers, which can lead to the following problems:

[0006] In on-board instruction, the safety instructor has to ride in each vehicle driven by multiple drivers, which places a heavy burden on the safety instructor. In addition, drivers tend to be more conscious of safe driving during on-board instruction, which may prevent the safety instructor from noticing driving behavior that needs to be corrected during normal work, and may prevent them from providing sufficient instruction and advice.

[0007] In addition, when using dashcams for guidance, the need to consistently evaluate footage of a large number of drivers places a heavy burden on safety instructors. Furthermore, it is not realistic to extract all driving operations that need improvement from the footage and then provide guidance and advice.

[0008] Therefore, in order to reduce the burden on safety instructors during safety training, a device has been proposed that automates part of the safety training using data from sensors installed in vehicles.

[0009] For example, Japanese Patent Application Laid-Open No. 2022-32244 (hereinafter referred to as Patent Document 1) provides a device and method that allows the safety know-how of multiple instructors to be easily shared with drivers by registering data recorded while driving, searching for similar scenes from past cases accumulated in a database, and displaying subjective information such as the instructor's knowledge and safety know-how registered for similar scenes in synchronization with camera footage of similar scenes, environmental information, and vehicle control information.

[0010] Japanese Patent Application Laid-Open No. 2022-32244

[0011] However, in Patent Document 1, because subjective information registered for similar scenes in a database is displayed, it does not take into consideration automatically generating advice to improve driving operations for the traffic scene that the driver actually drove in. Also, while it takes into consideration a form in which a safety instructor provides education after driving, it does not take into consideration a form in which support for driving operations or advice on areas for improvement is given to the driver while driving.

[0012] In addition, searching for similar scenes may extract scenes that differ from the perspective of how the driver should improve their driving, and may display subjective information that is not optimal for the effectiveness of driver education.

[0013] The present invention addresses at least one of the above-mentioned problems, and aims to provide a driving assistance device, a vehicle, a driving assistance system, a driving education system, and a driving assistance method that automatically generate advice information to assist the driver in driving operations based on sensor data during driving.

[0014] The present invention is characterized by having a sensor data receiving unit that receives sensor data from a sensor that measures the surroundings of the vehicle in which the driver is riding, a target object information generating unit that generates target object information for targets included in the sensor data, a map information acquiring unit that acquires map information of the driving environment of the vehicle, a driving operation related information generating unit that uses at least the map information and target object information to generate driving operation related information that is highly relevant to advice information to be notified to the driver in the driver's driving operation, an advice information generating unit that generates advice information to be notified to the driver based on the driving operation related information, and a notification unit that notifies the driver of the advice information generated by the advice information generating unit.

[0015] According to the present invention, advice information for assisting a driver in driving operations can be automatically generated based on sensor data during driving.

[0016] 7 is a configuration diagram showing a first system configuration of a sensor and a driving assistance device. FIG. 8 is a configuration diagram showing a second system configuration of a sensor and a driving assistance device. FIG. 9 is a configuration diagram showing a third system configuration of a sensor and a driving assistance device. FIG. 10 is a block diagram showing functional blocks of a driving assistance device according to an embodiment of the present invention. FIG. 11 is a block diagram showing functional blocks of a target object information generation unit shown in FIG. 2. FIG. 12 is an explanatory diagram illustrating an example of an image of sensor data in a sensor data reception unit. FIG. 13 is an explanatory diagram illustrating an example of output of a target object recognition unit shown in FIG. 3. FIG. 14 is an explanatory diagram illustrating an example of output of a first target object information generation unit shown in FIG. 3. FIG. 15 is an explanatory diagram illustrating an example of output of a second target object information generation unit shown in FIG. 3. FIG. 16 is an explanatory diagram illustrating an example of output of a target object information integration unit shown in FIG. 3. FIG. 17 is a block diagram showing functional blocks of a map information generation unit shown in FIG. 2. FIG. 18 is a block diagram showing functional blocks of a driving operation related information generation unit shown in FIG. 2. FIG. 19 is an explanatory diagram illustrating an example output (relevance) of an relevance degree calculation unit shown in FIG. 7. FIG. 19 is a block diagram showing functional blocks of an advice information generation unit shown in FIG. 2. FIG. 19 is an explanatory diagram illustrating a first form of information presentation to a driver in a notification unit shown in FIG. 2. FIG. 19 is an explanatory diagram illustrating a second form of information presentation to a driver in a notification unit shown in FIG. 2.

[0017] The present invention will be described in detail with reference to the drawings, but the present invention is not limited to the following embodiments and includes various modifications and applications within the technical concept of the present invention. Specific embodiments of the present invention will be described below with reference to the drawings.

[0018] [Description of System Configuration of Driving Assistance Device] First, a typical system configuration of the present invention will be briefly described with reference to FIGS. 1A to 1C.

[0019] 1A is a diagram showing the relationship (first system configuration) between a vehicle 120 such as a passenger car or truck, various sensors 110, and a driving assistance device 100. In this embodiment, the driving assistance device 100 and various sensors 110 are mounted on the vehicle 120. This is a system intended for trucks in the logistics industry, taxis in the transportation industry, and the like.

[0020] FIG. 1B is a diagram showing the mutual relationships (second system configuration) between the vehicle 120, various sensors 110, and the driving assistance device 100. In this embodiment, the various sensors 110 are mounted on the vehicle 120, while the driving assistance device 100 is provided in an external data control center and connected to the vehicle 120 via wireless communication. This is also a system intended for trucks in the logistics industry and taxis in the transportation industry. FIG. 1C is a diagram showing the mutual relationships (third system configuration) between the vehicle 120, various sensors 110, and the driving assistance device 100. In this embodiment, data to the driving assistance device 100 is provided using a sensor data storage medium (such as an SD card). This is a system intended for safety training centers in the logistics or transportation industries.

[0021] The various sensors 110 may be external sensors such as cameras and LiDAR, but may also be other vehicle sensors that detect the vehicle's running state, or external sensors such as cameras installed on roads, etc. Therefore, hereinafter, they may be simply referred to as "sensors."

[0022] As shown in FIG. 1A, the driving assistance device 100 is configured to receive data from a sensor 110 installed in a vehicle 120. The "data" is mainly image data obtained from a camera, a LIDAR, or the like, but other data can also be utilized. Specific examples of the "data" will also be given in the explanation of the driving assistance device 100 below. The same applies to FIGS. 1B and 1C described below.

[0023] The driving assistance device 100 includes a processing unit 10, a storage unit 11, an input unit 12, a display unit 13, an audio output unit 14, a communication unit 15, etc. The driving assistance device 100 in Fig. 1A is mounted on a vehicle 120, and enables a driver to receive driving assistance while performing driving operations.

[0024] The processing unit 10 constituting the driving assistance device 100 is a central processing unit (CPU) that executes various programs stored in RAM, HDD, etc. The storage unit 11 is an HDD, etc., that stores various data for the driving assistance device 100 to execute processes. The input unit 12 is a device for inputting instructions to a computer, such as a keyboard or mouse, and inputs instructions such as program startup.

[0025] The display unit 13 is a display or the like, and displays the execution status and execution results of processing by the driving assistance device 100. The audio output unit 14 is a speaker or the like, and outputs the execution results of processing by the driving assistance device 100 by audio. The communication unit 15 is a device that exchanges various data and commands with other devices via a network. The driving assistance devices 100 shown in Figures 1B and 1C have substantially the same configuration.

[0026] 1B, the driving assistance device 100 may receive data from the sensor 110 transmitted by wireless communication via an edge device 130. The edge device 130 refers to a point (network terminal device) that transmits data collected between a terminal and a network on the terminal side to a line.

[0027] The driving assistance device 100 is installed in an external data control center, and advice information for assisting the driver in driving is sent via wireless communication to an information notification device with a display function installed in the vehicle. An example of the information notification device is a navigation device.

[0028] The vehicle information notification device notifies the driver of advice information and includes a display unit 16 and a voice output unit 17. It also includes a communication unit 18 for wireless communication with a communication unit 15 of a data control center. The data control center may be a traffic management center, and may be a server operated by a company.

[0029] 1C , the driving assistance device 100 may acquire data from the sensor 110 via a sensor data storage medium 140 such as an SD card. The driving assistance device 100 is installed in a safety education center and can be used to review the driver's driving operation after driving.

[0030] In other words, the sensor data stored in the sensor data storage medium 140 is input into the driving assistance device 100, advice information on driving operations is created from the sensor data, and driving operations are reviewed, thereby providing education to raise safety awareness.

[0031] In this manner, the driving assistance device 100 shown in FIG. 1A is provided in the vehicle 120, while the driving assistance devices 100 shown in FIGS. 1B and 1C are configured to be disposed outside the vehicle 120.

[0032] By using the system configuration of Figures 1A and 1B, the driving assistance device 100 can provide driving assistance by automatically generating advice information to assist and improve driving operations in real time while the driver is driving.

[0033] In addition, by using the system configuration of FIG. 1C, the driving assistance device 100 can analyze sensor data after the driver has driven, automatically generating advice information to improve driving operations while driving, and allowing the driver to "reflect" on their driving operations.

[0034] [Description of Driving Assistance Device 100] Next, the configuration of the driving assistance device 100 will be described. In the following, the description will be given using the system shown in FIG. 1A as an example.

[0035] The driving assistance device 100 shown in Fig. 2 is a device that automatically generates advice information for assisting and improving the driver's driving operation by using data acquired from a sensor 110. Each functional unit in Fig. 2 is realized by a processing unit 20. Furthermore, the storage unit 11 shown in Fig. 1A stores driving knowledge 201, map information 202, inference parameters 203, etc., which will be described later. In the following description, the vehicle 120 is referred to as the "own vehicle" driven by the driver.

[0036] The sensor data receiving unit 1 has a function of acquiring various data necessary for generating advice information from sensors 110 mounted on the host vehicle 120. In the example of this embodiment, the host vehicle is a standard automobile, and the sensor information input to the sensor data receiving unit 1 includes image information from an external camera that captures images of the external world, image information from an internal camera that captures images of the interior of the vehicle, which are attached to the host vehicle, position and orientation information from a GNSS (Global Navigation Satellite System) that acquires position and orientation parameters of the host vehicle, and vehicle information from a vehicle sensor that acquires vehicle data via a CAN (Controller Area Network). Of course, it goes without saying that the information acquired in this embodiment is not limited to these examples.

[0037] The vehicle may be a compact car, a light car, a large special-purpose vehicle, a small special-purpose vehicle, a motorized bicycle, etc. As the type of camera for capturing images of the external and internal worlds (inside the vehicle), a monocular camera, a stereo camera, or an infrared camera may be used to acquire data such as images, distance images, and infrared images, or other images may be used. Instead of an external camera, a LiDAR (Light Detection and Ranging) may be used to acquire data such as distance and three-dimensional point clouds, or may be used in combination with an external camera.

[0038] The target information generating unit 2 has a function of estimating one or more pieces of information from the type, position, speed, orientation, behavior, and characteristics of external targets (e.g., people, vehicles, features, etc.) during driving using various data acquired from the sensor 110, and generating target information (see FIG. 5) that integrates this information.

[0039] The map information generation unit 3 has the function of estimating one or more pieces of information from the road information (road classification, legal speed limit, road width, etc.) during driving, traffic information (traffic congestion, construction, road regulations, etc.), and weather information (weather, road surface conditions, etc.) using data acquired from the sensor 110 or map information 202 described later, and generating map information that integrates this information.

[0040] The additional information acquisition unit 4 has a function of acquiring additional information, which is information about the driver or the driving operation of the driver. In this embodiment, the additional information is specifically one or more of the following information: the driver's driving tendency, past operation information such as past inappropriate driving operations, and current operation information such as inappropriate driving operations by the driver that occurred while driving.

[0041] The driving operation related information generation unit 5 has the function of generating driving operation related information that indicates the degree of relevance of the target information from the target information generation unit 2 and the map information from the map information generation unit 3 to determine whether it is necessary to generate advice information regarding the driver's driving operation.

[0042] Here, the relevance is an index calculated from one or more of the following information: (1) An "interference influence degree" that indicates the degree of possibility that each target included in the target information will physically interfere with the vehicle being operated by the driver (for example, the degree of possibility of a collision or an abnormal approach); (2) An "information importance degree" that indicates the degree to which at least each target included in the target information is important for generating advice information (for example, the degree to which the driver is not aware of a certain target); and (3) An "advice information correspondence degree" that indicates the degree to which the information is suitable for generating advice information for providing driving assistance or education, based on the correspondence between predetermined information (for example, words or phrases) stored in the memory unit 11 and the target information of the currently detected target (for example, the degree of correspondence / matching, such as whether the same words or phrases exist). However, in this embodiment, the combination conditions of the above-mentioned items (1), (2), and (3) are such that items (1) and (2) are OR conditions and item (3) is an AND condition, and the relevance is calculated by determining "((1) + (2)) x (3)".

[0043] Then, of the actual target information generated by the target information generation unit 2 and the local map information generated by the map information generation unit 3, information determined to be related to the driving operation advice information to be communicated to the driver is regarded as information with a high degree of relevance. In this way, the driving operation related information is information obtained by extracting the target information and map information that are deemed to be highly related from the actual target information and the local map information. Here, in this embodiment, this information is extracted as text information.

[0044] The advice information generation unit 6 has a function of generating advice information in text format (character information) from the driving operation-related information to educate the driver or to help with driving operations. The notification unit 7 has a function of presenting the generated text format advice information to the driver using the display unit 13. Since the audio output unit 14 is also provided, the advice information in text format can be read out and notified by audio, and furthermore, the advice information can be notified using both the display unit 13 and the audio output unit 14.

[0045] In the present embodiment, the following describes a case where text information (character information) is used as the target information, map information, and driving operation-related information, but the present invention is not limited to this example, and the target information or map information may be generated using, for example, structured data or numerical feature quantities. Next, the functions of each functional unit shown in FIG. 2 will be described.

[0046] [Description of Target Information Generator 2] Fig. 3 is a block diagram showing functional blocks of the target information generator 2 according to this embodiment. The target information generator 2 has a function of generating target information from sensor data. In this embodiment, the target information generator 2 includes a target recognition unit 21, a first target information generator 22, a second target information generator 23, and a target information integration unit 24.

[0047] The target information generating unit 2 may have only one of the first target information generating unit 22 and the second target information generating unit 23, or may further have a third target information generating unit or the like added. Here, the target information generated by the target information generating units 22 and 23 is formed of a "sentence" in text format.

[0048] The target recognition unit 21 has a function of estimating information about a target, and includes a target detection unit 211, a target position estimation unit 212, a target tracking unit 213, a target speed estimation unit 214, a target orientation estimation unit 215, a target feature estimation unit 216, and a target behavior estimation unit 217.

[0049] 4A and 4B, the functions of the target recognition unit 21 will be described. Fig. 4A shows an example of an external image captured as an input to the target recognition unit 21, and Fig. 4B shows an example of a target recognition result estimated by each function of the target recognition unit 21 for the external image.

[0050] <<Description of Target Detection Unit 211>> The target detection unit 211 has a function of estimating the type of target shown in an external image and its position in the image. For example, by using an image as input and performing inference using a neural network that has learned the type of target and a rectangle surrounding the target as training data, it is possible to recognize that, in the image of FIG. 4A , two people (Pe-0, Pe-1), one car (Au), and two houses (Hu) are shown around a road (Rd).

[0051] In this embodiment, people (Pe-0, Pe-1) and moving objects such as an automobile (Au) are used as targets for generating target information, and the target detection unit 211 generates target information from these people (Pe-0, Pe-1) and the automobile (Au). However, for example, from information showing two houses (Hu), it may be estimated that the area is a residential area as road information, which will be described later.

[0052] <<Description of Target Position Estimation Unit 212>> The target position estimation unit 212 has a function of estimating the position in three-dimensional space of a moving object (here, a person or a vehicle) detected by the target detection unit 211. A well-known method can be used as a method for estimating the position in three-dimensional space.

[0053] For example, by using external parameters such as the camera mounting position and camera parameters such as the camera's focal length and optical axis, the target can be determined by monocular ranging from the center position of the bottom end of the rectangle surrounding the target inferred by the target detection unit 211.

[0054] Furthermore, when a stereo camera, a LIDAR, or the like is used as the external camera, the position can be determined from the center of gravity of a three-dimensional point cloud of the target area. Note that the position in three-dimensional space can be estimated by other methods as well, not limited to these examples.

[0055] The position in three-dimensional space estimated in this way using a camera or LIDAR may be converted into a three-dimensional position based on the vehicle position from the positional relationship between the camera or LIDAR and the vehicle that has been determined in advance, or may be converted into a coordinate system with the reference position as the origin using the position and attitude of the vehicle relative to the reference position determined by GNSS.

[0056] <<Description of the target tracking unit 213>> The target tracking unit 213 has a function of linking a target estimated by the target detection unit 211 from an external world image at a certain time with a target estimated from a past external world image (assigning the same ID) when the two are the same.

[0057] Examples of linking methods include a method of extracting feature amounts of target regions from the current and past external world images and then judging based on the proximity of the distance between vectors when comparing the feature amounts, and a method of linking based on the target types in the estimation results of the current and past external world images being the same and the target positions calculated by the target position estimation unit 212. However, the methods are not limited to these.

[0058] <<Description of Target Speed ​​Estimation Unit 214>> The target speed estimation unit 214 has a function of estimating the speed of a target from the amount of movement of the target position, using information such as the time (time stamp) assigned to the sensor data, for targets determined to be the same target by the target tracking unit 213.

[0059] <<Description of the target orientation estimation unit 215>> The target orientation estimation unit 215 has a function of estimating the angle (direction) of the movement direction of a target, based on a change in the target position, for targets determined to be the same target by the target tracking unit 213.

[0060] The position, velocity and orientation of the target estimated by the target position estimation unit 212, the target velocity estimation unit 214 and the target orientation estimation unit 215 may be corrected using a Kalman filter or the like.

[0061] <<Description of Target Feature Estimation Unit 216>> The target feature estimation unit 216 has a function of estimating the features of a detected target using sensor data. Target features include appearance such as clothing, as well as attribute information such as age and gender. The appearance and attribute information may be estimated by pattern matching of the detected target or by a neural network. Furthermore, estimation of target features is not limited to these examples, and other methods may be used.

[0062] <<Description of the target behavior estimation unit 217>> The target behavior estimation unit 217 has a function of estimating the behavior of a detected target using sensor data. The behavior of a target is behavior information that indicates the movement status of a moving object, such as a person walking, a car driving, or a stopped car. Furthermore, the behavior information may include information that indicates the behavioral state of a person, such as whether they are sitting, talking, or making a phone call.

[0063] This target behavior may be inferred directly from sensor data using a neural network, or may be generated from speed information by the target speed estimation unit 214, or may be estimated from time-series changes in the posture of a person by separately estimating the posture. Furthermore, the method is not limited to these examples, and other methods may also be used.

[0064] <<Explanation of the First Target Information Generator 22>> The above-described functions of the target recognition unit 21 enable the generation of target recognition results for targets such as those shown in FIG. 4B . In FIG. 4B , two people and one vehicle are recognized as targets, and a target ID (Oj) is assigned to each of them. Corresponding to each target ID (Oj), the target type (Ojk), the target's current position (Ojp), the target's movement speed (Ojs), the target's orientation (Ojd), the target's behavior state (Oja), and the target's characteristics (Ojf) are linked. These target recognition results are sent to the first target information generator 22, which generates target information in text format for each target ID (Oj).

[0065] 5A shows an example of first target information generated by the first target information generation unit 22. In this embodiment, the first target information generation unit 22 generates text information as the first target information using the target recognition result generated by the target recognition unit 21.

[0066] As a method of generation, for example, if the target recognition information with target ID (Oj) "0" is obtained as "Person, Position (-5, 10), Speed ​​1.0 m / s, Orientation 0 degrees, Walking, Child, Male," then based on the rules, position X = -5 m → Left of the vehicle, position y = 10 m → 10 m forward, orientation 0 degrees (same direction as the vehicle's x-axis) → Turning right relative to the vehicle, Walking, Type → Person, text information such as "There is a person walking facing right relative to the vehicle at a position 5.0 m to the left of the vehicle and 10.0 m forward." can be generated as the first target information (Ojif1). Furthermore, if information such as "Child, Male" is obtained as target recognition information, text information such as "The person is a male child" can be added to the first target information.

[0067] Alternatively, as a method other than the rule-based method, a large language model (LLM) may be used to generate text information from target recognition information. For example, a prompt template, which is an instruction sentence for generating text information that is first target information based on the target recognition information, is prepared in advance, and the first target information can be generated by combining the target recognition information and inputting it into the LLM.

[0068] A prompt is an instruction to the LLM, such as, "Please use the information in the table below to create a sentence that describes a target that exists outside the vehicle (target recognition information shown in FIG. 4B). As an explanation for each column in the table, please... The output format is..." For target IDs (Oj) of "1" and "2," similar target information is generated as shown in FIG. 5A.

[0069] <<Explanation of Second Target Information Generator 23>> In response to the generation of first target information (Ojif1) by the first target information generator 22, the second target information generator 23 also generates second target information (Ojif2). Here, the first target information (Ojif1) by the first target information generator 22 is regarded as quantitative target information because specific numerical values ​​(position information) are used. On the other hand, the second target information (Ojif2) described below is regarded as qualitative target information because information for modifying the target (words and phrases) is used.

[0070] 5B shows an example of second target information (Ojif2) generated by the second target information generation unit 23 for the sensor data (image data) of FIG. 4A. While the first target information generation unit 22 in this embodiment generates the first target information based on the target recognition information shown in FIG. 4B, the second target information generation unit 23 generates the second target information (Ojif2) as text information directly from sensor data (image data in this case). Therefore, information such as position and speed is not reflected.

[0071] 5B shows an example in which text information modifying the first target information is generated from sensor data for each target ID. For example, when the target ID (Oj) is "0", text information such as "There is a child walking on the sidewalk to the left in front of your vehicle. The child is wearing red clothes and a hat" is generated. When the target ID (Oj) is "1" or "2", similar text information is generated as shown in FIG. 5B.

[0072] As a method for generating the second target information (Ojif2), for example, machine learning techniques such as captioning, which generates captions that describe a scene by inputting an image or video, dense captioning, which recognizes targets that appear in an image or video and generates captions for those targets, and VQA (Visual Question Answering), which generates answers to questions based on images or videos, can be applied.

[0073] Furthermore, the sensor data is not limited to images and videos, and machine learning techniques may be used to generate text information from distance information or a three-dimensional point cloud. In this embodiment, an example in which dense captioning is performed is shown.

[0074] <<Description of Target Information Integration Unit 24>> The first target information generated by the first target information generation unit 22 and the second target information generated by the second target information generation unit 23 are sent to the target information integration unit 24 and combined. Here, the first target information (Ojif1) includes numerical information such as specific position information, and therefore can be considered quantitative target information. The second target information (Ojif2) does not include specific numerical information, and therefore can be considered qualitative target information.

[0075] FIG. 5C shows an example in which the target information integrating unit 24 integrates the target information from the first target information generating unit 22 and the second target information generating unit 23 to generate integrated target information (Ojif3).

[0076] The target object information integration unit 24 has a function of comparing each "sentence" (which may be each phrase or word) included in the first target object information (Ojif1) with each "sentence" (which may be each phrase or word) included in the second target object information (Ojif2), deleting "sentences" (which may be each phrase or word) that contain overlapping or contradictory content, and generating text information as final integrated target object information (Ojif3).

[0077] However, there is a possibility that the first target information (Ojif1) and the second target information (Ojif2) may generate "sentences" with overlapping content. In such cases, only one of the "sentences" can be registered as the final integrated target information (Ojif3). Methods for determining content overlap include using a natural language processing technique to evaluate the similarity of sentences, or using an LLM to evaluate whether the content is identical.

[0078] Which "sentence" to keep may be determined by registering a priority function as a rule in advance, or by evaluating the amount of information contained in the "sentence" based on the number of words, etc., and adopting the one with the greater amount of information. Alternatively, the LLM may determine the overlapping and non-overlapping parts of the "sentences" of the first target information and the second target information, and then create a new "sentence" by removing only the overlapping parts.

[0079] On the other hand, there is a possibility that the first target information (Ojif1) and the second target information (Ojif2) may generate "sentences" with contradictory content. For example, for a person (Per-1) with a target ID (Oj) of "1" shown in Figures 5A and 5B, the first target information in Figure 5A states that the person is "stationary," while the second target information in Figure 5B states that the person is "walking."

[0080] In this example, the person (Per-1) with the target ID (Oj) of "1" is actually stationary, and in such a case, the first target information is left and the second target information is deleted. As a method for determining whether or not a "statement" contains a contradiction, a plurality of prompts may be prepared for determining whether or not two "statements" contain a contradiction regarding the target type, position, speed, direction, action, target characteristics, etc., and the LLM may be made to input a prompt containing the two "statements" to perform the determination.

[0081] Furthermore, as a method for determining which “sentence” to judge as correct and keep when a contradiction occurs, a rule such as “with regard to speed, priority is given to the first target information that generates the “sentence” based on the quantitative position of the target” may be determined in advance according to the characteristics of the first target information generating unit 22 and the second target information generating unit 23.

[0082] Alternatively, a machine learning method may be used to input sensor data and two "sentences," output the reliability of each "sentence," and then keep the one with the higher reliability. Furthermore, other methods may be used, without being limited to these examples.

[0083] As described above, the first target information generating unit 22 can generate first target information (Ojif1) including quantitative information about the target, while the second target information generating unit 23 can generate second target information (Ojif2) including qualitative information about the target and information about the scene other than the target.

[0084] Furthermore, when there are multiple pieces of target information, the target information integrating unit 24 can generate final integrated target information (Ojif3) after deleting overlaps and contradictions among them. As described above, the target information generating unit 2 can generate final target information for each target.

[0085] [Explanation of Map Information Generator 3] The map information generator 3 has a function of estimating one or more pieces of information from road information (road classification, legal speed limit, road width, etc.) during driving, traffic information (traffic congestion, construction, road regulations, etc.), and weather information (weather, road surface conditions, etc.), using data acquired from the sensor 110 or map information 202, and generating map information that integrates this information.

[0086] 6 is a diagram showing functional blocks of the map information generation unit 3 according to this embodiment. In this embodiment, the map information generation unit 3 will be described as having a map information acquisition unit 31 that acquires map information 202, a map information construction unit 32, and a map information integration unit 33. Note that the configuration may include only the map information acquisition unit 31 or the map information construction unit 32, and is not limited to this example.

[0087] <<Description of Map Information Acquisition Unit 31>> The map information acquisition unit 31 acquires map information 202 of the surroundings of the vehicle from position information such as GNSS acquired from the sensor data acceptance unit 1, and registers the acquired map information as first map information (similar to map information in a so-called navigation device). At this time, the map information 202 may be stored in a storage unit of the driving assistance device 100, or may be acquired from the cloud or the like.

[0088] <<Description of Map Information Construction Unit 32>> The map information construction unit 32 also has a function of constructing second map information using images, range images, three-dimensional point clouds, etc. acquired from the sensor data acceptance unit 1. For example, when images are used, road areas, white lines, walls, etc. may be estimated by semantic segmentation or the like, or road information may be estimated by recognizing signs by image recognition. Similarly, traffic information and weather information may be estimated by image recognition or the like, and then registered as map information; this is not a limitation.

[0089] <<Description of Map Information Integration Unit 33>> The map information integration unit 33 integrates the first map information and the second map information. As a method of integration, similar to the target object information integration unit 24, there is a method of combining the first map information and the second map information after removing overlapping and contradictory information. As similar to the target object information integration unit 24, which map information is to be used with priority when overlapping or contradictory map information occurs may be determined in advance based on a rule, such as "use the second map information for weather information," or the LLM may determine this.

[0090] The map information acquisition unit 31 acquires first map information, the map information construction unit 32 constructs second map information, and the map information integration unit 33 integrates the first map information and the second map information. As a result, the map information generation unit 3 can generate map information consisting of road information, traffic information, and weather information for the time of travel.

[0091] In the following embodiments, the explanation will be given assuming that text information such as "The road is a single lane, and is a residential road. The speed limit is 20 km / h. The weather is sunny" has been obtained as the map information related to FIG. 4A. [Explanation of the Additional Information Acquisition Unit 4] Returning to FIG. 2, the additional information acquisition unit 4 has a function of acquiring additional information, which is information related to the driver or the driver's driving operations. In this embodiment, the additional information is specifically one or more of "information on the driver's driving tendencies," "information on past inappropriate driving operations, etc.," and "information on inappropriate driving operations, etc., that occurred by the driver while driving."

[0092] The additional information may be received from another device / method that analyzes the driver's driving, or alternatively, the driving assistance device 100 may have an analysis function that generates the additional information.

[0093] In the following, a method for generating the supplementary information will be described, which uses the vehicle data, the external image, the internal image, and the GNSS information among the sensor data. In the following, an example of a method for estimating "information on the driver's inappropriate driving operation that occurred while driving" among the supplementary information will be described.

[0094] For example, vehicle data can be used to detect sudden acceleration, sudden braking, and sudden steering, and by combining it with map information, it can determine speed deviations, etc. Furthermore, trajectory information estimated from GNSS can be used to determine the vehicle's swaying or deviation when turning right or left.

[0095] In addition, by recognizing the driver using an internal image, it is possible to detect drowsiness or distraction. Furthermore, by integrating target recognition information from an external image, it is possible to determine whether the driver has correctly recognized an external target. Note that the present invention is not limited to these examples, and any method can be used as long as it can detect inappropriate driving operations by the driver.

[0096] In addition, the estimated inappropriate driving operations of the driver may be registered, or the items frequently detected as inappropriate driving operations may be registered as the driving tendencies of the driver to generate the supplementary information. The supplementary information acquisition unit 4 described above can acquire or generate supplementary information that is information about the driver or the driving operations of the driver.

[0097] In the following embodiments, it is assumed that the following information has been obtained as supplementary information: text information "The driver has not confirmed the target on the left side of the road," flag information indicating that the type of inappropriate driving operation is "unconfirmed," and the ID of the unconfirmed target is "0." Note that the sensor data, supplementary information, map information, and target information are each linked with a time, and the unrecognized target included in the supplementary information and the target included in FIG. 5C have corresponding target IDs.

[0098] [Description of Driving Operation-Related Information Generator 5] Fig. 7 is a diagram showing functional blocks of the driving operation-related information generator 5 according to this embodiment. The driving operation-related information generator 5 has a function of generating driving operation-related information that is estimated to be notified as advice information to the driver based on the target object information obtained by the target object information generator 2 shown in Fig. 3 and the map information obtained by the map information generator 3 shown in Fig. 6.

[0099] In this way, the driving operation related information is information extracted from the target information and map information, with information related to the advice information to be notified defined as information with a high degree of relevance.

[0100] 7, the driving operation related information generation unit 5 includes at least an interference influence degree calculation unit 51, an information importance degree calculation unit 52, an advice information correspondence degree calculation unit 53, a relevance degree calculation unit 54, and a related information extraction unit 55. Each functional unit will be described in detail below.

[0101] <<Description of Interference Influence Degree Calculation Unit 51>> The interference influence degree calculation unit 51 has a function of calculating the degree of interference that each target object exerts on the host vehicle. The degree of interference can be calculated by, for example, comparing a predicted future trajectory of the host vehicle calculated from the orientation, speed, size, etc. of the host vehicle with a predicted trajectory of the target calculated from the orientation, speed, and size of the target, thereby determining a risk of physical interference (e.g., collision, contact, abnormal approach, etc.) between the host vehicle and the target.

[0102] The interference influence level in this embodiment is the degree of interference that quantifies the risk of such collision, contact, or abnormal approach. The interference level may be evaluated in two stages, "score 0" and "score 1," or in multiple stages of three or more. Here, a lower score is considered to be on the safe side.

[0103] In the example of FIG. 5C , the person (Pe-1) with target ID “1” is stationary on the right side of the road, so the risk of a collision due to the predicted trajectory is low, and the degree of interference influence is calculated as “score 0.” On the other hand, the person (Pe-0) with target ID “0” is taking a route that intrudes into the road, and their predicted trajectories interfere with each other, posing a risk of collision with the vehicle, so the degree of interference influence is calculated as “score 1.” In addition, the vehicle with target ID “2” is also on a single-lane road, so they must pass each other, and there is a risk that their predicted trajectories will interfere with each other, so the degree of interference influence is calculated as “score 1.”

[0104] The interference influence calculation unit 51 determines that a target with a target ID (Oj) of “1” has a low interference influence (score 0), and that targets with target IDs of “0” and “2” have a high interference influence (score 1).

[0105] <<Description of Information Importance Calculation Unit 52>> The information importance calculation unit 52 has a function of calculating the importance of each piece of information (in this embodiment, “sentences”) included in the map information and the target information based on the incidental information. Specifically, the incidental information is one or more of “information on the driver’s driving tendency,” “information on past inappropriate driving operations, etc.,” and “information on inappropriate driving operations, etc., that occurred by the driver while driving.”

[0106] In this embodiment, the accompanying information includes text information such as "The driver has not confirmed the target on the left side of the road," as described above, flag information indicating that the type of inappropriate driving operation is unconfirmed or unrecognized, and information indicating that the unconfirmed or unrecognized target ID (Oj) is "0."

[0107] The "target object on the left side of the road" in the target object information (text information) is a target object with a target ID (Oj) of "0" and is a person (Pe-0). Since the person (Pe-0) is taking a route that leads to an intrusion toward the road, there is a risk of interference with the vehicle, and the person is considered to be target object information of high importance from a safety perspective. The information importance calculation unit 52 calculates the importance of each piece of information included in the map information and target object information based on the incidental information.

[0108] Therefore, the importance of the target information relating to the target having the target ID (Oj) of "0" is determined to be high importance of "score 1."

[0109] Furthermore, since flag information indicating that the driver is unidentified is obtained as supplementary information, the score of a "sentence" determined to correspond to "unidentified behavior" by machine learning or the like may be determined to be a high "score 1." Furthermore, the importance may be determined in advance according to the type and characteristics of the target included in the target information, and is not limited to these examples. A similar determination is made when the target ID (Oj) is "1" or "2."

[0110] <<Explanation of Advice Information Correspondence Calculation Unit 53>> The advice information correspondence calculation unit 53 has a function of calculating the degree to which each piece of information (in this embodiment, “sentences”) included in the map information and target information corresponds to the generation of advice information that provides driving assistance or education.

[0111] As a method of determining this degree, for example, words and phrases that are defined as important may be defined in advance in the driving knowledge 201, and if those words and phrases are included in each "sentence" of the map information and the landmark information, it may be determined that the degree of correspondence with the advice information is high.

[0112] Another method is to assign scores to words and phrases in the driving knowledge 201, and if a "sentence" to be judged contains a corresponding word or phrase, add up the score. Furthermore, calculation may be performed by directly judging whether or not advice information is important information using LLM.

[0113] Alternatively, the target information generating unit 2 may perform the above processing on the second target information after estimating that the target information generated from the first target information generating unit 22 has a high degree of correspondence with the advice information, thereby dividing the processing according to the function that generated the information.

[0114] For example, the target information for person (Pe-1) with target ID "1" in Figure 5C is expressed as follows: "(a) There is a stationary person 10.0 m to the right of the vehicle and 14.0 m ahead. (b) The person is an adult woman. (c) The woman is smiling. (d) The woman is carrying a backpack and a tennis racket." Then, the following judgment is made for each of these "sentences" (see Figure 8 for details).

[0115] When words such as "stationary," "walking," "driving," "single lane," "speed limit," "weather," etc. are stored in driving knowledge 201 with high scores, (a) is judged to have a high degree of correspondence to advice information (e.g., score 1) because it contains the word "stationary," and (b), (c), and (d) are judged to have a low degree of correspondence to advice information (e.g., score 0) because they do not contain the defined word.

[0116] As shown in Figure 8, similar estimation is performed for other target ID (Oj) values ​​of "0" and "2" and for each "sentence" in the map information. When the target ID (Oj) is "0", a "sentence" containing the word "walk" exists, and when the target ID (Oj) is "2", a "sentence" containing the word "drive" exists, and the map information contains "sentences" containing the words "single lane", "speed limit", and "weather". Therefore, each "sentence" containing these words is determined to have a high degree of correspondence with advice information (for example, a score of 1).

[0117] <<Explanation of the relevance calculation unit 54>> The relevance calculation unit 54 estimates the relevance, which is an index of how much each “sentence” included in the target information and map information is related to the advice information to be proposed, from the degree of interference influence, information importance, and degree of correspondence to advice information (score).

[0118] 8 shows the target ID (Oj), target information (Ojif3), interference influence (Ojinf), information importance (Ojimp), advice information correspondence (Ojadv), and relevance (Ojrv) for each target. ◯ indicates a high score (score 1), and × indicates a low score (score 0). The advice information correspondence is determined for each "sentence" that constitutes the target information and map information.

[0119] In Fig. 8, for each target ID, the target information (Oj), interference influence (Ojinf), information importance (Ojimp), advice information correspondence (Ojadv), and relevance (Ojrv) determined from these are indicated by "○" and "×". Here, the advice information correspondence (Ojadv) is determined for each "sentence" constituting the target information. A similar determination is made for map information. "○" may be considered as "score 1" and "×" as "score 0".

[0120] For example, the relevance (Ojrv) is calculated based on target information that has a high interference influence (Ojinf) or information importance (Ojimp), or both, and determining whether the advice information correspondence (Ojadv) for each "sentence" in the target information is equal to or greater than a predetermined threshold (score 1) or less than a predetermined threshold (score 0).

[0121] In Figure 8, in terms of interference influence degree (Ojinf), the person (Pe-0) and other vehicle (Au) whose target ID (Oj) is "0" and "2" are judged to have a high interference influence degree (Ojinf) (=○) as they are likely to interfere with the vehicle, while the person (Pe-1) whose target ID (Oj) is "1" is judged to have a low interference influence degree (Ojinf) (=×) as they are stationary and therefore are not likely to interfere with the vehicle.

[0122] Similarly, in terms of information importance (Ojimp), there is target information regarding a person (Pe-0) whose target ID (Oj) is “0”, but based on the aforementioned incidental information, it is determined that the driver does not recognize the person (Pe-0), so the information importance (Ojimp) is judged to be high (=○), and for target IDs “1” and “2”, based on the incidental information, the person (Pe-1) and other vehicle (Au) are recognized by the driver, so the information importance (Ojimp) is judged to be low (=×).

[0123] Then, in the advice information correspondence degree (Ojadv), the degree to which the information corresponds to the generation of advice information is calculated for "sentences" that make up the landmark information and map information. For example, for predetermined words (here, "stationary," "walking," "driving," "single lane," "speed limit," and "weather"), if there are words that correspond to each of the "sentences" that make up the landmark information and map information, the advice information correspondence degree (Ojadv) is determined to be high (= 0), and if not, the advice information correspondence degree (Ojadv) is determined to be low (= x).

[0124] That is, when the target ID (Oj) is "0", the "sentence" of the target information is made up of three "sentences", but the first "sentence" contains the word "walk", so the advice information correspondence degree (Ojadv) is determined to be high (= O). On the other hand, the other "sentences" do not contain the above-mentioned words, so the advice information correspondence degree (Ojadv) is determined to be low (= X). The same applies to other target information and map information.

[0125] Then, once the scores for the interference influence degree (Ojinf), information importance degree (Ojimp), and advice information correspondence degree (Ojadv) are obtained, the relevance degree of each target information and map information is calculated. In this case, the condition for advice information is that at least one of the scores for the interference influence degree (Ojinf) and the information importance degree (Ojimp) is high (=○). Therefore, target IDs (Oj) of "0" and "2" are targeted. For this target information, information that meets the condition of a high advice correspondence degree (=○) is determined to have a high relevance degree (Ojadv) (=○) and is derived.

[0126] On the other hand, for targets with a target ID (Oj) of "1", the scores for interference impact (Ojinf) and information importance (Ojimp) are low (=×), so even if the score for advice information correspondence is determined to be high (=○), the correlation will not be derived.

[0127] Here, the map information is calculated using only the advice information correspondence degree. In this embodiment, for ease of explanation, each item on the horizontal axis is treated as a discrete value of "high" or "low", but it may also be a continuous value. In this case, the relevance degree may be calculated by weighting the interference influence degree (Ojinf), importance degree (Ojimp), and advice information correspondence degree (Ojadv) and adding them up.

[0128] <<Description of Related Information Extraction Unit 55>> The related information extraction unit 55 generates, as target related information, information (○ information) that is estimated to have a degree of association equal to or greater than a threshold value for map information and target information with high scores (○).

[0129] In this way, the interference influence degree calculation unit 51 can extract target object information with a high degree of interference influence on the vehicle. Also, the information importance calculation unit 52 can extract map information and target object information with high importance based on the incidental information. Also, the advice information correspondence degree calculation unit 53 can extract map information and target object information with a high degree of advice information correspondence in order to generate advice information for providing driving assistance or education.

[0130] As described above, even if the target object information and map information contain unnecessary information not related to the advice information, the relevance calculation unit 54 and the related information extraction unit 55 can remove information with low relevance to generate driving operation-related information, thereby enabling the advice information generation unit 6 to generate high-quality advice information. [Explanation of Advice Information Generation Unit 6] Figure 9 is a diagram showing functional blocks of the advice information generation unit 6 according to this embodiment. The advice information generation unit 6 uses the incidental information and the driving operation-related information to generate text information of advice information that provides driving assistance or safety education to the driver. The advice information generation unit 6 includes a prompt generation unit 61 that creates a prompt, which is an instruction sentence for generating the text, and an inference unit 62 that generates advice information by performing inference such as LLM using the prompt information and inference parameters 203, which are weight parameters in machine learning.

[0131] <<Description of Prompt Generation Unit 61>> The prompt generation unit 61 generates a prompt to be input to the inference unit 62. As a method of generating a prompt, a template is selected from a plurality of prepared prompt templates based on accompanying information.

[0132] In this embodiment, since the information about inappropriate driving operation in which the driver did not confirm (recognize) the person (Pe-0) on the left side of the road is input, a template corresponding to "unconfirmed" or the like is selected. At this time, the following template, for example, is generated using the driving operation related information.

[0133] "The driver did not check for a person on the left side of the road. Below is a scene that may occur when the driver does not check for a person. Please generate advice to help the driver perform the correct driving maneuver. ⇒〈Scene Generation〉⇒There is a person (an object not identified by the driver) walking to the right of the vehicle, 5.0 m to the left and 10.0 m ahead. The road is a single-lane residential road. The speed limit is 20 km / h. The weather is clear. There is a vehicle (an object identified by the driver) traveling in the opposite direction to the vehicle, 3.0 m to the right and 20.0 m ahead." Alternatively, information such as the degree of interference impact (Ojinf) can be added to the prompt scene as shown below. "(Omitted) ...There is a person (an object not identified by the driver) walking to the right of the vehicle, 5.0 m to the left and 10.0 m ahead. There is a risk of collision if both vehicles are traveling straight ahead." ...(omitted)" The prompt templates shown above are examples for explaining this embodiment, and are not limited to these examples. For example, it is possible to specify an output format, embed more detailed supplementary information, map information, and landmark information, and then write an explanation of the embedded information so that the LLM can interpret it. Alternatively, it is possible to first summarize the scene I situation and then generate a prompt.

[0134] <<Description of Inference Unit 62>> The inference unit 62 generates advice information by inputting the generated prompt into a machine learning model such as an LLM having inference parameters 203. At this time, the advice information may be generated by machine learning that can input sensor data such as images as well as the text-based prompt as input.

[0135] Even when applying a machine learning model that can input text and images in this way, the quality of the advice information can be improved by generating advice information after explicitly generating driving operation-related information from target information and map information, as in the driving assistance device 100 of this embodiment. [Explanation of Notification Unit 7] Returning to Figure 2, the notification unit 7 has a function of displaying the advice information generated by the advice information generation unit 6 on a display device of the vehicle (for example, a display of a navigation device (not shown)) or a driving simulator at a safety education center.

[0136] Fig. 10A is a diagram showing an example of a screen when providing driving education after driving using advice information generated by the advice information generating unit 6. Fig. 10B is a diagram showing an example of providing assistance to a driver while driving using advice information generated by the advice information generating unit 6.

[0137] As shown in Fig. 10A, driving education can be provided by superimposing generated advice information on the driving video for a scene where an inappropriate driving operation occurred and displaying it on a display after the drive. In this case, a form of information presentation that is easier for the driver to understand can be considered by synchronously superimposing target recognition information and map information in addition to the driving video and advice information.

[0138] In Figure 10A, a rectangular frame (Rf) is displayed around the person (Pe-1) and an arrow indicating the direction of travel (Dt) is displayed, and an arrow indicating the direction of travel (Dt) is also displayed around the other vehicle (Au). Furthermore, a pop-up window is opened to display advice information (Adinf) stating, "This was a scene in a residential area where a child could have run out into the roadway, so it was necessary to check for oncoming vehicles, as well as the left side of the road, and to slow down. Please be especially careful of people around you when driving in residential areas."

[0139] The displayed image may have embedded therein icons such as a "time scroll function" and a "driving operation improvement scene function" as needed. Note that this image can also be displayed on the display inside the vehicle while driving.

[0140] In addition, advice information may be presented not only through images or text displayed on a screen, but also through audio or animation, and sensor data may be processed and presented in three-dimensional space using a VR (Virtual Reality) device rather than a display, allowing the driver to re-experience the driving situation.

[0141] On the other hand, while the driver is driving the vehicle, if an inappropriate driving operation occurs or a scene requiring attention occurs, as shown in FIG. 10B , people (Pe-1, PE-2), other vehicles (Au), and the driver's vehicle (Oc) are displayed, and a pop-up window is opened to display advice information (Adinf) such as, "There is a child ahead on the left, please be careful." In this case, the advice information is in short sentences to speed up the driver's recognition. Furthermore, by playing the generated advice information as audio, it becomes possible to provide safe driving assistance and safety education to the driver.

[0142] As shown in Figures 10A and 10B, even for the same scene, it is effective to change the text to be presented depending on whether post-driving education is being provided or assistance is being provided during driving. For example, in post-driving education, it is possible to present advice information that includes a variety of information as part of driving education, but when providing assistance during driving, it is necessary to present concise information to ensure safety. By specifying the format of the advice information to be output using a prompt from the advice information generation unit 6, the format and amount of information of the advice information can be flexibly changed, making it applicable to a variety of uses.

[0143] The above-described driving assistance device 100 can automatically generate advice information to assist the driver in driving operations based on sensor data during driving, thereby improving the driver's driving skills and improving safety through driving assistance for the driver.

[0144] It should be noted that the present invention is not limited to the above-described embodiments and includes various modifications. The above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace other configurations with respect to the configuration of each embodiment.

[0145] 1...sensor data receiving unit, 2...target object information generating unit, 3...map information generating unit, 4...accompanying information acquiring unit, 5...driving operation related information generating unit, 6...advice information generating unit, 7...notifying unit, 201...driving knowledge

Claims

1. A driving assistance device that generates advice information for a driver's driving operation, comprising: a sensor data receiving unit that receives sensor data from a sensor that measures the periphery of a vehicle in which the driver is riding; a target object information generating unit that generates target object information for targets included in the sensor data; a map information acquiring unit that acquires map information of the driving environment of the vehicle; a driving operation related information generating unit that uses at least the map information and the target object information to generate driving operation related information that is highly relevant to the advice information to be notified to the driver in relation to the driving operation of the driver; an advice information generating unit that generates the advice information to be notified to the driver based on the driving operation related information; and a notifying unit that notifies the driver of the advice information generated by the advice information generating unit.

2. A driving assistance device according to claim 1, wherein the driving operation related information generation unit comprises: an interference influence calculation unit that calculates an interference influence level indicating the physical interference effect on the vehicle caused by each of the targets included in the target information; an information importance calculation unit that calculates an information importance level indicating whether or not each of the targets included in the target information is important in generating the advice information; an advice information correspondence calculation unit that calculates an advice information correspondence level indicating the degree to which the target information is corresponding to the generation of the advice information, based on the correspondence between the pre-stored target information and the target information of the currently detected target; and a related information extraction unit that generates the driving operation related information based on the interference influence level, the information importance level, and the advice information correspondence level.

3. A driving assistance device according to claim 2, wherein the advice information generating unit generates text information based on the driving operation related information.

4. A driving assistance device according to claim 2, characterized in that the interference influence calculation unit calculates the interference influence using one or more of the type of the target, the position of the target, and the speed of the target contained in the sensor data.

5. A driving assistance device according to claim 2, wherein the information importance calculation unit calculates the importance of the target information using information relating to the driving operation of the driver.

6. A driving assistance device according to claim 5, wherein the information relating to the driver's driving operations is one or more of the following: information on the driver's driving tendencies, information on past inappropriate driving operations, and information on current inappropriate driving operations that have occurred while driving.

7. A driving assistance device according to claim 2, wherein the advice information correspondence degree calculation unit calculates the advice information correspondence degree by determining whether knowledge information to which information related to the advice information is added is included in the target information.

8. A driving assistance device according to claim 6, wherein the advice information generation unit generates text information as advice information to be notified to the driver based on information that the driver has performed an inappropriate driving operation, and the notification unit displays the text information as a pop-up on a display.

9. A driving assistance device according to claim 1, wherein the target information generation unit comprises a first target information generation unit, a second target information generation unit, and a target information integration unit, wherein the first target information generation unit generates first target information including quantitative information about the target, the second target information generation unit generates second target information including qualitative information about the target and information about the scene other than the target, and the target information integration unit integrates the first target information and the second target information to generate the target information.

10. A driving assistance device according to claim 9, wherein the target information integration unit compares the first target information with the second target information, and if there is overlap and / or contradiction in the predetermined information constituting each of the target information, eliminates one of the predetermined information to generate the target information.

11. A driving assistance device according to claim 10, wherein the target information is text information.

12. A driving assistance device according to claim 11, wherein the notification unit displays both the target object and the text information on a display.

13. A driving assistance device according to claim 8, wherein the notification unit is provided with a voice output unit, and the text information is converted into voice and notified from the voice output unit.

14. A vehicle that is driven by a driver and is equipped with a driving assistance device that generates advice information to assist the driver in their driving operation, wherein the driving assistance device is a driving assistance device as defined in any one of claims 1 to 12.

15. A driving assistance system comprising: a vehicle that is driven by a driver and is equipped with a notification unit that notifies advice information to assist the driver in driving; and a driving assistance device that generates the advice information to assist the driver in driving and transmits the advice information to the notification unit of the vehicle via wireless communication, wherein the driving assistance device has: a sensor data receiving unit that receives sensor data via wireless communication from a sensor that measures the periphery of the vehicle in which the driver is riding; a target information generating unit that generates target information for targets included in the sensor data; a map information acquiring unit that acquires map information of the driving environment of the vehicle; a driving operation related information generating unit that generates driving operation related information that is highly relevant to the advice information to be notified to the driver, using at least the map information and the target information in the driver's driving operation; an advice information generating unit that generates the advice information to be notified to the driver based on the driving operation related information; and a transmitting unit that transmits the advice information generated by the advice information generating unit to the notification unit via wireless communication.

16. A driver education system that generates advice information to educate a driver on driving operations, comprising: a sensor data storage medium that receives and stores sensor data from sensors that measure the surroundings of a vehicle in which the driver is riding; a sensor data reception unit that receives the sensor data from the sensor data storage medium; a target information generation unit that generates target information for targets included in the sensor data; a map information acquisition unit that acquires map information of the driving environment of the vehicle; a driving operation related information generation unit that uses at least the map information and the target information to generate driving operation related information that is highly relevant to advice information to be notified to the driver in relation to the driver's driving operations; an advice information generation unit that generates the advice information to be notified to the driver based on the driving operation related information; and a notification unit that notifies the driver of the advice information generated by the advice information generation unit.

17. A driving assistance method for a driving assistance device that generates advice information to assist a driver in driving operations, comprising the steps of: receiving sensor data from a sensor that measures the periphery of a vehicle in which the driver is riding; generating target object information for targets included in the sensor data; acquiring map information of the driving environment of the vehicle; generating driving operation related information that is highly relevant to the advice information to be notified to the driver using at least the map information and the target object information in relation to the driver's driving operations; generating the advice information to be notified to the driver based on the driving operation related information; and notifying the driver of the advice information.

Images