Driver assistance systems

Abstract

The system uses speech recognition to evaluate the driver's commentary while they are driving the vehicle, and provides appropriate information to the driver based on the evaluation results to support their driving. [Solution] The driving support system comprises a driver recognition processing unit that performs voice recognition on voice comments spoken by the vehicle driver and generates first traffic situation recognition information recognized by the driver; a vehicle recognition processing unit that generates second traffic situation recognition information recognized based on surrounding environment information acquired by a driving situation recognition function installed in the vehicle; and a comparison and determination processing unit that determines the validity of the voice comments by comparing the first traffic situation recognition information with the second traffic situation recognition information and calculates the degree of agreement by comparing the first traffic situation recognition information and the second traffic situation recognition information, and performs predetermined driving support to the driver according to the validity and degree of agreement.

Description

【Technical Field】 【0001】 The present invention relates to a driving support system that supports driving of a vehicle by a driver. 【Background Art】 【0002】 There are known preventive safety technologies that assist a driver's driving, such as an automatic brake that suppresses mis-start due to a driver stepping on the accelerator pedal and the brake pedal of a vehicle by mistake and reduces physical damage during a collision with another vehicle. There is also known an advanced driving support system (ADAS) that aims to prevent the occurrence of contact and collision accidents between a vehicle and other objects (four-wheeled vehicles, two-wheeled vehicles), traffic signs, pedestrians, obstacles, etc., and reduce damage. 【0003】 Commentary driving is recommended to prevent a driver from losing attention and to raise the driver's vigilance against dangers such as traffic accidents. Commentary driving is a driving method in which a driver speaks while driving a vehicle as if broadcasting the presence or absence of objects, the road and traffic conditions, etc. in real time. 【0004】 Patent Document 1 discloses a change operation support device that recognizes speech by a driver and supports changing the setting state of driving support control through the speech. Driving support control includes, for example, constant speed driving control (ACC), following control of a preceding vehicle, lane keeping control (LKA), and automatic driving control. 【Prior Art Documents】 【Patent Documents】 【0005】 【Patent Document 1】 Japanese Patent Application Laid-Open No. 2023-71877 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0006】 Patent Document 1 describes a system that performs driver assistance control (ACC, LKA, etc.) according to the settings of the vehicle's driver assistance control, for example, by recognizing the driver's voice to turn ADAS functions on or off and set target vehicle speed. However, due to misrecognition of the driver's speech, setting change requests that differ from the driver's intention may be made. In this case, Patent Document 1 notifies the driver of confirmation information that includes information regarding the setting change request and information regarding the driver's approval operation. Subsequently, the driver changes the setting state of the driver assistance control by performing an approval operation. However, there is a concern that the driver may perform the approval operation incorrectly. Furthermore, because it involves a complicated processing process of setting change requests by the driver's speech and the driver's approval operation, it is difficult for a driver to change the setting state of the driver assistance control at the appropriate time while driving the vehicle. 【0007】 To promote safe driving, commentary driving is sometimes implemented, in which the driver provides a running commentary on the traffic conditions around the vehicle while driving. Commentary driving provides information to maintain and improve the driver's safety awareness. It has the effect of increasing the driver's concentration and reducing the discrepancy between the driver's intentions regarding vehicle operation and actual driving operations. However, simply having the driver talk about the traffic conditions does not necessarily mean that the driver is concentrating on driving the vehicle. In addition, various landmarks appear on the road in the driver's field of vision, making it difficult for the driver to accurately describe various landmarks that affect driving while concentrating on driving. For this reason, commentary driving alone is insufficient as a preventive safety technique for drivers, and there is a need to develop technology that can appropriately support the driver's driving in response to commentary driving. 【0008】 To solve the above-mentioned problems, the embodiment of the present invention aims to provide a driving assistance system that performs voice recognition of the speech of a driver operating a vehicle, evaluates the commentary driving, and provides appropriate information to the driver in accordance with the evaluation results to assist driving. [Means for solving the problem] 【0009】 Embodiments of the present invention are driver recognition processing units that perform voice recognition on voice comments spoken by the driver of a vehicle and generate first traffic situation recognition information recognized by the driver; vehicle recognition processing units that generate second traffic situation recognition information recognized based on surrounding environment information acquired by a driving situation recognition function installed in the vehicle; and comparison and determination processing units that determine the validity of voice comments by comparing the first traffic situation recognition information with the second traffic situation recognition information and calculate the degree of agreement by comparing the first traffic situation recognition information and the second traffic situation recognition information, and perform predetermined driving assistance to the driver according to the validity and degree of agreement. [Effects of the Invention] 【0010】 According to an embodiment of the present invention, the system can perform voice recognition on the speech of a driver operating a vehicle, evaluate the commentary driving, and provide appropriate information to the driver based on the evaluation results to support driving. [Brief explanation of the drawing] 【0011】 [Figure 1] This is a block diagram showing a schematic configuration of a driver assistance system according to an embodiment of the present invention. [Figure 2] (A) A schematic diagram showing the positional relationship between the vehicle and the landmark, (B) A schematic diagram showing the positional relationship between the vehicle and the pedestrian, and (C) A schematic diagram showing the display of the traffic sign as seen from the vehicle. [Figure 3] (A) A schematic diagram showing the direction of travel of a pedestrian as seen from the vehicle, (B) A schematic diagram showing the driver's line of sight and the pedestrian's position, and (C) A schematic diagram showing the driver's line of sight and the pedestrian's position when the vehicle is reversing. [Figure 4] (A) A schematic diagram showing the suppression of driving force in response to the driver's driving operations, and (B) A schematic diagram showing the application of steering reaction force in response to the driver's steering wheel operation. [Figure 5] This is a block diagram of the driver assistance system according to the first embodiment of the present application. [Figure 6] This is a flowchart showing the first part of the driver assistance system according to the first embodiment of the present application. [Figure 7] This is a flowchart showing the second part of the driver assistance system according to the first embodiment of the present application. [Figure 8] This is a block diagram of a driver assistance system according to the second embodiment of the present application. [Figure 9] This is a flowchart showing the first part of the driver assistance system according to the second embodiment of the present application. [Figure 10] This is a flowchart showing the second part of the driver assistance system according to the second embodiment of the present application. [Modes for carrying out the invention] 【0012】 A driver assistance system according to an embodiment of the present invention will be described with reference to the accompanying drawings. The embodiment of the present invention can be applied to various vehicles such as fuel-powered vehicles, hybrid vehicles, and electric vehicles. The embodiment of the present invention relates to a system configuration that assumes the driver of the vehicle is performing commentary driving, which provides real-time commentary on the traffic conditions around the vehicle as perceived. 【0013】 The driver assistance system uses speech recognition to process voice comments spoken by the driver and constructs driver traffic situation recognition information based on the traffic conditions perceived by the driver. Furthermore, the driver assistance system constructs vehicle traffic situation recognition information, which indicates the traffic conditions perceived by the vehicle, from surrounding environmental information (e.g., probe data and GPS data) acquired from various in-vehicle sensors and communication devices. 【0014】 The driver assistance system compares the driver's traffic situation perception information with the vehicle's traffic situation perception information and the driver's driving inputs (e.g., operation of the accelerator / brake pedals, operation of the turn signals, steering wheel operation, etc.) and evaluates the degree of agreement. Based on the degree of agreement, it determines whether the driver's voice comments are appropriate in light of the traffic situation. Here, the degree of agreement between the driver's traffic situation perception information and the vehicle's traffic situation perception information may be expressed numerically and scored. A standard degree of agreement between the driver's traffic situation perception information and the vehicle's traffic situation perception information may be expressed as a predetermined numerical value, and points may be added or deducted depending on the agreement or disagreement for each of the multiple parameters. After the vehicle has been driven (or after the driving cycle has ended), the driver assistance system compiles the scores of multiple voice comments made by the driver and notifies the driver of the scoring result. A driving cycle refers to the period from the time the driver starts driving by starting the vehicle's engine, etc., to the time the vehicle's engine is stopped or driving has ended. For the sake of explanation, the degree of agreement between the driver's perception of traffic conditions and the vehicle's perception of traffic conditions will be expressed as a large number for higher agreement and a small number for lower agreement. Therefore, the driver's driving style will be evaluated by adding points to increase the degree of agreement or subtracting points to decrease it. 【0015】 Figure 1 is a block diagram illustrating the schematic configuration of a driver assistance system 10 according to an embodiment of the present invention. The driver assistance system 10 is equipped with a processor such as a CPU or GPU and a semiconductor memory device such as ROM or RAM, and realizes the functions of the driver assistance system 10 by executing a predetermined program. Alternatively, an in-vehicle computer system consisting of multiple electronic control units (ECUs) connected by an in-vehicle network (CAN) may be used as the driver assistance system 10. 【0016】 The driving support system 10 includes a driver recognition processing unit 11, a vehicle recognition processing unit 12, a comparison and determination processing unit 13, and an output unit 14. It is assumed that the vehicle is equipped with a driver monitoring system that includes an in-vehicle camera for capturing the driver's face image and a microphone for collecting the driver's voice comments. The in-vehicle camera is used to detect the direction of the driver's face and the line of sight. An inactivation switch for stopping the functions of the driving support system 10 may be provided. The inactivation switch may be a physical stop button or a software switch that responds to the driver's voice. 【0017】 The driver recognition processing unit 11 is connected to, for example, a driver monitoring system equipped with an in-vehicle camera and a microphone, and inputs driver information related to the driver's face image and the driver's voice comments during commentary driving. The driver information and voice comments reflect the intentions of the driver driving the vehicle and the traffic situation recognized by the driver. The driver recognition processing unit 11 generates the driver's traffic situation recognition information based on the driver information and the voice comments. 【0018】 The driver's intention refers to, for example, the driver's intention in driving operations related to the straight-ahead, forward, backward, left turn, right turn, stop, etc. of the vehicle. In this embodiment, it is assumed that the driver's intention is expressed in voice comments and the driver's face image. That is, the driver's intention is detected through the driver's speech content collected by the microphone and the driver's face image reflected in the in-vehicle camera. 【0019】 The traffic conditions perceived by the driver refer to the surrounding environment of the vehicle, formed by other vehicles, pedestrians, traffic signs, etc., that the driver sees. For example, the traffic conditions perceived by the driver, including the presence or absence of obstacles around the vehicle and the positional relationship between the vehicle and obstacles, are determined based on driver information such as the direction of the driver's face and gaze detected from the driver's face image captured by the in-vehicle camera. Furthermore, the driver's perceived traffic conditions are also reflected in the driver's voice comments during commentary driving. In addition to the gaze direction and face direction extracted from the driver's face image, the driver information may also reflect the direction of the driver's upper body, etc. 【0020】 The driver's voice comments are subjected to predetermined speech recognition processing. Various methods have been developed for this speech recognition processing. For example, specific words (keywords) may be extracted from the driver's spoken voice comments. Examples of specific words include those related to traffic conditions around the vehicle and driving behavior. For example, words related to traffic conditions around the vehicle include those related to traffic congestion due to the presence of other vehicles, the status of traffic lights, the presence or absence of pedestrians on crosswalks, the presence or absence of obstacles on the road, and the display of traffic signs. In addition, words related to driving behavior include those related to the vehicle's actions such as going straight, turning right, turning left, and stopping. 【0021】 As speech recognition processing for driver voice comments, natural language processing (NPL) using AI (such as deep learning) or machine learning may be applied. In conventional natural language processing, morphological analysis, syntactic analysis, semantic analysis, and contextual analysis are performed on natural language spoken by humans, so the processing process is complex and requires a large amount of computation. However, in the embodiment of this application, since natural language analysis is performed only on specific words spoken by the driver, it can be realized with a relatively simple processing process and does not necessarily require a large amount of computation. For this reason, speech recognition of driver voice comments can be implemented by installing a relatively small algorithm or machine learning model on the in-vehicle computer. 【0022】 To implement speech recognition processing of driver voice comments, for example, an IT vendor or automobile manufacturer may pre-create a speech recognition model that performs natural language processing and implement it in the in-vehicle computer. The speech recognition model may analyze the content of the driver's utterances and extract specific words. For example, the driver's pronunciation may be registered in advance, and the driver's utterances may be speech-recognized to extract specific words contained in the driver's utterances as a result of the speech recognition. Examples of specific words include words related to the type of object, the location of the object, and traffic signs. The type of object refers to other vehicles, traffic signs, obstacles, pedestrians, etc., and the location of the object refers to its position relative to the vehicle, such as to the right, left, front, or rear. Traffic signs refer to green lights, red lights, speed limits, stop lines, etc. 【0023】 The results of speech recognition processing on voice comments may be machine-learned. For example, a machine learning model may be generated by supervised learning based on the driver's past speech history. In this case, the driver's current speech may be input to the machine learning model, and specific words may be extracted from the driver's speech using a specific algorithm or a mathematical formula with multiple parameters. For this reason, the driver recognition processing unit 11 may be equipped with a speech recognition model or a machine learning model for performing speech recognition. 【0024】 In-vehicle systems utilize various in-vehicle technologies, such as in-vehicle sensors and communication devices. Examples of in-vehicle sensors include in-vehicle cameras (front and rear cameras), LiDAR (Light Detection and Ranging), millimeter-wave radar, and sonar. Examples of receiving devices that receive vehicle location information and traffic information include GPS (Global Positioning System) receivers, GNSS (Global Navigation Satellite System) receivers, and probe information receivers. If a vehicle is equipped with a car navigation system, it can acquire probe information held by various operators and government agencies. That is, location, speed, road conditions, weather, etc., can be monitored as the driving history of multiple vehicles, processed at a data center, and traffic conditions such as congestion information can be communicated and broadcast to users as probe information. Probe information receivers receive probe information and use it to recognize traffic conditions on the vehicle side. In addition, communication devices that communicate with external servers such as traffic monitoring systems operated by government agencies may be installed in the vehicle, and various information acquired by the communication device (for example, information from road traffic information communication systems) may be used to recognize traffic conditions. 【0025】 The vehicle recognition processing unit 12 acquires information about the vehicle's surrounding environment. Surrounding environment information is a general term for various information acquired by the vehicle, namely, various information obtained from on-board sensors, receiving devices, communication devices, etc. The vehicle's surrounding environment information reflects the vehicle's location information, the conditions around the vehicle, and traffic conditions. Based on the vehicle's surrounding environment information, the vehicle recognition processing unit 12 generates vehicle traffic condition recognition information. Vehicle traffic condition recognition information refers to information mechanically analyzed and recognized by the vehicle based on information obtained from various devices mounted on the vehicle, and is composed of multiple parameters, similar to the driver's traffic condition recognition information. Examples of multiple parameters include the vehicle type and location, the type and location of landmarks, time, weather, and traffic conditions. 【0026】 As described above, driver information and voice comments are input to the driver recognition processing unit 11, and information about the vehicle's surrounding environment is input to the vehicle recognition processing unit 12. In addition, the driver may input vehicle driving operations into the vehicle recognition processing unit 12. Examples of driving operation inputs include the driver's operation of the accelerator and brake pedals, the shift lever, steering wheel, and turn signal (indicator) operation. The vehicle recognition processing unit 12 generates vehicle traffic situation recognition information based on the vehicle's surrounding environment information and the driving operation inputs. 【0027】 The comparison and determination processing unit 13 has three inputs a, b, and c. The three inputs a, b, and c are the driver's traffic situation recognition information, the vehicle's traffic situation recognition information, and the driving operation input. If the driving operation input is input to the vehicle recognition processing unit 12, the comparison and determination processing unit 13 does not need to receive the driving operation input. The comparison and determination processing unit 13 evaluates the degree of agreement between at least the driver's traffic situation recognition information and the vehicle's traffic situation recognition information and generates the evaluation result. If the comparison and determination processing unit 13 has a driving operation input, the comparison and determination processing unit 13 evaluates the degree of agreement between the driver's traffic situation recognition information, the vehicle's traffic situation recognition information, and the driving operation input and generates the evaluation result. The method for evaluating the degree of agreement will be described later. 【0028】 The output unit 14 provides information based on the comparison and judgment processing unit 13's evaluation result of the degree of agreement. Instead of providing information, the system may, for example, perform accelerator suppression control or emergency stop using automatic braking. Information may be provided to the driver in the form of voice or text. In addition to notifying the driver of warnings and guidance information to attract the driver's attention, the system may also provide warnings to the driver by flashing a predetermined indicator installed in front of the driver's seat. For example, text information may be displayed on an in-vehicle monitor (e.g., the display of a car navigation system), or voice information may be emitted from the in-vehicle speaker. Information may also be provided to a terminal device such as a mobile terminal carried by the driver via a communication device. In this case, a communication link may be established in advance between the in-vehicle communication device and the driver's terminal device using Bluetooth communication or the like. Through this information provision, the system makes the driver aware of the differences between the driver's traffic situation perception information and the vehicle's traffic situation perception information as an evaluation result of the degree of agreement, thereby attracting the driver's attention. 【0029】 As part of providing information, the system may provide the driver with information related to driver error or misperception. For example, the system may provide the driver with pre-recorded voice guidance, guidance through the display of text and symbols, and flashing indicators. For example, it may be determined that the direction of travel of the vehicle estimated from the driver's traffic situation recognition information, which reflects the driver's voice comments, differs from the driving operation input actually performed by the driver. In this case, the driver assistance system 10 may, in addition to providing information to the driver, activate a misacceleration suppression function as a preventive safety function to suppress vehicle movement. Furthermore, the driver assistance system 10 may notify the driver to confirm the driver's driving operation input or to confirm the voice comments in order to correct whether the error was in the driving operation or the voice comments. Alternatively, the driver assistance system 10 may notify the driver to prompt them to make another voice comment. 【0030】 Next, we will explain the evaluation and scoring method for the degree of agreement between the driver's traffic situation awareness information, the vehicle's traffic situation awareness information, and the driving operation input. 【0031】 The driver visually identifies the type and location of landmarks and traffic signs, and then speaks aloud comments. The driver assistance system 10 compares the driver's traffic situation recognition information, constructed based on driver information and aloud comments, with the vehicle's traffic situation recognition information, constructed based on the vehicle's surrounding environment information, and evaluates the degree of agreement between the two. If the driver's traffic situation recognition information is valid for the vehicle's traffic situation recognition information, points are added to the degree of agreement. Validity means that the driver's traffic situation recognition information and the vehicle's traffic situation recognition information are identical, or that the difference between the two is within an acceptable range. Specific examples will be described later. 【0032】 The driver assistance system 10 uses information from the driver monitoring system to verify the likelihood (probability) of whether a voice comment was actually spoken by the driver based on their perception of the conditions inside and outside the vehicle. For example, if the driver's line of sight does not match the position of the object the driver spoke about, the driver's voice comment will not match the driver's perception, and a low likelihood will be verified. If the likelihood is low, the driver assistance system 10 will issue a warning to the driver. As a warning method, a predetermined warning message may be pre-recorded and automatically played to alert the driver. The purpose of verifying the likelihood is to exclude potentially false comments made by the driver from the voice recognition process. For example, if the driver makes a voice comment about the area behind the vehicle while reversing, even though the driver is not looking behind the vehicle using the rearview mirror, side mirrors, or rearview monitor, the voice comment will be considered a false comment. Voice comments with a low likelihood may be excluded from the driver's traffic situation recognition information. Furthermore, it is possible to consider not adding points for matching scores for voice comments with low likelihood, or deducting points for matching scores if false comments are spoken. 【0033】 Regarding the driver's voice comments, the driver assistance system 10 may continuously compare the driver's traffic situation recognition information with the vehicle's traffic situation recognition information and driving operation inputs, and score the degree of agreement for all items at regular time intervals. Items related to vehicle driving operations are expected to include items such as going straight, turning left, turning right, accelerating, decelerating, and stopping. When a traffic sign requires the vehicle to stop temporarily, the degree of agreement is scored based on whether or not the brake pedal has been operated as part of the driving operation input. 【0034】 The driver assistance system 10 may predefine specific situations and score the degree of agreement based on the relationship between the driver's traffic situation recognition information and the vehicle's traffic situation recognition information, the relationship between the driver's traffic situation recognition information and driving operation input, or combinations thereof. In specific situations, if the driver does not make a voice comment, or if the driver's voice comment is inappropriate, the degree of agreement may be reduced. Examples of specific situations include situations where a vehicle passes over a pedestrian crossing laid out on the road, situations where pedestrians or cyclists are present around the vehicle or on the pedestrian crossing when the vehicle turns right or left, and situations where the driver checks for safety to the left and right when the vehicle comes to a stop in accordance with a traffic sign. In specific situations, the driver assistance system 10 may prompt the driver to make a voice comment. In situations other than specific situations, the driver assistance system 10 does not need to prompt the driver to make a voice comment or to give a warning, so as not to interfere with the driver's driving. 【0035】 Next, we will explain an example of scoring the degree of agreement between the driver's traffic situation recognition information, including the driver's voice comments, the vehicle's traffic situation recognition information, and the driving operation input. Note that the degree of agreement may be calculated by adding or subtracting points based on a predetermined numerical value (for example, 100 points). In this case, a higher degree of agreement will be expressed as a numerical value greater than the baseline of 100 points, and a lower degree of agreement will be expressed as a numerical value less than the baseline of 100 points. 【0036】 (1) Examples of scoring based on the relationship between the vehicle and the target Figure 2(A) is a schematic diagram showing the positional relationship between the vehicle V and the landmark T. Figure 2(B) is a schematic diagram showing the positional relationship between the vehicle V and the pedestrian P. Figure 2(C) is a schematic diagram showing the display of the traffic sign S as seen from the vehicle V. 【0037】 Figure 2(A) determines whether the driver accurately recognizes the type and location of the object T in front of their vehicle V. For example, if object T is a preceding vehicle in front of vehicle V, and the driver utters the voice comment "There is a car," it is determined that the driver has accurately recognized the type of object T. In this case, the driver assistance system 10 (particularly the comparison and determination processing unit 13) adds points to the degree of agreement. On the other hand, if object T is a bicycle in front of vehicle V, and the driver utters the voice comment "There is a car," it is determined that the driver has misrecognized the type of object T. In this case, the driver assistance system 10 subtracts points from the degree of agreement. 【0038】 Figure 2(B) determines whether the driver accurately perceives the position of pedestrian P in front of their vehicle V. For example, if the driver utters the voice comment "There is a pedestrian on the left," it is determined that the driver accurately perceives the position of pedestrian P. In this case, the driver assistance system 10 adds points to the accuracy score. On the other hand, if the driver utters the voice comment "There is a pedestrian on the right," it is determined that the driver has misperceived the position of pedestrian P. In this case, the driver assistance system 10 deducts points from the accuracy score. 【0039】 Figure 2(C) shows how the system determines whether the driver accurately recognizes the traffic sign S in front of their vehicle V. The system can read the information displayed on the traffic sign, such as distance and speed, using a sign recognition function. For example, suppose the traffic sign S displays the number "60," indicating a speed limit of 60 km / h. In this case, if the driver says the voice comment, "This is a road with a 60 km / h speed limit," the system determines that the driver has accurately recognized the traffic sign S. In this case, the driver assistance system 10 adds points to the accuracy score. On the other hand, if the driver says the voice comment, "This is a road with an 80 km / h speed limit," the system determines that the driver has misrecognized the traffic sign S. In this case, the driver assistance system 10 deducts points from the accuracy score. 【0040】 (2) Examples of scoring based on the driver's visibility of pedestrians Figure 3(A) is a schematic diagram showing the direction of travel D of pedestrian P as seen from the vehicle V. Figure 3(B) is a schematic diagram showing the driver's line of sight directions L1 and L2 and the position of pedestrian P. Figure 3(C) is a schematic diagram showing the driver's line of sight directions L3 and L4 and the position of pedestrian P when the vehicle V is reversing. 【0041】 Figure 3(A) determines whether the driver accurately perceives the position and direction of travel D of pedestrian P. For example, suppose pedestrian P is moving in the direction of travel D in front of the vehicle V. When the driver utters the voice comment, "The pedestrian is walking to the right," it is determined that the driver accurately perceives the position and direction of travel D of pedestrian P. In this case, the driver assistance system 10 adds points for agreement. On the other hand, when the driver utters the voice comment, "The pedestrian is moving forward," although pedestrian P is moving forward from their perspective, this does not match the direction of travel D of pedestrian P as seen from the vehicle V. In this case, it is determined that the driver has misidentified the direction of travel D of pedestrian P, and the driver assistance system 10 deducts points for agreement. 【0042】 Figure 3(B) determines whether the driver has accurately seen pedestrian P located in front of their vehicle V. Here, it is assumed that pedestrian P is located to the left of the front of the vehicle. When the driver directs their gaze in the direction of their line of sight L1 and sees pedestrian P, and utters the voice comment "There is a pedestrian on the left," it is determined that the driver has accurately recognized the position of pedestrian P. In this case, the driver assistance system 10 adds points for agreement. On the other hand, when the driver directs their gaze in the direction of their line of sight L2, but utters the voice comment "There is a pedestrian on the left," the voice comment is correct, but the driver's line of sight L2 does not have sight of pedestrian P. Therefore, the driver assistance system 10 notifies the driver that "their line of sight is not aligned." The driver assistance system 10 may also deduct points for agreement due to the discrepancy between the position of pedestrian P indicated in the voice comment and the driver's line of sight L2. 【0043】 Figure 3(C) determines whether the driver accurately sees pedestrian P located behind the vehicle V when the vehicle V is reversing. When the vehicle V is reversing, if the driver sees pedestrian P by directing their gaze to the direction of sight L3 using the rearview mirror, side mirrors, and rearview monitor, and utters the voice comment "There is a pedestrian to the right rear," it is determined that the driver has accurately recognized the position of pedestrian P. In this case, the driver assistance system 10 adds points for agreement. On the other hand, if the driver's gaze is directed to the direction of sight L4, but they utter the voice comment "There is a pedestrian to the right rear," the voice comment is correct, but the driver's gaze direction L4 does not capture pedestrian P. Therefore, the driver assistance system 10 notifies the driver that "their gaze is not aligned." The driver assistance system 10 may also deduct points for agreement due to the discrepancy between the position of pedestrian P indicated in the voice comment and the driver's gaze direction L4. 【0044】 (3) Examples of scoring based on the driver's perception of driving operations Figure 4(A) is a schematic diagram showing the suppression of driving force in response to the driver's driving operations. Figure 4(B) is a schematic diagram showing the application of steering reaction force in response to the driver's steering wheel operation. As for driving force control, it is envisioned that the operation of the accelerator pedal will be suppressed or that braking force will be applied to the wheels. As for steering reaction force, it is envisioned that a load will be applied to the rotation axis of the steering wheel to make it difficult to drive, or that the driver's steering operation will be suppressed by a small motor embedded in the steering wheel. 【0045】 Figure 4(A) shows how the system determines whether the driver accurately understands their driving actions, particularly the shift lever and accelerator pedal operations. For example, if the driver operates the shift lever SL to the drive (D) position and says the voice comment "move forward," the system determines that the driver accurately understands the position of the shift lever SL. In this case, the driver assistance system 10 adds points to the accuracy score. On the other hand, if the driver operates the shift lever SL to the reverse (R) position but says the voice comment "move forward," the system determines that the driver has misunderstood the position of the shift lever SL. In this case, the driver assistance system 10 deducts points from the accuracy score and implements a drive force suppression DS for the driver's accelerator pedal operation. This prevents the vehicle V from accidentally moving backward due to the driver's misunderstanding of the shift lever operation. 【0046】 In Figure 4(A), when the driver presses the accelerator pedal AP to turn it on and utters the voice comment "accelerate," the system determines that the driver has accurately recognized the accelerator pedal operation. In this case, the driver assistance system 10 adds points to the degree of agreement. On the other hand, when the driver turns on the accelerator pedal AP but utters the voice comment "deceler," the system determines that the driver has misinterpreted the operation of the accelerator pedal AP. In this case, the driver assistance system 10 deducts points from the degree of agreement and implements a drive force suppression DS in response to the driver's accelerator pedal operation. This prevents the vehicle V from accelerating due to a misinterpretation of the driver's accelerator pedal operation. 【0047】 Figure 4(B) shows how the system determines whether the driver accurately understands the steering wheel operation. For example, if the driver turns the steering wheel H to the right and says the voice comment "turn right," the system determines that the driver accurately understands the steering wheel operation. In this case, the driver assistance system 10 adds points to the accuracy score. On the other hand, if the driver turns the steering wheel H to the right but says the voice comment "turn left," the system determines that the driver has misinterpreted the steering wheel operation. In this case, the driver assistance system 10 deducts points from the accuracy score and applies a steering reaction force SR to the steering wheel H in response to the driver's operation of the steering wheel H. This applies a load to the rightward turn of the steering wheel H, contrary to the driver's perception, and suppresses the driver's operation of the steering wheel H due to misinterpretation. 【0048】 Next, we will explain how the driver will be notified of the score of agreement calculated by the driver assistance system 10. The driver assistance system 10 will notify the driver of the score of the commentary driving after the driver has finished driving their own vehicle V. Along with notifying the driver of the score, the following information will be provided to the driver in order to maintain and improve the driver's awareness of safe driving. 【0049】 When the driver assistance system 10 deducts points for accuracy, it is assumed that a dangerous situation in terms of traffic safety has occurred. For example, this could be a situation where a preventive safety function such as an emergency automatic braking function is activated, or a situation where the driver's driving behavior is close to the threshold for activating a preventive safety function. As for the driver's driving behavior, it is conceivable to quantify driving operations that could lead to violations of safety standards, such as sudden starts, sudden acceleration, sudden braking, and inattention to the road ahead, and compare them with a predetermined threshold. In this case, the hazardous object markers and their behavior may be displayed on the in-vehicle monitor on a map indicating the location of the hazard. Examples of hazardous objects and markers include objects that have fallen from other vehicles near the vehicle's lane, collapsed parts of buildings on the side of the road, landslides in forests, river flooding, and people that suddenly appear from the vehicle's blind spot. Alternatively, it is conceivable to display actual footage of the driver's encounter on the in-vehicle monitor and provide the driver with comments explaining appropriate judgments and actions to deal with markers that the driver should recognize and hazards that should be anticipated. For explanatory comments, pre-recorded audio may be played, or text may be displayed. 【0050】 If the driver assistance system 10 deducts points for accuracy, it may record data related to the surrounding environment at the time of detection of hazardous objects or landmarks in order to provide the driver with information about dangerous situations. If the memory or storage capacity of the driver assistance system 10 is limited, it may be considered to selectively save data from more dangerous situations. Regarding the degree of danger, the severity of traffic safety for each dangerous item may be set in advance, and data from high-danger situations may be overwritten with data from low-danger situations. 【0051】 The scoring results and explanations notified to the driver may be output as audio via a speaker or other means, or displayed on an in-vehicle monitor or display device. Alternatively, data related to the scoring results and explanations may be transmitted to a terminal device such as a smartphone owned by the driver, so that the driver can receive the scoring results and explanations as audio playback or text display on the terminal device. 【0052】 Furthermore, in order to equalize the variability in scoring results due to vehicle mileage and driving time, the numerical values ​​related to the scoring results may be averaged for each mileage or driving time. Alternatively, statistics on the numerical values ​​of the scoring results for each mileage or driving time may be collected, and the average or median value may be calculated. 【0053】 Furthermore, the vehicle may be equipped with a switch or other control that allows the driver to arbitrarily disable the driver assistance functions of the driver assistance system 10. For this reason, the driver assistance system 10 may be configured to operate only when the driver activates the driver assistance functions by operating the switch. Since the driver does not always perform commentary driving, the driver can arbitrarily activate the driver assistance functions to perform commentary driving. 【0054】 [1] First Embodiment Next, the configuration and functions of the driver assistance system 100 according to the first embodiment of the present application will be described in detail. Figure 5 is a block diagram of the driver assistance system 100 according to the first embodiment of the present application. In Figure 5, three data d1 to d3 are provided to the driver assistance system 100 based on the driver's recognition of the traffic situation. Data d1 is the driver's voice comment collected by an on-board microphone or the like. Data d2 is a facial image showing the driver's gaze and face direction, captured by an on-board camera of the driver monitoring system. Data d3 is driving operations related to the accelerator, brake pedals, shift lever, etc., operated by the driver. The driver assistance system 100 also receives data d4 and data d5. Data d4 shows the traffic situation outside the vehicle detected by various sensing devices such as cameras, millimeter-wave radar, and LiDAR mounted on the vehicle. Data d5 is GPS signals and probe information obtained by monitoring the traffic situation with satellites or traffic monitoring systems. 【0055】 In the driver assistance system 100, the voice recognition unit 101 receives data d1 and performs voice recognition on the driver's voice comments. Subsequently, the natural language processing unit 106 performs natural language processing on the voice comments after voice recognition. As a result, as shown in Figure 1, the voice comments after voice recognition and natural language processing are output. The face recognition unit 102 receives data d2 and performs face recognition processing on the driver's face image. Subsequently, the face information reading unit 107 reads face information indicating the driver's gaze direction and face orientation from the face image after face recognition processing. As a result, as shown in Figure 1, the driver information after face recognition processing and face information reading is output. In addition, the driving operation input unit 103 receives data d3 and uses it as the driver's driving operation input. 【0056】 The sensing unit 104 receives data d4 and aggregates the detection results of various sensing devices mounted on the vehicle, such as images of the area around the vehicle and the driving status of other vehicles, to detect traffic conditions. The communication device 105 receives data d5 and detects a map of the area around the vehicle and traffic conditions. As a result, as shown in Figure 1, the output of the sensing unit 104 and the output of the communication device 105 are combined to output information about the vehicle's surrounding environment. 【0057】 The driver recognition processing unit 108 receives voice comments as the output of the natural language processing unit 106 and driver information as the output of the facial information reading unit 107. Subsequently, the driver recognition processing unit 108 generates driver traffic situation recognition information based on the voice comments and driver information. The vehicle recognition processing unit 109 receives vehicle surrounding environment information as the output of the sensing unit 104 and the communication device 105, and generates vehicle traffic situation recognition information. 【0058】 The comparison and judgment processing unit 110 receives driving operation input from the driving operation input unit 103, driver traffic situation recognition information from the driver recognition processing unit 108, and vehicle traffic situation recognition information from the vehicle recognition processing unit 109. The comparison and judgment processing unit 110 compares the driving operation input with the driver and vehicle traffic situation recognition information to determine the validity of the driver's driving operation. In other words, the comparison and judgment processing unit 110 scores the degree of agreement between the voice comment, the driving operation input, and the vehicle's surrounding environment information, and determines the validity of the driving operation. The comparison and judgment processing unit 110 may also store the comparison results of the driver and vehicle traffic situation recognition information for display on the in-vehicle monitor. 【0059】 The display processing unit 111 generates display data for displaying the scoring results of the commentary driving, that is, the score and evaluation of the degree of agreement. Subsequently, the display device 113, such as an in-vehicle monitor, displays the scoring results of the commentary driving based on the display data. Alternatively, the display data may be transmitted to a terminal device 120, such as a smartphone owned by the driver, which is linked with the driver assistance system 100, and the scoring results of the commentary driving may be displayed on the terminal device 120. 【0060】 The notification processing unit 112 notifies the driver of the scoring results for the commentary driving. Furthermore, if there are no voice comments from the driver, or if the validity of the driver's voice comments (i.e., the validity of the voice comments in relation to the driving operation input and traffic situation information) is low, the notification processing unit 112 prompts the driver to provide appropriate voice comments. For example, it may play an audio guidance or warning sound for the driver, or it may flash a predetermined indicator on the dashboard on the driver's side. Voice comments with low validity may be excluded from the driver's traffic situation recognition information. Additionally, points may not be added to the accuracy score for voice comments with low validity, or points may be deducted for accuracy. 【0061】 Next, the operation and processing of the driver assistance system 100 according to the first embodiment of the present application will be described in detail. Figures 6 and 7 are flowcharts of the driver assistance system 100 according to the first embodiment of the present application. The flowchart in Figure 6 includes steps S1 to S14, and the flowchart in Figure 7 includes steps S15 to S25. 【0062】 In Figures 6 and 7, steps S1 to S21 are repeated for each driving cycle (step S22). If the driver assistance function is turned off, the flow transitions from step S1 to step S21, and it is determined again whether the driving cycle has ended (step S22). 【0063】 In Figure 6, when the driver assistance function is turned on, the driver inputs driving operations, and the sensing unit 104 and communication device 105 acquire information about the vehicle's surrounding environment (steps S2-S3). The vehicle recognition processing unit 109 constructs vehicle traffic situation recognition information based on the vehicle's surrounding environment information (step S4). Meanwhile, the voice recognition unit 101 recognizes the driver's voice (step S5). However, if there is no voice comment from the driver, the flow moves from step S6 to step S12. In step S12, it is determined whether the vehicle's traffic situation recognition information is in a specific situation that requires a voice comment from the driver. If it is a specific situation, the notification processing unit 112 prompts the driver for a voice comment (step S13). 【0064】 If there is a voice comment from the driver, the flow proceeds from step S6 to step S7, where the natural language processing unit 106 performs natural language processing on the voice comment. The face recognition unit 102 recognizes the driver's face image, and the face information reading unit 107 reads the driver's gaze direction and face orientation (steps S8, S9). The driver recognition processing unit 108 compares the driver's voice comment with the driver's gaze direction and face orientation to determine the likelihood of the driver's voice comment (step S10). If the likelihood of the driver's voice comment is low, a warning is issued to the driver via the notification processing unit 112 (step S11). If the likelihood of the driver's voice comment is not low, the driver recognition processing unit 108 constructs the driver's traffic situation recognition information based on the voice comment and driver information (step S14). 【0065】 After step S13 or step S14 in Figure 6, the flow proceeds to step S15 in Figure 7, where the comparison and determination processing unit 110 compares the driver's traffic situation recognition information with the vehicle's traffic situation recognition information and the driving operation input. Based on the comparison results, the comparison and determination processing unit 110 determines the validity of the driver's traffic situation recognition information (step S16). Subsequently, the comparison and determination results are stored in a storage device or the like (step S17). 【0066】 If the comparison judgment result evaluation of the comparison judgment processing unit 110 is not low, the flow proceeds from step S18 to step S22. On the other hand, if the comparison judgment result evaluation is low, the driver assistance system 100 obtains the position of its own vehicle from the map data acquired by the communication device 105 and saves it to the storage device (step S19). The driver assistance system 100 also saves the driver and vehicle's traffic situation recognition information to the storage device (step S20). Furthermore, the driver assistance system 100 saves video data captured by the driver monitoring system and / or video data of the area around the vehicle captured by an in-vehicle camera, etc., to the storage device (step S21). The various information and data stored as described above will be used later for verification of vehicle driving by the driver. 【0067】 After the driving cycle ends, the driver assistance system 100 determines whether the comparison judgment results described above have been stored in the memory device (steps S22, S23). If the comparison judgment results have not been stored in the memory device, the flow shown in Figures 6 and 7 ends. On the other hand, if the comparison judgment results have been stored in the memory device, the driver assistance system 100 scores the degree of agreement derived from the comparison judgment results stored in the memory device, extracts important points for the safe driving of the vehicle (for example, hazardous materials or locations where hazards may occur), and creates an explanation for the driver. Subsequently, the display processing unit 111 outputs the scoring results of the driver's voice comments to the display device 113, and the notification processing unit 112 outputs the scoring results to the terminal device 120. This ends the flow shown in Figures 6 and 7. 【0068】 [2] Second embodiment Next, the configuration and functions of the driver assistance system 200 according to the second embodiment of the present application will be described. Figure 8 is a block diagram of the driver assistance system 200 according to the second embodiment of the present application. In Figure 8, the same reference numerals are used for parts that are the same as those in Figure 5, and their descriptions are omitted. The driver assistance system 200 shown in Figure 8 comprises a comparison and determination processing unit 110, followed by a confirmation processing unit 114 and a suppression control unit 115. The confirmation processing unit 114 confirms the driver's driving operations and voice comments based on the determination result of the comparison and determination processing unit 110. In addition, if the driver does not speak a voice comment, it prompts the driver to speak a voice comment. For example, it may play an audio such as a guidance sound or a warning sound to the driver, or it may flash a predetermined indicator provided on the dashboard on the driver's side. 【0069】 The suppression control unit 115 receives the driving operation input from the driving operation input unit 103 and the comparison judgment result from the comparison judgment processing unit 110, and suppresses acceleration / decision related to the driver's accelerator and brake pedal operations and forward / reverse movements related to the shift lever operation according to the traffic conditions. For example, in a situation where a vehicle is transitioning from a stopped state to a reverse state, the driver may have checked for safety behind the vehicle, but the transmission may be set to a forward gear with the shift lever in drive (D). Here, suppose the driver makes a voice comment saying "Rear, clear!" and has checked for safety behind the vehicle. In this case, when the driver presses the accelerator pedal, the driving assistance system 200 installed in the vehicle determines that the driver's voice comment and the driving operation are different. As a result, the suppression control unit 115 activates and suppresses the vehicle from moving forward. In addition, the confirmation processing unit 114 plays a pre-recorded voice to alert the driver in order to confirm the driver's driving operation. 【0070】 Next, the operation and processing of the driver assistance system 200 according to the second embodiment of the present application will be described. Figures 9 and 10 are flowcharts of the driver assistance system 200 according to the second embodiment of the present application. In the flowcharts of Figures 9 and 10, the same reference numerals are used for steps that are the same as those in the flowcharts of Figures 6 and 7, and their explanations are omitted. Specifically, the flowcharts of Figures 9 and 10 include steps S2 to S16, similar to those in Figures 6 and 7. For this reason, steps S101 to S104 will be described in the flowchart of Figure 9, and steps S105 to S112 will be described in the flowchart of Figure 10. 【0071】 The flows in Figures 9 and 10 are executed for each driving cycle (steps S101 to S112). In Figure 9, step S101 determines whether the driver assistance function of the driver assistance system 200 is off or off. If the driver assistance function is off, the flow proceeds to step S112. If the driver assistance function is not off, the sensing unit 104 and communication device 105 of the driver assistance system 200 acquire information about the vehicle's surrounding environment (step S102). Subsequently, the sensing unit 104 acquires the vehicle speed (step S103). In the driver assistance system 200, in order to activate suppression control by the suppression control unit 115, suppression control permission areas are pre-set according to the driver's voice comments during commentary driving. For example, in specific areas such as near intersections or parking lots, suppression control of the vehicle's driving force and braking force is permitted according to the validity of the driver's voice comments. Therefore, in step S104, it is determined whether the vehicle's current position is within the suppression control permission area based on the driver's voice comments. If it is not a suppression control permission area, the flow proceeds from step S104 to step S109. 【0072】 If the vehicle's current location is within the area where suppression control is permitted by voice comment, the flow moves from step S104 to step S2. From step S2 to step S16, the functions and processes described in Figures 6 and 7 are executed. 【0073】 In Figure 10, after step S12 or step S16, the flow proceeds to step S105, where it is determined whether the voice comment is appropriate for the driver's driving operation (i.e., driving operation input). If the voice comment is not appropriate, the confirmation processing unit 114 asks the driver to confirm the driving operation input (step S106). Subsequently, if the state of not receiving an appropriate voice comment from the driver continues for a certain period of time (for example, several minutes), the suppression control unit 115 starts suppression control for the driver's driving operation (for example, accelerator pedal operation) (steps S107, S108). 【0074】 If the vehicle's current position is not within the suppression control permitted area (step S104), or if the voice comment in response to the driving operation input is appropriate (step S105), the flow proceeds to step S109, where it is determined whether the driver assistance system 200 is performing suppression control based on the voice comment. If suppression control is being performed, the driver assistance system 200 notifies the driver that the suppression control has been released, and then releases the suppression control (steps S110, S111). After that, when it is determined that the driving cycle has ended (step S112), the flows in Figures 9 and 10 end. 【0075】 The driver assistance systems 10, 100, and 200 can be installed in vehicles to assist drivers, but can also be used as driving simulators for novice drivers, elderly drivers, and trainees, as well as for practical driving training in driver training courses. In other words, by scoring and evaluating the driver's commentary driving, it is possible to determine whether the driver is recognizing the surrounding environment of the vehicle and making accurate judgments when operating the vehicle. In addition to the driver's driving operations, it can also determine whether various settings of the vehicle are appropriate. For example, it can determine whether the initial settings at the start of driving and the route setting to the destination are appropriate. 【0076】 Next, the features and effects of the driver assistance systems 10, 100, and 200 according to the embodiment of the present application will be described. (1) The driving assistance system comprises a driver recognition processing unit that performs voice recognition on voice comments spoken by the vehicle's driver and generates first traffic situation recognition information recognized by the driver; a vehicle recognition processing unit that generates second traffic situation recognition information recognized based on surrounding environment information acquired by a driving situation recognition function installed in the vehicle; and a comparison and determination processing unit that determines the validity of the voice comments by comparing the first traffic situation recognition information with the second traffic situation recognition information and calculates the degree of agreement by comparing the first traffic situation recognition information with the second traffic information recognition information. The driving assistance system performs predetermined driving assistance to the driver according to the validity and degree of agreement. 【0077】 The driving situation recognition function includes, for example, a communication device that acquires traffic information by communicating with an in-vehicle camera or an external server, or a GPS system that detects the vehicle's position. Surrounding environment information refers to the surrounding environment of the vehicle, such as the presence or absence of people or landmarks around the vehicle, and traffic conditions such as traffic congestion caused by other vehicles. In addition to surrounding environment information, the vehicle recognition processing unit may generate second traffic situation recognition information using driving operation inputs such as accelerator / brake pedal operation, shift lever operation, and steering wheel operation. The first traffic information recognition information indicates the driver's perception of the traffic situation, and the validity of the voice comment is determined by whether the difference with the second traffic situation recognition information is within an acceptable range. The acceptable range of validity may be arbitrarily determined by the driver or the developer of the driving assistance system. Multiple indicators can be envisioned as the degree of agreement between the first and second traffic situation recognition information. For example, the degree of agreement may be scored for each of the vehicle's multiple behaviors, such as moving forward, backward, accelerating, decelerating, turning right, turning left, and going straight. The driver assistance system allows for a review of whether the driver's commentary driving is appropriate, thereby helping to maintain and improve the driver's awareness of safe driving. 【0078】 (2) The first traffic situation recognition information and the second traffic situation recognition information may each recognize objects located outside the vehicle. Examples of objects include other vehicles, pedestrians, traffic signs, and obstacles. 【0079】 (3) If the comparison and judgment processing unit determines that the validity of the voice comment is low, the first traffic situation recognition information that reflects the voice comment may be excluded from the calculation of the degree of agreement. This prevents the calculation of an inaccurate degree of agreement using the first traffic situation recognition information that reflects a voice comment with low validity. 【0080】 (4) If the comparison and judgment processing unit determines that the validity of the voice comment is low, the degree of agreement may be reduced. 【0081】 (5) When the objects recognized by the first traffic situation recognition information and the second traffic situation recognition information are of the same type, the comparison judgment processing unit may set the degree of agreement high. Here, the degree of agreement refers to an index related to the type of object. Depending on whether the object present around the vehicle is an obstacle, a road sign, or a pedestrian, the behavior of the vehicle and the driver's response will differ. For example, if the driver can take evasive action to avoid contact with an obstacle, safe driving can be achieved, but if the driver misjudges the type of object, they may not be able to take appropriate action. For this reason, the driving support system is designed to review the accuracy of the driver's identification of the type of object. This allows the system to grasp the driver's recognition ability and encourage the maintenance and improvement of the driver's awareness of safe driving. It also allows the system to review the accuracy of the driver's judgment of the type of traffic sign (speed limit, no overtaking, stop sign, etc.). 【0082】 (6) In relation to the above, when the targets recognized by the first traffic situation recognition information and the second traffic situation recognition information are of different types, the comparison judgment processing unit may set the degree of agreement low. 【0083】 (7) When the front-rear or left-right position of the object recognized by the first traffic situation recognition information and the second traffic situation recognition information are the same, the comparison judgment processing unit may set the degree of agreement high. Here, the degree of agreement refers to an index relating to the relative positional relationship between the vehicle and the object. In other words, it determines whether the driver is accurately aware that the object is located in front of or behind the vehicle, or that the object is located to the left or right of the vehicle. 【0084】 If the driver can know in advance the position from which obstacles and other targets will move relative to the vehicle, they can take evasive action to avoid contact with the targets and drive safely. On the other hand, if the driver misjudges the position of the targets, they may not be able to take appropriate action. Therefore, by reviewing the accuracy of the target positions, the driver assistance system can understand the driver's perception ability and encourage the maintenance and improvement of the driver's awareness of safe driving. 【0085】 (8) In connection with the above, if the front-to-back or left-to-right positions of the object recognized by the first traffic situation recognition information and the second traffic situation recognition information are different relative to the vehicle, the comparison and determination processing unit may lower the degree of agreement. 【0086】 (9) When the direction of movement of the object recognized by the first traffic situation recognition information and the second traffic situation recognition information is the same, the comparison judgment processing unit may set the degree of agreement high. Here, the degree of agreement refers to an index related to the direction of movement of the object. If it is known in advance that an object such as an obstacle is moving toward the vehicle, the driver can take evasive action to avoid contact with the object according to the direction of movement of the object and drive safely. On the other hand, if the driver misjudges the direction of movement of the object, they may not be able to take appropriate action. For this reason, the driving support system can grasp the driver's recognition ability by reviewing the accuracy of the direction of movement of the object and promote the maintenance and improvement of the driver's awareness of safe driving. 【0087】 (10) In relation to the above, when the direction of movement of the object recognized by the first traffic situation recognition information and the second traffic situation recognition information is different relative to the vehicle, the comparison judgment processing unit may lower the degree of agreement. 【0088】 (11) The driver recognition processing unit generates first traffic situation recognition information recognized by the driver by analyzing the driver's gaze direction in addition to speech recognition of the voice comment, and the comparison judgment processing unit may set the degree of agreement depending on whether the position of the object indicated by the voice comment included in the first traffic situation recognition information is in the driver's gaze direction. Here, the degree of agreement means an index relating to the relationship between the driver's gaze direction and the position of the object. For example, the driver's gaze direction is detected by a driver monitoring system installed in the vehicle. Even if the driver's voice comment accurately grasps the position of an object such as an obstacle or traffic sign, it is assumed that the driver's gaze does not necessarily see the object. In other words, even if the voice comment accidentally captures the position of the object accurately, if the driver's gaze direction does not capture the position of the object, the degree of agreement is set to prevent the degree of agreement from being increased due to accidental circumstances. This can promote the accuracy of commentary driving and the driver's recognition accuracy. 【0089】 (12) In the above, the comparison and determination processing unit may increase the degree of agreement when the location of the object indicated by the voice comment is in the direction of the driver's line of sight. 【0090】 (13) In the above, the comparison and judgment processing unit may give a predetermined notification to the driver if the location of the object indicated by the voice comment is not in the driver's line of sight. Here, the predetermined notification refers to a notification that prompts the driver to turn their line of sight towards the object. In other words, by having the driver give a voice comment for objects that they have clearly seen, the accuracy of the commentary driving and the driver's recognition accuracy can be improved. 【0091】 (14) The vehicle is equipped with a viewing device that allows the driver to see a direction different from the direction of the driver's line of sight, and the comparison judgment processing unit may set the degree of agreement depending on whether the driver can see the target in a direction different from the direction of the driver's line of sight through the viewing device. Examples of viewing devices include rearview mirrors, side mirrors, and rearview monitors mounted on the vehicle. In addition to the driver directly seeing the target, it is assumed that the driver will see the target indirectly through the viewing device. In particular, the accuracy of the driver's recognition can be improved by evaluating the driver's voice comments when the vehicle is reversing. 【0092】 (15) The first driving situation recognition information and the second driving situation recognition information may each reflect the driver's driving operations. Driver's driving operations include operating the accelerator and brake pedals and operating the shift lever. 【0093】 (16) When the direction of travel of the vehicle recognized by the first traffic situation recognition information and the second traffic situation recognition information are the same, the comparison judgment processing unit may set the degree of agreement to a high value. Here, the degree of agreement refers to an index related to the direction of travel corresponding to the operation of the vehicle's shift lever. Examples of shift lever operations include D range (forward), R range (reverse), P range (stop), etc. In this way, by reviewing the accuracy of the shift lever operation corresponding to the driver's voice comments, it is possible to encourage the maintenance and improvement of the driver's awareness of safe driving. 【0094】 (17) In relation to the above, when the direction of travel of the vehicle recognized by the first traffic situation recognition information and the second traffic situation recognition information are different, the comparison judgment processing unit may lower the degree of agreement. 【0095】 (18) When a driver operates an operator that controls the acceleration or deceleration of the vehicle, if the operator reflected in the first traffic situation recognition information and the second traffic situation recognition information is the same, the comparison judgment processing unit may set a high degree of agreement. Here, the degree of agreement refers to an index related to operators such as accelerator and brake pedals installed in the vehicle. For example, if the driver operates the accelerator pedal and also makes a voice comment indicating the operation of the accelerator pedal, the type of operator operated by the driver and the voice comment will match. In this case, points will be added to the degree of agreement. In other words, by reviewing the accuracy of the driver's accelerator and brake pedal operation, it is possible to encourage the maintenance and improvement of the driver's awareness of safe driving. 【0096】 (19) When an operator that accelerates or decelerates the vehicle is operated as part of the driver's driving operation, if the operator reflected in the first traffic situation recognition information and the second traffic situation recognition information are different, the comparison judgment processing unit may reduce the degree of agreement. For example, it is assumed that the driver made a voice comment that they were operating the accelerator pedal, but in reality they were operating the brake pedal. In this way, when there is a discrepancy between the operator that the driver actually operates and the voice comment, the degree of agreement is reduced. 【0097】 (20) When the steering element used to steer the vehicle is operated as part of the driver's driving operation, if the steering direction of the steering element reflected in the first traffic situation recognition information and the second traffic situation recognition information are the same, the comparison judgment processing unit may set a high degree of agreement. Here, the degree of agreement refers to an index related to the steering direction of the steering element such as the steering wheel. For example, if the driver makes an audio comment that they will turn left and actually turns the steering wheel to the left, the degree of agreement will be increased. In particular, by reviewing the accuracy of the driver's steering direction when the vehicle is reversing, it is possible to encourage the maintenance and improvement of the driver's awareness of safe driving. 【0098】 (21) When the steering mechanism used to steer the vehicle is operated as part of the driver's driving operation, if the steering direction of the steering mechanism reflected in the first traffic situation recognition information and the second traffic situation recognition information are different, the comparison judgment processing unit may reduce the degree of agreement. For example, if the driver makes a voice comment that they will turn left, but in reality the driver turns the steering mechanism such as the steering wheel to the right, the degree of agreement will be reduced. 【0099】 (22) The degree of agreement calculated by the comparison and judgment processing unit may be notified to the driver. The degree of agreement may be displayed on the navigation system or meters installed in the vehicle, or it may be transmitted to a terminal device such as a mobile terminal owned by the driver and displayed. 【0100】 (23) The comparison and judgment processing unit may set an index related to the degree of agreement for each predetermined specific situation. A specific situation is defined by either or both of a specific place and / or time. For example, it means a situation in which a driver should pay attention at an intersection or parking lot in the evening or at night. Examples of an index related to the degree of agreement include moving forward, backward, turning right, turning left, and stopping (parking) of a vehicle. By setting an index of the degree of agreement for each specific situation, the driver's attention will increase, and the driver will be able to concentrate more on driving operations. This will help maintain and improve the driver's awareness of safe driving. 【0101】 (24) The comparison and judgment processing unit may set an index related to the degree of agreement for each predetermined specific situation, and in a specific situation where the degree of agreement falls below a predetermined threshold, it may give a predetermined notification to the driver. This allows the driver to be notified in advance that they will encounter a specific situation that they are not comfortable with, so that they should pay attention. This helps to maintain and improve the driver's awareness of safe driving. 【0102】 For example, in situations where points were deducted for matching during past vehicle driving, or in situations where a danger to vehicle driving is anticipated based on past history and traffic conditions, the location and behavior of hazardous objects or targets may be displayed on the in-vehicle monitor on a map near the hazardous area. Dangerous situations include traffic conditions where preventive safety functions are activated, or situations where the driving state approaches a predetermined threshold (e.g., TTL (Time To Collision)) that triggers the preventive safety functions. In addition, external video footage of the vehicle during driving may be displayed on the in-vehicle monitor to explain the targets that the driver should recognize and the dangers that should be anticipated, enabling the driver to make appropriate judgments and perform appropriate driving operations. For example, the driver may be advised to slow down and drive carefully in places or times where danger is anticipated. 【0103】 Advice and explanations to drivers can be given at the end of a driving cycle, when the driver reflects on their driving style and provides feedback. Alternatively, advice and explanations can be given to the driver when they are about to drive through dangerous areas again. 【0104】 Furthermore, the system may provide the driver with information about situations in the past where the matching score was reduced, as well as situations that are anticipated to pose a danger to vehicle operation based on past history and traffic conditions. When such situations are detected, various data may be stored in a storage device. If the storage capacity of the storage device is limited, situations with a greater reduction in the matching score or situations that pose a greater danger may be selectively stored in the storage device. 【0105】 (25) If the degree of agreement calculated by the comparison and judgment processing unit is lower than a predetermined threshold, the driver may be notified that the first traffic situation recognition information and the second traffic situation recognition information indicate different driving operations. The predetermined threshold can also be set for each indicator of agreement (acceleration, deceleration, forward, reverse, etc.). By notifying the driver of the difference between the first and second traffic situation recognition information, the driver can become aware of errors in their driving operations or in the voice comments. This can provide an opportunity for the driver to correct the error, thereby helping to maintain and improve the driver's awareness of safe driving. 【0106】 (26) If the degree of agreement calculated by the comparison judgment processing unit is lower than a predetermined threshold, and the operator reflected in the first traffic situation recognition information and the second traffic situation recognition information corresponds to the vehicle moving forward or backward, the vehicle may further be provided with a suppression control unit that suppresses the vehicle's driving force. In particular, when the vehicle is moving forward or backward, accidents such as contact with or collision with an object may occur due to the driver's inattention. For this reason, considering the possibility that the driver may make a mistake in driving operations, if the degree of agreement is low, suppression control as a preventive safety function is activated. For example, when the shift lever is in drive (D) or reverse (R), the vehicle's driving force is automatically suppressed regardless of the driver's driving operation due to a decrease in the degree of agreement. This can reduce damage caused by the vehicle moving in a direction contrary to the driver's intention. In other words, it is possible to prevent accidents caused by discrepancies between the driver's perception and actual driving operations. When the shift lever is in a non-driving range, such as the P range or N range, it is assumed that serious personal or property damage is unlikely to occur, so the preventive safety function may not be activated. 【0107】 (27) If the degree of agreement calculated by the comparison judgment processing unit is lower than a predetermined threshold, and the operator reflected in the first traffic situation recognition information and the second traffic situation recognition information corresponds to vehicle acceleration, the system may further be provided with a suppression control unit that suppresses the vehicle's driving force. For example, if the driver has made a voice comment to decelerate but has mistakenly pressed the accelerator pedal, the vehicle will accelerate against the driver's intention. In this case, as a preventive safety function, the driving force of the vehicle is suppressed in response to the driver's erroneous operation of the accelerator pedal. This reduces damage caused by contact with or collision with an object due to the vehicle accelerating against the driver's intention. On the other hand, if the driver has made a voice comment to accelerate but has mistakenly pressed the brake pedal, it is unlikely that there will be significant damage, so there is no particular need to activate the preventive safety function. 【0108】 (28) If the degree of agreement calculated by the comparison judgment processing unit is lower than a predetermined threshold, the vehicle may further be provided with a suppression control unit that applies a steering reaction force to the steering element operated by the driver. For example, it is conceivable that the driver may utter a voice comment indicating that they will turn right, but mistakenly turn the steering element, such as the steering wheel, to the left. In this case, as a preventive safety function, by applying a steering reaction force to the steering element for turning to the left, damage caused by the vehicle moving in a steering direction contrary to the driver's intention can be reduced. The steering reaction force may be applied mechanically by applying a load in the turning direction of the steering element. Alternatively, a small motor that obstructs the rotation of the steering element's rotation axis may be provided to apply the steering reaction force to the steering element. 【0109】 While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols] 【0110】 10. Driver assistance systems 11. Driver Recognition Processing Unit 12. Vehicle Recognition Processing Unit 13 Comparison and Judgment Processing Unit 14 Output section 100, 200 driver assistance systems 101 Voice Recognition Unit 102 Face Recognition Unit 103 Operation Input Section 104 Sensing Unit 105 Communication equipment 106 Natural Language Processing Unit 107 Face Information Reading Unit 108 Driver Recognition Processing Unit 109 Vehicle Recognition Processing Unit 110 Comparison and Judgment Processing Unit 111 Display Processing Unit 112 Notification Processing Unit 113 Display device 114 Verification Processing Unit 115 Suppression Control Unit 120 Terminal devices AP Accelerator Pedal D Moving direction DS driving force suppression H Handle L1, L2, L3, L4 line of sight direction P Pedestrian S traffic sign SL Shift Lever SR steering reaction force T target V My vehicle

Claims

[Claim 1] A driver recognition processing unit that performs speech recognition on voice comments spoken by the vehicle driver and generates first traffic situation recognition information recognized by the driver, A vehicle recognition processing unit generates second traffic situation recognition information based on surrounding environment information acquired by a driving situation recognition function installed in the vehicle, The system includes a comparison and determination processing unit that determines the validity of the voice comment by comparing the first traffic situation recognition information with the second traffic situation recognition information, and calculates the degree of agreement by comparing the first traffic situation recognition information and the second traffic situation recognition information. A driver assistance system characterized by performing predetermined driver assistance for the driver in accordance with the aforementioned validity and degree of agreement. [Claim 2] The driving assistance system according to claim 1, characterized in that the first traffic situation recognition information and the second traffic situation recognition information each recognize an object located outside the vehicle. [Claim 3] The driving support system according to claim 2, characterized in that, if the comparison and determination processing unit determines that the validity of the voice comment is low, the first traffic situation recognition information reflecting the voice comment is excluded from the calculation of the degree of agreement. [Claim 4] The driving support system according to claim 2, characterized in that if the comparison and determination processing unit determines that the validity of the voice comment is low, the degree of agreement is reduced. [Claim 5] The driving support system according to claim 2, characterized in that when the objects recognized by the first traffic situation recognition information and the second traffic situation recognition information are of the same type, the comparison judgment processing unit increases the degree of agreement. [Claim 6] The driving support system according to claim 2, characterized in that when the first traffic situation recognition information and the second traffic situation recognition information recognize different types of targets, the comparison and determination processing unit lowers the degree of agreement. [Claim 7] The driving assistance system according to claim 2, characterized in that when the front-to-rear or left-to-right position of the object relative to the vehicle, as recognized by the first traffic situation recognition information and the second traffic situation recognition information respectively, is the same, the comparison determination processing unit increases the degree of agreement. [Claim 8] The driving assistance system according to claim 2, characterized in that when the front-to-rear or left-to-right position of the object recognized by the first traffic situation recognition information and the second traffic situation recognition information are different from the front-to-rear or left-to-right position of the object relative to the vehicle, the comparison judgment processing unit lowers the degree of agreement. [Claim 9] The driving support system according to claim 2, characterized in that when the direction of movement of the object relative to the vehicle recognized by the first traffic situation recognition information and the second traffic situation recognition information are the same, the comparison determination processing unit increases the degree of agreement. [Claim 10] The driving support system according to claim 2, characterized in that when the direction of movement of the object relative to the vehicle recognized by the first traffic situation recognition information and the second traffic situation recognition information are different, the comparison determination processing unit lowers the degree of agreement. [Claim 11] The driver recognition processing unit generates the first traffic situation recognition information recognized by the driver by analyzing the driver's gaze direction in addition to the voice recognition of the voice comment. The driver assistance system according to claim 2, characterized in that the comparison and determination processing unit sets the degree of agreement according to whether or not the position of the object indicated by the voice comment included in the first traffic situation recognition information is located in the driver's line of sight. [Claim 12] The driving assistance system according to claim 11, characterized in that the comparison and determination processing unit increases the degree of agreement when the position of the target indicated by the voice comment is in the direction of the driver's line of sight. [Claim 13] The driving assistance system according to claim 11, characterized in that the comparison and determination processing unit gives a predetermined notification to the driver when the location of the object indicated by the voice comment is not in the driver's line of sight. [Claim 14] The vehicle is equipped with a viewing device that causes the driver to see a direction different from the driver's line of sight, The driving support system according to claim 11, characterized in that the comparison and determination processing unit sets the degree of agreement according to whether or not the driver can see the target in a direction different from the line of sight through the viewing device. [Claim 15] The driving support system according to claim 1, characterized in that the first traffic situation recognition information and the second traffic situation recognition information each reflect the driving operations performed by the driver. [Claim 16] The driving assistance system according to claim 15, characterized in that when the direction of travel of the vehicle recognized by the first traffic situation recognition information and the second traffic situation recognition information are the same, the comparison determination processing unit increases the degree of agreement. [Claim 17] The driving assistance system according to claim 15, characterized in that when the first traffic situation recognition information and the second traffic situation recognition information each recognize different directions of travel of the vehicle, the comparison judgment processing unit lowers the degree of agreement. [Claim 18] The driving support system according to claim 15, characterized in that when an operator for accelerating or decelerating the vehicle is operated as a driving operation by the driver, and the operator reflected in the first traffic situation recognition information and the second traffic situation recognition information is the same, the comparison judgment processing unit increases the degree of agreement. [Claim 19] The driving support system according to claim 15, characterized in that when an operator for accelerating or decelerating the vehicle is operated as a driving operation by the driver, and the operator reflected in the first traffic situation recognition information and the second traffic situation recognition information are different, the comparison judgment processing unit lowers the degree of agreement. [Claim 20] The driving support system according to claim 15, characterized in that when the steering element for steering the vehicle is operated as a driving operation by the driver, and the steering direction of the steering element reflected in the first traffic situation recognition information and the second traffic situation recognition information is the same, the comparison judgment processing unit increases the degree of agreement. [Claim 21] The driving support system according to claim 15, characterized in that when the steering element for steering the vehicle is operated as a driving operation by the driver, and the steering direction of the steering element reflected in the first traffic situation recognition information and the second traffic situation recognition information are different, the comparison judgment processing unit lowers the degree of agreement. [Claim 22] The driving support system according to any one of claims 1 to 21, characterized in that it notifies the driver of the degree of agreement calculated by the comparison and determination processing unit. [Claim 23] The driving support system according to any one of claims 1 to 21, characterized in that the comparison and determination processing unit sets an index related to the degree of agreement for each specific situation predetermined in advance. [Claim 24] The comparison and determination processing unit sets an index related to the degree of agreement for each predetermined specific situation, and in the specific situation where the degree of agreement falls below a predetermined threshold, it provides a predetermined notification to the driver, characterized in that it is the driving support system according to any one of claims 1 to 21. [Claim 25] The driving support system according to any one of claims 1 to 21, characterized in that if the degree of agreement calculated by the comparison and determination processing unit is lower than a predetermined threshold, the system notifies the driver that the first traffic situation recognition information and the second traffic situation recognition information indicate different driving operations. [Claim 26] The driving support system according to claim 19, further comprising a suppression control unit that suppresses the driving force of the vehicle when the degree of agreement calculated by the comparison and determination processing unit is lower than a predetermined threshold, and the operator reflected in the first traffic situation recognition information and the second traffic situation recognition information corresponds to the vehicle moving forward or backward. [Claim 27] The driving support system according to claim 19, further comprising a suppression control unit that suppresses the driving force of the vehicle when the degree of agreement calculated by the comparison and determination processing unit is lower than a predetermined threshold, and the operator reflected in the first traffic situation recognition information and the second traffic situation recognition information corresponds to the acceleration of the vehicle. [Claim 28] The driving assistance system according to claim 21, further comprising a suppression control unit that applies a steering reaction force to the steering element operated by the driver when the degree of agreement calculated by the comparison determination processing unit is lower than a predetermined threshold.

Images