Control device and control method

Abstract

A control device according to the present embodiment is disposed in a moving body provided with an image capture device, and is provided with: an image acquisition means that acquires an image of the outside of the moving body captured by the image capture device; a recognition means that recognizes target objects outside the moving body on the basis of the image; an identification means that identifies, from among the recognized target objects, a risk object that poses a risk of close proximity to the moving body; and a notification control means that notifies a user of a voice notification using a natural language, including expressions representing the recognized target objects. On the basis of the risk object and the traveling state of the moving body, the notification control means provides either direct notification notifying the user of the existence of a risk due to the risk object, or indirect notification notifying the user of information suggesting the risk object.

Description

Control device and control method

[0001] The present invention relates to a control device and a control method.

[0002] A technology is known in the art that issues a warning to the driver and controls the brakes when it is predicted that a traveling vehicle may collide with a pedestrian or other pedestrian (Patent Document 1). Patent Document 1 also discloses that information for calling the driver's attention (information pointing in the direction of a user with a high risk score or a message displaying a warning) is projected onto the windshield of the vehicle regardless of whether a collision is predicted to occur.

[0003] International Publication No. 2022 / 239327

[0004] However, if an audible warning is issued when an outside pedestrian is approaching the vehicle, the driver of the vehicle can recognize the object of high collision risk and the actions to take to avoid the collision (such as slowing down) without any additional information. On the other hand, if no pedestrian is approaching the vehicle, the driver may not immediately understand what the audible warning is about, even if it is issued. Therefore, it is useful to display visual information (attention alert information) on the windshield to notify the driver of the object of the warning. However, if the number of alerts provided increases, the driver may feel uncomfortable or confused about what to do.

[0005] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to realize a technology that can provide appropriate notifications depending on the situation.

[0006] According to the present invention, there is provided a control device that is to be placed on a mobile body equipped with an imaging device, comprising: an image acquisition means that acquires an image of the outside of the mobile body captured by the imaging device; a recognition means that recognizes targets outside the mobile body based on the image; an identification means that identifies, among the recognized targets, risk targets that pose a risk of being in proximity to the mobile body; and a notification control means that notifies a user by voice notification using natural language that includes an expression representing the recognized target, wherein the notification control means notifies the user by either a direct notification that notifies the user of the existence of a risk posed by a risk target, or an indirect notification that notifies the user of information suggesting the risk target, based on the risk target and the traveling state of the mobile body.

[0007] According to the present invention, it is possible to provide an appropriate notification depending on the situation.

[0008] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0009] The accompanying drawings are included in the specification, constitute a part thereof, illustrate embodiments of the present invention, and together with the description thereof, are used to explain the principles of the present invention.

[0023] Figure 1 shows an example of the configuration of a vehicle according to an embodiment; Block diagram showing an example of the functional configuration of a control device according to an embodiment; Figure explaining one mode of notification output by a notification control unit according to an embodiment; Figure explaining the relationship between the state of a risk target according to an embodiment and the notification mode; Figure explaining the relationship between the state of a risk target according to an embodiment and the notification mode; Figure explaining an example when a risk target according to an embodiment is not identified; Figure explaining an example when a risk target according to an embodiment is identified; Figure explaining another example when a risk target according to an embodiment is identified; Flowchart showing a series of operations of notification processing according to an embodiment; Flowchart showing a series of operations of notification control processing according to an embodiment; Figure explaining an example of notification depending on whether a risk target according to an embodiment is visually recognized; Block diagram showing an example of the configuration of a notification control unit 205 according to an embodiment; Flowchart showing a series of operations of follow-up processing according to an embodiment; Flowchart showing another series of operations of follow-up processing according to an embodiment;

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.

[0011] <Vehicle Configuration Example> Fig. 1 is a block diagram of a vehicle 1 according to an embodiment of the present invention. Fig. 1 shows an outline of the vehicle 1 in a plan view and a side view. As an example, the vehicle 1 is a four-wheeled passenger car, but it may also be a two-wheeled vehicle or another type of vehicle. The vehicle 1 is an example of a moving body in this embodiment, and the moving body is not limited to a vehicle and may include other moving bodies such as a remotely controlled robot.

[0012] The vehicle 1 includes a vehicle control device 2 (hereinafter simply referred to as the control device 2) that controls the vehicle 1. The control device 2 includes multiple ECUs (Electronic Control Units) 20-29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), a memory such as a semiconductor memory, an interface with external devices, etc. The memory stores programs executed by the processor and data used by the processor for processing. Each ECU may include multiple processors, memories, interfaces, etc. For example, the ECU 20 includes a processor 20a and a memory 20b. The processor 20a executes instructions included in a program stored in the memory 20b, thereby performing processing by the ECU 20. Alternatively, the ECU 20 may include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for performing processing by the ECU 20. The same applies to the other ECUs.

[0013] The functions of each of the ECUs 20 to 29 will be described below. The number of ECUs and the functions they are responsible for can be designed as appropriate, and they can be subdivided or integrated more than in this embodiment. For example, one ECU (e.g., ECU 22) may also have the functions of other ECUs.

[0014] The ECU 20 executes control related to manual driving and automatic driving of the vehicle 1. In automatic driving, the ECU 20 automatically controls at least one of steering and acceleration / deceleration of the vehicle 1. Note that automatic driving by the ECU 20 may include automatic driving that does not require driving operation by the driver (also called automatic driving) and automatic driving that assists driving operation by the driver (also called driving assistance). The control of driving by the ECU 20 may include, for example, control to automatically stop or steer the vehicle to avoid a collision in place of driving by the driver.

[0015] The ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for steering the front wheels in response to a driver's operation (steering operation) of the steering wheel 31. The electric power steering device 3 also includes a motor that generates driving force for assisting the steering operation and automatically steering the front wheels, a sensor that detects the steering angle, etc. When the vehicle 1 is in an autonomous driving state, the ECU 21 automatically controls the electric power steering device 3 in response to an instruction from the ECU 20, and controls the traveling direction of the vehicle 1.

[0016] The ECUs 22 and 23 control the detection units that detect the vehicle's surroundings and process information on the detection results. The vehicle 1 includes, for example, one standard camera 40 and four fisheye cameras 41 to 44 as detection units that detect the vehicle's surroundings. The standard camera 40 and the fisheye cameras 42 and 44 are connected to the ECU 22. The fisheye cameras 41 and 43 are connected to the ECU 23. The ECUs 22 and 23 analyze images captured by the standard camera 40 and the fisheye cameras 41 to 44 to recognize the type and trajectory of targets in the images, as well as lane areas and dividing lines (such as white lines) on the road. The type, number, and mounting positions of the cameras in the vehicle 1 are not limited to those described in this embodiment and may be other configurations.

[0017] The standard camera 40 is attached to the center of the front of the vehicle 1 and captures the surroundings in front of the vehicle 1. The fisheye camera 41 is attached to the center of the front of the vehicle 1 and captures the surroundings in front of the vehicle 1. In FIG. 1 , the standard camera 40 and the fisheye camera 41 are shown aligned horizontally. However, the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this, and they may be aligned vertically, for example. Furthermore, at least one of the standard camera 40 and the fisheye camera 41 may be attached to the front of the roof of the vehicle 1 (for example, on the passenger compartment inside of the windshield). The fisheye camera 42 is attached to the center of the right side of the vehicle 1 and captures the surroundings to the right of the vehicle 1. The fisheye camera 43 is attached to the center of the rear of the vehicle 1 and captures the surroundings behind the vehicle 1. The fisheye camera 44 is attached to the center of the left side of the vehicle 1 and captures the surroundings to the left of the vehicle 1. The vehicle 1 may include a lidar (Light Detection and Ranging) or millimeter wave radar as a detection unit for detecting targets around the vehicle 1 and measuring the distance to the targets.

[0018] The ECU 22 controls the standard camera 40 and the fisheye cameras 42 and 44 and processes information on the detection results. The ECU 23 controls the fisheye cameras 41 and 43 and processes information on the detection results. By dividing the detection units that detect the vehicle's surroundings into two systems, the reliability of the detection results can be improved. In addition, the ECU 22 can detect the driver's head direction and line of sight using an image of the driver captured by a fisheye camera (not shown) installed in the vehicle cabin.

[0019] The ECU 24 controls the gyro sensor 5, the GPS sensor 24b, and the communication device 24c, and processes information on the detection results or communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. The path of the vehicle 1 can be determined based on the detection results of the gyro sensor 5, the wheel speed, etc. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information and traffic information to acquire this information. The ECU 24 can access a database 24a of map information built in memory, and the ECU 24 performs tasks such as searching for a route from the current location to a destination. The ECU 24, the map database 24a, and the GPS sensor 24b constitute a so-called navigation device.

[0020] The ECU 25 includes a communication device 25a for vehicle-to-vehicle communication. The communication device 25a performs, for example, wireless communication with other vehicles in the vicinity, and exchanges information between the vehicles.

[0021] The ECU 26 controls the power plant 6. The power plant 6 is a mechanism that outputs driving force to rotate the drive wheels of the vehicle 1 and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in response to a driving operation (accelerator operation or acceleration operation) by the driver detected by an operation detection sensor 7a provided on the accelerator pedal 7A, for example, and switches gears of the transmission based on information such as the vehicle speed detected by a vehicle speed sensor 7c.

[0022] The ECU 27 controls lighting devices (headlights, taillights, etc.) including turn signals 8 (blinkers). In the example of FIG. 1 , the turn signals 8 are provided at the front, door mirrors, and rear of the vehicle 1.

[0023] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to the driver and accepts information input from the driver. The audio output device 91 notifies the driver of information by audio, including speech, for example. The notification content is generated, for example, by the ECU 22 performing a notification control process described below and transmitted to the ECU 28 for output. The display device 92 notifies the driver of information by displaying an image. The display device 92 is disposed, for example, on the surface of the driver's seat and constitutes an instrument panel, etc. Note that, although audio and display are exemplified here, information may be notified by vibration or light. Information may also be notified by a combination of audio, display, vibration, and light. The input device 93 is a group of switches disposed in a position operable by the driver to issue instructions to the vehicle 1, but may also include an audio input device.

[0024] The ECU 29 controls the brake device 10 and a parking brake (not shown). The brake device 10 is, for example, a disc brake device provided on each wheel of the vehicle 1. It applies resistance to the rotation of the wheel to decelerate or stop the vehicle 1. The ECU 29 controls the operation of the brake device 10 in response to a driving operation (brake operation) by the driver detected, for example, by an operation detection sensor 7b provided on the brake pedal 7B. When the vehicle 1 is in an autonomous driving state, the ECU 29 automatically controls the brake device 10 in response to instructions from the ECU 20, and controls the deceleration and stopping of the vehicle 1. The brake device 10 and the parking brake can also be activated to maintain the vehicle 1 in a stopped state. Furthermore, if the transmission of the power plant 6 is equipped with a parking lock mechanism, this can also be activated to maintain the vehicle 1 in a stopped state.

[0025] <Example of Functional Configuration Implemented in ECU 22> Next, an example of functional configuration implemented in the ECU 22 will be described with reference to Fig. 2. Note that the example functional configuration shown in Fig. 2 illustrates an example of functional configuration implemented by the ECU 22 when the ECU 22 executes a program stored in its internal memory. The example functional configuration shown in Fig. 2 also focuses on a configuration related to notification processing, which will be described later. Therefore, the functions implemented in the ECU 22 are not limited to those shown in Fig. 2 and may include other functions.

[0026] The image acquisition unit 201 acquires an image of the outside of the vehicle 1 captured by the standard camera 40. Note that images captured by the standard camera 40 and the fisheye cameras 42 and 44 may also be acquired.

[0027] The target object recognition unit 202 recognizes targets outside the vehicle 1 based on the image acquired by the image acquisition unit 201. The targets include, for example, pedestrians and cyclists traveling on the road. The target object recognition unit 202 may recognize, for example, the types of targets in the image, lane areas and dividing lines (such as white lines) on the road, by inputting the image into, for example, one or more neural networks.

[0028] The risk assessment unit 203 identifies, among the recognized targets (e.g., pedestrians), targets that pose a risk of approaching the vehicle 1 as risk targets. First, the risk assessment unit 203 estimates the trajectory of the pedestrian's movement based on the image acquired by the image acquisition unit 201. The trajectory of the pedestrian's movement in the image may be estimated, for example, by inputting the image into one or more neural networks. A known technique can be used as a method for estimating the trajectory of the pedestrian's movement in the image based on the image. The trajectory of the pedestrian's movement may be estimated, for example, based on the body orientation and face orientation of the pedestrian estimated from the image.

[0029] Based on the estimated pedestrian's trajectory and the vehicle's movement trajectory (e.g., acquired from ECU 20), the risk assessment unit 203 calculates the predicted length of time until a collision between vehicle 1 and the pedestrian and the distance of the pedestrian from the trajectory on which vehicle 1 is traveling. Then, based on the calculated length of time and the distance between the trajectory and the pedestrian, calculates the risk of approaching vehicle 1 (e.g., the risk of a collision). The risk of approach may be a numerical value, for example, between 0 and 1, or may be expressed in several stages (e.g., zero, low, medium, high). For example, the risk assessment unit 203 may identify, as a risk target, a target (e.g., a pedestrian) whose risk of approach exceeds a predetermined risk threshold. Furthermore, if a risk target is not identified, the risk assessment unit 203 may determine that the existence of a risk target is unknown.

[0030] However, the method for identifying a risk object is not limited to this. The risk assessment unit 203 may identify, for example, a recognized target object present in an image as a risk object, regardless of whether the proximity risk is higher than the risk threshold. Furthermore, if it determines that a target object such as a pedestrian does not exist in the image, and further recognizes a scene in the image in which a target object that creates a blind spot, such as a parked vehicle, exists (i.e., a predetermined scene in which there is statistically a risk of a vehicle jumping out, etc.), it may determine that the presence of a risk object is unknown.

[0031] 5 schematically shows a state in which a pedestrian 503 is walking along a sidewalk as a vehicle 501 travels on a road 502. The pedestrian 503 is walking at a distance from the vehicle 1. In this case, the risk assessment unit 203 may determine that the risk of the vehicle 1 and the pedestrian 503 coming into close proximity is equal to or less than a predetermined threshold, and may determine that the presence of a risk object is unknown.

[0032] 6 also shows a schematic diagram of a situation in which a pedestrian 603 is walking on a sidewalk as a vehicle 601 travels on a road 602, and an obstacle may cause the pedestrian's predicted trajectory to approach the vehicle's travel trajectory. The pedestrian 603 is walking at a distance from the vehicle 601 (distance 604 is large). In this case, the risk assessment unit 203 may determine, for example, that the risk of the vehicle 601 and the pedestrian 603 approaching each other is greater than a predetermined threshold, and may determine that the pedestrian 603 is a risk target.

[0033] 7 schematically illustrates a situation in which a pedestrian 703 is walking on the sidewalk as a vehicle 701 travels on a road 702, and an obstacle may cause the pedestrian's predicted trajectory to approach the vehicle's travel trajectory. In the example shown in FIG. 7, the pedestrian 703 is walking close to the vehicle 701 (distance 704 is small). Therefore, the risk of the vehicle 701 and the pedestrian 703 approaching each other is higher than the risk of the vehicle 601 and the pedestrian 603 approaching each other. Therefore, the risk assessment unit 203 determines, for example, that the risk of the vehicle 701 and the pedestrian 703 approaching each other is higher than a predetermined threshold, and determines the pedestrian 703 to be a risk target.

[0034] The gaze estimation unit 204 estimates whether the driver is viewing a pedestrian using an image captured by a camera inside the vehicle and including the driver (for convenience, referred to as a rear-view image). The gaze estimation unit 204 estimates the driver's facial direction and gaze direction using, for example, the rear-view image. The driver's facial direction and gaze direction may be estimated, for example, by inputting the rear-view image into one or more neural networks. The gaze estimation unit 204 estimates whether the driver is viewing a pedestrian, for example, based on the trajectory of the pedestrian (position as seen from the vehicle 1) obtained by the risk assessment unit 203 and the driver's facial direction and gaze direction.

[0035] The notification control unit 205 notifies the driver of a voice notification using natural language (including an expression expressing the recognized target) based on the result of identifying the risk target and the degree of proximity between the risk target and the vehicle 1. The notification control unit 205 of this embodiment outputs either a direct notification or an indirect notification as the voice notification using natural language to the driver.

[0036] FIG. 3 shows one mode of notification output by the notification control unit 205 of this embodiment. The notification control unit 205 outputs one of an indirect notification 301, a direct notification 302, and an alarm sound 303, for example, depending on the degree of proximity between the vehicle and a risk object (e.g., a pedestrian) (e.g., the time until the predicted proximity occurs, or the distance from the vehicle 1 to the position where the predicted proximity occurs). The example shown in FIG. 3 shows that the alarm sound 303, the direct notification 302, and the indirect notification 301 are output in order of the shortest time until the risk object and the vehicle approach each other. The time range satisfying the condition for outputting the indirect notification 301 may be wider than the time range satisfying the condition for outputting the direct notification 302. By outputting the indirect notification for a longer period of time than the direct notification, the presence of a target that requires attention can be suggested to the driver at a relatively early stage.

[0037] FIG. 4A also shows the relationship between the state of a risk object and the notification mode. For example, an audible alarm 303 is output when the risk object is about to be approached. Furthermore, when a risk object is identified, a direct notification 302 is output (under certain conditions, such as the time until approach occurs). For example, in the risk object situation shown in FIG. 7, a direct notification 302 is output. Furthermore, an indirect notification 301 is output when a risk object is not identified (e.g., the situation shown in FIG. 5) and when a risk object is identified and under certain conditions (e.g., the situation shown in FIG. 6, such as a long time until approach occurs). Note that the example shown in FIG. 4A illustrates an example in which an indirect notification 301 is output when a risk object is not identified (when the risk object is unknown). However, when the risk object is unknown, none of the indirect notification 301, direct notification 302, and audible alarm 303 may be output. That is, an indirect notification 301 may be output only under certain conditions when a risk object is identified (e.g., when the time until approach occurs is longer than a threshold or the distance to the risk object is longer than a threshold). If the notification control unit 205 determines that there is no risk, for example, if there is no object in the image, it does not output a notification.

[0038] An example of a notification mode will be specifically described. For example, when a risk object is identified and the time until the vehicle 1 approaches the identified risk object is less than or equal to a first time (e.g., 8 seconds), the notification control unit 205 sends a direct notification 302 to the driver. The direct notification may include, for example, an expression representing the target and an expression prompting the driver to initiate risk avoidance. For example, the direct notification may include an expression such as, "There is a pedestrian ahead on the left. He may be coming this way. Drive slowly." That is, the direct notification may include an expression representing the location of the risk object (e.g., ahead on the left), an expression representing the type of the target object that is the risk object (e.g., a pedestrian), and an expression prompting the driver to initiate risk avoidance (e.g., drive slowly). The expression prompting the driver to initiate risk avoidance may include an expression of a behavior the driver should cause the vehicle to exhibit (e.g., slow down). The direct notification may also include at least one of an expression representing the predicted future behavior of the target (e.g., he may be coming this way) and an expression indicating the reason for starting risk avoidance (e.g., a pedestrian is approaching from the front left). In this way, simply sounding an alarm when there is a certain amount of time until the approaching object prevents the driver from intuitively understanding the risk. However, a direct notification allows the driver to specifically identify the object to be aware of and take the necessary action. In the above example, the direct notification was described as including, as an example, a representation of the object and a representation that prompts the driver to initiate risk avoidance. However, the direct notification may include various representations as long as it notifies the user of the presence of a risk due to a specific object. For example, the direct notification may include a representation indicating the presence of a risk due to a risk object (specific object), such as "There is a pedestrian crossing the road." For example, the direct notification may include a representation indicating the location of the risk object and a representation indicating the presence of a risk due to the risk object, such as "There is a pedestrian approaching from the left front." The direct notification may also include a representation indicating the presence of a risk due to the risk object and a representation indicating the presence of a risk due to the risk object, such as "There is a pedestrian crossing the road. Drive slowly."

[0039] The notification control unit 205 outputs an audible alarm 303 when the time until the vehicle 1 approaches the identified risk object is equal to or shorter than a second time that is shorter than the first time (i.e., the vehicle is about to approach the risk object). The audible alarm 303 does not need to include natural language, for example. In other words, when the vehicle is about to approach a pedestrian who is a risk object (e.g., 2 seconds before), the driver can understand by hearing the output audible alarm and looking in the direction of travel that the vehicle may collide with the pedestrian and that the vehicle should slow down or steer (i.e., the risk is obvious). Therefore, generating the audible alarm 303 allows the driver to take action to avoid the risk.

[0040] The notification control unit 205 issues an indirect notification 301 to the driver when (a risk object has been identified) the time until the identified risk object and the vehicle come into close proximity is longer than a predetermined first time. Alternatively, the notification control unit 205 may issue an indirect notification 301 to the driver when the time until the identified risk object and the vehicle come into close proximity is longer than a predetermined first distance. The notification control unit 205 may also issue a direct notification when the distance until the identified risk object and the vehicle come into close proximity is equal to or shorter than the first distance. Furthermore, the notification control unit 205 issues an indirect notification 301 to the driver when a risk object has not been identified in the risk evaluation result by the risk evaluation unit 203. Risk states that output such an indirect notification are, for example, the states described above in FIGS. 5 and 6 .

[0041] For example, an indirect notification may include an expression representing a target but not an expression prompting the driver to initiate risk avoidance. An indirect notification may include an expression such as, "There are a lot of delivery bicycles these days." For example, an indirect notification may include an expression indicating the presence of a target. An indirect notification may not include an expression indicating the target's location like a direct notification, but may simply indicate the presence of the target. An indirect notification may not include an expression identifying the target, but may include a general expression that does not identify the target. Because an indirect notification does not include an expression indicating the target's location or an expression prompting the driver to initiate risk avoidance, it simply notifies the driver of the presence of the target and does not induce the driver to take excessive risk avoidance actions. Furthermore, if an alarm is simply sounded when the driver is approaching a target with a long time remaining, the driver may not intuitively understand the risk and may find the repeated alarms annoying. In contrast, by using the above-described indirect notification, the driver can continue driving while being aware of potential risks. In the above example, an indirect notification was described as including an expression representing a target but not an expression prompting the driver to initiate risk avoidance. The indirect notification may include expressions that include information suggesting a risk object without identifying a specific risk object (individual), such as "There are a lot of delivery bicycles these days" or "There are a lot of children going to school." Information suggesting a risk object may include, for example, at least one of the following: characteristics of the risk object (object) without identifying the individual, information that can identify the risk object, and information other than the content of the risk associated with the risk object. Information other than the content of the risk associated with the risk object may be information associated with current events or trending topics. The ECU 22 can acquire and use information, such as news published on the Internet or social media, via the communication device 24c. For example, the indirect notification may include information associated with current events, such as "It seems like there has been an increase in children running out into the street recently," without identifying the individual child.

[0042] Furthermore, outputting a warning sound or a direct notification may cause the driver to feel negative. Therefore, as shown in FIGS. 3 and 4A , after notifying the driver of a direct or indirect notification, the notification control unit 205 may include an expression showing empathy or acceptance of the user in the evaluation notification 304. The expression showing empathy may include, for example, an expression such as, "That kind of behavior is annoying, isn't it?" By providing a notification including an expression showing empathy or acceptance of the user after notifying the driver of a direct or indirect notification, the notification control unit 205 can increase the user's acceptance of the notification. The expression for increasing the user's acceptance may include an expression praising the user or an expression giving advice to the user. The notification control unit 205 may also notify the driver of an evaluation notification 304 including an expression evaluating the user's behavior in accordance with a user operation (such as deceleration or steering) received after notifying the driver of a direct or indirect notification. At this time, the notification control unit 205 may include an expression praising the user or an expression giving advice to the user in the evaluation notification 304, depending on the time from when the direct notification or indirect notification is given until when a user operation is accepted. For example, if a user operation is accepted within a predetermined time after the direct notification or indirect notification is given, an expression praising the driver may be included in the notification, and if not, an expression giving advice to the user may be included in the notification.

[0043] Referring to FIG. 4B , another example of the relationship between the state of a risk object and the notification mode will be described. For example, an alarm sound 303 is output when a risk object is about to be approached. Furthermore, when a risk object is identified, a direct notification 302 is output (under certain conditions, such as the time until approach occurs and the visibility of the risk object). For example, when the time until approach occurs is equal to or less than a threshold (or the distance to the risk object is equal to or less than a threshold) and the risk object is not visible, a direct notification 302 is output. An indirect notification 301 is issued when a risk object is not identified and under certain conditions when a risk object is identified (under certain conditions, such as the time until approach occurs and the visibility of the risk object). For example, an indirect notification 301 may be issued when a risk object is not identified and when the time until approach occurs is longer than a threshold (or the distance to the risk object is longer than a threshold) and the risk object is visible. On the other hand, an indirect notification 301 may be issued only when the time until approach occurs is longer than a threshold (or the distance to the risk object is longer than a threshold) and the risk object is visible. If the risk target is unknown, none of the indirect notification 301, the direct notification 302, and the warning sound 303 may be sent. The evaluation notification 304 shown in Fig. 4B is the same as the evaluation notification 304 described with reference to Fig. 4A.

[0044] As an example, the notification control unit 205 generates a direct notification utterance by inputting the type and location of the recognized risk object, the relative distance from the vehicle 1, etc. into a trained utterance generation algorithm. For example, the utterance generation algorithm is trained with direct notification data, which is a set of various risk objects, relative distances, etc., collected in advance, and example utterances for direct notification, as training data. Also, as an example, the notification control unit 205 generates an indirect notification utterance by inputting the presence or absence of a risk object, the type and location of the risk object, the relative distance from the vehicle 1, etc., into a trained utterance generation algorithm. For example, the utterance generation algorithm is trained with indirect notification data, which is a set of various risk objects, relative distances, the presence or absence of a risk object, etc., collected in advance, and example utterances for indirect notification, as training data. Note that a direct notification or an indirect notification may be output using a single utterance generation algorithm.

[0045] Similarly, the notification control unit 205 generates a notification including an expression that praises the user or an expression that gives advice to the user by using an utterance generation model trained using praise utterance data or advice utterance data.

[0046] <Example of Notification Generation Using Large-Scale Language Model (LLM)> In addition to the above example, a specific example of the case where the notification control unit 205 generates an utterance using a large-scale language model will be described with reference to FIG. 11 . Here, an example of generating an indirect notification using a large-scale language model (LLM) will be described. FIG. 11 shows a configuration example of the case where the notification control unit 205 uses a large-scale language model to generate the utterance content of the indirect notification. Note that in the example described below, a large-scale language model is used to generate the utterance content of the indirect notification, but a large-scale language model may also be used to generate the utterance content of the direct notification, or to generate both the indirect notification and the direct notification.

[0047] The direct notification utterance generation unit 1101 generates the utterance content for the direct notification. For example, the direct notification utterance generation unit 1101 receives risk scene information generated by the risk assessment unit 203 as input and generates the utterance content for the direct notification. The risk scene information indicates the content of the risk expected from the scene. For example, the risk scene information includes information indicating a risk scene in which a bicycle, which is a risk target, changes lanes to avoid a parked vehicle and collides with the vehicle itself. The direct notification utterance generation unit 1101 generates, for example, an utterance such as "The bicycle on the left may come this way" as the utterance content for the direct notification.

[0048] The indirect notification utterance generation unit 1103 generates the utterance content for the indirect notification. The indirect notification utterance generation unit 1103 generates the utterance content for the indirect notification by, for example, inputting the prompt generated by the prompt generation unit 1102 and a driving scene image into a large-scale language model. The driving scene image is an image acquired by the image acquisition unit 201. The prompt generation unit 1102 inputs, for example, attributes of traffic participants (recognized in the image) output from the target recognition unit 202, information on the position and attributes of the risk object output from the risk assessment unit 203, and Internet information to generate a prompt for the LLM. The Internet information may include, for example, at least one of current news, trend information, and buzzwords on the Internet. The prompt includes an instruction (i.e., a prompt) for the LLM (i.e., the indirect notification utterance generation unit 1103) to generate an utterance expressing the risk object, taking into account, for example, current news, seasons, trends, buzzwords, etc. The prompt generation unit 1102 generates a prompt such as, for example, "Generate a conversation about a bicycle driven by a person wearing red clothing in the image, taking into consideration the winter season and the latest news." The prompt generation unit 1102 may generate a prompt including recent current news and buzzwords by inputting Internet information. By including recent current news and buzzwords in the prompt, it is possible to allow the large-scale language model to generate utterance content that is more closely related to the current news and buzzwords.

[0049] In this embodiment, the prompt generation unit 1102 receives information on the location and attributes of the risk object output from the risk assessment unit 203 and generates a prompt for the LLM. The risk assessment unit 203 is implemented as a risk estimation model, and may be implemented as a trained risk estimation model separate from the LLM of the indirect notification utterance generation unit 1103, for example, as a model-based or deep neural network model. That is, the risk assessment unit 203 as a risk estimation model identifies a risk object through estimation, and the prompt generation unit 1102 generates a prompt based on information on the identified risk object. In this manner, the generated prompt is refined for the risk object, and the natural language sentence of the indirect notification generated by the LLM can be made more natural and appropriate. This suppresses the generation of sentences that would require filtering (in the indirect notification selection unit 1104) from the indirect notification generated by the LLM, thereby making the generated indirect notification more appropriate.

[0050] In this embodiment, at least one of current news, trending information, and buzzwords on the Internet extracted and generated via the Internet by the Internet information extraction LLM 1006 is input to the prompt generation unit 1102 as Internet information. The Internet information extraction LLM 1006 may be an LLM separate from the LLM used in the indirect notification utterance generation unit 1103. By acquiring Internet information using an LLM separate from the LLM used in the indirect notification utterance generation unit 1103, the information input to the prompt generation unit 1102 can be refined and the proportion of noisy or ambiguous information input to the prompt generation unit 1102 can be reduced. The prompt generation unit 1102 generates a prompt suitable for generating an indirect notification, thereby making the natural language sentence of the indirect notification generated by the LLM of the indirect notification utterance generation unit 1103 more natural and appropriate. This suppresses the generation of sentences that require filtering (in the indirect notification selection unit 1104) from the indirect notification generated by the LLM, thereby making the generated indirect notification more appropriate. Predefined prompts such as "What's hot in the news these days?", "What's trending these days?", and "What's a popular phrase these days?" may be input to the internet information extraction LLM 1006. Alternatively, a generative model may be used separately to generate prompts to input to the internet information extraction LLM 1006.

[0051] Note that the configuration for inputting the Internet information from the Internet information extraction LLM 1006 to the prompt generation unit 1102 is not limited to this example. For example, a module for selecting information that further filters the Internet information from the Internet information extraction LLM 1006 may be interposed. Filtering of the Internet information can be performed, for example, using a learning model that has previously learned the terms, expressions, and sentences to be filtered.

[0052] The prompt generation unit 1102 may be configured with a trained model trained using training data in which the above-mentioned input data and the prompt to be generated are one set. When the indirect notification utterance generation unit 1103 generates an utterance content, the generated utterance content is input to the indirect notification selection unit 1104. The indirect notification utterance generation unit 1103 may generate multiple utterance contents for one set of a prompt and a driving scene image.

[0053] The indirect notification selection unit 1104 filters only utterance contents that satisfy predetermined conditions from the (plural) utterance contents generated by the indirect notification utterance generation unit 1103, and outputs, for example, one randomly selected utterance content from the filtered utterance contents as an indirect notification. For example, the indirect notification selection unit 1104 determines utterance contents that are equivalent to or highly similar to a direct notification (such as those containing expressions that prompt the driver to start risk avoidance) as utterance contents that do not satisfy the predetermined conditions and excludes them from the indirect notification. For example, the indirect notification selection unit 1104 determines utterance contents that include predetermined negative expressions that suggest an accident as utterance contents that do not satisfy the predetermined conditions and excludes them from the indirect notification. The indirect notification selection unit 1104 also determines utterance contents that have already been used for notification within a predetermined time or utterance contents that use predetermined unnatural wording as utterance contents that do not satisfy the predetermined conditions and excludes them from the indirect notification. That is, the indirect notification selection unit 1104 selects and outputs one utterance content from the filtered-out utterance contents that do not include these utterance contents. The indirect notification output from the indirect notification selection unit 1104 includes, for example, an utterance content such as "That's a trendy green bicycle, but it looks cold because it's winter."

[0054] 11 illustrates an example in which internet information is input to the prompt generation unit 1102. However, instead of being input to the prompt generation unit 1102, the internet information may be input to the indirect notification utterance generation unit 1103. Large-scale language models are often trained using a large amount of information on the internet, but there are cases in which information about the latest current events, trends, and buzzwords is not included in the training data. Therefore, inputting internet information (in addition to prompts) into the large-scale language model makes it easier to generate utterance content associated with the latest current events, trends, etc.

[0055] <Series of Operations in Notification Processing in Vehicle> Next, a series of operations in notification processing in the vehicle will be described with reference to Fig. 8. This processing is realized, for example, by the processor 20a of the ECU 22 of the control device 2 executing a program in the memory 20b.

[0056] In S801, the image acquisition unit 201 acquires an image of the outside of the vehicle 1 taken by, for example, the standard camera 40. In S802, the target recognition unit 202 recognizes a target outside the vehicle 1 based on the image acquired by the image acquisition unit 201.

[0057] In S803, the risk assessment unit 203 identifies, among the recognized targets (e.g., pedestrians), targets that pose a risk of approaching the vehicle 1 as risk targets. In S804, the notification control unit 205 performs a notification control process, which will be described later. When the notification control unit 205 completes the notification control process, it ends this series of operations.

[0058] Next, a series of operations related to the notification control process will be described with reference to Fig. 9. The notification control process described below is executed by the notification control unit 205.

[0059] In S901, the notification control unit 205 determines whether a risk object exists. For example, the notification control unit 205 may determine that a risk object exists when the risk object has been identified by the risk assessment unit 203 as described above. If the notification control unit 205 determines that a risk object exists, the process proceeds to S902; otherwise, the notification control unit 205 terminates the series of operations of the notification control process. Note that when determining that a risk object does not exist, for example, if the target object in the image is recognized by the target object recognition unit 202 but the notification control unit 205 determines that a risk object does not exist (i.e., the risk object is unknown), the process may proceed to S906 to generate an utterance sentence for indirect notification.

[0060] In S902, the notification control unit 205 determines whether the time until the vehicle 1 and the risk target come close to each other is longer than a first time threshold (first time). If the notification control unit 205 determines that the time until the vehicle 1 and the risk target come close to each other is longer than the first time threshold (first time), the process proceeds to S906; otherwise, the process proceeds to S904. The notification control unit 205 may determine whether the distance until the vehicle 1 and the risk target come close to each other is longer than a first distance threshold (first distance). If the notification control unit 205 determines that the time until the vehicle 1 and the risk target come close to each other is longer than the first distance threshold (first distance), the process proceeds to S906; otherwise, the process proceeds to S904. If there is a grace period in terms of time or distance before the risk target comes close to each other, the notification control unit 205 generates an utterance sentence for indirect notification.

[0061] In S903, the notification control unit 205 determines whether the driver is viewing the risk target. For example, the notification control unit 205 determines whether the driver is viewing the risk target based on the estimation result by the line-of-sight estimation unit 204 as to whether the driver is viewing the risk target.

[0062] For example, FIG. 10 shows a state in which a pedestrian 1003 is crossing a road while a vehicle 1001 is traveling on a road 1002. In the example shown on the left side of FIG. 10, the driver's line of sight 1010 is directed toward the pedestrian 1003, and the driver is visually checking the pedestrian 1003 crossing the road 1002. In such a case, the notification control unit 205 determines that the driver is visually checking the risk object. The notification control unit 205 can prevent the driver from outputting a notification when the driver is visually checking the risk object. By not outputting a notification even when the driver is visually checking the risk object, unnecessary notifications to the driver can be suppressed. On the other hand, in the example shown on the right side of FIG. 10, the driver's line of sight 1011 is not directed toward the pedestrian 1003, and the driver is not visually checking the pedestrian 1003. In such a case, the notification control unit 205 determines that the driver is not visually checking the risk object. The notification control unit 205 proceeds with processing to output a direct notification or a warning sound if the driver does not visually recognize the risk object (i.e., if there is a high possibility that the driver is unaware of the risk object). If the notification control unit 205 determines that the driver is visually recognizing the risk object, the processing proceeds to S908, and if not, the processing proceeds to S904.

[0063] In the example shown in Figure 9, if the driver is viewing the risk target, no notification is made. However, the notification control unit 205 may be configured to issue an indirect notification to the user if it determines that the driver is viewing the risk target. In this case, the notification control unit 205 may proceed to S906.

[0064] In S904, the notification control unit 205 determines whether the time until the vehicle 1 and the risk object come close to each other is longer than a second time threshold (second time). In this case, the second time threshold is smaller than the first time threshold. If the time until the vehicle 1 and the risk object come close to each other is longer than the second time threshold (second time), the notification control unit 205 proceeds to S905; otherwise, the notification control unit 205 proceeds to S907. The notification control unit 205 may also determine whether the distance until the vehicle 1 and the risk object come close to each other is longer than a second distance threshold (second distance). In this case, the second distance threshold is smaller than the first distance threshold. If the notification control unit 205 determines that the time until the vehicle 1 and the risk object come close to each other is longer than the second distance threshold (second distance), the notification control unit 205 proceeds to S905; otherwise, the notification control unit 205 proceeds to S907.

[0065] In S905, the notification control unit 205 generates and selects a direct notification using the configuration for generating utterances for direct notification described above with reference to Fig. 11, and notifies the driver (user). In addition, in S906, the notification control unit 205 generates and selects an indirect notification using the configuration for generating utterances for indirect notification described above, and notifies the driver (user). In S907, the notification control unit 205 outputs a predetermined warning sound.

[0066] In step S908, the notification control unit 205 executes the follow-up process, and then ends the series of operations of the notification control process. The follow-up process will be described in detail later.

[0067] As described above, the notification control unit 205 issues a direct notification or an indirect notification based on the result of identifying the risk target, the risk target, and the vehicle's driving state. Here, the direct notification includes an expression representing the target and an expression prompting the user to start risk avoidance, while the indirect notification includes an expression representing the target but does not include an expression prompting the user to start risk avoidance. In this way, it is possible to provide an appropriate notification depending on the situation.

[0068] Next, a series of operations in the follow-up processing for notifying evaluation notification will be described. The follow-up processing is processing for notifying the evaluation notification 304 after the notification control unit 205 notifies the driver of a direct or indirect notification. Note that in the above-described embodiment, an example was described in which the evaluation notification 305 is an expression showing sympathy for the user or an expression accepting the user (including an expression praising the user). However, the evaluation notification 305 is not limited to the above examples and may include an utterance that calms the user or an utterance that praises the user.

[0069] In the following explanation, a specific example of the process (referred to as follow-up processing) for notifying evaluation notification 304 will be explained using an example in which evaluation notification 305 includes a speech that calms the user or a speech that praises the user.

[0070] 12 shows an example of the follow-up process. Note that the series of operations of the follow-up process is realized, for example, by the processor 20a of the ECU 22 of the control device 2 executing a program in the memory 20b. Furthermore, this process is started in a state where a direct notification or an indirect notification has already been made.

[0071] In S1201, the ECU 22 calculates the amount of operation that will achieve ideal vehicle behavior based on the risk target behavior prediction result. For example, a bicycle, which is a risk target, is traveling, and there is a parked vehicle ahead of the traveling bicycle. In this case, the risk target behavior prediction includes, for example, a prediction by the risk assessment unit 203 that the risk target will avoid the parked vehicle and enter the lane in which the host vehicle is traveling. Furthermore, the amount of operation that will achieve ideal vehicle behavior includes, in a time series, the ideal amount of operation of the steering wheel and the ideal amount of operation of the brakes to avoid the risk of colliding with the risk target.

[0072] In step S1202, the ECU 22 (via one or more ECUs) acquires the user's vehicle operation amounts, including, for example, the actual steering operation amount and braking operation amount by the user over time.

[0073] In S1203, the ECU 22 determines whether the sum (in the time direction) of the differences between the operation amount calculated in S1201 and the operation amount acquired in S1202 is greater than a predetermined threshold value. If the ECU 22 determines that the sum of the differences is greater than the predetermined threshold value, the process proceeds to S1204; otherwise, the process proceeds to S1205.

[0074] In S1204, the notification control unit 205 notifies the user of an utterance that calms the user. A case in which the sum of the differences between the operation amount calculated in S1201 and the operation amount acquired in S1202 becomes large includes a situation in which the user hastily avoids a risk when the user becomes close to a risk object. Examples of utterances that calm the user include utterances such as, "That bicycle just now was really annoying," "I'm relieved that I was able to deal with a sudden sudden jump out," or "It's annoying that there are so many bicycles suddenly jumping out recently."

[0075] In S1205, the notification control unit 205 notifies the user with an utterance praising the user. A situation in which the sum of the differences between the operation amount calculated in S1201 and the operation amount acquired in S1202 is small includes a situation in which the user was able to operate the vehicle with ease from an early stage. Examples of utterances praising the user include utterances such as, "You noticed the risk in advance, as expected," "It was good that you let the bicycle go first," or "It was good that you slowed down in advance." When the notification control unit 205 completes the processing of S1204 or S1205, it ends this series of operations.

[0076] In this way, by performing the follow-up process, it is possible to output a predetermined notification (in addition to the direct or indirect notification) with different utterance content depending on the user's behavior after the direct or indirect notification. In the example shown in Figure 12, the user's behavior is evaluated based on the difference between the amount of operation required to achieve ideal vehicle behavior, which is required for the risk target behavior prediction result, and the amount of operation performed by the user on the vehicle.

[0077] In the follow-up process described with reference to FIG. 12 , the content of the utterance is changed based on the difference in the amount of operation. However, the follow-up process may be based on other determinations. For example, the content of the utterance may be controlled depending on whether the user took the risk avoidance action after a direct notification or after an indirect notification. An example of such a follow-up process will be described with reference to FIG. 13 . The series of operations of the follow-up process shown in FIG. 13 is realized, for example, by the processor 20 a of the ECU 22 of the control device 2 executing a program in the memory 20 b.

[0078] In S1301, the ECU 22 determines whether the user has taken risk avoidance behavior. The determination of whether the user has taken risk avoidance behavior can be made using any known technology. For example, the ECU 22 may calculate a time series of operation amounts that realize ideal vehicle behavior based on the behavior prediction result of the risk target, and determine that the user has taken risk avoidance behavior if the time series of operation amounts of the vehicle performed by the user are similar to the time series of operation amounts that realize ideal vehicle behavior. If the ECU 22 determines that the user has taken risk avoidance behavior, the process proceeds to S1302; otherwise, the ECU 22 can terminate the follow-up process and return to the original process.

[0079] In S1302, the notification control unit 205 determines whether the risk avoidance behavior by the user is avoidance due to an indirect notification. If the risk avoidance behavior by the user is performed within a predetermined time after the indirect notification, the notification control unit 205 determines that the risk avoidance behavior is avoidance due to an indirect notification and proceeds to S1305; otherwise, the notification control unit 205 proceeds to S1303.

[0080] In S1303, the notification control unit 205 determines whether the risk avoidance behavior by the user is avoidance due to a direct notification. If the risk avoidance behavior by the user is performed after the direct notification, the notification control unit 205 determines that it is avoidance due to a direct notification and proceeds to S1304, but if not, proceeds to S1305. Cases where it is determined that the risk avoidance behavior is not avoidance due to a direct notification include, for example, a case where the risk avoidance behavior was not performed within a predetermined time after the indirect notification, but the behavior was performed before the direct notification.

[0081] In S1304, the notification control unit 205 notifies the user of an utterance that calms the user. An example of an utterance that calms the user is the same as the example described with reference to FIG. 12. In S1305, the notification control unit 205 notifies the user of an utterance that praises the user. An example of an utterance that praises the user is also the same as the example described with reference to FIG. 12. When the notification control unit 205 completes the processing of S1204 or S1205, it ends this series of operations. Even when performing such follow-up processing, it is possible to further output a predetermined notification with different utterance content depending on the user's behavior after notifying the direct notification or indirect notification.

[0082] Summary of the embodiment 1. A control device (e.g., 2) of the above embodiment is a control device disposed in a moving body (e.g., 1) equipped with an imaging device (e.g., 40), and includes: image acquisition means (e.g., 201) that acquires an image of the outside of the moving body captured by the imaging device; recognition means (e.g., 202) that recognizes targets outside the moving body based on the image; identification means (e.g., 203) that identifies a risk target that poses a risk of being in proximity to the moving body among the recognized targets; and notification control means (e.g., 205) that notifies a user of a voice notification using a natural language that includes an expression representing the recognized target, and the notification control means notifies the user of either a direct notification (e.g., 302) that notifies the user of the existence of a risk posed by the risk target, or an indirect notification (e.g., 301) that notifies the user of information suggesting the risk target, based on the risk target and the traveling state of the moving body.

[0083] According to this embodiment, it is possible to provide appropriate notifications depending on the situation.

[0084] 2. In the above embodiment, when a risk object is identified in the risk object identification result, the notification control means issues the indirect notification in accordance with the identified risk object and a traveling state of the mobile body.

[0085] According to this embodiment, when an alarm sound is simply sounded, the driver is unable to intuitively understand the risk and may find the frequent alarm sounds annoying, whereas the driver can continue driving while being aware of the potential risks.

[0086] 3. In the above embodiment, the notification control means issues the direct notification when a risk object is identified in the risk object identification result and the identified risk object and the moving body satisfy a predetermined proximity condition.

[0087] According to this embodiment, simply sounding an alarm does not allow the user to intuitively understand the risk content, but by providing a direct notification, the user can specifically grasp what they need to be careful of and take the necessary action.

[0088] 4. In the above embodiment, the device further includes estimation means (e.g., 204) for estimating whether the user is visually recognizing a risk object, and the notification control means issues the direct notification when a risk object is identified in the risk object identification result, and the identified risk object and the moving body satisfy a predetermined proximity condition, and when it is estimated that the user is not visually recognizing the risk object.

[0089] According to this embodiment, it is possible to appropriately output a notification when there is a high possibility that the person is unaware of the risk object.

[0090] 5. In the above embodiment, when a risk object is identified in the risk object identification result, and the identified risk object and the moving object satisfy a predetermined proximity condition, and when it is estimated that the user is viewing the risk object, the notification control means performs control so as not to output the direct notification.

[0091] According to this embodiment, unnecessary notifications to the user can be suppressed by not notifying the user even when the user is viewing the risk object.

[0092] 6. In the above embodiment, when a risk object is identified in the risk object identification result, the notification control means issues the indirect notification when the time until the identified risk object and the moving body come close to each other is longer than a first time or the distance until the identified risk object and the moving body come close to each other is longer than a first distance, and issues the direct notification when the time until the identified risk object and the moving body come close to each other is equal to or shorter than the first time or the distance until the identified risk object and the moving body come close to each other is equal to or shorter than the first distance.

[0093] According to this embodiment, if it is a long time before the risk object and the moving body come close to each other, the indirect notification allows the user to continue driving while being aware of the potential risk, and if it is a short time before the risk object and the moving body come close to each other, the direct notification allows the user to pay attention to the specific risk object.

[0094] 7. In the above embodiment, the notification control means outputs an alarm sound when the time until the identified risk object and the moving object come close to each other is equal to or shorter than a second time that is shorter than a first time, or when the distance until the identified risk object and the moving object come close to each other is equal to or shorter than a second distance that is shorter than the first distance.

[0095] According to this embodiment, an intuitive notification of risk can be given in a situation where the user can understand the situation and the actions to be taken by looking in the direction of travel.

[0096] 8. In the above embodiment, the notification control means issues the indirect notification when no risk object is identified in the risk object identification result.

[0097] According to this embodiment, when an alarm sound is simply sounded, the driver is unable to intuitively understand the risk and may find the frequent alarm sounds annoying, whereas the driver can continue driving while being aware of the potential risks.

[0098] 9. In the above embodiment, the direct notification includes at least one of an expression representing a position of the target that is the risk object, an expression representing a type of the target that is the risk object, and an expression that prompts the user to start risk avoidance.

[0099] According to this embodiment, the user can specifically grasp the objects that require attention and take the necessary action.

[0100] 10. In the above embodiment, the direct notification further includes at least one of an expression representing a predicted future behavior of the target and an expression representing a reason for starting the risk avoidance.

[0101] According to this embodiment, the user can easily understand what situation will occur if he or she continues driving.

[0102] 11. In the above embodiment, the indirect notification includes, as the expression representing the target, an expression representing that a target exists.

[0103] According to this embodiment, it is possible to continue driving while being aware of potential risks.

[0104] 12. In the above embodiment, the notification control means includes a large-scale language model that generates utterance content of the indirect notification, and a prompt generation means that generates a prompt that is an instruction to the large-scale language model to generate utterance expressing a risk target, and the large-scale language model generates the utterance content of the indirect notification based on the image acquired by the image acquisition means and the prompt generated by the prompt generation means.

[0105] According to this embodiment, it is possible to generate indirect notifications including natural speech content from a large-scale language model, and further, by inputting the generated prompt, it is possible to provide appropriate instructions to obtain desired speech content from the large-scale language model.

[0106] 13. In the above embodiment, the prompt generation means generates the prompt based on internet information including at least one of current news, trends, and buzzwords obtained via the internet, attributes of traffic participants recognized in an image, and information on the location and attributes of the identified risk object.

[0107] According to this embodiment, it is possible to generate prompts that take into account the latest trends on the Internet.

[0108] 14. In the above embodiment, the identification means is configured with a trained estimation model that is separate from the large-scale language model, and the prompt generation means generates the prompt using information on the location and attributes of the risk object identified by the identification means, which is the estimation model.

[0109] According to this embodiment, the generated prompts are refined for the risk target, and the natural language sentences of the indirect notification generated by the large-scale language model (LLM) can be made more natural and appropriate, thereby suppressing the generation of sentences that must be filtered for the indirect notification generated by the LLM, and making the generated indirect notification more appropriate.

[0110] 15. In the above embodiment, the system further includes a second large-scale language model that is separate from the large-scale language model and that outputs the Internet information, and the prompt generation means generates the prompt using at least one of current news, trends, and buzzwords acquired via the Internet by the second large-scale language model.

[0111] According to this embodiment, the information input to the prompt generation means can be refined, and the proportion of noisy or ambiguous information input to the prompt generation means can be reduced. The prompt generation means generates prompts suitable for generating indirect notifications, which makes the natural language sentences of the indirect notifications generated by the large-scale language model (LLM) more natural and appropriate. This suppresses the generation of sentences that require filtering of the indirect notifications generated by the LLM, making the generated indirect notifications more appropriate.

[0112] 16. In the above embodiment, the prompt generation means generates the prompt based on attributes of the traffic participants recognized in the image and information on the location and attributes of the identified risk object, and the large-scale language model generates the utterance content of the indirect notification based on internet information (e.g., including at least one of current news, trends, and buzzwords) acquired via the internet, the image acquired by the image acquisition means, and the prompt generated by the prompt generation means.

[0113] According to this embodiment, the large-scale language model can generate utterances that take into account the latest trends on the Internet.

[0114] 17. In the above embodiment, the notification control means excludes utterance content that does not satisfy a predetermined condition from the utterance content of the plurality of indirect notifications generated by the large-scale language model.

[0115] According to this embodiment, it is possible to extract only utterance contents that are appropriate as indirect notification from among a plurality of utterance contents generated by a large-scale language model.

[0116] 18. In the above embodiment, the notification control means further outputs a predetermined notification, the content of which varies depending on an action of the user after the direct notification or the indirect notification is given.

[0117] According to this embodiment, it is possible to increase the user's acceptance of notifications.

[0118] 19. In the above embodiment, the behavior of the user is evaluated based on a difference between an amount of operation for realizing the behavior of a moving object, which is determined based on a result of the behavior prediction of the risk target, and an amount of operation by the user on the moving object.

[0119] According to this embodiment, the user can be notified of the utterance depending on whether the operation the user has taken to avoid the risk is appropriate.

[0120] 20. In the above embodiment, after notifying the user of the direct notification or the indirect notification, the notification control means further outputs a predetermined notification including an expression evaluating the user's behavior.

[0121] According to this embodiment, it is possible to give the user feedback on risk aversion.

[0122] 21. In the above embodiment, the predetermined notification includes an expression showing sympathy for the user or an expression accepting the user.

[0123] According to this embodiment, after a direct or indirect notification is given, an evaluation notification that is considerate to the user can be output.

[0124] 22. In the above embodiment, the notification control means outputs a calming utterance when the user takes a risk-avoidance action after notifying the direct notification, and outputs a praising utterance when the user takes a risk-avoidance action after notifying the indirect notification.

[0125] According to this embodiment, in a situation where a direct notification is likely to make a user feel negative, it is possible to output a speech that calms the user's feelings. Also, from the indirect notification stage, i.e., at a relatively early stage, it is possible to recognize the presence of a target that requires attention and provide a speech that makes the user feel positive if the user is able to avoid the risk.

[0126] 23. In the above embodiment, a mobile object including the above control device is provided.

[0127] According to this embodiment, a mobile device is provided that is capable of providing appropriate notifications depending on the situation.

[0128] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0129] 1...vehicle, 2...control device, 21-29...ECU

Claims

1. A control device to be placed on a moving body equipped with an imaging device, comprising: an image acquisition means for acquiring an image of the outside of the moving body by the imaging device; a recognition means for recognizing targets outside the moving body based on the image; an identification means for identifying risk targets among the recognized targets that pose a risk of being in close proximity to the moving body; and a notification control means for notifying a user of a voice notification in a natural language including an expression expressing the recognized target, wherein the notification control means provides either a direct notification to notify the user of the existence of a risk posed by a risk target, or an indirect notification to notify the user of information suggesting the risk target, based on the risk target and the traveling state of the moving body.

2. The control device described in claim 1, characterized in that when a risk object is identified in the risk object identification result, the notification control means notifies the indirect notification depending on the identified risk object and the driving state of the mobile body.

3. The control device described in claim 1, characterized in that the notification control means issues the direct notification when a risk object is identified in the risk object identification result and the identified risk object and the moving body satisfy a predetermined proximity condition.

4. The control device described in claim 1, further comprising an estimation means for estimating whether the user is viewing a risk object, and the notification control means issues the direct notification when a risk object is identified in the risk object identification result and the identified risk object and the moving body satisfy a predetermined proximity condition, and it is further estimated that the user is not viewing the risk object.

5. The control device described in claim 4, characterized in that the notification control means controls so as not to output the direct notification when a risk object is identified in the risk object identification result, and when the identified risk object and the moving body satisfy a predetermined proximity condition, and when it is further estimated that the user is viewing the risk object.

6. The control device described in claim 1, characterized in that, when a risk object is identified in the risk object identification result, the notification control means issues the indirect notification when the time until the identified risk object and the moving body come into close proximity is longer than a first time or the distance until the identified risk object and the moving body come into close proximity is longer than a first distance, and issues the direct notification when the time until the identified risk object and the moving body come into close proximity is equal to or shorter than the first time or the distance until the identified risk object and the moving body come into close proximity is equal to or shorter than the first distance.

7. The control device described in claim 6, characterized in that the notification control means outputs an alarm sound when the time until the identified risk object and the moving body come into close proximity is equal to or shorter than a second time shorter than a first time, or when the distance until the identified risk object and the moving body come into close proximity is equal to or shorter than a second distance shorter than the first distance.

8. The control device according to claim 1, characterized in that the notification control means issues the indirect notification when no risk object is identified in the risk object identification result.

9. The control device described in claim 1, characterized in that the direct notification includes at least one of an expression representing the position of the target that is the risk object, an expression representing the type of the target that is the risk object, and an expression that causes the user to initiate risk avoidance.

10. The control device described in claim 9, characterized in that the direct notification further includes at least one of an expression representing a predicted future behavior of the target and an expression representing a reason for initiating the risk avoidance.

11. The control device according to claim 1, characterized in that the indirect notification includes an expression representing the presence of a target as an expression representing a target.

12. The control device described in claim 1, characterized in that the notification control means includes a large-scale language model that generates the spoken content of the indirect notification, and a prompt generation means that generates a prompt that is an instruction to the large-scale language model to generate an utterance expressing a risk target, and the large-scale language model generates the spoken content of the indirect notification based on an image acquired by the image acquisition means and a prompt generated by the prompt generation means.

13. The control device according to claim 12, characterized in that the prompt generation means generates the prompt based on Internet information obtained via the Internet, attributes of traffic participants recognized in an image, and information on the location and attributes of the identified risk object.

14. The control device described in claim 12, characterized in that the identification means is composed of a trained estimation model that is separate from the large-scale language model, and the prompt generation means generates the prompt using information on the location and attributes of the risk object identified by the identification means, which is the estimation model.

15. The control device according to claim 13, further comprising a second large-scale language model separate from the large-scale language model for outputting the Internet information, and the prompt generation means generates the prompt using information obtained via the Internet by the second large-scale language model.

16. The control device described in claim 12, characterized in that the prompt generation means generates the prompt based on attributes of a traffic participant recognized in an image and information on the location and attributes of the identified risk object, and the large-scale language model generates the speech content of the indirect notification based on Internet information acquired via the Internet, images acquired by the image acquisition means, and the prompt generated by the prompt generation means.

17. The control device according to claim 12, characterized in that the notification control means excludes utterance contents that do not satisfy predetermined conditions from the utterance contents of the multiple indirect notifications generated by the large-scale language model.

18. The control device according to claim 1, characterized in that the notification control means further outputs a predetermined notification having different speech content depending on the user's action after the direct notification or the indirect notification is given.

19. The control device described in claim 18, characterized in that the user's behavior is evaluated based on the difference between the amount of operation required to realize the behavior of a moving object, which is determined based on the predicted behavior of the risk object, and the amount of operation performed by the user on the moving object.

20. The control device described in claim 18, characterized in that the notification control means, after notifying the user of the direct notification or the indirect notification, further outputs the specified notification including an expression evaluating the user's behavior, and the specified notification includes an expression showing sympathy for the user or an expression accepting the user.

21. The control device described in claim 18, characterized in that the notification control means outputs calming utterances when the user takes risk-avoidance action after notifying the direct notification, and outputs praising utterances when the user takes risk-avoidance action after notifying the indirect notification.

22. A control method in which each step is executed by a control device disposed in a moving body equipped with an imaging device, comprising: acquiring an image of the outside of the moving body with the imaging device; recognizing targets outside the moving body based on the image; identifying risk targets that pose a risk of being in close proximity to the moving body from among the recognized targets; and notifying a user of a voice notification using a natural language including an expression expressing the recognized target, wherein the notification includes either a direct notification that notifies the user of the existence of a risk posed by a risk target, or an indirect notification that notifies the user of information suggesting the risk target, based on the risk target and the traveling state of the moving body.