Control device and control method

Abstract

A control device according to the present embodiment is disposed in a moving body provided with an image capture device, and is provided with: an image acquisition means that acquires an image of the outside of the moving body captured by the image capture device; a recognition means that recognizes target objects outside the moving body on the basis of the image; an identification means that identifies, from among the recognized target objects, a risk object that poses a risk of close proximity to the moving body; and a notification control means that notifies a user of a voice notification using a natural language, including expressions representing the recognized target objects. On the basis of the risk object and the traveling state of the moving body, the notification control means provides either direct notification notifying the user of the existence of a risk due to the risk object, or indirect notification notifying the user of information suggesting the risk object.

Description

[[Technical Field]]

[0001] The present invention relates to a control device and a control method. [[Background Art]]

[0002] Conventionally, there is known a technology for notifying a driver of a warning and controlling a brake when it is predicted that a traveling vehicle may collide with a pedestrian such as a person (Patent Document 1). Further, Patent Document 1 discloses that information for alerting the driver (information indicating a direction where a user with a high risk score is present, and a message indicating a warning) is projected onto the windshield of the vehicle regardless of whether a collision risk is predicted or not. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] International Publication No. 2022 / 239327 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] By the way, when an external pedestrian is approaching the vicinity of a traveling vehicle and a warning sound is issued, the driver of the vehicle can recognize the target with high collision probability and the operation to avoid collision (such as deceleration) even without additional information. On the other hand, when a pedestrian does not approach the vicinity of the vehicle, even if a warning sound is issued, the driver may not be able to instantly recognize what the warning is for, so it is useful to display visual information (alert information) on the windshield to notify the target of the warning. However, an increase in the number of alerts provided may cause the driver to feel uncomfortable or become confused about what action to take.

[0005] The present invention has been made in view of the above problem, and an object thereof is to implement a technology capable of providing appropriate notification according to situations. [[Means for Solving the Problem]]

[0006] According to the present invention, A control device disposed on a mobile body equipped with an imaging device, Image acquisition means for acquiring an image of the outside of the moving object captured by the aforementioned imaging device, Based on the aforementioned image, a recognition means for recognizing an external target of the moving object, A means for identifying risk targets among the recognized targets that pose a risk due to their proximity to the moving object, The system includes a notification control means that notifies the user of a voice notification using natural language, which includes an expression representing a recognized target, The notification control means notifies the user of either a direct notification that indicates the presence of a risk caused by the risk object, or an indirect notification that indicates information suggesting the presence of the risk object, based on the risk object and the driving state of the moving body. death, The notification control means includes a large-scale language model that generates the content of the indirect notification, and a prompt generation means that generates a prompt which is an instruction to cause the large-scale language model to generate an utterance that represents the risk object. The large-scale language model generates the speech content of the indirect notification based on the image acquired by the image acquisition means and the prompt generated by the prompt generation means. The prompt generation means generates the prompt based on internet information obtained via the internet, the attributes of traffic participants recognized in the image, and the location and attribute information of the identified risk target. A control device is provided that is characterized by doing so. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to provide appropriate notifications depending on the situation.

[0008] Other features and advantages of the present invention will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar components are given the same reference numeral. [Brief explanation of the drawing]

[0009] The attached drawings are included in the specification and constitute part thereof, illustrating embodiments of the present invention and are used together with the description to explain the principles of the present invention. [Figure 1] A diagram showing an example of the vehicle configuration according to the embodiment. [Figure 2] Block diagram showing an example of the functional configuration of the control device according to the embodiment. [Figure 3] Diagram illustrating one aspect of a notification output by a notification control unit according to an embodiment [Figure 4A] Diagram explaining the relationship between the status of a risk target and the notification mode according to an embodiment [Figure 4B] Diagram explaining the relationship between the status of a risk target and the notification mode according to an embodiment [Figure 5] Diagram illustrating an example where a risk target according to an embodiment is not specified [Figure 6] Diagram illustrating an example where a risk target according to an embodiment is specified [Figure 7] Diagram illustrating another example where a risk target according to an embodiment is specified [Figure 8] Flowchart showing a sequence of operations of notification processing according to an embodiment [Figure 9] Flowchart showing a sequence of operations of notification control processing according to an embodiment [Figure 10] Diagram explaining an example of a notification depending on whether a risk target according to an embodiment is visually recognized or not [Figure 11] Block diagram showing a configuration example of a notification control unit 205 according to an embodiment [Figure 12] Flowchart showing a sequence of operations of follow-up processing according to an embodiment [Figure 13] Flowchart showing another sequence of operations of follow-up processing according to an embodiment DETAILED DESCRIPTION OF EMBODIMENTS

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined arbitrarily. In addition, the same or similar configurations are denoted by the same reference numerals, and repeated descriptions are omitted.

[0011] <Configuration Example of Vehicle> FIG. 1 is a block diagram of a vehicle 1 according to one embodiment of the present invention. In FIG. 1, an outline of the vehicle 1 is shown by a plan view and a side view. The vehicle 1 is a four-wheeled passenger car as an example, but may be a two-wheeled vehicle or another type of vehicle. The vehicle 1 is an example of the moving object according to the present embodiment, and the moving object is not limited to vehicles, and may include other moving objects such as remotely operated robots.

[0012] The vehicle 1 includes a vehicle control device 2 (hereinafter simply referred to as control device 2) that controls the vehicle 1. The control device 2 includes a plurality of ECUs (Electronic Control Units) 20 to 29 communicatively connected via an in-vehicle network. Each ECU includes a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), a memory such as a semiconductor memory, and an interface for connecting to external devices. The memory stores programs executed by the processor, data used by the processor for processing, and the like. Each ECU may include a plurality of processors, memories, interfaces, and the like. For example, the ECU 20 includes a processor 20a and a memory 20b. Processing by the ECU 20 is executed when the processor 20a executes instructions included in a program stored in the memory 20b. Alternatively, the ECU 20 may include a dedicated integrated circuit such as an ASIC (Application Specific Integrated Circuit) for executing processing by the ECU 20. The same applies to other ECUs.

[0013] Functions and the like handled by each of the ECUs 20 to 29 will be described below. The number of ECUs and the functions handled by the ECUs can be designed as appropriate, and can be further subdivided or integrated compared to the present embodiment. For example, one ECU (e.g., ECU 22) may also have the functions of other ECUs.

[0014] The ECU 20 performs control related to the manual and automatic driving of Vehicle 1. In automatic driving, it automatically controls at least one of the steering and acceleration / deceleration of Vehicle 1. Automatic driving by the ECU 20 may include automatic driving that does not require driving operations by the driver (which may also be called autonomous driving) and automatic driving that assists driving operations by the driver (which may also be called driver assistance). The driving control by the ECU 20 may include, for example, control that automatically stops or steers the vehicle to avoid a collision in place of the driver's driving.

[0015] The ECU 21 controls the electric power steering system 3. The electric power steering system 3 includes a mechanism that steers the front wheels in response to the driver's steering input (steering operation) to the steering wheel 31. The electric power steering system 3 also includes a motor that provides driving force to assist the steering operation or to automatically steer the front wheels, as well as sensors that detect the steering angle. When the vehicle 1 is in an automated driving state, the ECU 21 automatically controls the electric power steering system 3 in response to instructions from the ECU 20 to control the direction of travel of the vehicle 1.

[0016] ECUs 22 and 23 control the detection unit that detects the surrounding conditions of the vehicle and process the information of the detection results. Vehicle 1 includes, for example, one standard camera 40 and four fisheye cameras 41 to 44 as a detection unit that detects the surrounding conditions of the vehicle. The standard camera 40 and fisheye cameras 42 and 44 are connected to ECU 22. Fisheye cameras 41 and 43 are connected to ECU 23. By analyzing the images captured by the standard camera 40 and fisheye cameras 41 to 44, ECUs 22 and 23 can recognize the type and movement trajectory of objects in the images, as well as the lane areas and road markings (white lines, etc.) on the road. Note that the type, number, and mounting positions of the cameras in vehicle 1 are not limited to the example of this embodiment and may be in other configurations.

[0017] The standard camera 40 is mounted in the center of the front of the vehicle 1 and captures the surrounding area in front of the vehicle 1. The fisheye camera 41 is mounted in the center of the front of the vehicle 1 and captures the surrounding area in front of the vehicle 1. In Figure 1, the standard camera 40 and the fisheye camera 41 are shown aligned horizontally. However, the arrangement of the standard camera 40 and the fisheye camera 41 is not limited to this, and they may be aligned vertically, for example. Also, at least one of the standard camera 40 and the fisheye camera 41 may be mounted on the front of the roof of the vehicle 1 (for example, on the interior side of the front windshield). The fisheye camera 42 is mounted in the center of the right side of the vehicle 1 and captures the surrounding area to the right of the vehicle 1. The fisheye camera 43 is mounted in the center of the rear of the vehicle 1 and captures the surrounding area behind the vehicle 1. The fisheye camera 44 is mounted in the center of the left side of the vehicle 1 and captures the surrounding area to the left of the vehicle 1. Vehicle 1 may include a lidar (Light Detection and Ranging) or millimeter-wave radar as a detection unit for detecting targets around Vehicle 1 or measuring the distance to targets.

[0018] ECU22 controls the standard camera 40 and the fisheye cameras 42 and 44 and processes the information from their detection results. ECU23 controls the fisheye cameras 41 and 43 and processes the information from their detection results. By dividing the detection unit that detects the surrounding conditions of the vehicle into two systems, the reliability of the detection results can be improved. In addition, ECU22 can use images of the driver taken by a fisheye camera (not shown) installed inside the vehicle to detect the direction of the driver's head and gaze.

[0019] The ECU 24 controls the gyro sensor 5, GPS sensor 24b, and communication device 24c, and processes the detection results or communication results. The gyro sensor 5 detects the rotational motion of the vehicle 1. The vehicle 1's path can be determined based on the detection results of the gyro sensor 5 and the wheel speed, etc. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c communicates wirelessly with a server that provides map information and traffic information to acquire this information. The ECU 24 can access the map information database 24a built in memory, and the ECU 24 performs route searching from the current location to the destination, etc. The ECU 24, map database 24a, and GPS sensor 24b constitute a so-called navigation device.

[0020] The ECU25 is equipped with a communication device 25a for vehicle-to-vehicle communication. The communication device 25a, for example, communicates wirelessly with other nearby vehicles to exchange information between vehicles.

[0021] The ECU 26 controls the power plant 6. The power plant 6 is a mechanism that outputs driving force to rotate the drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. For example, the ECU 26 controls the engine output in response to the driver's driving operation (accelerator operation or acceleration operation) detected by the operation detection sensor 7a provided on the accelerator pedal 7A, or switches the gear of the transmission based on information such as vehicle speed detected by the vehicle speed sensor 7c.

[0022] The ECU27 controls the lighting devices (headlights, taillights, etc.), including the turn signals 8. In the example shown in Figure 1, the turn signals 8 are located on the front, door mirrors, and rear of the vehicle 1.

[0023] The ECU 28 controls the input / output device 9. The input / output device 9 outputs information to a driver and accepts input of information from the driver. The audio output device 91 notifies the driver of information by, for example, audio including utterance. The content of the notification is generated by, for example, the ECU 22 through a notification control process described below, and is output by being transmitted to the ECU 28. The display device 92 notifies the driver of information by displaying an image. The display device 92 is disposed, for example, on a surface of a driver's seat, and constitutes an instrument panel or the like. Although audio and display are exemplified herein, information may be notified by vibration or light. Further, information may be notified by combining a plurality of audio, display, vibration, or light. The input device 93 is a switch group that is disposed at a position operable by the driver and is used for issuing instructions to the vehicle 1, but may also include an audio input device.

[0024] The ECU 29 controls the brake device 10 and a parking brake (not shown). The brake device 10 is, for example, a disc brake device, which is provided on each wheel of the vehicle 1, and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheels. The ECU 29 controls the operation of the brake device 10 in accordance with, for example, the driver's driving operation (brake operation) detected by the operation detection sensor 7b provided on the brake pedal 7B. When the driving state of the vehicle 1 is an autonomous driving state, the ECU 29 automatically controls the brake device 10 in accordance with instructions from the ECU 20, and controls the deceleration and stopping of the vehicle 1. The brake device 10 and the parking brake can also be operated to maintain the stopped state of the vehicle 1. Further, when the transmission of the power plant 6 includes a parking lock mechanism, the parking lock mechanism can also be operated to maintain the stopped state of the vehicle 1.

[0025] <Example of functional configuration implemented in ECU 22> Next, with reference to Figure 2, an example of a functional configuration implemented in the ECU22 will be described. The functional configuration example shown in Figure 2 illustrates an example of a functional configuration implemented by the ECU22 executing a program stored in its internal memory. Furthermore, the functional configuration example shown in Figure 2 focuses on the configuration related to notification processing, which will be discussed later. Therefore, the functions implemented in the ECU22 are not limited to those shown in Figure 2 and may include other functions.

[0026] The image acquisition unit 201 acquires images of the exterior of the vehicle 1 captured by the standard camera 40. Alternatively, images captured by the standard camera 40, as well as the fisheye cameras 42 and 44, may also be acquired.

[0027] The object recognition unit 202 recognizes objects outside the vehicle 1 based on the image acquired by the image acquisition unit 201. Objects include, for example, pedestrians and people riding bicycles on the road. The object recognition unit 202 may also recognize, for example, the type of object in the image, as well as lane areas and road markings (white lines, etc.) on the road, by inputting the image into, for example, one or more neural networks.

[0028] The risk assessment unit 203 identifies targets (e.g., pedestrians) that pose a risk due to their proximity to vehicle 1 as risk targets. First, the risk assessment unit 203 estimates the trajectory of the pedestrians based on the images acquired by the image acquisition unit 201. The trajectory of the pedestrians in the image may be estimated, for example, by inputting the image into one or more neural networks. Known techniques can be used to estimate the trajectory of the pedestrians in the image based on the image. The trajectory of the pedestrians may be estimated, for example, based on the orientation of the pedestrian's body and face estimated from the image.

[0029] The risk assessment unit 203 calculates the length of the predicted time until vehicle 1 and the pedestrian collide, and the distance of the pedestrian from the trajectory of vehicle 1, based on the estimated pedestrian trajectory and the vehicle's movement trajectory (obtained, for example, from the ECU 20). Then, based on these calculated lengths of time and the distance between the trajectory and the pedestrian, it calculates the risk of proximity between the pedestrian and vehicle 1 (e.g., collision risk). The proximity risk may be a numerical value, for example, between 0 and 1, or it may be expressed in stages (e.g., zero, low, medium, high). The risk assessment unit 203 may identify a target (e.g., a pedestrian) as a risk target if the proximity risk exceeds a predetermined risk threshold. In addition, if no risk target is identified, the risk assessment unit 203 may determine that the existence of a risk target is unknown.

[0030] However, the method for identifying risk targets is not limited to this method. The risk assessment unit 203 may identify, for example, an object present in the image and recognized as a risk target, not only when the proximity of a risk is higher than the risk threshold. Furthermore, if it determines that there are no objects such as pedestrians in the image, and then recognizes a scene in the image where there are objects that create blind spots, such as parked vehicles (i.e., a predetermined scene in which statistically there is a risk of sudden appearances or other incidents), it may determine that the presence of a risk target is unknown.

[0031] For example, Figure 5 schematically shows a situation where a vehicle 501 is traveling on a road 502, and a pedestrian 503 is walking along the sidewalk. The pedestrian 503 is walking at a distance from the vehicle 1. In this case, the risk assessment unit 203 may determine, for example, that the risk of the vehicle 1 and the pedestrian 503 being in close proximity is below a predetermined threshold, and therefore determine that the presence of a risk object is unknown.

[0032] Furthermore, Figure 6 schematically illustrates a situation where, as vehicle 601 travels along road 602, a pedestrian 603 is walking on the sidewalk, and an obstacle may cause the pedestrian's predicted trajectory to be close to the vehicle's trajectory. The pedestrian 603 is walking at a distance from vehicle 1 (the distance 604 is large). In this case, the risk assessment unit 203 may, for example, determine that the risk of vehicle 601 and pedestrian 603 being close together is greater than a predetermined threshold, and determine that pedestrian 603 is a risk target.

[0033] Figure 7 schematically illustrates a scenario where a vehicle 701 is traveling on a road 702, and a pedestrian 703 is walking on the sidewalk, but an obstacle may cause the pedestrian's predicted trajectory to be close to the vehicle's trajectory. In the example shown in Figure 7, the pedestrian 703 is walking close to the vehicle 701 (the distance 704 is small). Therefore, the risk of proximity between the vehicle 701 and the pedestrian 703 is higher than the risk of proximity between the vehicle 601 and the pedestrian 603. Accordingly, the risk assessment unit 203 determines, for example, that the risk of proximity between the vehicle 701 and the pedestrian 703 is greater than a predetermined threshold, and determines that the pedestrian 703 is a risk target.

[0034] The gaze estimation unit 204 uses an image captured by a camera inside the vehicle, which includes the driver (referred to as the rear-view image for convenience), to estimate whether the driver is seeing the pedestrian. For example, the gaze estimation unit 204 uses the rear-view image to estimate the direction of the driver's face and the direction of their gaze. The direction of the driver's face and the direction of their gaze may be estimated, for example, by inputting the rear-view image into one or more neural networks. For example, the gaze estimation unit 204 estimates whether the driver is seeing the pedestrian based on the pedestrian's trajectory (position as seen from vehicle 1) obtained by the risk assessment unit 203, and the direction of the driver's face and gaze.

[0035] The notification control unit 205 notifies the driver of an audio notification using natural language (including an expression representing the recognized target) based on the identification result of the risk target and the degree of proximity between the risk target and vehicle 1. In this embodiment, the notification control unit 205 outputs either a direct notification or an indirect notification as an audio notification using natural language to the driver.

[0036] Figure 3 shows one mode of notification output by the notification control unit 205 of this embodiment. The notification control unit 205 outputs one of the following: an indirect notification 301, a direct notification 302, or an alarm sound 303, depending on the degree of proximity between the vehicle and the risk object (e.g., a pedestrian) (e.g., the time until the predicted proximity occurs, or the distance from vehicle 1 to the location where the predicted proximity occurs). In the example shown in Figure 3, the alarm sound 303, direct notification 302, and indirect notification 301 are output in order from shortest to longest time until the risk object and the vehicle become close. The time window that satisfies the conditions for outputting the indirect notification 301 may be wider than the time window that satisfies the conditions for outputting the direct notification 302. By outputting indirect notifications for a longer period of time than direct notifications, the presence of a target that requires attention can be suggested to the driver at a relatively early stage.

[0037] Figure 4A also shows the relationship between the state of the risk object and the notification method. For example, an alarm sound 303 is output when the system is in a state immediately before approaching the risk object. Also, if the risk object is identified, a direct notification 302 is output (under certain conditions, such as the time until proximity occurs). For example, a direct notification 302 is output in the risk object situation shown in Figure 7. Furthermore, an indirect notification 301 is issued when the risk object is not identified (for example, the situation shown in Figure 5) and under certain conditions when the risk object is identified (for example, the situation shown in Figure 6, such as a long time until proximity occurs). Note that the example shown in Figure 4A shows an example where an indirect notification 301 is issued when the risk object is not identified (when the risk object is unknown). However, if the risk object is unknown, none of the indirect notification 301, direct notification 302, or warning sound 303 may be issued. In other words, the indirect notification 301 may be issued only under certain conditions when the risk object is identified (for example, when the time until proximity occurs is longer than a threshold, or when the distance to the risk object is longer than a threshold). The notification control unit 205 does not output a notification if it determines that there is no risk, such as when there are no objects in the image.

[0038] Examples of notification modes will be explained in detail. For example, if a risk target has been identified and the time until the identified risk target and vehicle 1 come into close proximity is less than or equal to a first time (e.g., 8 seconds), the notification control unit 205 will notify the driver of a direct notification 302. The direct notification may include, for example, an expression representing the target and an expression prompting the driver to begin risk avoidance. For example, the direct notification may include an expression such as, "Pedestrian on the left front. They may be coming this way. Proceed slowly." That is, the direct notification may include an expression representing the location of the risk target (e.g., left front), an expression representing the type of target that is the risk target (e.g., pedestrian), and an expression prompting the driver to begin risk avoidance (e.g., proceed slowly). The expression prompting the driver to begin risk avoidance may include an expression of the action the driver should take with the vehicle (e.g., slow down). The direct notification may also include at least one of the following: an expression representing the expected future behavior of the target (e.g., they may be coming this way), and an expression representing the reason for beginning risk avoidance (e.g., a pedestrian is approaching from the left front). Thus, simply sounding an alarm when there is some time before the object is close makes it difficult to intuitively understand the nature of the risk. However, direct notification allows the driver to specifically identify the object they need to pay attention to and take the necessary actions. In the above example, direct notification was described as including, for one example, an expression indicating a target and an expression prompting the driver to begin risk avoidance. However, direct notification may include various expressions as long as it notifies the user of the existence of a risk caused by a specific target. For example, direct notification may include an expression indicating the existence of a risk caused by a risk object (a specific target), such as "There is a pedestrian crossing the road." For example, direct notification may include an expression indicating the location of the risk object and an expression indicating the existence of a risk caused by the risk object, such as "There is a pedestrian approaching from the left front." Direct notification may include an expression indicating the existence of a risk caused by a risk object and an expression prompting the driver to begin risk avoidance, such as "There is a pedestrian crossing the road. Proceed slowly."

[0039] The notification control unit 205 outputs an alarm sound 303 if the time until the vehicle 1 approaches the identified risk target is less than or equal to the second time, which is shorter than the first time (i.e., the state immediately before approaching the risk target). The alarm sound 303 does not have to include, for example, natural language. That is, if the vehicle is immediately approaching a pedestrian who is a risk target (for example, 2 seconds ago), the driver can hear the outputted alarm sound and look in the direction of travel to understand that there is a possibility of the vehicle colliding with the pedestrian and that the vehicle should slow down or steer (i.e., the risk is obvious). Therefore, if the alarm sound 303 is generated, the driver can take action to avoid the risk.

[0040] The notification control unit 205 notifies the driver of an indirect notification 301 if (a risk target has been identified) the time until the vehicle comes into close proximity with the identified risk target is longer than a predetermined first time. Alternatively, the notification control unit 205 may also notify the driver of an indirect notification 301 if the time until the vehicle comes into close proximity with the identified risk target is longer than a predetermined first distance. Furthermore, the notification control unit 205 may also notify the driver of a direct notification when the distance until the vehicle comes into close proximity with the identified risk target is less than or equal to the first distance. In addition, the notification control unit 205 notifies the driver of an indirect notification 301 if the risk target has not been identified in the risk assessment results from the risk assessment unit 203. The risk states that trigger such indirect notifications are, for example, the states described above in Figures 5 and 6.

[0041] Indirect notifications, for example, include expressions that indicate a target but do not include expressions that prompt the driver to initiate risk avoidance. Indirect notifications include expressions such as, "There have been a lot of delivery bicycles lately." For example, indirect notifications include expressions that indicate the presence of a target. Indirect notifications do not include expressions that indicate the location of the target, as in direct notifications, and may simply indicate the presence of the target. Indirect notifications do not include expressions that identify the individual target, as in direct notifications, and may include general expressions that do not identify the individual target. Because indirect notifications do not include expressions that indicate the location of the target or expressions that prompt the driver to initiate risk avoidance, they only convey the presence of a target and do not cause the driver to take excessive risk avoidance actions. Also, if an alarm sounds when there is a long time until approach, the driver may not be able to intuitively understand the nature of the risk and may find the frequent warning sounds unpleasant. In contrast, by using the indirect notifications described above, the driver can continue driving while being aware of the potential risks. In the above example, indirect notification was explained as including an expression that represents a target but does not include an expression that prompts the driver to initiate risk avoidance. Indirect notification may include an expression that suggests a risk target without specifying a particular risk target (individual), such as "There seem to be a lot of delivery bicycles lately" or "There seem to be a lot of children going to school." Information suggesting a risk target may include, for example, the characteristics of a risk target (target) whose individual is not specified, information that can identify the risk target, and information other than the content of the risk related to the risk target. Information other than the content of the risk related to the risk target may be information associated with current events or trending topics. The ECU22 can acquire and use information such as news and SNS published on the internet via the communication device 24c. For example, an indirect notification may include information associated with a current events expression, such as "It seems that there has been an increase in children running out into the road recently," but without specifying an individual child.

[0042] Furthermore, drivers may feel negatively when warning sounds or direct notifications are output. For this reason, as shown in Figures 3 and 4A, the notification control unit 205 may include an evaluation notification 304 that includes expressions of empathy or acceptance for the user after directly or indirectly notifying the driver. Expressions of empathy for the user may include, for example, "That kind of behavior is annoying, isn't it?" The notification control unit 205 can increase the user's acceptance of the notification by issuing a notification that includes expressions of empathy or acceptance for the user after directly or indirectly notifying the driver. Expressions to increase user acceptance may include expressions that praise the user or expressions that offer advice to the user. The notification control unit 205 may also issue an evaluation notification 304 that includes expressions evaluating the user's actions in response to user operations (such as deceleration or steering) received after directly or indirectly notifying the driver. In this case, the notification control unit 205 may include in the evaluation notification 304 expressions praising the user or giving advice to the user, depending on the time elapsed between the direct or indirect notification and the acceptance of user action. For example, if user action is accepted within a predetermined time after the direct or indirect notification is issued, the notification may include expressions praising the driver; otherwise, it may include expressions giving advice to the user.

[0043] Referring to Figure 4B, other examples of the relationship between the state of a risk object and the notification method will be explained. For example, an alarm sound 303 is output when the state is immediately before approaching a risk object. Also, if the risk object is identified, a direct notification 302 is output (under certain conditions such as the time until proximity occurs and the visibility of the risk object). For example, if the time until proximity occurs is less than or equal to a threshold (or the distance to the risk object is less than or equal to a threshold) and the risk object is not visible, a direct notification 302 is output. An indirect notification 301 is issued when the risk object is not identified and under certain conditions when the risk object is identified (under certain conditions such as the time until proximity occurs and the visibility of the risk object). For example, an indirect notification 301 may be issued when the risk object is not identified and when the time until proximity occurs is longer than a threshold (or the distance to the risk object is longer than a threshold) and the risk object is visible. On the other hand, an indirect notification 301 may be issued only when the time until proximity occurs is longer than a threshold (or the distance to the risk object is longer than a threshold) and the risk object is visible. Furthermore, if the risk target is unknown, none of the indirect notification 301, direct notification 302, or warning sound 303 may be issued. The evaluation notification 304 shown in Figure 4B is the same as the evaluation notification 304 explained with reference to Figure 4A.

[0044] The notification control unit 205 generates direct notification utterances by inputting, for example, the type and location of the recognized risk target, the relative distance from vehicle 1, etc., into a trained speech generation algorithm. For example, the speech generation algorithm is trained using direct notification data, which consists of sets of direct notification utterances and various risk targets and relative distances collected in advance. The notification control unit 205 also generates indirect notification utterances by inputting, for example, the presence or absence of a risk target, the type and location of the risk target, the relative distance from vehicle 1, etc., into a trained speech generation algorithm. For example, the speech generation algorithm is trained using indirect notification data, which consists of sets of indirect notification utterances and various risk targets and relative distances, the presence or absence of a risk target, etc., collected in advance. Note that a single speech generation algorithm may output either direct or indirect notifications.

[0045] Similarly, the notification control unit 205 generates notifications that include expressions praising the user or giving advice to the user, by using a speech generation model that has been trained using praise speech data and advice speech data.

[0046] <Example of notification generation using a Large-Scale Language Model (LLM)> Beyond the examples described above, specific examples of cases where the notification control unit 205 generates utterances using a large-scale language model will be explained with reference to Figure 11. Here, an example of generating indirect notifications using a large-scale language model (LLM) will be described. Figure 11 shows an example configuration in the notification control unit 205 when a large-scale language model is used to generate the utterance content of indirect notifications. In the examples described below, a large-scale language model is used to generate the utterance content of indirect notifications, but it may also be used to generate the utterance content of direct notifications, or to generate both indirect and direct notifications.

[0047] The direct notification speech generation unit 1101 generates the content of a direct notification speech. For example, the direct notification speech generation unit 1101 takes risk scene information generated by the risk assessment unit 203 as input and generates speech content related to the direct notification. The risk scene information indicates the content of the risk that can be expected from the scene. For example, the risk scene information includes information indicating a risk scene such as a bicycle, which is the object of the risk, changing lanes to avoid a parked vehicle and colliding with the vehicle. The direct notification speech generation unit 1101 generates, for example, the speech content of the direct notification, "The bicycle on the left may come out this way."

[0048] The indirect notification speech generation unit 1103 generates the content of an indirect notification speech. The indirect notification speech generation unit 1103 generates the content of an indirect notification speech by inputting, for example, a prompt generated by the prompt generation unit 1102 and a driving scene image into a large-scale language model. The driving scene image is an image acquired by the image acquisition unit 201. The prompt generation unit 1102 generates a prompt for the LLM by inputting, for example, the attributes of traffic participants (recognized in the image) output from the target recognition unit 202, the location and attribute information of the risk target output from the risk assessment unit 203, and internet information. The internet information may include, for example, current events news, trending information, and popular words on the internet. The prompt includes an instruction (i.e., a prompt) to the LLM (i.e., the indirect notification speech generation unit 1103) to generate an utterance that expresses the risk target, taking into consideration, for example, current events news, season, trends, popular words, etc. The prompt generation unit 1102 generates a prompt such as, "Consider the winter season and the latest news, generate a conversation about a bicycle being ridden by a person in red clothing in the image." The prompt generation unit 1102 may also generate prompts that include recent current events and slang by inputting internet information. By including recent current events and slang in the prompts, the large-scale language model can generate utterances that are more closely related to current events and slang.

[0049] In this embodiment, the prompt generation unit 1102 receives information on the location and attributes of the risk target output from the risk assessment unit 203 and generates a prompt for LLM. The risk assessment unit 203 is implemented as a risk estimation model and may be implemented as a trained risk estimation model separate from the LLM of the indirect notification speech generation unit 1103, for example, as a model-based or deep neural network model. That is, the risk assessment unit 203 as a risk estimation model identifies the risk target through estimation, and then the prompt generation unit 1102 generates a prompt based on the information of the identified risk target. In this way, the generated prompt is refined with respect to the risk target, and the natural language text of the indirect notification generated by LLM can be made into a more natural and appropriate text. This suppresses the generation of texts that need to be filtered (in the indirect notification selection unit 1104) from the indirect notification generated by LLM, and makes the generated indirect notification more appropriate.

[0050] Furthermore, in this embodiment, the Internet Information Extraction LLM 1006 inputs at least one of the following as Internet information to the prompt generation unit 1102: current events news, trending information, and trending words extracted and generated via the Internet. The Internet Information Extraction LLM 1006 may be a separate LLM from the one used in the indirect notification speech generation unit 1103. By acquiring Internet information using a separate LLM from the one used in the indirect notification speech generation unit 1103, the information input to the prompt generation unit 1102 can be made more refined, and the proportion of noisy or ambiguous information input to the prompt generation unit 1102 can be reduced. The prompt generation unit 1102 generates prompts suitable for generating indirect notifications, making the natural language text of the indirect notification generated by the LLM of the indirect notification speech generation unit 1103 more natural and appropriate. This suppresses the generation of texts that need to be filtered (in the indirect notification selection unit 1104) from the indirect notification generated by the LLM, and makes the generated indirect notifications more appropriate. The Internet Information Extraction LLM1006 may be input with predetermined prompts such as "Tell me the latest news," "Tell me the latest trends," and "Tell me the latest buzzwords." Alternatively, a separate generative model may be used to generate prompts for input to the Internet Information Extraction LLM1006.

[0051] Note that the configuration for inputting Internet information from the Internet information extraction LLM 1006 to the prompt generation unit 1102 is not limited to this example. For example, a module for selecting information that further filters the Internet information from the Internet information extraction LLM 1006 may be interposed. Filtering of Internet information can be performed, for example, using a learning model that has been pre-trained on terms, expressions, and sentences to be filtered.

[0052] The prompt generation unit 1102 may consist of a trained model trained using training data in which the above-mentioned input data and the prompt to be generated constitute one set. When the indirect notification speech generation unit 1103 generates speech content, the generated speech content is input to the indirect notification selection unit 1104. The indirect notification speech generation unit 1103 can generate multiple speech contents for one set of prompt and driving scene image.

[0053] The indirect notification selection unit 1104 filters out the (multiple) utterances generated by the indirect notification utterance generation unit 1103 to only those that meet predetermined conditions, and outputs one utterance, for example, randomly selected from the filtered utterances, as an indirect notification. The indirect notification selection unit 1104 excludes utterances that are equivalent to or highly similar to direct notifications (for example, those containing expressions that prompt the driver to begin risk avoidance) from indirect notifications by determining that they do not meet the predetermined conditions. The indirect notification selection unit 1104 also excludes utterances that contain predetermined negative expressions that evoke an accident by determining that they do not meet the predetermined conditions. The indirect notification selection unit 1104 also excludes utterances that have already been used for notification within a predetermined time, or utterances that are predetermined unnatural expressions, by determining that they do not meet the predetermined conditions. In other words, the indirect notification selection unit 1104 selects and outputs one utterance from the filtered utterances that do not contain these utterances. The indirect notification output by the indirect notification selection unit 1104 includes, for example, utterances such as, "That's a trendy green bicycle, but it looks cold because it's winter."

[0054] In the example shown in Figure 11, internet information is input to the prompt generation unit 1102. However, instead of inputting the internet information to the prompt generation unit 1102, it may also be input to the indirect notification utterance generation unit 1103. Large-scale language models are often trained using a large amount of information from the internet, but the latest current events, trends, and slang may not be included in the training data. Therefore, by inputting internet information (in addition to prompts) into the large-scale language model, it becomes easier to generate utterances that associate the latest current events, trends, etc.

[0055] Next, with reference to Figure 8, a series of operations for notification processing in the vehicle will be described. This process is achieved, for example, by the processor 20a of the ECU 22 of the control unit 2 executing a program in memory 20b.

[0056] In S801, the image acquisition unit 201 acquires an image of the outside of the vehicle 1, for example, using a standard camera 40. In S802, the target recognition unit 202 recognizes a target outside the vehicle 1 based on the image acquired by the image acquisition unit 201.

[0057] In S803, the risk assessment unit 203 identifies targets (e.g., pedestrians) that pose a risk due to their proximity to vehicle 1 as risk targets. In S804, the notification control unit 205 performs the notification control processing described later. Once the notification control processing is complete, the notification control unit 205 terminates this series of operations.

[0058] Next, with reference to Figure 9, a series of operations related to the notification control process will be described. The notification control process described below is executed by the notification control unit 205.

[0059] In S901, the notification control unit 205 determines whether a risk target exists. For example, the notification control unit 205 may determine that a risk target exists if the risk target has been identified by the risk assessment unit 203 as described above. If the notification control unit 205 determines that a risk target exists, it proceeds to S902; otherwise, it terminates the series of operations of the notification control process. In cases where it determines that a risk target does not exist, for example, if the target recognition unit 202 has recognized a target in the image, but the notification control unit 205 determines that no risk target exists (i.e., the risk target is unknown), it may proceed to S906 to generate an indirect notification speech statement.

[0060] In S902, the notification control unit 205 determines whether the time until vehicle 1 and the risk object come into close proximity is longer than the first time threshold (first time). If the notification control unit 205 determines that the time until vehicle 1 and the risk object come into close proximity is longer than the first time threshold (first time), it proceeds to S906; otherwise, it continues the process. S903 The notification control unit 205 may determine whether the distance until vehicle 1 and the risk object are close together is longer than the first distance threshold (first distance). distance If it is determined that the value is longer than the threshold for the first distance (first distance), the process proceeds to S906; otherwise, the process continues. S903 The process proceeds as follows. The notification control unit 205 generates an indirect notification speech statement if there is a time or distance leeway before approaching the risk target.

[0061] In S903, the notification control unit 205 determines whether the driver has seen the risk object. For example, the notification control unit 205 determines whether the driver has seen the risk object based on the estimation result of the gaze estimation unit 204.

[0062] For example, Figure 10 shows a situation where a vehicle 1001 is traveling on a road 1002 and a pedestrian 1003 is crossing the road. In the example shown on the left side of Figure 10, the driver's line of sight 1010 is directed towards the pedestrian 1003, and the driver is seeing the pedestrian 1003 crossing the road 1002. In such a case, the notification control unit 205 determines that the driver is seeing the hazard. The notification control unit 205 can choose not to output a notification when the driver is seeing the hazard. By not sending notifications even when the driver is seeing the hazard, unnecessary notifications to the driver can be suppressed. On the other hand, in the example shown on the right side of Figure 10, the driver's line of sight 1011 is not directed towards the pedestrian 1003, and the driver is not seeing the pedestrian 1003. In such a case, the notification control unit 205 determines that the driver is not seeing the hazard. The notification control unit 205 proceeds to output a direct notification or warning sound if the driver has not seen the risk object (i.e., there is a high probability that the driver has not noticed the risk object). If the notification control unit 205 determines that the driver has seen the risk object, it proceeds to process S908; otherwise, it proceeds to process S904.

[0063] In the example shown in Figure 9, no notification is sent if the driver has visually identified the risk object. However, the notification control unit 205 may also send an indirect notification to the user if it determines that the driver has visually identified the risk object. In this case, the notification control unit 205 may proceed to S906.

[0064] In S904, the notification control unit 205 determines whether the time until vehicle 1 and the risk object approach each other is longer than the second time threshold (second time). In this case, the second time threshold is smaller than the first time threshold. If the time until vehicle 1 and the risk object approach each other is longer than the second time threshold (second time), the notification control unit 205 proceeds to S905; otherwise, it proceeds to S907. The notification control unit 205 may also determine whether the distance until vehicle 1 and the risk object approach each other is longer than the second distance threshold (second distance). In this case, the second distance threshold is smaller than the first distance threshold. If the notification control unit 205 determines that the time until vehicle 1 and the risk object approach each other is longer than the second distance threshold (second distance), it proceeds to S905; otherwise, it proceeds to S907.

[0065] In S905, the notification control unit 205 generates and selects a direct notification using the direct notification speech generation configuration described above, with reference to Figure 11, and notifies the driver (user). In S906, the notification control unit 205 generates and selects an indirect notification using the indirect notification speech generation configuration described above and notifies the driver (user). In S907, the notification control unit 205 outputs a predetermined warning sound.

[0066] In S908, the notification control unit 205 performs follow-up processing, and then terminates the series of operations of the notification control processing. Details of the follow-up processing will be described later.

[0067] As explained above, the notification control unit 205 is configured to provide direct or indirect notifications based on the identification of the risk target and the driving status of the risk target and the vehicle. Here, direct notifications include expressions representing the target and expressions prompting the user to initiate risk avoidance, while indirect notifications include expressions representing the target but do not include expressions prompting the user to initiate risk avoidance. In this way, it becomes possible to provide appropriate notifications depending on the situation.

[0068] Next, a series of operations for the follow-up process that notifies the evaluation notification will be described. The follow-up process is the process by which the notification control unit 205 notifies the driver of the evaluation notification 304 after notifying the driver directly or indirectly. In the above embodiment, the evaluation notification 304 However, the explanation was given using examples of expressions that show empathy for the user or expressions that accept the user (including expressions that praise the user). However, evaluation notification 304 This is not limited to the examples above, but may include utterances that soothe the user or utterances that praise the user.

[0069] The following explanation concerns evaluation notifications. 304 However, evaluation notifications that include calming or praising remarks for the user 304 Using the case of notifying an evaluation notification 304 as an example, we will explain a specific example of the process for notifying an evaluation notification 304 (referred to as the follow-up process).

[0070] Figure 12 shows an example of follow-up processing. The series of operations in the follow-up process are implemented, for example, by the processor 20a of the ECU 22 of the control device 2 executing a program in memory 20b. Furthermore, this process begins after direct or indirect notification has already been given.

[0071] In S1201, the ECU 22 determines the control variables that will achieve ideal vehicle behavior based on the predicted behavior of the risk target. For example, suppose a bicycle, which is the risk target, is traveling, and there is a parked vehicle in front of the bicycle. In this case, the predicted behavior of the risk target includes, for example, the prediction by the risk assessment unit 203 that the risk target will avoid the parked vehicle and enter the lane in which the vehicle is traveling. Furthermore, the control variables that will achieve ideal vehicle behavior include the ideal steering inputs and brake inputs in a time series to avoid the risk of collision with the risk target.

[0072] In S1202, ECU22 acquires user input to the vehicle (via one or more ECUs). User input includes, for example, the actual steering wheel and brake inputs performed by the user over time.

[0073] In S1203, ECU22 determines whether the sum of the differences (in the time direction) between the manipulated variable obtained in S1201 and the manipulated variable obtained in S1202 is greater than a predetermined threshold. If ECU22 determines that the sum of these differences is greater than the predetermined threshold, it proceeds to S1204; otherwise, it proceeds to S1205.

[0074] In S1204, the notification control unit 205 notifies the user of a calming utterance. Situations in which the sum of the differences between the operation amount obtained in S1201 and the operation amount obtained in S1202 is large include situations in which risk avoidance was performed quickly when approaching a risk target. Calming utterances include, for example, "That bicycle just now was really annoying," "I'm relieved that I was able to react to the sudden appearance," or "It's troublesome that bicycles are suddenly appearing more and more these days."

[0075] In S1205, the notification control unit 205 notifies the user of an utterance praising the user. A situation in which the sum of the differences between the operation amounts obtained in S1201 and the operation amounts obtained in S1202 is small includes a situation in which the user was able to operate the vehicle with ample margin from an early stage. Utterances praising the user include, for example, "It's impressive that you noticed the risk in advance," "It was good that you let the bicycle go first," or "It was good that you slowed down in advance." The notification control unit 205 terminates this series of operations after completing the processing in S1204 or S1205.

[0076] In this way, by performing follow-up processing, it is possible to output predetermined notifications (in addition to the direct or indirect notification) with different utterances depending on the user's actions after a direct or indirect notification has been issued. In the example shown in Figure 12, the user's actions are evaluated based on the difference between the amount of operation required to achieve the ideal vehicle behavior, which is determined based on the risk target's behavior prediction result, and the amount of operation performed by the user on the vehicle.

[0077] In the follow-up process described with reference to Figure 12, the example given was one where the utterance content differs based on the difference in the amount of interaction. However, the follow-up process may be based on other judgments. For example, the utterance content may be controlled depending on whether the user took risk avoidance action after direct notification or after indirect notification. An example of such follow-up process will be explained with reference to Figure 13. The series of operations of the follow-up process shown in Figure 13 are realized, for example, by the processor 20a of the ECU 22 of the control device 2 executing a program in the memory 20b.

[0078] In S1301, the ECU22 determines whether the user has taken risk-avoidance action. This determination can be made using any known technique. For example, the ECU22 may determine the time-series manipulation amount that would achieve ideal vehicle behavior based on the risk target behavior prediction result, and if the user's time-series manipulation amount for the vehicle is similar to the time-series manipulation amount that would achieve ideal vehicle behavior, it may determine that the user has taken risk-avoidance action. If the ECU22 determines that the user has taken risk-avoidance action, it proceeds to S1302; otherwise, it can terminate the follow-up process and return to the original process.

[0079] In S1302, the notification control unit 205 determines whether the user's risk avoidance action is an avoidance due to indirect notification. If the user's risk avoidance action is performed within a predetermined time after the indirect notification, the notification control unit 205 determines that it is an avoidance due to indirect notification and proceeds to S1305; otherwise, it proceeds to S1303.

[0080] In S1303, the notification control unit 205 determines whether the user's risk avoidance action is an avoidance due to a direct notification. If the user's risk avoidance action was performed after the notification of a direct notification, the notification control unit 205 determines that it is an avoidance due to a direct notification and proceeds to S1304; otherwise, it proceeds to S1305. Cases where it is determined that it is not an avoidance due to a direct notification include, for example, cases where the risk avoidance action was not performed within a predetermined time after the notification of an indirect notification, but the action was performed before the direct notification was issued.

[0081] In S1304, the notification control unit 205 notifies the user of a calming utterance. An example of a calming utterance is the same as the example described with reference to Figure 12. In S1305, the notification control unit 205 is Yu The system notifies the user of an utterance praising them. An example of an utterance praising the user is the same as the example explained with reference to Figure 12. The notification control unit 205 terminates this series of operations after completing the processing of S1204 or S1205. Even when performing such follow-up processing, the system can output a predetermined notification with different utterance content depending on the user's actions after the direct or indirect notification.

[0082] <Summary of Embodiments> 1. The control device of the above embodiment (for example, 2) is: A control device disposed on a mobile body (e.g., 1) equipped with an imaging device (e.g., 40), Image acquisition means (e.g., 201) that acquires an image of the outside of the moving object captured by the imaging device, Based on the aforementioned image, a recognition means (e.g., 202) recognizes an external target of the moving object, A means for identifying risk targets (e.g., 203) that have a risk of being in close proximity to the moving object among the recognized targets, The system includes a notification control means (e.g., 205) that notifies the user of a voice notification using natural language, which includes an expression representing the recognized target, The notification control means notifies the user of either a direct notification (e.g., 302) that notifies the user of the presence of a risk caused by the risk object, or an indirect notification (e.g., 301) that notifies the user of information suggesting the presence of the risk object, based on the risk object and the driving state of the moving body.

[0083] According to this embodiment, it becomes possible to provide appropriate notifications depending on the situation.

[0084] 2. In the above embodiment, If a risk target is identified in the result of identifying the risk target, the notification control means will notify the indirect notification according to the identified risk target and the driving state of the moving body.

[0085] According to this embodiment, while simply sounding an alarm may not allow for an intuitive understanding of the risk and the frequent alarms may be perceived as unpleasant, this embodiment allows drivers to continue driving while being aware of potential risks.

[0086] 3. In the above embodiment, The notification control means issues the direct notification if the risk target has been identified in the result of identifying the risk target, and the identified risk target and the moving body satisfy predetermined proximity conditions.

[0087] According to this embodiment, simply sounding an alarm does not allow users to intuitively understand the nature of the risk, but direct notification allows users to specifically identify what they need to pay attention to and take the necessary actions.

[0088] 4. In the above embodiment, The system further includes estimation means (e.g., 204) for estimating whether the user is visually inspecting the risk object, The notification control means issues the direct notification when a risk target has been identified in the result of identifying the risk target, and when the identified risk target and the moving object satisfy predetermined proximity conditions, and it is further estimated that the user has not seen the risk target.

[0089] According to this embodiment, it is possible to appropriately output notifications when there is a high probability that the user is unaware of the risk.

[0090] 5. In the above embodiment, The notification control means controls the output of the direct notification if, in the result of identifying the risk target, the risk target has been identified, and the identified risk target and the moving object satisfy predetermined proximity conditions, and it is further estimated that the user has visually observed the risk target.

[0091] According to this embodiment, unnecessary notifications to the user can be suppressed by notifying the user even when the user has visually identified the risk object.

[0092] 6. In the above embodiment, The notification control means, when a risk target is identified in the result of identifying the risk target, notifies the indirect notification when the time until the identified risk target and the moving body approach each other is longer than a first time or the distance until the identified risk target and the moving body approach each other is longer than a first distance, and notifies the direct notification when the time until the identified risk target and the moving body approach each other is less than or equal to the first time or the distance until the identified risk target and the moving body approach each other is less than or equal to the first distance.

[0093] According to this embodiment, if there is a long time until the risk object and the moving object come into close proximity, indirect notification allows the user to continue driving while being aware of the potential risk, and if there is a short time until the risk object and the moving object come into close proximity, direct notification allows the user to focus their attention on the specific risk object.

[0094] 7. In the above embodiment, The notification control means outputs an alarm sound if the time until the identified risk target and the moving body come into close proximity is less than or equal to a second time which is shorter than a first time, or if the distance until the identified risk target and the moving body come into close proximity is less than or equal to a second distance which is shorter than a first distance.

[0095] According to this embodiment, in situations where the user can understand the situation and the actions they should take by looking in the direction they are going, intuitive notifications about risks can be provided.

[0096] 8. In the above embodiment, The notification control means issues the indirect notification if no risk target is identified in the results of the risk target identification.

[0097] According to this embodiment, while simply sounding an alarm may not allow for an intuitive understanding of the risk and the frequent alarms may be perceived as unpleasant, this embodiment allows drivers to continue driving while being aware of potential risks.

[0098] 9. In the above embodiment, The direct notification includes at least one of the following: an expression indicating the location of the target subject to risk, an expression indicating the type of target subject to risk, and an expression prompting the user to initiate risk avoidance.

[0099] According to this embodiment, the user can specifically identify the object that requires attention and take the necessary actions.

[0100] 10. In the above embodiment, The direct notification further includes at least one of the following: a statement describing the expected future behavior of the target; and a statement describing the reasons for initiating risk avoidance.

[0101] According to this embodiment, the user can easily understand what will happen if they continue driving.

[0102] 11. In the above embodiment, The aforementioned indirect notification includes, as an expression representing a target, an expression indicating the existence of a target.

[0103] According to this embodiment, it is possible to continue driving while being aware of potential risks.

[0104] 12. In the above embodiment, The notification control means includes a large-scale language model that generates the content of the indirect notification utterance, and a prompt generation means that generates a prompt which is an instruction to cause the large-scale language model to generate an utterance that represents the risk object. The large-scale language model generates the content of the indirect notification based on the image acquired by the image acquisition means and the prompt generated by the prompt generation means.

[0105] According to this embodiment, it is possible to generate indirect notifications containing natural speech content from a large-scale language model, and furthermore, by inputting the generated prompt, appropriate instructions can be given to obtain desired speech content from the large-scale language model.

[0106] 13. In the above embodiment, The prompt generation means generates the prompt based on internet information including at least one of current events, trends, and popular words obtained via the internet, the attributes of traffic participants recognized in the image, and information on the location and attributes of the identified risk target.

[0107] According to this embodiment, prompts can be generated that take into account the latest trends on the internet.

[0108] 14. In the above embodiment, The aforementioned identification means consists of a trained estimation model that is separate from the large-scale language model. The prompt generation means generates the prompt using the location and attribute information of the risk target identified by the identification means, which is the estimation model.

[0109] According to this embodiment, the generated prompts are refined for the risk target, and the natural language text of the indirect notifications generated by the Large-Scale Language Model (LLM) can be made more natural and appropriate. This suppresses the generation of text that would need to be filtered in the indirect notifications generated by the LLM, making the generated indirect notifications more appropriate.

[0110] 15. In the above embodiment, The system further includes a second large-scale language model, separate from the aforementioned large-scale language model, for outputting the aforementioned internet information. The prompt generation means generates the prompt using at least one of current events, trends, and slang obtained via the Internet by the second large-scale language model.

[0111] According to this embodiment, the information input to the prompt generation means can be made more refined, and the proportion of noisy or ambiguous information input to the prompt generation means can be reduced. By generating prompts suitable for generating indirect notifications using the prompt generation means, the natural language text of the indirect notifications generated by the large-scale language model (LLM) can be made more natural and appropriate. This suppresses the generation of texts that need to be filtered in the indirect notifications generated by the LLM, and makes the generated indirect notifications more appropriate.

[0112] 16. In the above embodiment, The prompt generation means generates the prompt based on the attributes of the traffic participant recognized in the image and the location and attribute information of the identified risk target. The large-scale language model generates the content of the indirect notification based on internet information obtained via the internet (including, for example, current events, trends, and popular words), images obtained by the image acquisition means, and prompts generated by the prompt generation means.

[0113] According to this embodiment, the large-scale language model can generate speech content that takes into account the latest trends on the internet.

[0114] 17. In the above embodiment, The notification control means excludes utterances that do not meet predetermined conditions from the utterances of multiple indirect notifications generated by the large-scale language model.

[0115] According to this embodiment, it is possible to extract only the utterances that are appropriate as indirect notifications from among the multiple utterances generated by a large-scale language model.

[0116] 18. In the above embodiment, The notification control means further outputs a predetermined notification whose content differs depending on the user's actions after the direct notification or the indirect notification has been issued.

[0117] According to this embodiment, it is possible to increase user acceptance of notifications.

[0118] 19. In the above embodiment, The user's actions are evaluated based on the difference between the amount of manipulation required to realize the behavior of the moving object, which is determined from the behavior prediction result of the risk target, and the amount of manipulation performed by the user on the moving object.

[0119] According to this embodiment, the user can be notified of the appropriateness of the actions taken by the user to avoid risk through speech.

[0120] 20. In the above embodiment, The notification control means, after notifying the user of the direct notification or the indirect notification, further outputs a predetermined notification that includes an expression evaluating the user's actions.

[0121] According to this embodiment, it is possible to provide the user with feedback on risk avoidance.

[0122] 21. In the above embodiment, The aforementioned notification shall include expressions that show empathy for the user or expressions that accept the user.

[0123] According to this embodiment, after sending a direct or indirect notification, an evaluation notification that is tailored to the user can be output.

[0124] 22. In the above embodiment, The notification control means outputs a calming utterance if the user takes risk-avoidance action after the direct notification is issued, and outputs a praise utterance if the user takes risk-avoidance action after the indirect notification is issued.

[0125] According to this embodiment, in situations where direct notification is likely to cause the user to feel negative emotions, it is possible to output utterances that soothe the user's feelings. Furthermore, if the presence of a target requiring attention is recognized from the indirect notification stage, i.e., at a relatively early stage, and risk avoidance can be taken, it is possible to provide utterances that make the user feel positive emotions.

[0126] 23. In the above embodiment, A mobile body including the above-mentioned control device is provided.

[0127] According to this embodiment, a mobile body is provided that can provide appropriate notifications depending on the situation.

[0128] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention. [Explanation of Symbols]

[0129] 1...Vehicle, 2...Control device, 21-29...ECU

Claims

1. A control device disposed on a mobile body equipped with an imaging device, Image acquisition means for acquiring an image of the outside of the moving object captured by the aforementioned imaging device, Based on the aforementioned image, a recognition means for recognizing an external target of the moving object, A means for identifying risk targets among the recognized targets that pose a risk due to their proximity to the moving object, The system includes a notification control means that notifies the user of a voice notification using natural language, which includes an expression representing a recognized target, The notification control means notifies the user of either a direct notification that informs the user of the presence of a risk caused by the risk object, or an indirect notification that provides information suggesting the presence of the risk object, based on the risk object and the driving state of the moving body. The notification control means includes a large-scale language model that generates the content of the indirect notification, and a prompt generation means that generates a prompt which is an instruction to cause the large-scale language model to generate an utterance that represents the risk object. The large-scale language model generates the speech content of the indirect notification based on the image acquired by the image acquisition means and the prompt generated by the prompt generation means. The control device is characterized in that the prompt generation means generates the prompt based on internet information obtained via the internet, attributes of traffic participants recognized in the image, and location and attribute information of the identified risk target.

2. The control device according to claim 1, wherein the notification control means notifies the indirect notification according to the identified risk target and the driving state of the moving body when a risk target has been identified in the result of identifying the risk target.

3. The control device according to claim 1, wherein the notification control means notifies the direct notification when the risk target is identified in the result of identifying the risk target and the identified risk target and the moving body satisfy predetermined proximity conditions.

4. The system further comprises estimation means for estimating whether the user is visually inspecting the risk object. The control device according to claim 1, wherein the notification control means provides a direct notification when a risk target has been identified in the result of identifying the risk target, and the identified risk target and the moving object satisfy predetermined proximity conditions, and it is further estimated that the user has not visually inspected the risk target.

5. The control device according to claim 4, wherein the notification control means controls the output of the direct notification when a risk target has been identified in the result of identifying the risk target, and when the identified risk target and the moving object satisfy predetermined proximity conditions, and it is further estimated that the user has visually inspected the risk target.

6. The control device according to claim 1, wherein the notification control means, when a risk target is identified in the result of identifying the risk target, notifies the indirect notification when the time until the identified risk target and the moving body approach each other is longer than a first time or the distance until the identified risk target and the moving body approach each other is longer than a first distance, and notifies the direct notification when the time until the identified risk target and the moving body approach each other is less than or equal to the first time or the distance until the identified risk target and the moving body approach each other is less than or equal to the first distance.

7. The control device according to claim 6, characterized in that the notification control means outputs an alarm sound when the time until the identified risk target and the moving body come into close proximity is less than or equal to a second time which is shorter than a first time, or when the distance until the identified risk target and the moving body come into close proximity is less than or equal to a second distance which is shorter than a first distance.

8. The control device according to claim 1, wherein the notification control means notifies the indirect notification if the risk target is not identified in the result of identifying the risk target.

9. The control device according to claim 1, characterized in that the direct notification includes at least one of the following: an expression indicating the location of the target that is the risk object, an expression indicating the type of target that is the risk object, and an expression instructing the user to initiate risk avoidance.

10. The control device according to claim 9, wherein the direct notification further includes at least one of the following: an expression representing the expected future behavior of the target and an expression representing the reason for initiating risk avoidance.

11. The control device according to claim 1, characterized in that the indirect notification includes an expression indicating the existence of a target as an expression representing a target.

12. A control device disposed on a mobile body equipped with an imaging device, Image acquisition means for acquiring an image of the outside of the moving object captured by the aforementioned imaging device, Based on the aforementioned image, a recognition means for recognizing an external target of the moving object, A means for identifying risk targets among the recognized targets that pose a risk due to their proximity to the moving object, The system includes a notification control means that notifies the user of a voice notification using natural language, which includes an expression representing a recognized target, The notification control means notifies the user of either a direct notification that informs the user of the presence of a risk caused by the risk object, or an indirect notification that provides information suggesting the presence of the risk object, based on the risk object and the driving state of the moving body. The notification control means includes a large-scale language model that generates the content of the indirect notification, and a prompt generation means that generates a prompt which is an instruction to cause the large-scale language model to generate an utterance that represents the risk object. The large-scale language model generates the speech content of the indirect notification based on the image acquired by the image acquisition means and the prompt generated by the prompt generation means. The aforementioned identification means consists of a trained estimation model that is separate from the large-scale language model. The control device is characterized in that the prompt generation means generates the prompt using information on the location and attributes of the risk target identified by the identification means, which is the estimation model.

13. A control device disposed on a mobile body equipped with an imaging device, Image acquisition means for acquiring an image of the outside of the moving object captured by the aforementioned imaging device, Based on the aforementioned image, a recognition means for recognizing an external target of the moving object, A means for identifying risk targets among the recognized targets that pose a risk due to their proximity to the moving object, The system includes a notification control means that notifies the user of a voice notification using natural language, which includes an expression representing a recognized target, The notification control means notifies the user of either a direct notification that informs the user of the presence of a risk caused by the risk object, or an indirect notification that provides information suggesting the presence of the risk object, based on the risk object and the driving state of the moving body. The notification control means includes a large-scale language model that generates the content of the indirect notification, and a prompt generation means that generates a prompt which is an instruction to cause the large-scale language model to generate an utterance that represents the risk object. The large-scale language model generates the speech content of the indirect notification based on the image acquired by the image acquisition means and the prompt generated by the prompt generation means. The prompt generation means generates the prompt based on internet information obtained via the internet, the attributes of traffic participants recognized in the image, and the location and attribute information of the identified risk target. The system further includes a second large-scale language model, separate from the aforementioned large-scale language model, for outputting the aforementioned internet information. The control device is characterized in that the prompt generation means generates the prompt using information obtained via the Internet by the second large-scale language model.

14. A control device disposed on a mobile body equipped with an imaging device, Image acquisition means for acquiring an image of the outside of the moving object captured by the aforementioned imaging device, Based on the aforementioned image, a recognition means for recognizing an external target of the moving object, A means for identifying risk targets among the recognized targets that pose a risk due to their proximity to the moving object, The system includes a notification control means that notifies the user of a voice notification using natural language, which includes an expression representing a recognized target, The notification control means notifies the user of either a direct notification that informs the user of the presence of a risk caused by the risk object, or an indirect notification that provides information suggesting the presence of the risk object, based on the risk object and the driving state of the moving body. The notification control means includes a large-scale language model that generates the content of the indirect notification, and a prompt generation means that generates a prompt which is an instruction to cause the large-scale language model to generate an utterance that represents the risk object. The large-scale language model generates the speech content of the indirect notification based on the image acquired by the image acquisition means and the prompt generated by the prompt generation means. The prompt generation means generates the prompt based on the attributes of the traffic participant recognized in the image and the location and attribute information of the identified risk target. The control device is characterized in that the large-scale language model generates the content of the indirect notification based on internet information obtained via the internet, images obtained by the image acquisition means, and prompts generated by the prompt generation means.

15. The control device according to claim 1, characterized in that the notification control means excludes utterances that do not satisfy predetermined conditions from among the utterances of a plurality of indirect notifications generated by the large-scale language model.

16. The control device according to claim 1, wherein the notification control means further outputs a predetermined notification whose content differs depending on the user's actions after the direct notification or the indirect notification has been issued.

17. The control device according to claim 16, wherein the user's actions are evaluated based on the difference between the amount of operation required to realize the behavior of the moving object, which is determined in relation to the behavior prediction result of the risk target, and the amount of operation performed by the user on the moving object.

18. The control device according to claim 16, wherein the notification control means further outputs a predetermined notification including an expression that evaluates the user's actions after notifying the user of the direct notification or the indirect notification, and the predetermined notification includes an expression that shows empathy for the user or an expression that accepts the user.

19. A control device disposed on a mobile body equipped with an imaging device, Image acquisition means for acquiring an image of the outside of the moving object captured by the aforementioned imaging device, Based on the aforementioned image, a recognition means for recognizing an external target of the moving object, A means for identifying risk targets among the recognized targets that pose a risk due to their proximity to the moving object, The system includes a notification control means that notifies the user of a voice notification using natural language, which includes an expression representing a recognized target, The notification control means notifies the user of either a direct notification that informs the user of the presence of a risk caused by the risk object, or an indirect notification that provides information suggesting the presence of the risk object, based on the risk object and the driving state of the moving body. The notification control means further outputs a predetermined notification whose content differs depending on the user's actions after the direct notification or the indirect notification has been issued. The notification control means is characterized by outputting a calming utterance when the user takes risk-avoidance action after the direct notification is issued, and outputting a praise utterance when the user takes risk-avoidance action after the indirect notification is issued.

20. A control method in which each step is executed by a control device arranged on a mobile body equipped with an imaging device, The imaging device acquires an image of the outside of the moving object, Based on the aforementioned image, the external target of the moving object is recognized, Among the recognized targets, identify the risk targets that pose a risk due to their proximity to the moving object, This includes notifying the user with a voice notification using natural language, which includes an expression representing the recognized target, The notification described above includes either a direct notification informing the user of the existence of a risk caused by the risk object, or an indirect notification informing the user of information suggesting the presence of the risk object, based on the risk object and the driving state of the moving object. The notification described above includes generating a prompt which is an instruction to cause the large-scale language model that generates the content of the indirect notification to generate an utterance that represents the risk object, The large-scale language model generates the speech content of the indirect notification based on the acquired image and the generated prompt. A control method for generating the prompt, comprising generating the prompt based on internet information obtained via the internet, attributes of traffic participants recognized in an image, and location and attribute information of identified risk targets.

Images