Notification method, notification device, program, and notification system

Abstract

This technology provides a way to reduce confusion and stress experienced by people with color vision deficiency through warning notifications. [Solution] The exemplary notification method involves a first artificial intelligence model generating attention points that a person with normal color vision would notice in a camera image, a second artificial intelligence model generating attention points that a person with color vision deficiency would notice in the camera image, and by comparing the generated attention points that a person with normal color vision would notice with the attention points that a person with color vision deficiency would notice, an object to be alerted is identified, and a notification is made to alert the identified object to be alerted.

Description

Technical Field

[0001] The present invention relates to a notification method, a notification device, a program, and a notification system.

Background Art

[0002] Conventionally, there has been a technique for detecting pedestrians and obstacles from the image of a camera attached to the front of a vehicle and issuing a notification to alert the driver (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the driver is a color vision defective person, if a notification for alerting an object that the driver has not perceived is issued, the driver will be confused or stressed because the reason for attention is unknown.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique capable of reducing the confusion and stress experienced by color vision defective persons due to alert notifications.

Means for Solving the Problems

[0006] An exemplary notification method of the present invention generates, by a first artificial intelligence model, attention points that a person with normal color vision notices with respect to a camera image, generates, by a second artificial intelligence model, attention points that a color vision defective person notices with respect to the camera image, identifies an object to be alerted by comparing the generated attention points that the person with normal color vision notices and the attention points that the color vision defective person notices, and issues a notification for alerting the identified object to be alerted.

Effects of the Invention

[0007] According to an exemplary example of the present invention, it is possible to identify objects that are difficult for colorblind individuals to perceive and require attention. By appropriately providing notifications to alert individuals to the identified objects (in a manner that allows colorblind individuals to understand the reason for the attention required), it is possible to reduce confusion and stress experienced by colorblind individuals as a result of the notification. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of the notification system according to the first embodiment. [Figure 2] Diagram showing an example of the configuration of a navigation system. [Figure 3] A diagram showing an example server configuration. [Figure 4] Notification processing flowchart [Figure 5] A diagram showing an example of a notice or display for an object requiring a warning. [Figure 6] A diagram showing an example of the server configuration according to the third embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0010] <First Embodiment> [Notification System] Figure 1 is a diagram showing an example configuration of the notification system SYS1 according to the first embodiment. As shown in Figure 1, the notification system SYS1 includes a navigation device 1 and a server SV1.

[0011] In this embodiment, the navigation device 1 is used inside the vehicle V1. The navigation device 1 may be a device that is permanently installed in the vehicle V1, or it may be a portable device that can be moved in and out of the vehicle V1. In addition to the navigation device 1, the vehicle V1 is also equipped with a drive recorder DR1. The drive recorder DR1 photographs the area in front of the vehicle V1 and records the captured images as needed.

[0012] Navigation device 1 guides the user along a route and, when the color blindness alert function is turned on, identifies objects requiring attention using attention location information generated by LLM (Large Language Models) in server SV1, and issues a notification alerting the user to the identified objects. In other words, navigation device 1 is an example of a notification device that alerts the user to objects requiring attention, which are identified by comparing attention locations noticed by people with normal color vision with attention locations noticed by people with color blindness in the camera image. The color blindness alert function can be turned on or off by the user through operation on the control device 6, which will be described later.

[0013] Server SV1 includes an LLM as described above. The LLM in Server SV1 is a multimodal LLM capable of image recognition. Server SV1 inputs the question text sent from Navigation Device 1 via the network (Internet) NT1 into the LLM and generates an answer text corresponding to the question text. Server SV1 transmits the generated answer text to Navigation Device 1.

[0014] [Navigation system] Figure 2 shows the configuration of the navigation device 1. The navigation device 1 comprises a controller 2, a memory 3, and a communication unit 4.

[0015] Controller 2 includes a processor that performs calculations and other processing. The processor may include, for example, a CPU (Central Processing Unit). Controller 2 may consist of one processor or multiple processors. If it consists of multiple processors, they just need to be connected to each other so that they can communicate with one another. Controller 2 also includes computer components such as memory (RAM (Random Access Memory), ROM (Read Only Memory), etc.) necessary for program execution.

[0016] As its functions, the controller 2 includes an acquisition unit 21, a destination setting unit 22, a route search unit 23, a timing unit 24, a route guidance unit 25, a prompt generation unit 26, a comparison unit 27, a notification unit 28, and an output control unit 29.

[0017] Also, each of the functional units 21 to 29 may be realized by the processor executing arithmetic processing according to one program. However, for example, it may be a configuration realized by the processor executing arithmetic processing according to separate programs for each functional unit. Also, each of the functional units 21 to 29 may be realized by software, that is, by causing the processor to execute a program as described above. However, it may also be realized by other methods. At least a part of each of the functional units 21 to 29 may be realized using, for example, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. That is, each of the functional units 21 to 29 may be realized by hardware using a dedicated IC or the like. Also, each of the functional units 21 to 29 may be realized by using software and hardware in combination. Also, each of the functional units 21 to 29 is a conceptual component. The functions executed by one component may be distributed among multiple components. Also, the functions of multiple components may be integrated into one component.

[0018] The acquisition unit 21 acquires various images and information. For example, the acquisition unit 21 acquires a camera image (a captured image captured by the drive recorder DR1) from the drive recorder DR1. Also, for example, the acquisition unit 21 acquires map information 32 from the memory 3. Also, for example, the acquisition unit 21 acquires information input and selected by the user from the operation device 6. Also, for example, the acquisition unit 21 acquires position information indicating the current position of the navigation device 1 from the position information acquisition device 5. Also, for example, the acquisition unit 21 acquires the response of the server SV1 via the communication unit 4.

[0019] The destination setting unit 22 sets the destination of the guidance route. Specifically, for example, the destination setting unit 22 extracts candidates for the destination that match the search conditions for the destination input by the user via the operation device 6. Further, the destination setting unit 22 sets the destination candidate selected by the user via the operation device 6 as the destination.

[0020] The route search unit 23 searches for a route (guidance route) from the departure point (the current position of the navigation device 1 at the time of route search) to the destination. Specifically, for example, the route search unit 23 uses the destination set by the destination setting unit 22, the position information indicating the current position of the navigation device 1 acquired by the position information acquisition device 5, and the map information 32 stored in the memory 3 to search for a route (guidance route) from the departure point to the destination.

[0021] The timing unit 24 measures time information indicating the current time. For example, the timing unit 24 measures time using the time data included in the GPS information received by a GPS (Global Positioning System) receiver described later, or using a timer that counts the clock signal used in a computer. Further, the timing unit 24 calculates the required time from the departure point to the destination. Specifically, for example, the timing unit 24 calculates the required time from the departure point to the destination based on the distance of the route from the departure point to the destination based on the map data, information such as the legal speed limit of each road link in the guidance route, etc.

[0022] The route guidance unit 25 guides the route (guidance route) from the departure point to the destination. Specifically, for example, the route guidance unit 25 causes the video output device 7 to display an image in which the guidance route is superimposed on the map image near the current position of the navigation device 1 via the output control unit 29, and outputs voice guidance regarding route guidance such as a right turn instruction and a left turn instruction to the voice output device 8 when turning right or left, thereby guiding the user the route (guidance route) from the departure point to the destination.

[0023] The prompt generation unit 26 uses the prompt template 33 to generate a prompt appropriate for performing the color blindness warning function, and transmits the generated prompt to the server SV1 via the communication unit 4.

[0024] The comparison unit 27 identifies the object to be alerted by comparing two responses sent from the server SV1, namely, the points of interest that a person with normal color vision would notice in the camera image and the points of interest that a person with color vision deficiency would notice in the camera image.

[0025] The notification unit 28 issues a notification to alert the user to the object identified by the comparison unit 27. The notification by the notification unit 28 may be a display notification, an audio notification, or a combination of display and audio.

[0026] The output control unit 29 controls the output from the video output device 7 (video output) and the output from the audio output device 8 (audio output).

[0027] Memory 3 is composed of a non-volatile storage medium such as flash memory or a hard disk drive. Memory 3 stores various programs 31, map information 32, prompt templates 33, and color information 34. Part or all of Memory 3 may be composed of a portable recording medium and be detachable from the main body of the navigation device 1.

[0028] Program 31 includes, in addition to a general navigation program, a program that provides notifications to alert users to objects identified by comparing the points of attention that normal colorblind individuals notice with those of colorblind individuals in camera images. Generally, the program is written to memory 3 at the factory during the manufacturing of the navigation device 1, but it is also possible to download the program from a server and write it to memory 3, or to have the navigation device 1 read a storage medium such as an optical disc or memory card on which the program is written and have the navigation device 1 write it to memory 3, or to use a data writer that reads the program from these storage media and writes it to the memory 3 of the navigation device 1.

[0029] Map information 32 is map-related data used for route guidance, etc., and includes road information, facility information, etc. Road information includes various information about each road (each link: unit road), such as distance of the road link, legal speed limit, road type, etc. Facility information includes event venue information. Map data is divided into predetermined areas called meshes (for example, 100m square), and each mesh is filed separately. Map data for the area used for processing, etc., is selected from map information 32 and used.

[0030] Prompt template 33 is a template (standard phrase) for a prompt that is appropriate for serving as a warning function for people with color blindness.

[0031] Color information 34 is color information that people with color vision deficiency cannot perceive. Furthermore, color information 34 is a color conversion data table in which each color is associated with a corresponding conversion color. By using this data table, for example, by replacing the color of each pixel in a camera image with the conversion color, it is possible to generate images that are easily recognizable by people with color vision deficiency.

[0032] The communication unit 4 transmits and receives arbitrary signals with the server SV1. Although the controller 2 can transmit and receive arbitrary information with the server SV1 using the communication unit 4, the description of the communication unit 4 may be omitted below.

[0033] The location information acquisition device 5, the operation device 6, the video output device 7, and the audio output device 8 are connected to the navigation device 1 via an in-vehicle network or the like.

[0034] The location information acquisition device 5 acquires location information indicating the current location of the navigation device 1. For example, a GPS receiver can be used as the location information acquisition device 5. The location information includes, for example, latitude and longitude information.

[0035] The operating device 6 is a device that allows the user to input commands, information, etc., to operate the navigation device 1. For example, the operating device 6 can be a touch panel, mechanical buttons, etc. The operating device 6 may be configured by placing a touch panel on the screen of the video output device 7. Alternatively, the operating device 6 may be configured by arranging mechanical buttons around the screen of the video output device 7.

[0036] The video output device 7 is a display device that outputs video based on video signals sent from the output control unit 29, and its screen faces the interior of the vehicle. For example, the video output device 7 can be a liquid crystal display, an organic EL (Electro-Luminescence) display, or a head-up display that projects images onto the vehicle's windshield. The navigation device 1 can display, for example, maps, road information, vehicle location, and notifications from the notification unit 28 on the screen of the video output device 7. Note that the video output device 7 is not limited to a device installed in the vehicle. For example, the navigation device 1 may have wireless communication functions such as Wi-Fi® communication or Bluetooth® communication, and the navigation device 1 may communicate wirelessly with a portable device brought into the vehicle, with the display unit of the portable device being used as the video output device 7.

[0037] The audio output device 8 is a device that outputs audio based on an audio signal sent from the output control unit 29, and faces the vehicle interior. For example, a speaker can be used as the audio output device 8. The navigation device 1 can provide audio notifications from the audio output device 8, such as maps, road information, vehicle location, and notifications from the notification unit 28. Note that the audio output device 8 is not limited to a device installed in the vehicle. For example, the navigation device 1 may have wireless communication functions such as Wi-Fi communication or Bluetooth communication, and the navigation device 1 may communicate wirelessly with a portable device brought into the vehicle interior, and the audio output unit of the portable device may be used as the audio output device 8.

[0038] [server] Figure 3 shows an example configuration of server SV1. Server SV1 comprises a communication unit 110, a memory 120, and a controller 130.

[0039] The communication unit 110 transmits and receives arbitrary signals to and from the navigation device 1. The controller 130 can also transmit and receive arbitrary information to and from the navigation device 1 using the communication unit 110; however, the description of the communication unit 110 may be omitted below.

[0040] Memory 120 is configured to include non-volatile memory such as ROM or flash memory, and volatile memory such as RAM. Memory 120 stores program 121 and LLM 122. Program 121 is a program that implements various functions of server SV1. LLM 122 is a model trained on training data that does not take color vision characteristics into consideration. In other words, LLM 122 is a model trained on training data that includes many combinations of images and attention points (correct data) that a person with normal color vision would notice within the image. In this embodiment, since training with training data that takes color vision characteristics into consideration is not required, the preparation of training data becomes easier.

[0041] The controller 130 comprehensively controls the operation of each component in the server SV1. The controller 130 is equipped with a processing unit including a CPU as a hardware resource. The controller 130 has an LLM execution unit 131.

[0042] When the LLM execution unit 131 receives a prompt from the navigation device 1, it uses the prompt as input data to execute the LLM 122 and causes the LLM 122 to generate a response. Once the LLM 122 generates a response, the controller 130 transmits the response generated by the LLM 122 to the navigation device 1.

[0043] [Notification Processing] Figure 4 is a flowchart of the notification process executed by controller 2. This notification process is achieved when controller 2 executes program 31 as described above. The notification process shown in Figure 4 starts when power is supplied to navigation device 1 while the color blindness warning function of navigation device 1 is turned on, or when the color blindness warning function is switched from off to on while power is being supplied to navigation device 1. The notification process shown in Figure 4 ends when power is stopped to navigation device 1, or when the color blindness warning function is switched from on to off.

[0044] First, in step S10, the controller 2 (acquisition unit 21) acquires the camera image, which is the image captured by the drive recorder DR1. Once the processing in step S10 is complete, the process proceeds to step S20.

[0045] In step S20, the controller 2 (prompt generation unit 26) generates a first prompt for the LLM122 to generate and output attention points that a person with normal color vision would notice in the camera image acquired in step S10, and sends it to the server SV1. Once the processing in step S20 is complete, the process proceeds to step S30.

[0046] An example of the first prompt generated in step S20 is a prompt that includes the camera image acquired in step S10 and the question, "You are the driver. You can see the scenery shown in the attached image. Please tell us what points to pay attention to for safe driving. Please answer in the format of answering in order the type of area to pay attention to, the location of the area to pay attention to in the image, the color of the area to pay attention to, and the reason why you should pay attention to it." This question is stored in memory 3 as one of the prompt templates 33. By including control statements to control the output format, such as "Please answer in the format of answering in the format of answering in order the type of area to pay attention to, the location of the area to pay attention to in the image, the color of the area to pay attention to, and the reason why you should pay attention to it," in both the first prompt and the second prompt described later, the comparison processing by the comparison unit 27 becomes easier, and an improvement in the accuracy of the comparison results can be expected.

[0047] In step S30, controller 2 checks whether it has received a response to the first prompt. If it has not received a response to the first prompt, it returns to step S30; if it has received a response to the first prompt, it proceeds to step S40.

[0048] In step S40, the controller 2 stores the answer to the first prompt in memory 3. Once the processing in step S40 is complete, the process proceeds to step S50.

[0049] In step S50, the controller 2 (prompt generation unit 26) generates a converted image by transforming the camera image acquired in step S10 to simulate the color vision of a person with color blindness. It also generates a second prompt for the LLM 122 to generate and output points of interest that a person with color blindness would notice in the camera image acquired in step S10, and sends it to the server SV1. Image transformation can be performed by replacing the color of each pixel in the camera image with the transformed color using a color transformation data table in which each color of the color information 34 is associated with the transformed color corresponding to each color. Once the processing in step S50 is complete, the process proceeds to step S60.

[0050] An example of the second prompt generated in step S50 is a prompt containing the above converted image and the question: "You are the driver. You can see the scenery shown in the attached image. Please tell us what points to pay attention to for safe driving. Please answer in the following order: type of point of attention, location of the point of attention in the image, color of the point of attention, and reason for paying attention." This question is stored in memory 3 as one of the prompt templates 33.

[0051] In step S60, controller 2 checks whether it has received a response to the second prompt. If it has not received a response to the second prompt, it returns to step S60; if it has received a response to the second prompt, it proceeds to step S70.

[0052] In step S70, the controller 2 stores the answer to the second prompt in memory 3. Once the processing in step S70 is complete, the process proceeds to step S80.

[0053] In step S80, the controller 2 (comparison unit 27) identifies objects to be alerted by comparing the response to the first prompt (points of attention noticed by people with normal color vision) and the response to the second prompt (points of attention noticed by people with color vision deficiency). Specifically, the controller 2 (comparison unit 27) takes the difference between the response to the first prompt (points of attention noticed by people with normal color vision) and the response to the second prompt (points of attention noticed by people with color vision deficiency) to extract points of attention that people with normal color vision notice but people with color vision deficiency do not, and identifies the extracted points of attention as objects to be alerted (objects that should be alerted more strongly). Once the processing in step S80 is completed, the process proceeds to step S90.

[0054] For example, if the response to the first prompt is "The type of object to pay attention to is a car. The location of the object to pay attention to in the image is in the center. The color of the object to pay attention to is red. The reason to pay attention to it is that the brake lights are on," and the response to the second prompt is "There is no type of object to pay attention to. There is no location of the object to pay attention to in the image. There is no color for the object to pay attention to. There is no reason to pay attention to it," (as in the first example), then the difference between the two is "The type of object to pay attention to is a car. The location of the object to pay attention to in the image is in the center. The color of the object to pay attention to is red. The reason to pay attention to it is that the brake lights are on." In this first example, the red car in the center of the image with its brake lights on is identified as the object to pay attention to.

[0055] For example, the answer to the first prompt might be: "The type of object to pay attention to is a car. The location of the first object to pay attention to on the image is the center. The color of the first object to pay attention to is red. The reason to pay attention to the first object to pay attention to is that the brake lights are on. The type of object to pay attention to is a pedestrian. The location of the second object to pay attention to on the image is the left edge. The color of the second object to pay attention to is white. The reason to pay attention to the second object to pay attention to is that the pedestrian is crossing a crosswalk." And the answer to the second prompt might be: "The type of object to pay attention to is a pedestrian. The location of the object to pay attention to on the image is the left edge. The color of the object to pay attention to is white. Note In the second example, if the reason for paying attention to the area is "The person is crossing a pedestrian crossing," the controller 2 (comparison unit 27) takes the difference between the two and identifies the content "The type of area to pay attention to is a pedestrian. The location of the area to pay attention to in the image is at the far left. The color of the area to pay attention to is white. The reason for paying attention to the area to pay attention to is that the person is crossing a pedestrian crossing." as the same answer, and identifies the content "The type of area to pay attention to is a car. The location of the area to pay attention to in the image is in the center. The color of the area to pay attention to is red. The reason for paying attention to the area is that the brake lights are on." as a different answer. In this second example, the red car in the center of the image with its brake lights on is identified as the object to be warned about.

[0056] In step S90, the controller 2 (notification unit 28) issues a notification to alert the user to the object requiring attention. Once the processing in step S90 is complete, the process proceeds to step S100.

[0057] In the first example above, when providing an audio notification, the reason for the attention-grabbing object, namely "brake lights on," should be read aloud, and when providing a display notification, the message "brake lights on" should be displayed. If the reason for the attention-grabbing object is "the brake lights on the red car in the center," the controller 2 (notification unit 28) should refer to the color information 34 and exclude color information that colorblind individuals cannot see, resulting in "brake lights on" or "brake lights on the car in the center." By excluding color information that colorblind individuals cannot see in the notification of the attention-grabbing object, it is possible to reduce the confusion and stress experienced by colorblind individuals due to the warning notification.

[0058] In the second example above, only different responses are notified. When providing an audio notification, the reason for paying attention to the object to be noticed, i.e., "brake lights on," should be read aloud. When providing a display notification, the message "brake lights on" should be displayed. If the reason for paying attention to the object to be noticed is "the brake lights on the red car in the center," then the controller 2 (notification unit 28) should refer to the color information 34 and exclude color information that colorblind individuals cannot see, resulting in "brake lights on" or "brake lights on the car in the center." By excluding color information that colorblind individuals cannot see in the notification of the object to be noticed, it is possible to reduce the confusion and stress experienced by colorblind individuals due to the warning notification.

[0059] In the second example above, instead of only notifying the user of different answers, both different and identical answers may be notified, and the different answers may be highlighted. Examples of highlighting different answers include highlighting them with size, color, etc. (making them more noticeable than the same answers), and highlighting them with increased volume or added warning sounds (making them more attention-grabbing than the same answers).

[0060] When notifying information about an object to be alerted, obtained in the form of a sentence that is the difference between the answer to the first prompt (the point of attention noticed by a person with normal color vision) and the answer to the second prompt (the point of attention noticed by a person with color vision deficiency), and linking it to an object (car) on the camera image or converted image, it is sufficient, for example, for the object (car) on the camera image or converted image to be identified by grounding using a VLM (Vision Language Model). The VLM (Vision Language Model) may be provided on the navigation device 1 side or on the server SV1 side.

[0061] In a display notification linked to an object (car) on a camera image or converted image that contains information about an object to be warned about, color information that a person with color vision deficiency cannot see may be converted to color information using colors that a person with color vision deficiency can see before notification. By converting color information that a person with color vision deficiency cannot see to color information using colors that a person with color vision deficiency can see, it is possible to reduce the confusion and stress experienced by people with color vision deficiency due to warning notifications. For example, the controller 2 (notification unit 28) can refer to the color information 34 and convert the color information "red car brake lights on" into a dot pattern using colors that a person with color vision deficiency can see, and then provide the output control unit 29 with display data for superimposing the dot pattern DP1 onto the camera image or converted image F1, as shown in Figure 5. When highlighting an object to be warned about based on the difference in responses, for example, image processing can be performed to increase the brightness and contrast of the relevant area.

[0062] In step S100, the controller 2 checks whether a certain acquisition cycle has elapsed since the last time a camera image, which is a captured image taken by the drive recorder DR1, was acquired. If the acquisition cycle has not elapsed, the controller returns to step S100; if the acquisition cycle has elapsed, the controller returns to step S10 to acquire the next camera image.

[0063] <Second Embodiment> In the second embodiment, the contents of LLM122, the first prompt, and the second prompt differ from those of the first embodiment described above. The differences between this embodiment and the first embodiment will be explained below, and the similarities with the first embodiment will not be explained.

[0064] In this embodiment, LLM122 is a model trained on training data that takes color vision characteristics into account. Specifically, LLM122 is a model trained on training data that includes a large number of combinations of images, attention points that a person with normal color vision would notice within the images (ground truth data for people with normal color vision), and data indicating that the data is for a person with normal color vision, and training data that includes a large number of combinations of images, attention points that a person with color vision deficiency would notice within the images (ground truth data for people with color vision deficiency), and data indicating that the data is for a person with color vision deficiency.

[0065] In this embodiment, the first prompt generated and output by the LLM122 for the camera image acquired in step S10, which indicates areas of interest that a person with normal color vision would notice, includes the camera image acquired in step S10 and information indicating that a person with normal color vision is viewing it.

[0066] An example of the first prompt generated in step S20 is a prompt that includes the camera image acquired in step S10 and the question: "You are a driver with normal color vision. You can see the scenery shown in the attached image. Please tell us what points to pay attention to for safe driving. Please answer in the following order: type of area to pay attention to, location of the area to pay attention to in the image, color of the area to pay attention to, and reason for paying attention to it."

[0067] In this embodiment, the second prompt generated and output by the LLM122 for the camera image acquired in step S10, which indicates the points of interest that a person with color blindness would notice, includes the camera image acquired in step S10 and information indicating what the person with color blindness is looking at. Unlike the first embodiment, this embodiment does not require a converted image.

[0068] An example of the second prompt generated in step S50 is a prompt that includes the camera image acquired in step S10 and the question: "You are a driver with type 1 dichromacy. You can see the scenery shown in the attached image. Please tell us what points to pay attention to for safe driving. Please answer in the following order: type of area to pay attention to, location of the area to pay attention to in the image, color of the area to pay attention to, and reason for paying attention to it."

[0069] <Third Embodiment> In the third embodiment, the number and content of the LLMs differ from those of the first embodiment described above. The differences between this embodiment and the first embodiment will be explained below, and the similarities to the first embodiment will not be explained.

[0070] Figure 6 shows an example configuration of server SV1 according to the third embodiment. The memory 120 of server SV1 shown in Figure 6 stores LLM122A and LLM122B. LLM122A is a model trained with training data for normal color vision characteristics. In other words, LLM122A is a model trained with training data that includes many combinations of images and attention points (ground truth data) that a person with normal color vision would notice within the image. LLM122B is a model trained with training data for abnormal color vision characteristics. In other words, LLM122A is a model trained with training data that includes many combinations of images and attention points (ground truth data) that a person with color vision deficiency would notice within the image.

[0071] When the LLM execution unit 131 receives a first prompt from the navigation device 1, it uses the first prompt as input data to execute LLM 122A and causes LLM 122A to generate a response. Once LLM 122A has generated a response, the controller 130 transmits the response to the first prompt generated by LLM 122A to the navigation device 1.

[0072] When the LLM execution unit 131 receives the second prompt sent from the navigation device 1, it uses the second prompt as input data to execute LLM 122B and causes LLM 122B to generate a response. Once LLM 122B has generated a response, the controller 130 transmits the response to the second prompt generated by LLM 122B to the navigation device 1. The first and second prompts are the same as in the first embodiment.

[0073] <Notes, etc.> The various technical features disclosed in the embodiments for carrying out the invention as specified herein can be modified in various ways without departing from the spirit of the technical creation. Furthermore, the multiple embodiments and modifications disclosed in the embodiments for carrying out the invention as specified herein may be combined to the extent possible.

[0074] For example, in the notification process described above, the processing order of steps S20-S40 and steps S50-S70 may be reversed.

[0075] For example, an artificial intelligence model other than LLM may be used instead of LLM. If an artificial intelligence model other than LLM is used, the first prompt and an image may be input to the artificial intelligence model instead of the first prompt. The image input to the artificial intelligence model may have tag information attached to it, such as information indicating that it is being viewed by a person with normal color vision or information indicating that it is being viewed by a person with color vision deficiency.

[0076] In the embodiments described above, the explanation assumed that the person with color vision deficiency was a person with type 1 dichromacy. However, other types of color vision deficiency, such as type 2 dichromacy and type 3 dichromacy, may also be treated as color vision deficiencies. The type of color vision deficiency can be selected, and the comparison unit may compare the points of attention noticed by the person with the selected type of color vision deficiency with the points of attention noticed by a person with normal color vision. [Explanation of symbols]

[0077] 1...Notification device DR1... Dashcam SV1... Server SYS1...Notification System

Claims

1. The first artificial intelligence model generates points of interest that a person with normal color vision would notice in a camera image. The second artificial intelligence model generates points of interest that a person with color vision deficiency would notice in the aforementioned camera image. By comparing the points of attention that a person with normal color vision would notice with the points of attention that a person with color vision deficiency would notice, the object to be highlighted can be identified. A notification method for issuing a notice to warn about the identified object that requires attention.

2. The notification method according to claim 1, wherein the notification is generated by excluding and / or converting color information that the person with color vision deficiency cannot see.

3. The first artificial intelligence model and the second artificial intelligence model are the same artificial intelligence model. The aforementioned identical artificial intelligence model is a model trained on training data that does not take color vision characteristics into account. The notification method according to claim 1 or 2, wherein the same artificial intelligence model generates points of interest that a person with normal color vision would notice when the camera image is input, and generates points of interest that a person with color vision deficiency would notice when a converted image is input, which is obtained by converting the camera image to simulate the color vision of a person with color vision deficiency.

4. The first artificial intelligence model and the second artificial intelligence model are the same artificial intelligence model. The aforementioned identical artificial intelligence model is a model trained on training data that takes color vision characteristics into account. The notification method according to claim 1 or 2, wherein the same artificial intelligence model generates points of interest that a person with normal color vision would notice when the camera image and information indicating that a person with normal color vision is looking at it are input, and generates points of interest that a person with color vision deficiency would notice when the camera image and information indicating that a person with color vision deficiency is looking at it are input.

5. The first artificial intelligence model is a model trained on training data of normal color vision characteristics, The aforementioned second artificial intelligence model is a model trained on training data of abnormal color vision characteristics. The first artificial intelligence model generates attention points that a person with normal color vision would notice when the camera image is input, The notification method according to claim 1 or 2, wherein the second artificial intelligence model generates attention points that the colorblind person will notice when the camera image is input.

6. The notification method according to claim 1 or claim 2, wherein the first artificial intelligence model and the second artificial intelligence model are multimodal LLMs capable of image recognition.

7. The first artificial intelligence model retrieves the points of interest that a person with normal color vision would notice in a camera image. Regarding the aforementioned camera image, the second artificial intelligence model was used to obtain the points of interest that a person with color vision deficiency would notice. By comparing the points of attention noticed by the aforementioned person with normal color vision with the points of attention noticed by the aforementioned person with color vision deficiency, the object to be highlighted can be identified. A notification device that issues a notification to alert the identified object to the warning.

8. To obtain the points of interest that a person with normal color vision would notice in a camera image from the first artificial intelligence model, The second artificial intelligence model will acquire the points of interest that a person with color vision deficiency would notice in the aforementioned camera image, By comparing the points of attention noticed by the aforementioned person with normal color vision with the points of attention noticed by the aforementioned person with color vision deficiency, the object to be highlighted can be identified. To issue a notice to warn about the identified objects that require attention, A program that causes a computer to execute something.

9. A notification system comprising a notification device and a server including a first artificial intelligence model and a second artificial intelligence model, The notification device is The first artificial intelligence model retrieves the points of interest that a person with normal color vision would notice in a camera image. The second artificial intelligence model obtains the points of interest that a person with color vision deficiency would notice in the aforementioned camera image. By comparing the points of attention noticed by the aforementioned person with normal color vision with the points of attention noticed by the aforementioned person with color vision deficiency, the object to be highlighted can be identified. A notification system that issues a notification to alert the identified object of concern.

Images