Operation assistance method, operation assistance device, program, and computer-readable recording medium

Abstract

When a plurality of warning lights of a vehicle are lit, appearance information pertaining to the appearance of the warning lights and wording uttered by a user to request to describe the warning lights are included in utterance content of the user of the vehicle, and when it is determined that there are a plurality of warning lights matching the appearance information, the present invention sequentially outputs a plurality of instances of speech pertaining to warning content of the plurality of warning lights that match the appearance information, and in at least a portion of the period in which the instances of speech are output, executes directing processing for emphasizing a warning light corresponding to an instance of speech.

Description

Operation support method, operation support device, program, and computer-readable recording medium

[0001] The present invention relates to an operation assistance method, an operation assistance device, a program for operation assistance, and a computer-readable recording medium on which the program is recorded.

[0002] A vehicle status display system is known that, when a vehicle status display provided on a panel in front of the driver is illuminated, asks the driver whether or not to output explanatory information related to the vehicle status display, and if the driver affirms that explanatory information should be output, outputs explanatory information related to the illuminated vehicle status display (Patent Document 1). The vehicle status displays in Patent Document 1 include warning lights such as a brake warning light, an engine warning light, and a low fuel warning light.

[0003] Japanese Patent Application Laid-Open No. 2021-194996

[0004] In the above-described conventional technology, when a plurality of vehicle status indicators are illuminated and the driver responds affirmatively to an inquiry, explanatory information is output for the plurality of vehicle status indicators. In the case where the vehicle status indicators are warning lights of the vehicle and the explanatory information is audio information explaining the contents of the warning lights of the vehicle, in the above-described conventional technology, audio explaining the contents of the warning lights is output sequentially in response to the driver's affirmative response to the inquiry.

[0005] However, since the vehicle user cannot always accurately identify the warning light being explained from the content of the audio, the above-mentioned conventional technology has the problem that when audio explaining the contents of multiple warning lights is output sequentially, it is difficult for the vehicle user to understand which warning light the output audio corresponds to.

[0006] The problem that the present invention aims to solve is to provide an operation assistance method, an operation assistance device, a program, and a computer-readable recording medium that allow a vehicle user to easily understand which warning light the output voice corresponds to.

[0007] The present invention solves the above problem by, when multiple warning lights on a vehicle are illuminated, the vehicle user's speech includes appearance information regarding the appearance of the warning lights and words by which the user requests an explanation of the warning lights, and when it is determined that there are multiple warning lights that match the appearance information, sequentially outputting audio regarding the warning content of the multiple warning lights that match the appearance information, and executing a performance process that emphasizes the warning light corresponding to the audio for at least a portion of the period during which the audio is output.

[0008] According to the present invention, it is easy for a vehicle user to understand which warning light the output sound corresponds to.

[0009] FIG. 1 is a block diagram showing an example of an embodiment of an operation assistance system according to the present invention. FIG. 2 is a diagram showing an example of a warning light displayed on an instrument panel of a vehicle. FIG. 3 is a diagram showing an example of performance processing by the operation assistance device of FIG. 1. FIG. 4 is a diagram showing another example of performance processing by the operation assistance device of FIG. 1. FIG. 5 is a block diagram showing another example of an embodiment of an operation assistance system according to the present invention. FIG. 6 is a flowchart showing an example of a processing procedure in the operation assistance system of FIG. 1. FIG. 7 is a flowchart showing an example of a processing procedure related to performance processing in the operation assistance system of FIG. 1. FIG. 8 is a flowchart showing another example of a processing procedure related to performance processing in the operation assistance system of FIG. 1. FIG. 9 is a flowchart showing yet another example of the processing procedure related to performance processing in the operation assistance system of FIG. 1.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [Configuration of Operation Assistance System] The operation assistance system assists the user's operation using a voice assistant that recognizes the voice of a vehicle user (hereinafter simply referred to as the user), interacts with the user, and responds appropriately to the user's questions and requests or performs other actions. The user includes a vehicle occupant, and the user's operation includes all operations on the devices that make up the vehicle (such as the steering wheel and seat belt) and devices associated with the vehicle (such as an information terminal connected to the vehicle so as to be able to communicate with the vehicle). The user is not particularly limited as long as the user can use the vehicle appropriately, and the vehicle is not particularly limited as long as the vehicle can be equipped with the operation assistance system 10.

[0012] As shown in Fig. 1, the operation assistance system 10 includes an on-board sensor 11, a display device 12, a microphone 13, a speaker 14, and an operation assistance device 20. These devices are connected via a CAN (Controller Area Network) or other on-board LAN, and can exchange information with each other. The operation assistance device 20 is also connected to an external server via the network, and exchanges information with the external server as needed. The network refers to a telecommunications network such as the Internet, and the communication format is not particularly limited.

[0013] The on-board sensors 11 detect the status of the components of the vehicle, specifically, whether the components of the vehicle are operating (functioning) properly. The on-board sensors 11 include a hydraulic sensor that detects the hydraulic pressure of the vehicle's hydraulic system, a current sensor and a voltage sensor that detect the status of the battery, a sensor that detects the operating status of actuators such as motors, and a door sensor that detects the open / closed status of the vehicle doors. The on-board sensors 11 also include sensors that detect the running status of the vehicle, such as a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. These sensors are appropriately arranged within a range that allows them to appropriately detect the status of the detection target.

[0014] The display device 12 provides information to the vehicle user. Examples of the display device 12 include a display on an instrument panel provided in front of the driver's seat of the vehicle, a liquid crystal display provided on the dashboard of the vehicle, a projector such as a head-up display, etc. The display device 12 may also include an input device (e.g., a touch panel) for the occupant to input instructions to the operation assistance device 20.

[0015] The microphone 13 acquires sounds within the vehicle cabin as audio data. Examples of the microphone 13 include, but are not limited to, a small stand microphone, a directional close-talking microphone, a lavalier microphone that can be attached to the roof or seat of the vehicle, and the like. The sounds within the vehicle cabin include the voice of the user (utterance), noises such as road noise, and the output sound of the speaker 14. The microphones 13 can be installed in appropriate positions within a range that allows appropriate detection of sounds within the vehicle cabin, and the number of microphones 13 is determined appropriately depending on the detection range of the microphones 13 and the size of the vehicle cabin.

[0016] The speaker 14 converts electrical signal energy into acoustic energy and emits it into space. The speaker 14 may be, but is not limited to, an integrated or separate vehicle speaker. The speaker 14 may be installed in an appropriate position within a range where information can be transmitted to the user by voice, and the number of speakers 14 is determined appropriately depending on the settable range of the output volume of the speaker 14 and the size of the vehicle interior. The microphone 13 and speaker 14 may also be integrated like a headset.

[0017] The operation assistance device 20 controls the devices constituting the operation assistance system 10 to cooperate with each other, and performs user operation assistance (hereinafter also referred to simply as operation assistance). The operation assistance device 20 is, for example, a computer, and includes a CPU (Central Processing Unit) which is a processor, a ROM (Read Only Memory) in which programs are stored, and a RAM (Random Access Memory) which functions as an accessible storage device. The CPU is an operating circuit for executing the programs stored in the ROM and realizing the functions of the operation assistance device 20.

[0018] In the following description, the program executed by the operation assistance device 20 will be described as a program stored in ROM, but the program may be stored outside the operation assistance device 20. For example, the program executed by the operation assistance device 20 may be stored on a computer-readable recording medium. In this case, by connecting the recording medium to the operation assistance device 20 using a predetermined standard, the operation assistance device 20 can execute the program recorded on the recording medium. Examples of such recording media include disk media and flash memory.

[0019] [Functions of the Operation Assistance Device] A program for realizing operation assistance is stored in the ROM of the operation assistance device 20, and operation assistance is performed by the CPU of the operation assistance device 20 executing the program. For convenience, Fig. 1 shows a warning unit 21, a determination unit 22, an output unit 23, and a performance unit 24 as functional blocks for performing operation assistance.

[0020] The warning unit 21 acquires the detection results of the on-board sensors 11 and monitors whether any abnormality has occurred in the equipment constituting the vehicle based on the detection results. If the warning unit 21 determines that no abnormality has occurred in the equipment constituting the vehicle, it continues monitoring the status of the equipment. On the other hand, if the warning unit 21 determines that an abnormality has occurred in the equipment constituting the vehicle, it displays a warning light on the display device 12 and notifies the user of the vehicle abnormality (e.g., vehicle malfunction).

[0021] For example, when the detection result of the seat belt sensor determines that the seat belt is not fastened, and the vehicle speed obtained from the vehicle speed sensor exceeds a predetermined speed (e.g., 10 to 15 km / h), the warning unit 21 turns on the seat belt warning light on the instrument panel and outputs a warning sound (e.g., a buzzer sound) from the speaker 14.

[0022] The warning unit 21 displays a warning light when a vehicle abnormality is detected (for example, when a device constituting the vehicle is not operating normally), and does not display a warning light when no vehicle abnormality is detected. The warning light is different from an indicator light that is displayed in response to a user operation (for example, a turn signal indicator light that is displayed in response to operation of a turn signal lever) or an indicator light that is displayed when the vehicle (devices constituting the vehicle) is operating normally.

[0023] When the warning unit 21 turns on the warning light, the determination unit 22 executes a determination process to generate an explanatory text (hereinafter also simply referred to as an explanatory text) regarding the warning content of the warning light using the voice assistant function. The warning content includes, for example, details of an abnormality that has occurred in the vehicle (or an apparatus that constitutes the vehicle), how to deal with the abnormality, etc. The output unit 23 generates an explanatory text according to the determination result of the determination unit 22, reads the generated explanatory text aloud using the voice assistant's reading function, and outputs it from the speaker 14.

[0024] For example, the determination unit 22 acquires voice data from the microphone 13 and performs frequency analysis on the voice data to extract the content of the user's speech as voice data. When a voice input operation to the voice assistant is to be accepted when the user operates a predetermined switch (or only while the user is operating the predetermined switch), the determination unit 22 acquires voice data from the microphone 13 when the user operates the predetermined switch. The predetermined switch is, for example, a button provided on the steering wheel.

[0025] The determination unit 22 performs speech recognition processing such as STT (Speech-to-Text) on the extracted voice data to acquire the user's utterance (hereinafter simply referred to as the utterance). The utterance includes at least one of the user's intent and a named entity. The user's intent is what the user is thinking, such as checking how to operate a device, inputting device operations, and acquiring information such as a destination. The named entity includes proper nouns such as people's names, organization names, place names, and product names, as well as dates, time expressions, quantities, shapes, sizes, proportions, and colors. The determination unit 22 performs named entity extraction processing on the acquired utterance to extract a predetermined named entity from the utterance. For example, if the user utters, "What is that red warning light displayed on the instrument panel?", the determination unit 22 identifies the user's intent to check the warning content of the red warning light and extracts "red warning light" as a named entity.

[0026] The determination unit 22 may also identify the illuminated warning lights from the extracted named entities. For example, the determination unit 22 may set a characteristic expression indicating the characteristics of each warning light in advance, compare the extracted named entities with the characteristic expression, and identify the warning lights corresponding to the characteristic expression that matches the named entity (or is similar to the named entity). The determination unit 22 detects the state of the identified warning lights from the CAN signal. That is, the determination unit 22 determines whether the warning lights are illuminated and detects the state of the warning lights.

[0027] The number of warning lights identified by the determination unit 22 does not necessarily have to be one, but may be multiple. A named entity being similar to a characteristic entity means that the named entity matches a similar entity that is similar to the characteristic entity. The similar entity is set in advance together with the characteristic entity. The similar entity can be set appropriately within a range that allows the warning light identified by the named entity to be appropriately identified. Turning on the warning light includes displaying the warning light on the display device 12. If there is no warning light that corresponds to an expression that matches the extracted named entity (or is similar to the named entity), the determination unit 22 may end the processing or may acquire audio data from the microphone 13 again.

[0028] The determination unit 22 determines whether the named entity includes the name of a warning light. If it is determined that the named entity includes the name of a warning light, the output unit 23 generates an explanatory sentence. For example, when the battery warning light indicating an abnormality in the battery is on and the user utters, "The battery warning light is on. What should I do?", the output unit 23 generates an explanatory sentence such as, "The battery may be damaged, so please stop the vehicle immediately in a safe place." This explanatory sentence is generated by combining, for example, warning content and a countermeasure for the warning that are registered in advance for the battery warning light.

[0029] On the other hand, if it is determined that the named entity does not include the name of a warning light, the determination unit 22 determines whether or not the named entity includes an expression indicating the color of the warning light. If it is determined that the named entity includes an expression indicating the color of the warning light, the output unit 23 generates an explanation for one or more warning lights corresponding to the color indicated in the utterance content.

[0030] For example, suppose that a battery warning light and a brake warning light indicating that an abnormality has occurred in the function of the vehicle's brakes are illuminated, the battery warning light has a red rectangular appearance, and the brake warning light has a red circular appearance. In this case, when the user utters, "What does the red warning light indicate?", the output unit 23 generates an explanatory sentence such as, "The rectangular warning light indicates that an abnormality has occurred in the vehicle's battery. On the other hand, the circular warning light indicates that an abnormality has occurred in the function of the vehicle's brakes."

[0031] If it is determined that the named entity does not include the name of the warning light or an expression indicating the color of the warning light, the output unit 23 does not generate a description. Furthermore, in order to reduce the number of descriptions to be generated, the output unit 23 determines whether the number of warning lights corresponding to the color indicated by the named entity is equal to or greater than a predetermined number, and if it is determined that the number of warning lights is equal to or greater than the predetermined number, it is not necessary to generate a description. The predetermined number can be set appropriately depending on the processing capacity of the operation assistance device 20, and is, for example, 3 to 5. Note that if it is determined that the number of warning lights corresponding to the color indicated by the named entity is less than the predetermined number, the output unit 23 generates a description.

[0032] When generating an explanatory text, the output unit 23 may select an explanatory text registered in advance for each warning light, or may generate the explanatory text using a known generation AI. The output unit 23 may output the generated explanatory text to the speaker 14 as read-out audio data, or may output the generated explanatory text to the speaker 14 as text information and have a control device (not shown) of the speaker 14 read it out.

[0033] The above-described frequency analysis process, speech recognition process, speech content acquisition process, named entity extraction process, and description generation process using the generation AI may be executed by an external server connected via a network to the operation assistance device 20. That is, the operation assistance device 20 may transmit voice data and the like to the external server, execute each process on the server, and receive the processed data from the server.

[0034] When multiple warning lights are illuminated simultaneously and there are multiple warning lights corresponding to an expression that matches (or is similar to) the extracted named entity, the output unit 23 generates an explanation for each warning light, generates audio information using a reading function, and outputs the audio to the speaker 14. The output unit 23 causes the speaker 14 to sequentially output the audio. The order in which the audio (explanations) are output is not particularly limited, and for example, the output unit 23 may cause the speaker 14 to output the audio in order from shortest to oldest, or may output the audio in random order.

[0035] The output unit 23 may also output a sound (explanatory text) according to a grace period for the user to respond to the warning content. For example, the output unit 23 outputs a sound (explanatory text) explaining a warning content that requires the user to respond immediately (a short grace period) before a sound (explanatory text) explaining a warning content that does not require the user to respond immediately (a long grace period). The grace period is set in advance for each warning light, and for example, a relatively short grace period is set for warning content related to braking and steering of the vehicle.

[0036] When sounds are output sequentially from the speaker 14, the user may not immediately understand which warning light the output sound corresponds to. Therefore, in the operation assistance device 20 of this embodiment, the function of the performance unit 24 instructs the display device 12 to execute performance processing for the warning light corresponding to the sound being output during at least a portion of the period during which sounds related to the warning contents of the multiple warning lights are output sequentially. The execution of the performance processing will be described below.

[0037] The determination unit 22 determines, for example, from a CAN signal, whether multiple warning lights are illuminated. If it is determined that only one warning light is illuminated, the performance unit 24 does not output an instruction to execute performance processing to the display device 12 (i.e., does not cause the display device 12 to execute performance processing). On the other hand, if it is determined that multiple warning lights are illuminated, the determination unit 22 determines whether the utterance content includes appearance information regarding the appearance of the warning light and a statement by which the user requests an explanation of the warning light (hereinafter also referred to as a request statement).

[0038] The appearance information is an expression that represents the appearance of the warning light from among the named entities extracted by the determination unit 22, and includes an expression that represents the shape of the warning light, an expression that represents the color (hue, brightness, and saturation) of the warning light, an expression that represents the characters displayed on the warning light, etc. Expressions that represent the shape include "round," "rectangular," and "a picture of an engine," expressions that represent the color include "red," "yellow," and "orange," and expressions that represent the characters include "plus (+)," "minus (-)," and "ABS."

[0039] The request phrase includes not only expressions in which the user directly requests an explanation of the warning content, such as "Please tell me the warning content of the warning light," but also expressions in which the user is struggling to understand the warning content and is indirectly requesting an explanation of the warning content, such as "I don't know which warning light it is," "I don't understand the warning content," and "I don't know what to do." The determination unit 22 performs natural language processing such as morphological analysis on the acquired utterance content to extract words. The determination unit 22 matches the extracted words (e.g., predicates and objects) with the request phrase and searches for the request phrase in the utterance content.

[0040] Furthermore, the determination unit 22 may determine whether the utterance content includes a request wording based on the user's intention acquired from the utterance content. For example, if the user's intention is to confirm how to operate a device, the determination unit 22 determines that the utterance content includes a request wording. On the other hand, if the user's intention is to acquire information about a destination, the determination unit 22 determines that the utterance content does not include a request wording.

[0041] If it is determined that the utterance does not include at least one of the appearance information and the request wording, the production unit 24 does not output an instruction to execute the production process to the display device 12. On the other hand, if it is determined that the utterance includes the appearance information and the request wording, the determination unit 22 determines whether there are multiple warning lights that match the appearance information. For example, the determination unit 22 identifies warning lights that have characteristics that match the expression that describes the appearance of the warning light, and records the number of identified warning lights.

[0042] If multiple expressions describing the appearance of a warning light are set, the determination unit 22 may identify warning lights that have a degree of match with the appearance information that is equal to or greater than a predetermined value, and record the number of identified warning lights. The degree of match may be calculated, for example, by setting expressions that indicate characteristics for each warning light in advance, and calculating the percentage of the pre-set expressions that match the expressions (words) extracted from the appearance information. The predetermined value may be set as appropriate within a range that allows accurate identification of the warning light corresponding to the appearance information.

[0043] If it is determined from the number of recorded warning lights that there is only one warning light that matches the appearance information, the performance unit 24 does not output an instruction to execute performance processing to the display device 12. On the other hand, if it is determined from the number of recorded warning lights that there is multiple warning lights that match the appearance information, the performance unit 24 instructs the display device 12 to execute performance processing for the warning light corresponding to the sound being output. The instruction to execute performance processing by the performance unit 24 (hereinafter also referred to as a performance instruction) is output during at least a part of the period during which sound is output from the speaker 14 for the multiple warning lights that match the appearance information.

[0044] The order of determining whether the speech content includes the appearance information and the request wording (hereinafter also referred to as the first determination) and determining whether there are multiple warning lights that match the appearance information (hereinafter also referred to as the second determination) is not particularly limited. That is, the first determination and the second determination may be performed simultaneously (in parallel), the second determination may be performed after the first determination, or the first determination may be performed after the second determination.

[0045] The effect processing is a process in which a warning light that is the subject of the effect processing is displayed more emphatically (more conspicuously) than warning lights that are not the subject of the effect processing. The effect processing includes at least one of a process of flashing the warning light, a process of increasing the brightness of the warning light, a process of periodically changing the color of the warning light, a process of highlighting the edges of the warning light, and a process of enlarging the size of the warning light. In addition, if the warning light that is the subject of the effect processing is displayed on an instrument panel located in front of the driver's seat of the vehicle, the effect processing also includes a process of displaying the warning light that is the subject of the effect processing on a display device 12 other than the instrument panel (e.g., a liquid crystal display provided on the dashboard). Note that while the warning light that is the subject of the effect processing is displayed on a display device 12 other than the instrument panel, the warning light that is the subject of the effect processing may or may not be displayed on the instrument panel.

[0046] While the effect processing is being executed, the effect unit 24 may maintain the state of warning lights that are not the target of the effect processing. There are three states of warning lights: a lighting state in which the warning light is lit, a lighting state in which effect processing is being executed on the warning light, and an off state in which the warning light is not lit. Furthermore, the state of the warning light may be the state of the display part of the warning light, and maintaining the state of the warning light may mean maintaining the state (size, brightness, saturation, etc.) of the display part of the warning light.

[0047] 2A is a diagram showing an example of warning lights displayed on a vehicle instrument panel. The warning light shown in the upper part of Fig. 2A is a battery warning light 31 that is displayed when an abnormality occurs in the battery, and the warning light shown in the lower part is a brake warning light 32 that is displayed when an abnormality occurs in the brakes. For example, as shown in Fig. 2B, when processing is executed to enlarge the size of the battery warning light 31, the effect unit 24 maintains the size of the display portion of the brake warning light 32 that is not the target of the effect processing during the execution of the effect processing, and does not execute effect processing for the brake warning light 32.

[0048] When executing the performance processing, the performance unit 24 may display an image of a vehicle part or function corresponding to the warning content. Alternatively or in addition, when executing the performance processing, the performance unit 24 may display the name of the vehicle part or function corresponding to the warning content. For example, when executing processing to highlight the edge of the brake warning light 32 shown in Figure 2A as shown in Figure 2C, the word "brake" is displayed in the upper left corner of the brake warning light 32.

[0049] When it is determined that the speech content includes appearance information and a request phrase, the production unit 24 may extract, from a plurality of lit warning lights, candidates for warning lights that may match the appearance information. For example, when a plurality of characteristic expressions that represent the appearance of warning lights are set, the production unit 24 extracts, as candidates, warning lights that match at least one of the predetermined characteristic expressions extracted from the appearance information.

[0050] Furthermore, the production unit 24 may calculate the probability that the extracted candidates match the appearance information. The probability is calculated, for example, as the percentage of characteristic expressions preset for each warning light that match the expression extracted from the appearance information. When the probability is calculated, the production unit 24 instructs the display device 12 to perform a production process that emphasizes warning lights with a high probability more than warning lights with a low probability.

[0051] If the performance unit 24 determines that the utterance content includes the appearance information and the request wording, it may extract warning lights that match the appearance information (or have a degree of match equal to or greater than a predetermined value) from the multiple warning lights that are currently turned on, and calculate a priority for notifying the user of the warning content for the extracted warning lights. The priority is preset for each warning light according to the grace period until the warning content is notified to the user. For example, the priority of warning lights related to the vehicle's steering function and braking function is set higher than the priority of other warning lights (e.g., warning lights related to the shift position or the remaining fuel level). The performance unit 24 instructs the display device 12 to perform a performance process for warning lights with high priority that emphasizes the warning lights more than warning lights with low priority.

[0052] If the judgment unit 22 determines that the speech content includes appearance information and a request wording and that there are multiple warning lights that match the appearance information, the output unit 23 may calculate a priority for notifying the user of the warning content of the warning lights for the multiple warning lights that match the appearance information, and output warning content with a higher priority before warning content with a lower priority.

[0053] The determination unit 22 determines whether multiple warning lights are on, for example, from a CAN signal. If it is determined that only one warning light is on, the output unit 23 does not calculate the priority. On the other hand, if it is determined that multiple warning lights are on, the determination unit 22 determines whether the utterance content includes appearance information and a request statement.

[0054] If it is determined that the utterance does not include at least one of the appearance information and the request wording, the output unit 23 does not calculate the priority. On the other hand, if it is determined that the utterance includes the appearance information and the request wording, the determination unit 22 determines whether there are multiple warning lights that match the appearance information. For example, the determination unit 22 identifies warning lights that have characteristics that match the expression that describes the appearance of the warning light, and records the number of identified warning lights.

[0055] If it is determined from the number of recorded warning lights that there is only one warning light that matches the appearance information, the output unit 23 does not calculate a priority. On the other hand, if it is determined from the number of recorded warning lights that there are multiple warning lights that match the appearance information, the output unit 23 calculates priorities for notifying the user of the warning content of the multiple warning lights that match the appearance information.

[0056] The warning lights that match the appearance information include warning lights that match the appearance information as well as warning lights that have a degree of match with the appearance information that is equal to or greater than a predetermined value. The priority of notifying the user of warning content is an index that determines the order in which multiple warning content is notified to the user; for example, warning content with a higher priority is notified to the user first, and warning content with a lower priority is notified to the user thereafter.

[0057] The priority may be preset for each warning light according to the grace period until the user is notified of the warning content. For example, the priority of warning lights related to the vehicle's steering function and braking function may be set higher than the priority of other warning lights (e.g., warning lights related to the shift position or the remaining fuel level). The output unit 23 may also acquire information regarding the type of the illuminated warning light and calculate the priority according to the type. Examples of the types include warning lights related to user operations, warning lights related to vehicle parts and / or functions, and warning lights related to the environment around the vehicle. For example, the priority of warning lights related to vehicle parts and / or functions may be calculated to be higher than the priority of warning lights related to user operations.

[0058] For warning lights whose priorities have been calculated, the output unit 23 outputs information related to warning content with a higher priority in preference to information related to warning content with a lower priority. Specifically, the output unit 23 outputs first information related to warning content of a warning light with a first priority in preference to second information related to warning content of a warning light with a second priority lower than the first priority. For example, in a case where a brake warning light indicating that an abnormality has occurred in the function of the vehicle's brakes and a fuel level warning light indicating that the remaining fuel is low are both illuminated, if a higher priority is set for the brake warning light, the output unit 23 outputs information related to the abnormality in the function of the brakes (first information) before information related to the remaining fuel (second information).

[0059] The information (first information and second information) about the warning content of the warning light is not particularly limited as long as it is information about the warning content, and examples thereof include audio information in which the warning content (or explanatory text) is read out using a read-out function, image information showing the warning content (or explanatory text) in text, pictogram information that allows the warning content to be intuitively understood, image information showing parts and / or functions related to the warning content, etc. When the first information and the second information are visual information such as images, part of the period in which the first information is output and part of the period in which the second information is output may overlap.

[0060] The information regarding the warning content of the warning light may include status information and response information. The status information is information regarding at least one of the component and / or function corresponding to the warning light and the conditions under which the warning light is turned on (hereinafter also referred to as the lighting conditions). For example, when the battery warning light is turned on, the component corresponding to the warning light is the battery, and when the brake warning light is turned on, the function corresponding to the warning light is the braking function. The conditions under which the warning light is turned on are preset for each warning light so that the user can be appropriately notified of vehicle abnormalities.

[0061] The countermeasure information is information relating to at least one of the reason why the warning light is on and the countermeasure for the warning content. The countermeasure information relating to the reason why the warning light is on is generated, for example, based on the lighting conditions. The countermeasure for the warning content may be registered in advance for each warning light (or warning content), or may be generated based on the lighting conditions.

[0062] When there are multiple candidates for the lighting condition, the output unit 23 may compare the detected vehicle state with the lighting condition and identify which lighting condition is satisfied. Alternatively or in addition, the output unit 23 may compare the detected vehicle state with the lighting condition, identify which lighting condition is satisfied, and generate a countermeasure for eliminating the identified lighting condition (so that the lighting condition is not satisfied).

[0063] For example, if a door warning light indicating that the vehicle departed with one of its doors not completely closed is on, the output unit 23 identifies which door is not completely closed based on the detection result of the vehicle's door sensor.The output unit 23 then notifies the user that the reason the door warning light is on is that the identified door is not completely closed, and generates action information urging the user to close the identified door.

[0064] When the output unit 23 outputs the second information after the first information, the content of the second information may be simplified compared to when only the second information is output. Specifically, when the output unit 23 outputs the second information after the first information, the output unit 23 outputs the second information including status information but not action information. For example, when the brake warning light and the fuel level warning light are on, if the output unit 23 is to output information regarding the remaining fuel level (second information) after completing the output of information regarding a malfunction in the brake function (first information), the output unit 23 outputs only information indicating that the remaining fuel level is low (status information) to the display device 12 and / or the speaker 14 as the second information, but does not output the action (action information) to refuel.

[0065] The output unit 23 may output first information including status information and handling information regardless of the utterance content. On the other hand, the output unit 23 may output second information that differs depending on the utterance content. Specifically, the output unit 23 may determine whether the utterance content includes at least one of a statement requesting status information and a statement requesting handling information, and may output second information including the status information if it is determined that the utterance content includes a statement requesting status information. If it is determined that the utterance content includes a statement requesting handling information, the output unit 23 may output second information including handling information. If it is determined that the utterance content includes a statement requesting status information and a statement requesting handling information, the output unit 23 may output second information including the status information and handling information.

[0066] Words requesting status information include expressions in which the user directly requests status information, such as "What function corresponds to the warning light?", as well as expressions in which the user indirectly requests status information, such as "I don't understand the warning content." Words requesting response information include expressions in which the user directly requests response information, such as "Please tell me why the warning light is on," as well as expressions in which the user indirectly requests response information, such as "I don't know how to deal with it." Determining whether the utterance content contains these words is performed in the same way as determining whether the utterance content contains request words.

[0067] For example, when the brake warning light and the fuel level warning light are on, if it is determined that the speech content includes wording requesting status information regarding the fuel level warning light and wording requesting action information regarding the fuel level warning light, the output unit 23 outputs to the display device 12 and / or speaker 14 information indicating that the remaining fuel level is low (status information) and the action to take (action information) of refueling.

[0068] The output unit 23 may calculate a lower priority for a warning light that is lit for a longer period than for a warning light that is lit for a shorter period. If a warning light is lit for a longer period, this indicates that the vehicle can be used without immediately responding to the abnormality notified by the warning light, so the output unit 23 calculates a relatively lower priority for the warning light.

[0069] When it is determined that the utterance content includes appearance information and a request phrase, the output unit 23 may extract, from a plurality of illuminated warning lights, candidates for warning lights that may match the appearance information. For example, when a plurality of characteristic expressions that represent the appearance of warning lights are set, the output unit 23 extracts, as candidates, warning lights that match at least one of the predetermined characteristic expressions extracted from the appearance information.

[0070] The output unit 23 may also calculate the degree of certainty that the extracted candidates match the appearance information. The degree of certainty is calculated, for example, as the proportion of characteristic expressions preset for each warning light that match the expression extracted from the appearance information. When the degree of certainty is calculated, the output unit 23 calculates a higher priority for warning lights with a higher degree of certainty than for warning lights with a lower degree of certainty.

[0071] The output unit 23 may calculate the priority of a warning light whose warning content has a large impact on the running of the vehicle to be higher than the priority of a warning light whose warning content has a small impact on the running of the vehicle. For example, since the steering function and braking function have a relatively large impact on the running of the vehicle and the remaining fuel amount has a relatively small impact on the running of the vehicle, the output unit 23 calculates the priority of the brake warning light to be higher than the priority of the remaining fuel warning light.

[0072] So far, we have explained the rendering process. When notifying the user of the content of a warning, if the user is notified of countermeasure information that corresponds to the actual state of the vehicle, the user can easily understand how to resolve the warning and can quickly respond to the warning. Therefore, in the operation assistance device 20 of this embodiment, if the determination unit 22 determines that the utterance content includes a statement requesting countermeasure information, the output unit 23 outputs countermeasure information based on a control signal that controls the state of the vehicle and the type of the illuminated warning light. The output of the countermeasure information will be described below.

[0073] When the warning unit 21 turns on the warning light of the vehicle, the determination unit 22 determines whether the content of the utterance by the vehicle user includes a statement requesting response information regarding at least one of the reason why the warning light is on and how to deal with the warning content of the warning light. For example, the determination unit 22 acquires a control signal of the CAN bus to which the warning light is connected and determines whether the warning light is on. The determination unit 22 also determines whether the content of the user utterance includes an expression indicating that the user is directly or indirectly requesting response information. The determination of whether the content of the utterance includes a statement requesting response information is performed in the same manner as the determination of whether the content of the utterance includes a request statement.

[0074] If it is determined that the speech content does not include a statement requesting response information, the output unit 23 does not output the response information. On the other hand, if it is determined that the speech content includes a statement requesting response information, the output unit 23 outputs response information based on a control signal that controls the state of the vehicle and the type of the warning light that is turned on.

[0075] The control signal for controlling the state of the vehicle is a control signal input to on-board devices (such as a drive unit, a steering unit, a safety device, and a navigation system). The output unit 23 acquires the control signal from, for example, a CAN bus to which the on-board devices are connected, and detects the state of the vehicle. The type of the lit warning light is a classification according to the warning content of the warning light, and examples of the types include a warning light related to user operation, a warning light related to vehicle parts and / or functions, and a warning light related to the environment around the vehicle.

[0076] The corrective action information based on the control signal and the type of warning light is corrective action information that is in line with the actual (or current) state of the vehicle and is information that prompts the user to change the state of the vehicle so that the warning content is resolved (so that the lighting condition is no longer met). If the type of warning light is a warning light related to user operation, for example, corrective action information is output that prompts the user to perform an operation to resolve the warning content. If the type of warning light is a warning light related to a vehicle part and / or function, for example, corrective action information is output that prompts the user to replace (maintain) a part to resolve the warning content. If the type of warning light is a warning light related to the environment around the vehicle, for example, corrective action information is output that prompts the user to change the environment around the vehicle to resolve the warning content.

[0077] As an example, when the seat belt warning light is on, indicating that the vehicle has departed without the user fastening their seat belt, and it is detected that the user sitting in the passenger seat is not fastening their seat belt, the output unit 23 generates an explanatory message urging the user sitting in the passenger seat to fasten their seat belt, and uses the reading function to generate audio information from the explanatory message and output it to the speaker 14.

[0078] The output unit 23 acquires a control signal input to the display device 12 from a CAN bus to which the display device 12, on which the seat belt warning light is displayed, and detects from the acquired control signal that the seat belt warning light is on. The output unit 23 also acquires a signal from a reed switch sensor (on-board sensor 11) provided on the tongue of the seat belt, and identifies a user who is not wearing a seat belt.

[0079] When multiple warning lights are turned on by the warning unit 21, the determination unit 22 determines whether the utterance content includes appearance information regarding the appearance of the warning lights and a statement requesting handling information, and may also determine whether there are multiple warning lights that match the appearance information. Specifically, the determination unit 22 determines whether multiple warning lights are turned on based on a CAN signal or the like. If it is determined that only one warning light is turned on, the output unit 23 does not calculate the priority. On the other hand, if it is determined that multiple warning lights are turned on, the determination unit 22 determines whether the utterance content includes appearance information and a statement requesting handling information.

[0080] Regarding the appearance information, the determination unit 22 determines, for example, whether the utterance content includes an expression describing the appearance of the warning light. Similarly to determining whether the utterance content includes a request statement, the determination unit 22 also determines whether the utterance content includes a statement requesting response information. If it is determined that the utterance content does not include at least one of the appearance information and the statement requesting response information, the output unit 23 does not calculate the priority. On the other hand, if it is determined that the utterance content includes the appearance information and the statement requesting response information, the determination unit 22 determines whether there are multiple warning lights that match the appearance information.

[0081] If it is determined that there is only one warning light that matches the appearance information, the output unit 23 does not calculate a priority. On the other hand, if it is determined that there are multiple warning lights that match the appearance information, the output unit 23 calculates priorities for notifying the user of the warning contents of the multiple warning lights that match the appearance information. The priorities are set in advance for each warning light according to, for example, the grace period until the warning contents are notified to the user. Furthermore, the output unit 23 may obtain information regarding the type of the lit warning light and calculate the priority according to the type.

[0082] The output unit 23 outputs countermeasure information for a warning content with a higher priority than countermeasure information for a warning content with a lower priority. Specifically, the output unit 23 outputs first countermeasure information for a warning content of a warning light calculated to have a first priority over second countermeasure information for a warning content of a warning light calculated to have a second priority lower than the first priority. For example, when a brake warning light and a low fuel warning light are on and a higher priority is set for the brake warning light, the output unit 23 outputs countermeasure information (first countermeasure information) urging the user to perform brake maintenance before countermeasure information (second countermeasure information) urging the user to refuel.

[0083] The output unit 23 may output first response information including status information regardless of the content of the utterance. On the other hand, the output unit 23 may output second response information that differs depending on the content of the utterance. Specifically, the output unit 23 may determine whether the content of the utterance includes a statement requesting status information, and if it is determined that the content of the utterance includes a statement requesting status information, may output the second response information including status information. The determination of whether the content of the utterance includes a statement requesting status information is performed in the same manner as the determination of whether the content of the utterance includes a request statement.

[0084] For example, when the brake warning light and the fuel level warning light are on and it is determined that the speech content includes wording requesting status information regarding the fuel level warning light, the output unit 23 outputs to the display device 12 and / or speaker 14 status information indicating that the remaining fuel level is low and action information urging the user to refuel.

[0085] If the output unit 23 determines that the speech content includes appearance information and a statement requesting response information, it may extract candidates for warning lights that may match the appearance information from a plurality of warning lights, calculate the probability that the extracted candidates match the appearance information, and calculate the priority of warning lights with a high probability to be higher than the priority of warning lights with a low probability.

[0086] When the output unit 23 outputs the second action information after the first action information, the output unit 23 may output a simplified version of the second action information compared to when only the second action information is output. For example, when the brake warning light and the remaining fuel warning light are on and the output unit 23 outputs action information (first action information) that prompts the user to perform brake maintenance followed by action information (second action information) that prompts the user to refuel, the output unit 23 displays a pictogram indicating refueling on the display device 12 as the second action information, but does not output audio information from the speaker 14 that prompts the user to refuel.

[0087] 3 is a block diagram showing another example of an embodiment of an operation assistance system 10 according to the present invention. As shown in FIG. 3, the operation assistance system 10 includes an on-board sensor 11, a display device 12, a microphone 13, a speaker 14, and an operation assistance device 20, as well as a control device 30, a communication device 40, and a database 50. Each device except for the database 50 is connected via a CAN or other on-board LAN, and can exchange information with each other. The database 50 is communicably connected to the operation assistance device 20 via the communication device 40 and a network (not shown).

[0088] The control device 30 is a device different from the operation assistance device 20, and is connected to the display device 12 on which a warning light is displayed and to the vehicle's on-board sensor 11, and controls the display of the display device 12. That is, the control device 30 controls the illumination of the warning light. The control device 30 is, for example, a computer, and, like the operation assistance device 20, includes a CPU which is a processor, a ROM in which a program is stored, and a RAM which functions as an accessible storage device.

[0089] The communication device 40 is a communication interface compatible with various communication standards, and is a device for exchanging information between the operation assistance device 20 and the database 50. The communication device 40 is not particularly limited as long as it is a device that can communicate with other devices via a network, and a known device can be used. The database 50 is a server installed in a remote location away from the vehicle, and stores response information and the like.

[0090] The control device 30 shown in Fig. 3 is connected to the display device 12 via a CAN bus. The control device 30 outputs an instruction to turn on the warning light to the display device 12 via the CAN bus, and can detect information about the warning light (e.g., information about the type of the warning light that is turned on) from a control signal on the CAN bus. The control device 30 is also connected to the on-board sensor 11 and the microphone 13 via the CAN bus. The control device 30 can obtain the detection results of the on-board sensor 11 and the audio data obtained by the microphone 13 from the control signal on the CAN bus. The control device 30 can detect the state of the vehicle from this information.

[0091] The CAN bus connecting the control device 30 and the display device 12 may be the same as or different from the CAN bus connecting the control device 30, the in-vehicle sensor 11, and the microphone 13. Furthermore, the CAN bus connecting the control device 30 and the display device 12 differs from the CAN bus connecting the operation assistance device 20 and the control device 30, and the operation assistance device 20 and the display device 12 are not directly connected. In other words, the operation assistance device 20 is connected to the display device 12 via the control device 30.

[0092] The control device 30 detects the type of warning light from a control signal of the CAN bus connecting the control device 30 and the display device 12, and when it detects the state of the vehicle from the control signal of the CAN bus connecting the control device 30 and the on-board sensor 11, outputs a response instruction associated with response information corresponding to the type of warning light and the state of the vehicle to the operation assistance device 20. The response instruction is, for example, a number (ID number or index) assigned to each piece of response information. When a response instruction is input, the operation assistance device 20 obtains the response information associated with the response instruction from multiple pieces of response information registered in advance, and outputs the obtained response information.

[0093] For example, if the detection result of the on-board sensor 11 detects that the passenger door is not completely closed and the display device 12 detects that a door warning light is on, indicating that the vehicle departed with one of the vehicle doors not completely closed, the control device 30 outputs a countermeasure instruction corresponding to the countermeasure information urging the user to close the passenger door to the operation assistance device 20. When the countermeasure instruction is input, the operation assistance device 20 acquires, from the database 50 (via the communication device 40), countermeasure information urging the user to close the passenger door, which is associated with the countermeasure instruction. The operation assistance device 20 generates audio information from the countermeasure information acquired from the database 50 using a read-out function and outputs the audio information to the speaker 14.

[0094] 4 and 5A to 5C, the procedure for processing information by the operation assistance device 20 will be described. The process described below is executed at predetermined time intervals (e.g., every 0.1 to 1 millisecond) by a processor (CPU) included in the operation assistance device 20.

[0095] FIG. 4 is a flowchart showing an example of a processing procedure executed in the operation assistance system 10 when an explanatory text indicating the warning content of the warning light is output as audio from the speaker 14.

[0096] First, in step S1, the operation assistance device 20 acquires voice data from the microphone 13, and then in step S2, performs processes such as frequency analysis, voice recognition, and named entity extraction on the voice data to acquire the content of the utterance. In step S3, the operation assistance device 20 identifies a warning light that explains the content of the warning from the user's intention and the named entity, and then in step S4, acquires the state of the identified warning light from a signal on the CAN bus to which the warning light is connected.

[0097] In step S5, the operation assistance device 20 determines whether the utterance content includes the name of a warning light. If it is determined that the utterance content includes the name of a warning light, the operation assistance device 20 proceeds to step S6, where the operation assistance device 20 selects a description that has been registered in advance for each warning light or causes an external server to generate a description using a generation AI. Then, in step S7, the operation assistance device 20 reads out the description and outputs it as sound from the speaker 14.

[0098] On the other hand, if it is determined in step S5 that the utterance does not include the name of the warning light, the operation assistance device 20 proceeds to step S8, where it determines whether the utterance includes an expression related to the color of the warning light. If it is determined that the utterance does not include an expression related to the color of the warning light, the operation assistance device 20 ends the process without generating an explanatory sentence. On the other hand, if it is determined that the utterance includes an expression related to the color of the warning light, the operation assistance device 20 proceeds to step S9.

[0099] In step S9, the operation assistance device 20 determines whether the number of warning lights corresponding to the color is equal to or greater than a predetermined number. If it is determined that the number of warning lights corresponding to the color is equal to or greater than the predetermined number, the operation assistance device 20 ends the process without generating an explanatory text. On the other hand, if it is determined that the number of warning lights corresponding to the color is less than the predetermined number, the operation assistance device 20 proceeds to step S6.

[0100] Next, FIG. 5A is a flowchart showing an example of a processing procedure executed in the operation assistance system 10 when performing a rendering process.

[0101] First, in step S11, the operation assistance device 20 determines whether multiple warning lights are illuminated. If it is determined that only one warning light is illuminated, the process ends. On the other hand, if it is determined that multiple warning lights are illuminated, the process proceeds to step S12. In step S12, the operation assistance device 20 determines whether appearance information and a request statement are included in the utterance content. If it is determined that at least one of the appearance information and the request statement is not included in the utterance content, the process ends. On the other hand, if it is determined that the appearance information and the request statement are included in the utterance content, the process proceeds to step S13.

[0102] In step S13, the operation assistance device 20 starts generating explanatory text for the multiple warning lights, and in the following step S14, identifies a warning light that explains the warning content from the user's intention and the named entity, and in the following step S15, outputs a performance instruction to the display device 12 so that the warning light to be explained is in a performance state in which performance processing has been executed. In step S16, the operation assistance device 20 outputs the generated explanatory text as sound from the speaker 14, and in the following step S17, stops outputting the sound to the speaker 14.

[0103] In step S18, the operation assistance device 20 outputs an instruction to the display device 12 to turn on the warning light to be explained. In the following step S19, the operation assistance device 20 determines whether there are any warning lights that have not yet been explained. If it is determined that there are any warning lights that have not yet been explained, the process proceeds to step S13. On the other hand, if it is determined that all warning lights have been explained, the process ends.

[0104] Next, Fig. 5B is a flowchart showing another example of the processing procedure executed in the operation assistance system 10 when executing the rendering processing. In the flowchart shown in Fig. 5B, step S15 of the flowchart shown in Fig. 5A is replaced with steps S15a and S15b. In the following, explanations of the processing shown in Fig. 5B that overlaps with the processing shown in Fig. 5A will be omitted.

[0105] In step S15a, the operation assistance device 20 calculates the probability that the warning light to be explained matches the appearance information, and in the subsequent step S15b, if the probability is low, it outputs a performance instruction to the display device 12 to make the warning light less emphasized than when the probability is high.

[0106] Next, Fig. 5C is a flowchart showing another example of the processing procedure executed in the operation assistance system 10 when executing the rendering processing. In the flowchart shown in Fig. 5C, step S15 of the flowchart shown in Fig. 5A is replaced with steps S15c and S15d. In the following, explanations of the processing shown in Fig. 5C that overlaps with the processing shown in Fig. 5A will be omitted.

[0107] In step S15c, the operation assistance device 20 calculates the priority of notifying the user of the warning content for each warning light, and in the subsequent step S15d, outputs a performance instruction for a normal performance state when the priority is high. Specifically, when the priority is high, the operation assistance device 20 outputs a performance instruction to the display device 12 for emphasizing the warning light more than when the accuracy is low.

[0108] According to the present embodiment, in an operation assistance method executed by an operation assistance device 20, when multiple warning lights of a vehicle are illuminated, the operation assistance device 20 determines whether a speech content of a user of the vehicle includes appearance information regarding the appearance of the warning lights and a statement by the user requesting an explanation of the warning lights, and determines whether multiple warning lights match the appearance information. When it is determined that the speech content includes the appearance information and the statement and that multiple warning lights match the appearance information, the operation assistance device 20 sequentially outputs sounds related to the warning contents of the multiple warning lights that match the appearance information, and executes a performance process that highlights the warning lights corresponding to the sounds for at least a portion of the time period during which the sounds are output. This makes it easier for the vehicle user to understand which warning lights the output sounds correspond to.

[0109] In the operation assistance method of this embodiment, the effect processing includes at least one of a processing for flashing the warning light, a processing for increasing the brightness of the warning light, a processing for periodically changing the color of the warning light, a processing for highlighting the edge of the warning light, a processing for enlarging the size of the warning light, and a processing for displaying the warning light displayed on an instrument panel located in front of the driver's seat of the vehicle on a display device other than the instrument panel, thereby enabling the effect processing to be executed smoothly.

[0110] In the operation assistance method of this embodiment, the operation assistance device 20 maintains the state of the warning lights that are not the target of the effect processing while the effect processing is being executed, thereby making it possible to further emphasize the warning lights that are the target of the effect processing.

[0111] In the operation assistance method of this embodiment, when the rendering process is executed, the operation assistance device 20 displays an image of the vehicle part or function corresponding to the warning content, which makes it easier for the vehicle user to understand which warning light the output sound corresponds to.

[0112] In the operation assistance method of this embodiment, when the performance process is executed, the operation assistance device 20 displays the name of the vehicle part or function corresponding to the warning content, which makes it easier for the vehicle user to understand which warning light the output sound corresponds to.

[0113] In the operation assistance method of this embodiment, when the operation assistance device 20 determines that the utterance content includes the appearance information and the text, it extracts warning light candidates that may match the appearance information from the plurality of warning lights, calculates the probability that the candidates match the appearance information, and executes the effect processing to emphasize the warning lights with a high probability more than the warning lights with a low probability. In this way, effect processing according to the probability can be executed.

[0114] In the operation assistance method of this embodiment, when the operation assistance device 20 determines that the utterance content includes the appearance information and the text, it extracts the warning lights that match the appearance information from the plurality of warning lights, calculates the priority for notifying the user of the warning content for the extracted warning lights, and executes the effect processing to emphasize the warning lights with higher priority than the warning lights with lower priority. This allows the effect processing to be executed to more emphasize the warning lights with higher priority.

[0115] Furthermore, according to this embodiment, when multiple warning lights of a vehicle are illuminated, the operation assistance device 20 determines whether a speech made by a user of the vehicle includes appearance information about the appearance of the warning lights and a request from the user for an explanation of the warning lights, determines whether multiple warning lights match the appearance information, and, if it is determined that the speech includes the appearance information and the request and that multiple warning lights match the appearance information, sequentially outputs sounds related to the warning contents of the multiple warning lights that match the appearance information, and executes a performance process that highlights the warning lights corresponding to the sounds for at least a portion of the time period during which the sounds are output, thereby making it easier for the vehicle user to understand which warning lights the output sounds correspond to.

[0116] Furthermore, according to this embodiment, there is provided a program for causing a computer to execute the following steps: when multiple warning lights of a vehicle are illuminated, determine whether a speech content of the vehicle user includes appearance information regarding the appearance of the warning lights and a request from the user for an explanation of the warning lights, and determine whether multiple warning lights match the appearance information; and, when it is determined that the speech content includes the appearance information and the request and that multiple warning lights match the appearance information, sequentially output sounds related to the warning contents of the multiple warning lights that match the appearance information, and execute a performance process that highlights the warning lights corresponding to the sounds for at least a portion of the time period during which the sounds are output. This makes it easier for the vehicle user to understand which warning lights the output sounds correspond to.

[0117] REFERENCE SIGNS LIST 10... Operation assistance system 11... In-vehicle sensor 12... Display device 13... Microphone 14... Speaker 20... Operation assistance device 21... Warning unit 22... Determination unit 23... Output unit 24... Performance unit 31... Battery warning light 32... Brake warning light

Claims

1. An operation assistance method executed by an operation assistance device, wherein, when multiple warning lights of a vehicle are illuminated, the operation assistance device determines whether or not a speech made by a user of the vehicle includes appearance information about the appearance of the warning lights and a statement by the user requesting an explanation of the warning lights, and determines whether or not there are multiple warning lights that match the appearance information; and when it is determined that the speech includes the appearance information and the statement and that there are multiple warning lights that match the appearance information, the operation assistance device sequentially outputs sounds related to the warning contents of the multiple warning lights that match the appearance information, and executes a performance process that emphasizes the warning lights corresponding to the sounds for at least a portion of the period during which the sounds are output.

2. The operation assistance method according to claim 1, wherein the effect processing includes at least one of processing for flashing the warning light, processing for increasing the brightness of the warning light, processing for periodically changing the color of the warning light, processing for highlighting the edge of the warning light, processing for enlarging the size of the warning light, and processing for displaying the warning light displayed on an instrument panel located in front of the driver's seat of the vehicle on a display device other than the instrument panel.

3. An operation assistance method as described in claim 1 or 2, wherein the operation assistance device maintains the state of the warning lights that are not the target of the effect processing while the effect processing is being executed.

4. An operation assistance method according to any one of claims 1 to 3, wherein the operation assistance device displays an image relating to a part or function of the vehicle corresponding to the warning content when the performance processing is executed.

5. An operation assistance method according to any one of claims 1 to 4, wherein the operation assistance device displays the name of a part or function of the vehicle corresponding to the warning content when the performance processing is executed.

6. An operation assistance method according to any one of claims 1 to 5, wherein, when the operation assistance device determines that the speech content includes the appearance information and the wording, it extracts candidates for the warning light that may match the appearance information from the plurality of warning lights, calculates the probability that the candidates match the appearance information, and executes the performance processing to emphasize the warning lights with a high probability more than the warning lights with a low probability.

7. An operation assistance method according to any one of claims 1 to 6, wherein, when it is determined that the appearance information and the text are included in the utterance content, the operation assistance device extracts from the plurality of warning lights those warning lights that match the appearance information, calculates a priority for notifying the user of the warning content for the extracted warning lights, and executes the performance processing to highlight the warning lights with a higher priority than the warning lights with a lower priority.

8. An operation assistance device that, when multiple warning lights of a vehicle are illuminated, determines whether the speech of a user of the vehicle includes appearance information regarding the appearance of the warning lights and a statement by the user requesting an explanation of the warning lights, and determines whether there are multiple warning lights that match the appearance information; and, when it is determined that the speech includes the appearance information and the statement and that there are multiple warning lights that match the appearance information, sequentially outputs audio regarding the warning content of the multiple warning lights that match the appearance information, and executes a performance process that emphasizes the warning light corresponding to the audio for at least a portion of the period during which the audio is output.

9. A program that causes a computer to execute the following steps: when multiple warning lights of a vehicle are illuminated, determine whether the speech of the vehicle user includes appearance information regarding the appearance of the warning lights and a statement by the user requesting an explanation of the warning lights, and determine whether there are multiple warning lights that match the appearance information; and when it is determined that the speech includes the appearance information and the statement and that there are multiple warning lights that match the appearance information, sequentially output audio regarding the warning content of the multiple warning lights that match the appearance information, and execute a performance process that emphasizes the warning light corresponding to the audio for at least a portion of the period during which the audio is output.

10. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following steps: when multiple warning lights of a vehicle are illuminated, determining whether the speech of the vehicle user includes appearance information regarding the appearance of the warning lights and a statement by the user requesting an explanation of the warning lights, and determining whether there are multiple warning lights that match the appearance information; and when it is determined that the speech includes the appearance information and the statement and that there are multiple warning lights that match the appearance information, sequentially outputting audio regarding the warning content of the multiple warning lights that match the appearance information, and executing a performance process that emphasizes the warning lights corresponding to the audio for at least a portion of the period during which the audio is output.

Images