Voice recognition method and voice recognition device

Abstract

In the present invention, a controller (9): acquires speech content of an occupant of a vehicle (S6); acquires an input operation signal generated by the occupant operating an operation input device of the vehicle (S4); and, on the basis of the speech content and the input operation signal, estimates a target component, which is a component mentioned in the speech content from among a plurality of components that constitute the vehicle, and outputs information regarding the target component (S7).

Description

Speech recognition method and speech recognition device

[0001] The present invention relates to a speech recognition method and a speech recognition device.

[0002] Patent Document 1 proposes a technique for operating an in-vehicle device and highlighting an operating section of the in-vehicle device when a voice command to the in-vehicle device is received from a vehicle occupant.

[0003] Japanese Patent Application Laid-Open No. 2020-097378

[0004] The technology described in Patent Document 1 can inform the occupant of the location of an operation input device that accepts operation input from the occupant to in-vehicle equipment, but cannot inform the occupant of the name or purpose of the operation input device. An object of the present invention is to inform the occupant of information related to an operation input device that accepts operation input from the occupant to in-vehicle equipment.

[0005] In one aspect of the present invention, a speech recognition method acquires an utterance from a vehicle occupant, acquires an input operation signal generated by the occupant operating an operation input device of the vehicle, and, based on the utterance and the input operation signal, estimates a target component that is a component mentioned in the utterance among multiple components that configure the vehicle, and outputs information about the target component. For example, in a case where, in order to start operation of a driving assistance function of a vehicle, it is necessary to press a first switch that switches the driving assistance function on and off among a group of steering switches provided on a steering wheel and then a second switch that starts operation of the driving assistance function, the method may estimate that the component mentioned in the utterance is the steering switch group based on the input operation signal generated by the occupant pressing the first switch and the occupant's utterance, "What should I do next?", and output an explanatory message about how to use the steering switch group, "Please press the second switch," as information about the steering switch group.

[0006] According to the present invention, it is possible to notify the occupant of information relating to an operation input device that accepts an operation input from the occupant to an in-vehicle device.

[0007] 1 is a schematic configuration diagram of an example of a vehicle equipped with a voice recognition device of an embodiment; FIG. 2 is a block diagram showing an example of the functional configuration of the controller of FIG. 1; FIG. 3 is a flowchart of an example of a voice recognition method of a first embodiment; FIG. 4 is a flowchart of a voice recognition method of a first modified example; FIG. 5 is a flowchart of a voice recognition method of a second modified example; and FIG. 6 is a flowchart of an example of a voice recognition method of a second embodiment.

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the drawings are schematic and may differ from the actual product. Furthermore, the embodiments of the present invention shown below are examples of devices and methods for embodying the technical concept of the present invention, and the technical concept of the present invention does not limit the structure, arrangement, etc. of component parts to those described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims.

[0009] (First embodiment) (Configuration) Fig. 1 is a schematic configuration diagram of an example of a vehicle equipped with a voice recognition device according to an embodiment. The vehicle 1 is equipped with an in-vehicle device 2, a plurality of operation input devices 3, a voice recognition device 4, a push-to-talk (PTT) switch 5, a speaker 6, and a display device 7. The in-vehicle device 2 is various devices mounted on the vehicle 1. The in-vehicle device 2 may be, for example, an air conditioning system, an audio device, an interior light, a glove box, a console lamp, an in-vehicle infotainment (IVI) system, or a navigation system.

[0010] The operation input device 3 is a device that accepts operation inputs from the occupant to the in-vehicle equipment 2. The operation input device 3 may be, for example, a push switch, a click switch, a toggle switch, a rocker switch, a magnetic non-contact switch, a capacitive non-contact switch, a jog dial, a jog lever, a knob, a slide bar, a dial controller, or a touch panel. The push switch may be, for example, an alternate push switch that maintains the contact state even after the button is pressed and released, or a momentary push switch that returns to the state before the button was pressed when released.

[0011] A jog dial is an operation input device that accepts selection or adjustment operations by rotating an operation unit such as a dial or wheel, and also accepts operations by pressing the operation unit. A jog lever is an operation input device that accepts selection operations by tilting a lever, and also accepts operations by pressing the lever. A dial controller is an operation input device that accepts selection or adjustment operations by rotating a dial, selection operations by tilting a dial, operations by pressing the dial, and operations on the touchpad on the dial (e.g., character input, etc.).

[0012] The voice recognition device 4 recognizes speech uttered by an occupant of the vehicle 1 and outputs a guidance message answering the occupant's question regarding the operation input device 3. The voice recognition device 4 includes a microphone 8 and a controller 9. The microphone 8 is a voice input device that acquires speech input from the occupant. The controller 9 is an electronic control unit (ECU) that executes a voice recognition process to recognize the speech uttered by the occupant. The controller 9 includes a processor 9a and peripheral components such as a storage device 9b. The processor 9a may be, for example, a central processing unit (CPU) or a micro-processing unit (MPU). The storage device 9b may include a semiconductor storage device, a magnetic storage device, an optical storage device, or the like. The storage device 9b may include memories such as a read-only memory (ROM) and a random access memory (RAM), as well as a register and a cache memory. The functions of the controller 9 described below are realized, for example, by the processor 9a executing a computer program stored in the storage device 9b.

[0013] The PTT switch 5 is an operation input device that allows the occupant to instruct the start of voice recognition processing by the voice recognition device 4. As will be described later, the PTT switch 5 may be omitted when the start of voice recognition processing is instructed by a wake-up word, a dedicated voice command, or by operating the operation input device 3 other than the PTT switch 5. The speaker 6 is an information presentation device that outputs a voice message generated by the voice recognition device 4. The display device 7 is an information presentation device that displays text messages, images, symbols, and figures generated by the voice recognition device 4.

[0014] 2 is a block diagram showing an example of the functional configuration of the controller 9 in FIG. 1 . The controller 9 includes a voice recognition unit 10, an input operation signal acquisition unit 11, an operation determination unit 12, a response generation unit 13, and a device control unit 14. When the voice recognition device 4 is activated, the voice recognition unit 10 maintains a first standby mode until a predetermined voice recognition start event occurs. The voice recognition start event may be the voice input of a common wake-up word (e.g., "Hello XX") for starting the voice recognition process, or the input of a dedicated voice command (e.g., "I want to ask about the switch?") for accepting a voice question regarding the operation input device 3. Alternatively, the voice recognition start event may be the operation of the PTT switch 5.

[0015] When a voice recognition start event occurs, the voice recognition unit 10 starts voice recognition processing. The voice recognition unit 10 recognizes voice input from the occupant acquired by the microphone 8 and converts it into linguistic information such as text. The voice recognition unit 10 analyzes the linguistic information using natural language processing to acquire the content of the user's utterance. For example, the voice recognition unit 10 extracts keywords (e.g., "switch," "lever," "dial," etc.) that refer to the operation input device 3 from the content of the utterance.

[0016] Furthermore, the voice recognition unit 10 may extract the type of question regarding the operation input device 3 from the utterance content. For example, if the utterance content is "What is this switch?", the voice recognition unit 10 may determine that the type of question from the occupant is a "question regarding the name" of the operation input device 3. Furthermore, if the utterance content is "Which switch does XX?" or "Where is the switch for XX?", the voice recognition unit 10 may determine that the type of question from the occupant is a "question regarding the use and location" of the operation input device 3.

[0017] Furthermore, for example, when the utterance content is "Is this switch XX correct?", the voice recognition unit 10 may determine that the type of question from the occupant is "confirming the name" of the operation input device 3. Furthermore, when the utterance content is "I want to do XX, is this switch okay?", the voice recognition unit 10 may determine that the type of question from the occupant is "confirming the purpose and position" of the operation input device 3. The voice recognition unit 10 outputs the acquired utterance content to the operation determination unit 12.

[0018] The input operation signal acquisition unit 11 acquires, for each of the multiple operation input devices 3, an input operation signal generated by an occupant operating the operation input device 3. The input operation signal acquisition unit 11 determines whether or not the input operation signal satisfies a predetermined operation determination condition for each operation input device 3. When an operation input device 3 that satisfies the operation determination condition is found, the input operation signal acquisition unit 11 generates an operation detection signal that identifies the operation input device 3 that satisfies the operation determination condition. The operation detection signal may include identification information of the operation input device 3 that satisfies the operation determination condition.

[0019] For example, the input operation signal acquiring unit 11 determines that the operation determination condition is satisfied in the following cases: (1) When a push switch or click switch is pressed, when a jog dial or a dial controller dial is pressed, or when a jog lever lever is pressed, or (2) When a toggle switch, rocker switch, a jog lever lever, or a dial controller dial is pushed to a position where it is in one of the operation states.

[0020] (3) When a magnet is removed from a magnetic contactless switch. (4) When a change in capacitance is detected by waving a hand over a capacitive contactless switch or placing an object on it. (5) When a jog dial or the dial of a dial controller is rotated. (6) When a knob is rotated. (7) When the bar of a slide bar is slid. (8) When a change in capacitance is detected on the touch pad on the top of the dial of a dial controller. (9) When the state of the Graphical User Interface (GUI) on the touch panel screen is changed or a selection operation is performed by touching the surface of the touch panel or sliding a finger on it.

[0021] Note that there are operation input devices 3 that can accept multiple types of operations using a single operation unit. For example, a jog dial can accept a selection operation or adjustment operation by rotating an operation unit such as a dial or wheel, and an operation of pressing the operation unit. A jog lever can accept a selection operation by tilting the lever and an operation of pressing the lever. A dial controller can accept a selection operation or adjustment operation by rotating the dial, a selection operation by tilting the dial, an operation of pressing the dial, and an operation on a touchpad on the top surface of the dial (e.g., character input, etc.). In such an operation input device 3, different operation detection signals may be generated for different types of operations. For example, the operation detection signal may include identification information for identifying the type of operation. The input operation signal acquisition unit 11 outputs the input operation signal and the operation detection signal acquired from the operation input device 3 to the action determination unit 12.

[0022] The operation determination unit 12 switches the operation of the voice recognition device 4 depending on the acquisition result of the occupant's utterance content and the acquisition result of the input operation signal from the operation input device 3. That is, when the input operation signal acquisition unit 11 acquires the input operation signal from the operation input device 3 and the voice recognition unit 10 acquires the utterance content including a question regarding the operation input device 3, the operation determination unit 12 outputs a response generation command to the response generation unit 13 to generate a guidance message answering the utterance content, and causes the response generation unit 13 to output a guidance message answering the estimated question regarding the operation input device 3. On the other hand, when the input operation signal acquisition unit 11 acquires the input operation signal from the operation input device 3 but the voice recognition unit 10 does not acquire the utterance content including a question regarding the operation input device 3, the operation determination unit 12 outputs the input operation signal acquired from the operation input device 3 to the device control unit 14. The device control unit 14 controls the in-vehicle device 2 depending on the input operation signal.

[0023] Specifically, when the input operation signal acquisition unit 11 acquires an input operation signal from the operation input device 3 and the voice recognition unit 10 acquires an utterance content including a question about the operation input device 3, the operation determination unit 12 estimates the operation input device 3 mentioned in the utterance content of the occupant from among the multiple operation input devices 3 constituting the vehicle 1, based on the utterance content acquired by the voice recognition unit 10 and the input operation signal acquired by the input operation signal acquisition unit 11. For example, the operation input device 3 mentioned in the utterance content is estimated based on the utterance content acquired by the voice recognition unit 10 and the operation detection signal output by the input operation signal acquisition unit 11. The operation input device 3 is an example of a "component ... constituting a vehicle" as defined in the claims.

[0024] In the first embodiment, when an utterance including a question about the operation input device 3 is acquired after an input operation signal output from a certain operation input device 3 is acquired, the operation determination unit 12 estimates that the operation input device 3 that output the input operation signal is the operation input device 3 mentioned in the occupant's utterance. For example, when an utterance including a question about the operation input device 3 is acquired before a predetermined time has elapsed after the acquisition of the input operation signal, the operation determination unit 12 may estimate that the operation input device 3 that output the input operation signal is the operation input device 3 mentioned in the occupant's utterance. Furthermore, the operation determination unit 12 may determine that an input operation signal has been acquired when, for example, an operation detection signal is received from the input operation signal acquisition unit 11.

[0025] When the operation determination unit 12 estimates the operation input device 3 mentioned in the utterance content, it outputs a response generation command to the response generation unit 13 to generate a guidance message responding to the utterance content. For example, the response generation command may include identification information of the estimated operation input device 3 and identification information of the type of question included in the occupant's utterance content (e.g., "question regarding name," "question regarding use and location," "confirmation of name," or "confirmation of use and location"). Based on the response generation command received from the operation determination unit 12, the response generation unit 13 outputs a guidance message including a voice or an image representing information about the estimated operation input device 3 from the speaker 6 or the display device 7 as a response to the question included in the occupant's utterance content. In this case, the controller 9 may stop outputting the input operation signal acquired from the operation input device 3 to the device control unit 14 until the response generation unit 13 outputs the guidance message. In other words, the controller 9 may stop controlling the in-vehicle device 2 even if an input operation signal is acquired by operating the operation input device 3.

[0026] For example, the response generation unit 13 may generate a voice guidance message representing information about the estimated operation input device 3 and output it from the speaker 6. Alternatively, the response generation unit 13 may generate a textual guidance message, image, symbol, or graphic representing information about the estimated operation input device 3 and output it from the display device 7. Specific examples of messages generated by the response generation unit 13 are described below. (Example 1) If the operation input device 3 is a volume control switch for audio equipment, an operation detection signal is output when the switch is pressed. In this case, the operation input device 3 may be, for example, a push switch, a click switch, a jog lever (when pressed), or a dial controller (when pressed). When the utterance content is "What is this switch?", the voice recognition unit 10 determines that the type of question is a "question about the name." The response generation unit 13 outputs a guidance message including information about the name and how to use it, such as "This is a volume control switch. Press + to increase the volume and - to decrease the volume."

[0027] When the utterance content is "Which switch controls the volume?", the voice recognition unit 10 determines that the type of question is "Question about use and location." The response generation unit 13 outputs a guidance message including information about the use, location, and usage method, saying, "Volume can be controlled using the switch with + and - on the left side of the steering wheel. Press + to increase the volume and - to decrease the volume." When the utterance content is "Is this switch the volume control switch?", the voice recognition unit 10 determines that the type of question is "Confirm name." The response generation unit 13 outputs a guidance message saying, "Yes, that's right. Press + to increase the volume and - to decrease the volume." When the utterance content is "I want to adjust the volume. Is this button okay?", the voice recognition unit 10 determines that the type of question is "Confirm use and location." The response generation unit 13 outputs a guidance message saying, "Yes, that's right. Press + to increase the volume and - to decrease the volume."

[0028] (Example 2) If the operation input device 3 is an item selection switch of a navigation device, an operation detection signal is output when the lever is pushed to a position that results in one of the operation states. In this case, the operation input device 3 may be, for example, a toggle switch, a rocker switch, a jog lever (when the lever is pushed), or a dial controller (when the dial is pushed). When the utterance content is "What is this switch?", the voice recognition unit 10 determines that the type of question is a "question about the name." The response generation unit 13 outputs a guidance message including information about the name and usage: "This is an item selection switch. You can focus on the item you want to select by pushing or tilting it up and down / left and right." When the utterance content is "Which is the cursor / item selection switch?", the voice recognition unit 10 determines that the type of question is a "question about the purpose and location." For example, the response generation unit 13 outputs a guidance message including information about the purpose, location, and usage: "Item selection can be operated using the round knob-shaped dial on the console. You can focus on the item you want to select by tilting or pushing it up and down / left and right, or by rotating the dial left and right."

[0029] When the utterance content is "Is this switch a cursor movement / item selection switch?", the voice recognition unit 10 determines that the type of question is "confirm the name." For example, the response generation unit 13 outputs a guidance message such as "Yes, that's right. You can focus on the item you want to select by tilting / pressing it up / down / left / right, or by rotating the dial left / right." When the utterance content is "I want to select an item. Is this button okay?", the voice recognition unit 10 determines that the type of question is "confirm the purpose and position." For example, the response generation unit 13 outputs a guidance message such as "Yes, that's right. You can focus on the item you want to select by tilting / pressing it up / down / left / right, or by rotating the dial left / right."

[0030] (Example 3) When the operation input device 3 is a glove box open / close interlocking switch, an operation detection signal is output when the magnet is removed from the magnetic non-contact switch that is the open / close interlocking switch. When the utterance content is "What is the switch that turns on the light in the storage compartment in front of the passenger seat?", the voice recognition unit 10 determines that the type of question is a "question about the name." The response generation unit 13 outputs a guidance message including information about the name and how to use it, such as "This is the glove box open / close interlocking switch. When the box is opened, the light turns on, and when it is closed, the light turns off." When the utterance content is "Which switch turns on the light in the glove box?", the voice recognition unit 10 determines that the type of question is a "question about the purpose and location." The response generation unit 13 outputs a guidance message including information about the purpose, location, and how to use it, such as "The glove box is a drawer in front of the passenger seat. It can be operated by opening and closing the glove box lid. When the box is opened, the light turns on, and when it is closed, the light turns off."

[0031] When the utterance content is "Is this the right switch to turn on the glove box light?", the voice recognition unit 10 determines that the type of question is "confirm the name." The response generation unit 13 outputs a guidance message saying "Yes, that's right. The light turns on when you open the box, and turns off when you close it." When the utterance content is "I want to turn on the glove box light, but where is it?", the voice recognition unit 10 determines that the type of question is "confirm its purpose and location." The response generation unit 13 outputs a guidance message saying "The glove box is a drawer in front of the passenger seat. It can be operated by opening and closing the glove box lid. The light turns on when you open the box, and turns off when you close it."

[0032] (Example 4) If the operation input device 3 is a capacitive non-contact switch that turns a console lamp on and off, an operation detection signal is output when a change in capacitance is detected by waving a hand over the capacitive non-contact switch or placing an object over it. If the utterance content is "What is this switch?", the voice recognition unit 10 determines the type of question to be a "question about the name." The response generation unit 13 outputs a guidance message including information about the name and usage, such as "This is the switch for the console lamp inside the vehicle. You can turn it on and off by waving your hand over it." If the utterance content is "Which switch is the console lamp inside the vehicle?", the voice recognition unit 10 determines the type of question to be a "question about the purpose and location." The response generation unit 13 outputs a guidance message including information about the purpose, location, and usage, such as "The console lamp can be operated with a switch on the center console. You can turn it on and off by waving your hand over it." If the utterance content is "Is this switch a console lamp switch?", the voice recognition unit 10 determines the type of question to be "confirm the name." The response generation unit 13 outputs the guidance message "Yes, that's right. You can turn it on and off by waving your hand." When the utterance content is "I want to turn on the console lamp, is this button okay?", the voice recognition unit 10 determines that the type of question is "Confirm use and location." The response generation unit 13 outputs the guidance message "Yes, that's right. You can turn it on and off by waving your hand."

[0033] (Example 5) If the operation input device 3 is a volume control dial of an audio device, an operation detection signal is output when the occupant rotates the dial. In this case, the operation input device 3 may be, for example, a jog dial (when rotated) or a dial controller (when rotated). When the utterance content is "What is this dial?", the voice recognition unit 10 determines that the type of question is a "question about the name." The response generation unit 13 outputs a guidance message including information about the name and how to use it: "This is a volume control dial. Rotate the dial to the left to decrease the volume, and rotate it to the right to increase the volume." When the utterance content is "Which dial controls the volume?", the voice recognition unit 10 determines that the type of question is a "question about the purpose and location." The response generation unit 13 outputs a guidance message including information about the purpose, location, and how to use it: "You can adjust the volume using the round knob-shaped dial located at the bottom left of the IVI screen. Rotate the dial to the left to decrease the volume, and rotate it to the right to increase the volume."

[0034] When the utterance content is "Is this dial the right volume control dial?", the voice recognition unit 10 determines that the type of question is "confirm the name." The response generation unit 13 outputs a guidance message, "Yes, that's right. Turn the dial to the left to decrease the volume, and turn it to the right to increase the volume." When the utterance content is "I want to adjust the volume. Is this button okay?", the voice recognition unit 10 determines that the type of question is "confirm the purpose and location." The response generation unit 13 outputs a guidance message, "Yes, that's right. Turn the dial to the left to decrease the volume, and turn it to the right to increase the volume."

[0035] (Example 6) If the operation input device 3 is an air volume control knob for an air conditioner, an operation detection signal is output when the occupant turns the knob. If the utterance content is "What is this switch?", the voice recognition unit 10 determines that the type of question is a "question about the name." The response generation unit 13 outputs a guidance message including information about the name and how to use it, such as "This is the air volume control switch. Turn it to the left to decrease the air volume, and turn it to the right to increase the air volume." If the utterance content is "Which switch controls the air volume?", the voice recognition unit 10 determines that the type of question is a "question about the purpose and location." The response generation unit 13 outputs a guidance message including information about the purpose, location, and how to use it, such as "The air volume can be controlled with the knob on the left side below the IVI. Turn it to the left to decrease the air volume, and turn it to the right to increase the air volume."

[0036] When the utterance content is "Is this switch the air volume adjustment switch?", the voice recognition unit 10 determines that the type of question is "Confirm the name." The response generation unit 13 outputs a guidance message "Yes, that's right. Turn it left to decrease the air volume, and turn it right to increase the air volume." When the utterance content is "I want to adjust the air volume. Is this button okay?", the voice recognition unit 10 determines that the type of question is "Confirm the purpose and location." The response generation unit 13 outputs a guidance message "Yes, that's right. Turn it left to decrease the air volume, and turn it right to increase the air volume."

[0037] (Example 7) If the operation input device 3 is a slide bar used as an interior light switch, an operation detection signal is output when the occupant slides the bar. When the utterance content is "What is this switch?", the voice recognition unit 10 determines the type of question to be a "question about the name." The response generation unit 13 outputs a guidance message including information about the name and usage: "This is the interior light switch. You can operate it by sliding it to the left for off, to the center for door interlocking, and to the right for on." When the utterance content is "Which switch is used for the interior lights?", the voice recognition unit 10 determines the type of question to be a "question about the purpose and location." The response generation unit 13 outputs a guidance message including information about the purpose, location, and usage: "The interior lights can be operated with the slide switch near the ceiling rearview mirror. You can operate it by sliding it to the left for off, to the center for door interlocking, and to the right for on." When the utterance content is "Is this switch the interior light switch?", the voice recognition unit 10 determines the type of question to be a "confirm the name." The response generation unit 13 outputs the guidance message "Yes, that's right. You can operate them by sliding to the left for off, to the center for door interlocking, and to the right for on." When the utterance content is "I want to use the interior light. Is this button okay?", the voice recognition unit 10 determines that the type of question is "Confirm use and position." The response generation unit 13 outputs the guidance message "Yes, that's right. You can operate them by sliding to the left for off, to the center for door interlocking, and to the right for on."

[0038] (Example 8) If the operation input device 3 is a dial controller used for input operations to a navigation device or an audio device, an operation detection signal is output when a change in capacitance of the touchpad on the top surface of the dial is detected. When the utterance content is "What is this switch?", the voice recognition unit 10 determines that the type of question is a "question about the name." The response generation unit 13 outputs a guidance message including information about the name and how to use it: "This is a dial controller. You can manually input text on the surface of the dial. You can also select items and adjust the volume by rotating the knob left and right, and tilting it back and forth left and right." When the utterance content is "Which switch do you manually input text into?", the voice recognition unit 10 determines that the type of question is a "question about the purpose and location." The response generation unit 13 outputs a guidance message including information about the purpose, location, and how to use it: "You can manually input text on the surface of the dial. You can also select items and adjust the volume by rotating the knob left and right, and tilting it back and forth left and right."

[0039] When the utterance content is "Is this the switch that can input characters?", the voice recognition unit 10 determines that the type of question is "Confirm the name." The response generation unit 13 outputs a guidance message saying, "Yes, that's right. You can manually input characters on the surface of the dial. You can also select items and adjust the volume by rotating the knob left and right and tilting it back and forth left and right / pressing it." When the utterance content is "I want to input characters manually, but is this button okay?", the voice recognition unit 10 determines that the type of question is "Confirm the purpose and location." The response generation unit 13 outputs a guidance message saying, "Yes, that's right. You can manually input characters on the surface of the dial. You can also select items and adjust the volume by rotating the knob left and right and tilting it back and forth left and right / pressing it."

[0040] (Example 9) If the operation input device 3 is a touch panel on the IVI screen, an operation detection signal is output when the occupant touches the surface of the touch panel or slides a finger thereon, thereby switching the state of the GUI on the touch panel or performing a selection operation. When the utterance is "What's this switch?", the voice recognition unit 10 determines that the question type is a "question about the name." The response generation unit 13 outputs a guidance message including information about the name: "This is the IVI setting icon. You can configure settings related to language settings, navigation, telephone, etc." When the utterance is "Which switch is used to configure the IVI?", the voice recognition unit 10 determines that the question type is a "question about use and location." The response generation unit 13 outputs a guidance message including information about the use, location, and usage: "You can configure the IVI settings using the gear icons in the upper right / left corner of the IVI screen. You can configure settings related to language settings, navigation, telephone, etc."

[0041] When the utterance content is "Is this switch the correct setting switch for the IVI?", the voice recognition unit 10 determines that the type of question is "Confirm the name." The response generation unit 13 outputs a guidance message saying "Yes, that's right. You can make settings related to language settings, navigation, telephone, etc." When the utterance content is "I want to set up the IVI. Is this button okay?", the voice recognition unit 10 determines that the type of question is "Confirm the purpose and location." The response generation unit 13 outputs a guidance message saying "Yes, that's right. You can make settings related to language settings, navigation, telephone, etc."

[0042] Note that there are cases where a single operation input device 3 can accept multiple types of operations, such as a jog dial, a jog lever, or a dial controller. When different names and uses are assigned to different types of operations of the operation input device 3, the response generation unit 13 may generate a guidance message including information on the different names and uses for the single operation input device 3. For example, as shown in (Example 1) above, when a push operation of the dial controller is performed and a speech content including a question from the occupant to the operation input device 3 is acquired, a guidance message may be generated informing the user that the name and use of the dial controller are a "volume adjustment switch" and "volume adjustment," respectively. On the other hand, as shown in (Example 2) above, when the dial of the dial controller is pushed to a position corresponding to one of the operation states (when operating the lever), and a speech content including a question from the occupant to the operation input device 3 is acquired, a guidance message may be generated informing the user that the name and use of the dial controller are an "item selection switch" and "focus on an item to be selected," respectively.

[0043] Furthermore, when a single operation input device 3 can accept multiple types of operations, a unique purpose may be assigned to a combination or sequence of a series of different types of operations. For example, a first purpose may be assigned when the dial controller is tilted while being rotated, and a second purpose may be assigned when the dial controller is pressed down while being tilted. In this case, when a series of different types of operations are performed on the operation input device 3, if speech content including a question from the occupant directed to the operation input device 3 is acquired, a guidance message may be generated that notifies the occupant of the purpose assigned to the combination or sequence of these operations.

[0044] If the occupant wishes to interrupt the guidance message while the response generation unit 13 is outputting the guidance message (i.e., after the start of outputting the guidance message but before completion of output), the occupant can perform a predetermined interrupt instruction operation. For example, the occupant may perform the interrupt instruction operation by again operating the operation input device 3 mentioned in the utterance, by operating an operation input device other than the operation input device mentioned in the utterance among the multiple operation input devices 3, by long-pressing the PTT switch 5, or by speaking a specific keyword (e.g., "interrupt the guidance"). Upon receiving the interrupt instruction operation, the response generation unit 13 interrupts the output of the guidance message. Furthermore, the operation determination unit 12 outputs an input operation signal acquired from the operation input device 3 to the device control unit 14. The device control unit 14 controls the in-vehicle device 2 in accordance with the input operation signal.

[0045] Even if an input operation signal is acquired, if the occupant's speech content is not acquired within a predetermined period (e.g., three seconds), the voice recognition unit 10 terminates the voice recognition process. In this case, the operation determination unit 12 does not estimate the operation input device 3 based on the occupant's speech, and the response generation unit 13 does not output a guidance message including information about the operation input device 3, but instead outputs a termination guide message "Voice recognition is terminated" to inform the occupant that the voice recognition process is terminated. In addition, the operation determination unit 12 outputs the input operation signal acquired from the operation input device 3 to the device control unit 14. The device control unit 14 controls the in-vehicle device 2 in accordance with the input operation signal.

[0046] Furthermore, even if the voice recognition unit 10 detects a voice recognition start event, if the input operation signal acquisition unit 11 does not acquire an input operation signal within a predetermined period of time, the operation determination unit 12 does not estimate the operation input device 3 based on the occupant's speech. The response generation unit 13 outputs an end guide message without outputting a guidance message including information about the operation input device 3. Furthermore, if the voice recognition unit 10 acquires an input operation signal before detecting a voice recognition start event (i.e., before the voice recognition process is started), the operation determination unit 12 does not estimate the operation input device 3 based on the occupant's speech, and outputs the input operation signal acquired from the operation input device 3 to the device control unit 14. As a result, the response generation unit 13 does not output a guidance message including information about the operation input device 3, and the device control unit 14 controls the in-vehicle device 2 in accordance with the input operation signal.

[0047] (Operation) FIG. 3 is a flowchart of an example of the voice recognition method of the first embodiment. In step S1, the voice recognition unit 10 determines whether a voice recognition start event has occurred. If a voice recognition start event has occurred (step S1: Y), the process proceeds to step S4. If a voice recognition start event has not occurred (step S1: N), the process proceeds to step S2. In step S2, the input operation signal acquisition unit 11 determines whether an input operation signal has been acquired. If an input operation signal has been acquired (step S2: Y), the process proceeds to step S3. If an input operation signal has not been acquired (step S2: N), the process proceeds to step S12. In step S3, the operation determination unit 12 outputs the input operation signal to the device control unit 14. The device control unit 14 controls the in-vehicle device 2 in accordance with the input operation signal. Thereafter, the process proceeds to step S12.

[0048] In step S4, the input operation signal acquisition unit 11 determines whether or not an input operation signal has been acquired. If an input operation signal has been acquired (step S4: Y), the process proceeds to step S6. If an input operation signal has not been acquired (step S4: N), the process proceeds to step S5. In step S5, the response generation unit 13 outputs an end guide message. Thereafter, the process proceeds to step S12. In step S6, the operation determination unit 12 determines whether or not the occupant's utterance content has been acquired. If the utterance content has been acquired (step S6: Y), the process proceeds to step S7. If the utterance content has not been acquired (step S6: N), the process proceeds to step S9.

[0049] In step S7, the operation determination unit 12 estimates the operation input device 3 mentioned in the occupant's utterance, out of the multiple operation input devices 3 constituting the vehicle 1, based on the utterance content and the input operation signal. The response generation unit 13 outputs a guidance message including information about the estimated operation input device 3. In step S8, the operation determination unit 12 determines whether the occupant has performed an interrupt instruction operation. If an interrupt instruction operation has been performed (step S8: Y), the process proceeds to step S9. If an interrupt instruction operation has not been performed (step S8: N), the process proceeds to step S11. In step S9, the response generation unit 13 outputs an end guidance message. In step S10, the operation determination unit 12 outputs the input operation signal to the device control unit 14. The device control unit 14 controls the in-vehicle device 2 in accordance with the input operation signal. Thereafter, the process proceeds to step S12.

[0050] In step S11, the response generation unit 13 determines whether or not the output of the guidance message has been completed. If the output of the guidance message has been completed (step S11: Y), the process proceeds to step S12. If the output of the guidance message has not been completed (step S11: N), the process returns to step S7. In step S12, the controller 9 determines whether or not the ignition (IGN) switch of the vehicle has been turned off. If the IGN switch has not been turned off (step S12: N), the process returns to step S1. If the IGN switch has been turned off (step S12: Y), the process ends.

[0051] (First Modification) In the first modification, when the operation input device 3 (i.e., an operation input device other than the PTT switch 5) is operated, it is determined that a voice recognition start event has occurred, and the voice recognition process is started. That is, an input operation signal is acquired before starting the voice recognition process. For example, the voice recognition unit 10 may determine that a voice recognition start event has occurred and start the voice recognition process when it receives an operation detection signal from the input operation signal acquisition unit 11. If the occupant wishes to end the voice recognition process even after operating the operation input device 3 (e.g., if the occupant does not need the guidance message of the operation input device 3 and wishes to immediately operate the in-vehicle device 2), the occupant can perform a predetermined interrupt instruction operation. Furthermore, for example, when a predetermined operation (such as pressing and holding a button, repeatedly pressing a button, or rotating a dial left and right) is received, the operation of the operation device may be interrupted for a certain period of time (without issuing an input operation signal) and only wait for voice.

[0052] For example, the occupant may perform the interrupt instruction operation by again operating the operation input device 3 mentioned in the utterance, or by operating an operation input device other than the operation input device mentioned in the utterance among the multiple operation input devices 3. When the interrupt instruction operation is received, the response generation unit 13 interrupts the voice recognition. Furthermore, the operation determination unit 12 outputs the input operation signal acquired from the operation input device 3 to the equipment control unit 14. The equipment control unit 14 controls the in-vehicle equipment 2 in accordance with the input operation signal.

[0053] FIG. 4 is a flowchart of the voice recognition method of the first modified example. In step S20, the input operation signal acquisition unit 11 determines whether or not an input operation signal has been acquired. If an input operation signal has been acquired (step S20: Y), the process proceeds to step S21. If an input operation signal has not been acquired (step S20: N), the process proceeds to step S28. In step S21, the operation determination unit 12 determines whether or not the occupant has performed an interrupt instruction operation. If an interrupt instruction operation has been performed (step S21: Y), the process proceeds to step S22. If an interrupt instruction operation has not been performed (step S21: N), the process proceeds to step S24. In step S22, the response generation unit 13 outputs an end guide message. In step S23, the operation determination unit 12 outputs the input operation signal to the device control unit 14. The device control unit 14 controls the in-vehicle device 2 in accordance with the input operation signal. Thereafter, the process proceeds to step S28. The processes in steps S24 to S28 are similar to the processes in steps S6 to S8, S11, and S12 in FIG. 1, respectively.

[0054] (Second Modification) In the second modification, similar to the first modification, when the operation input device 3 (i.e., an operation input device other than the PTT switch 5) is operated, it is determined that a voice recognition start event has occurred, and the voice recognition process is started. That is, an input operation signal is acquired before the voice recognition process is started. In the second modification, when the input operation signal is acquired and then the occupant's speech content is acquired, the in-vehicle device 2 is controlled in accordance with the input operation signal, and a guidance message regarding the operation input device 3 mentioned in the occupant's speech content is output.

[0055] FIG. 5 is a flowchart of the voice recognition method of the second modified example. In step S30, the input operation signal acquisition unit 11 determines whether or not an input operation signal has been acquired. If an input operation signal has been acquired (step S30: Y), the process proceeds to step S31. If an input operation signal has not been acquired (step S30: N), the process proceeds to step S37. In step S31, the operation determination unit 12 outputs the input operation signal to the device control unit 14. The device control unit 14 controls the in-vehicle device 2 in accordance with the input operation signal. In step S32, the operation determination unit 12 determines whether or not the occupant's utterance content has been acquired. If the utterance content has been acquired (step S32: Y), the process proceeds to step S34. If the utterance content has not been acquired (step S6: N), the process proceeds to step S33. In step S33, the response generation unit 13 outputs an end guide message. Thereafter, the process proceeds to step S37. The processes in steps S34 to S37 are similar to the processes in steps S7, S8, S11 and S12 in FIG. 1, respectively.

[0056] (Third Modification) The voice recognition unit 10 may determine whether the type of question is a "question about how to use" the operation input device 3. For example, when the utterance content is "How do I use this switch?", the voice recognition unit 10 may determine that the type of question from the occupant is a "question about how to use" the operation input device 3. When the type of question is a "question about how to use" the operation input device 3, the response generation unit 13 may output a guidance message including information about how to use the operation input device 3.

[0057] Furthermore, for example, when the occupant's utterance after receiving the operation detection signal from the input operation signal acquisition unit 11 is a question, the voice recognition unit 10 may determine that the type of question is a "question about how to use" the operation input device 3. For example, assume that in order to start the operation of a driving assistance function of the vehicle 1, it is necessary to press a first switch of a steering switch group provided on the steering wheel that switches the driving assistance function on and off, and then press a second switch that starts the operation of the driving assistance function. In this case, when the occupant's utterance after operating the first operation input device is a question such as "What should I do next?", the voice recognition unit 10 may determine that the type of question from the occupant is a "question about how to use" the operation input device 3. Then, an explanatory message regarding how to use the steering switch group, "Please press the second switch," may be output.

[0058] (Fourth Modification) The voice recognition unit 10 may extract an operation instruction for the in-vehicle device 2 from the utterance content. For example, if the utterance content is "move this" or "set this to XX," the voice recognition unit 10 may determine that the utterance content from the occupant is an operation instruction for the in-vehicle device 2. When the operation determination unit 12 receives an operation detection signal from the input operation signal acquisition unit 11 and acquires the utterance content including an operation instruction for the in-vehicle device 2, the operation determination unit 12 may estimate that the operation input device 3 mentioned in the occupant's utterance content (i.e., the operation input device 3 used to operate the in-vehicle device 2 to be operated) is the operation input device 3 that output the input operation signal. Then, the operation determination unit 12 outputs a control signal to the device control unit 14 to operate the in-vehicle device 2 in accordance with the operation instruction in the utterance content. The device control unit 14 controls the in-vehicle device 2 in accordance with the control signal from the operation determination unit 12.

[0059] For example, when the operation determination unit 12 acquires an utterance content including an operation instruction for the in-vehicle device 2 after outputting a guidance message regarding the operation input device 3 that output the input operation signal as described above, the operation determination unit 12 may estimate that the operation input device 3 mentioned in the utterance content including the operation instruction is the operation input device 3 of the guidance message. The in-vehicle device operated by the operation input device 3 may then be operated according to the operation instruction in the utterance content. For example, assume that the in-vehicle device 2 is an interior light, the operation input device 3 is an interior light switch, and the occupant operates the interior light switch. In response to the utterance content "What is this switch?", the guidance message "This is the interior light switch. You can operate them by sliding them to the left for off, the center for door interlocking, and the right for on" is output as described above, and then the occupant utters "Set this to on." Then, the operation determination unit 12 may estimate that the operation input device 3 mentioned in the utterance content including the operation instruction is the interior light switch and that the in-vehicle device 2 to be operated is the interior light, and control the interior light to be on.

[0060] Second Embodiment In the first embodiment, when an utterance including a question about the operation input device 3 is acquired after an input operation signal is acquired, the operation input device 3 mentioned in the occupant's utterance is estimated, and a guidance message about the estimated operation input device 3 is output. In contrast, in the second embodiment, when an input operation signal is acquired after an utterance including a question about the operation input device 3 is acquired, the operation input device 3 mentioned in the occupant's utterance is estimated, and a guidance message about the estimated operation input device 3 is output. In the voice recognition unit 10 of the second embodiment as well, the voice input of a wake-up word, the input of a dedicated voice command for receiving a question (for example, "I want to ask about the switch?"), and the operation of the PTT switch 5 may be detected as a voice recognition start event.

[0061] Alternatively, the voice recognition unit 10 of the second embodiment may constantly recognize voice input from the occupant acquired by the microphone 8, analyze the speech content by natural language processing, and determine whether a question regarding the operation input device 3 (e.g., "What's this switch?", "Which switch does XX?", "Where is the switch for XX?", "Is this switch XX correct?", or "I want to do XX, is this switch okay?") has been input. When a question regarding the operation input device 3 is input, the operation determination unit 12 transitions to a standby mode in which the input operation signal acquisition unit 11 monitors acquisition of an input operation signal. When an input operation signal is acquired in the standby mode, the operation determination unit 12 estimates the operation input device 3 mentioned in the speech content of the occupant. The response generation unit 13 outputs a guidance message regarding the estimated operation input device 3.

[0062] 6 is a flowchart of an example of a voice recognition method according to the second embodiment. The processes of steps S40 to S42 are the same as the processes of steps S1 to S3 in FIG. 1. If a voice recognition start event occurs (step S40: Y), the process proceeds to step S43. In step S43, the operation determination unit 12 determines whether or not the occupant's speech content has been acquired. If the occupant's speech content has been acquired (step S43: Y), the process proceeds to step S44. If the occupant's speech content has not been acquired (step S43: N), the process proceeds to step S45.

[0063] In step S44, the input operation signal acquisition unit 11 determines whether or not an input operation signal has been acquired. If an input operation signal has been acquired (step S44: Y), the process proceeds to step S46. If an input operation signal has not been acquired (step S44: N), the process proceeds to step S45. In step S45, the response generation unit 13 outputs an end guide message. Thereafter, the process proceeds to step S51. The processes in steps S46 to S51 are the same as steps S7 to S12 in FIG. 3.

[0064] (Effects of the embodiment) (1) In the speech recognition method, the speech content of the occupant of the vehicle 1 is acquired, an input operation signal generated by the occupant operating the operation input device 3 of the vehicle 1 is acquired, and a target component that is a component mentioned in the speech content among multiple components that make up the vehicle 1 is estimated based on the speech content and the input operation signal, and information about the target component is output. This makes it possible to notify the occupant of information about the operation input device 3 that accepts operation inputs from the occupant to the in-vehicle equipment 2.

[0065] (2) For example, the utterance content may be acquired after the input operation signal is acquired. This allows the operation input device 3 that generated the input operation signal to be estimated as the target component. (3) For example, if the utterance content is not acquired even after a predetermined time has elapsed after the input operation signal is acquired, the in-vehicle device 2 may be controlled in accordance with the input operation signal. This allows the in-vehicle device 2 to be controlled in the same way as if the operation input device 3 were operated normally when the utterance content is not acquired.

[0066] (4) For example, it may be possible to determine whether a voice recognition process for acquiring the contents of an occupant's speech has started, and if an input operation signal has been acquired before the voice recognition process has started, to control the in-vehicle device 2 in accordance with the input operation signal without outputting information about the target component. In this way, if the voice recognition process has not started, the in-vehicle device 2 can be controlled in the same way as if the operation input device 3 had been operated normally.

[0067] (5) For example, it may be possible to determine whether a voice recognition process for acquiring the contents of an occupant's speech has started, and if an input operation signal is acquired before the voice recognition process starts, to control an in-vehicle device in accordance with the input operation signal and output information about the target component. In this way, even if the voice recognition process is started based on the operation of the operation input device 3, it is possible to achieve both the control of the in-vehicle device 2 and the voice recognition process.

[0068] (6) For example, the input operation signal may be acquired after acquiring the speech content. This allows the operation input device 3 that generated the input operation signal to be estimated as the target component. (7) For example, if an utterance by an occupant or an operation of the operation input device is detected while information about the target component is being output, the output of information about the target component may be interrupted and control of the in-vehicle device 2 may be executed in accordance with the input operation signal. This allows control of the in-vehicle device 2 to be started immediately when the information about the target component is no longer needed.

[0069] (8) For example, the target component may be the operation input device 3. For example, the target component may be a switch, lever, dial, knob, slide bar, or touch panel. This makes it possible to notify the occupant of information about the operation input device 3. (9) For example, it may be determined whether the content of the utterance is a question about the name, method of use, or purpose, and if it is determined that the content of the utterance is a question about the name, method of use, or purpose, the name, method of use, or purpose of the target component may be output as information about the target component. This makes it possible to notify the occupant of the name, method of use, or purpose of the operation input device 3. (10) For example, it may be possible to output a sound or an image representing information about the target component. This makes it possible to notify the occupant of information about the operation input device 3.

[0070] 1...vehicle, 2...in-vehicle equipment, 3...operation input device, 4...voice recognition device, 5...push-to-talk switch, 6...speaker, 7...display device, 8...microphone, 9...controller, 9a...processor, 9b...storage device, 10...voice recognition unit, 11...input operation signal acquisition unit, 12...action determination unit, 13...response generation unit, 14...equipment control unit

Claims

1. Acquire the speech content of the vehicle occupant, acquiring an input operation signal generated by the occupant operating an operation input device of the vehicle; estimating a target component that is a component mentioned in the utterance content among a plurality of components that configure the vehicle based on the utterance content and the input operation signal; Outputting information about the target configuration; A voice recognition method characterized by, when an utterance by the occupant or an operation of the operation input device is detected while information about the target structure is being output, interrupting the output of the information about the target structure and executing control of an in-vehicle device in accordance with an input operation signal obtained from the operation input device.

2. 2. The speech recognition method according to claim 1, wherein the speech content is acquired after the input operation signal is acquired.

3. 3. The speech recognition method according to claim 2, wherein, when the speech content is not acquired even after a predetermined time has elapsed since the acquisition of the input operation signal, control of an in-vehicle device is executed in accordance with the input operation signal.

4. determining whether a voice recognition process for acquiring the speech of the occupant has been started; When the input operation signal is acquired before the voice recognition process is started, control of an in-vehicle device in accordance with the input operation signal is executed without outputting information about the target configuration.

3. The speech recognition method according to claim 2.

5. determining whether a voice recognition process for acquiring the speech of the occupant has been started; when the input operation signal is acquired before the voice recognition process is started, the control of an in-vehicle device is executed in response to the input operation signal, and information regarding the target component is output.

3. The speech recognition method according to claim 2.

6. 2. The speech recognition method according to claim 1, wherein the input operation signal is acquired after the speech content is acquired.

7. 2. The speech recognition method according to claim 1, wherein the target component is the operation input device.

8. 2. The speech recognition method according to claim 1, wherein the target component is a switch, a lever, a dial, a knob, a slide bar, or a touch panel.

9. determining whether the spoken content is a question about a name, a method of use, or an application; When it is determined that the content of the utterance is a question regarding a name, a method of use, or an application, the name, the method of use, or the application of the target component is output as information regarding the target component.

2. The speech recognition method according to claim 1.

10. 2. The speech recognition method according to claim 1, further comprising the step of outputting a voice or an image representing information about the target component.

11. A process of acquiring speech content of a vehicle occupant; A process of acquiring an input operation signal generated by an occupant operating an operation input device of the vehicle; A process of estimating a target component, which is a component mentioned in the utterance content, among a plurality of components constituting the vehicle, based on the utterance content and the input operation signal; outputting information about the target configuration; a process of interrupting the output of the information about the target configuration and controlling an in-vehicle device according to an input operation signal acquired from the operation input device when an utterance by the occupant or an operation of the operation input device is detected during the output of the information about the target configuration; A speech recognition device comprising: a controller that executes the above-mentioned steps.