Control device and method of operating the control device

Abstract

To improve user convenience. [Solution] The control device includes an input unit that receives instructions from the user through speech, a display unit that displays information to the user, and a control unit that, when in a first operating mode in which the acceptance of instructions from the user through speech is stopped, receives a predetermined instruction to switch to a second operating mode in which instructions from the user through speech are received, causes the control device to display first information prompting the user to speak and second information other than the first information on the display unit in different manners.

Description

Technical Field

[0001] The present disclosure relates to a control device and an operation method of the control device.

Background Art

[0002] There is known an in-vehicle device having an agent function that recognizes requests included in the speech of a vehicle occupant and provides a response or service according to the request. For example, Patent Document 1 discloses an in-vehicle device that identifies an instruction including a so-called wake-up word for activating the agent function.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a control device for a vehicle that provides a service according to speech, there is room for improving user convenience.

[0005] Hereinafter, a control device and the like that enable improvement of user convenience will be disclosed.

Means for Solving the Problems

[0006] The control device in the present disclosure includes an input unit that receives an instruction by the user's speech, a display unit that displays information to the user, and a control unit that causes the display unit to display first information that prompts the user to speak and second information other than the first information in different modes when receiving a predetermined instruction to shift to a second operation mode in which an instruction by the user's speech is received in a first operation mode in which reception of an instruction by the user's speech is stopped. It has.

[0007] Another aspect of the present disclosure is a method for operating a control device having an input unit for receiving instructions by a user's utterance and a display unit for displaying information to the user, wherein when in a first operating mode in which the reception of instructions by the user's utterance is stopped, the control device receives a predetermined instruction to transition to a second operating mode in which instructions by the user's utterance are received, and the control device displays first information prompting the user to utter and second information other than the first information in different manners on the display unit. [Effects of the Invention]

[0008] The control devices and other devices described in this disclosure will enable improved user convenience. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing an example of the configuration of an information provision system. [Figure 2] This is a diagram showing an example of the configuration of a control device. [Figure 3] This is a flowchart illustrating an example of the operation procedure of a control device. [Figure 4A] This is an example of information display in a control device. [Figure 4B] This is an example of information display in a control device. [Figure 5A] This is an example of information display in a control device. [Figure 5B] This is an example of information display in a control device. [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the drawings.

[0011] Figure 1 shows an example configuration of a vehicle control system in one embodiment. The vehicle control system 1 has one or more server devices 10 and in-vehicle devices 12 mounted on a vehicle 13, each connected to the other via a network 11 for information communication. The server devices 10 are, for example, one or more server computers belonging to a cloud computing system or other computing system, which function as servers implementing various functions. The vehicle 13 is a passenger car, commercial vehicle, etc., and is an internal combustion engine vehicle, hybrid vehicle (HEV; Hybrid Electric Vehicle), plug-in hybrid vehicle (PHEV; Plug-in Hybrid Electric Vehicle), etc. The in-vehicle devices 12 have computers with communication functions and information processing functions, and provide various information or control the operation of the vehicle 13 in response to the operation of the user of the vehicle 13. The user may be the driver of the vehicle 13 or a passenger. The network 11 includes, for example, a mobile communication network, the Internet, an ad hoc network, a LAN (Local Area Network), a MAN (Metropolitan Area Network), or other networks or any combination thereof.

[0012] In this embodiment, the in-vehicle device 12, acting as a "control device," receives instructions from the user of the vehicle 13 through speech or operation, and, while communicating with the server device 10 as appropriate, provides the user with various information through voice output or image display, and controls equipment such as the vehicle 13's sound system, air conditioning, windows, sunroof, and seats, thereby providing convenience to the user. When the in-vehicle device 12 is in a first operating mode (hereinafter referred to as the standby mode) in which it has stopped receiving instructions from the user through speech, and receives a predetermined instruction (hereinafter referred to as the transition instruction) to transition to a second operating mode (hereinafter referred to as the speech-responsive mode) in which it accepts instructions from the user through speech, it displays first information prompting the user to speak (hereinafter referred to as speech-responsive information) and second information other than the speech-responsive information (hereinafter referred to as operation-responsive information) in different manners. Here, the transition instruction is the user uttering a predetermined wake-up word or performing a predetermined operation on a physical switch, etc. The switch is provided, for example, on the steering wheel, the dashboard, etc., and receives operations such as pressing or pressing. Furthermore, the standby mode is an operating mode in which, in addition to transition instructions, the device accepts instructions from the user through operations on the touch panel, physical switches, etc., and provides information and control according to the instructions. When the in-vehicle device 12 receives a transition instruction while in standby mode, it displays or hides operation-response information in a different manner from voice-response information, for example, using a color or brightness that is less visible than that of voice-response information, and then transitions to voice-response mode. Therefore, it becomes easier for the user to understand that they should speak according to the voice-response mode, and unnecessary confusion for the user can be avoided. In voice-response mode, the in-vehicle device 12 accepts instructions by voice in addition to operations by the user, and provides information and control according to the instructions. In this way, according to this embodiment, it is possible to improve user convenience.

[0013] Figure 2 shows an example configuration of the in-vehicle device 12. The in-vehicle device 12 includes a communication unit 121, a storage unit 122, a control unit 123, a positioning unit 124, an input unit 125, and an output unit 126. These may be configured as a single control device, or as two or more control devices, or as a control device and other devices such as a communication device. The control device includes, for example, an ECU (Electronic Control Unit). The communication device includes, for example, a DCM (Data Communication Module). Each part is connected to each other or to the equipment of the vehicle 13 via an in-vehicle network compliant with standards such as CAN (Controller Area Network) to enable information communication. The in-vehicle device 12 may also include an information processing device such as a smartphone or tablet terminal as part of its configuration.

[0014] The communication unit 121 includes modules that support mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), and modules that support in-vehicle LANs such as CAN. The in-vehicle device 12 is connected to the network 11 via the communication unit 121 through a nearby router device or a mobile communication base station, and communicates information with other devices via the network 11, or communicates information with various parts of the vehicle 13 via the in-vehicle LAN.

[0015] The storage unit 122 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. Semiconductor memories are, for example, RAM (Random Access Memory) or ROM (Read Only Memory). RAM is, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). ROM is, for example, EEPROM (Electrically Erasable Programmable ROM). The storage unit 122 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 122 stores information used in the operation of the control unit 123 and information obtained by the operation of the control unit 123.

[0016] The control unit 123 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit), or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like. While controlling each part of the in-vehicle device 12, the control unit 123 executes information processing related to the operation of the in-vehicle device 12.

[0017] The functions of the control unit 123 are realized by executing a control / processing program on the processor included in the control unit 123. The control / processing program is a program for causing a computer to execute the processing of the steps included in the operation of the control unit 123, so as to cause the computer to realize the functions corresponding to the processing of those steps. That is, the control / processing program is a program for causing a computer to function as the control unit 123. Also, some or all of the functions of the control unit 123 may be realized by the dedicated circuit included in the control unit 123.

[0018] The positioning unit 124 includes one or more GNSS (Global Navigation Satellite System) receivers. The GNSS includes, for example, at least any one of GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System), BeiDou, GLONASS (Global Navigation Satellite System), and Galileo. The positioning unit 124 sends the positioning result to the control unit 123, and the position information of the in-vehicle device 12 is obtained by the control unit 123.

[0019] The input unit 125 includes one or more input interfaces. The input interfaces include, for example, a microphone that receives voice input from the user's speech, physical keys, capacitive keys, pointing devices, a touch screen provided integrally with a display, and the like. Further, the input interface includes an interface with various input devices arranged inside the vehicle. The various input devices include a camera that images the inside of the vehicle and a physical switch for shift instructions. The switch for shift instructions is arranged on a console panel, a steering wheel, or the like. The input unit 125 receives operations for inputting various information including the user's spoken voice, and sends the input information to the control unit 123 or sends the captured image to the control unit 123.

[0020] The output unit 126 includes one or more output interfaces. The output interfaces include, for example, a speaker and a display. The display of the output unit 126 corresponds to the "display unit" of the present embodiment. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, and is installed on, for example, a dashboard or the like that is easily visible to the user in the driver's seat or the passenger seat, or at any location that is easily visible to the user in the rear seat. Further, the output interface includes an interface for sending control signals to each equipment such as the air conditioner, window, sunroof, seat, etc. of the vehicle 13. The output unit 126 outputs the information obtained by the operation of the control unit 123 by voice or displays it by an image.

[0021] FIG. 3 is a flowchart for explaining the operation procedure of the in-vehicle device 12. Each step in FIG. 3 is a step of information processing executed by the control unit 123 of the in-vehicle device 12.

[0022] The procedure shown in Figure 3 is executed when the control unit 123 receives a transition instruction. The transition instruction is made by the user uttering a wake-up word, such as a pre-set phrase like "Activate dialogue mode" or "Hey, XX" (where XX is the manufacturer name or system name of the vehicle 13). Alternatively, the transition instruction is made by the user operating a switch for transition instructions. When the input unit 125 sends the voice data of the wake-up word uttered via the microphone to the control unit 123, the control unit 123 obtains the transition instruction by detecting the wake-up word from the uttered voice through voice recognition processing. Also, when the input unit 125 receives input from an operation on the switch for transition instructions and sends the received information to the control unit 123, the control unit 123 obtains the transition instruction by receiving the information from the control unit 123. Upon obtaining the transition instruction, the control unit 123 stores the transition instruction information corresponding to the transition instruction in the storage unit 122 and starts the procedure shown in Figure 3.

[0023] In S31, the control unit 123 acquires transition instruction information. The transition instruction information includes information for determining whether the transition instruction was made by a wake-up word or by operating a switch for transition instructions. The control unit 123 reads and acquires the transition instruction information stored in the memory unit 122.

[0024] In S32, the control unit 123 determines whether or not the system is currently in standby mode. The control unit 123 determines whether or not the system is in standby mode by referring to the information indicating the operating mode stored in the memory unit 122. If the system is in standby mode (Yes in step S32), the control unit 123 proceeds to step S34. If the system is not in standby mode, i.e., in speech response mode (No in step S32), the control unit 123 proceeds to step S36.

[0025] In S33, the control unit 123 determines whether or not to display operation-corresponding information. The control unit 123 determines whether or not to display operation-corresponding information on the display of the output unit 126. The control unit 123 displays information on the display corresponding to the user's operation on the touch panel or various physical switches integrated with the display. The display of the output unit 126 displays, for example, screen 40a as shown in Figure 4A, or screen 40b as shown in Figure 4B.

[0026] Screen 40a in Figure 4A includes a map for providing navigation. Screen 40a includes notifications 41 and 42 corresponding to operation-dependent information and objects 43 and 44 corresponding to voice-dependent information, which are displayed superimposed on the map. Notifications 41 and 42 are notifications corresponding to various events that occur when providing navigation, and prompt the user to perform tap operations or other actions to activate actions related to each notification. For example, notification 41 is a banner that notifies the communication unit 121 of an incoming call, and when the user taps notification 41, the control unit 123 activates hands-free calling. Notification 42 is a push notification displayed by the control unit 123 in response to the communication unit 121 receiving information indicating congestion due to vehicle-to-vehicle communication or the distribution of road traffic information, and when the user taps notification 42, the control unit 123 displays further details of the congestion information. Furthermore, object 43 is a pop-up notification object in response to the congestion information notification 42, in which the control unit 123 proposes an alternative route to the user, and is an object that prompts the user to give instructions by voice. Object 44 is an icon object that prompts user instructions through verbal utterances by indicating that it is capable of receiving user utterances.

[0027] Screen 40b in Figure 4B is a menu screen for providing users with various services, including multimedia. Screen 40b includes object 46 corresponding to operation information and objects 45 and 47 corresponding to voice response information. Object 46 is an object that prompts the user to tap to launch interface screens such as maps, messages, and function settings. Object 45 is an object that prompts the user to give voice commands to launch phone functions and various audio applications. Furthermore, object 47 is an icon object that prompts the user to give voice commands by indicating that it is possible to receive user voice commands.

[0028] Returning to Figure 3, in step S33, if the control unit 123 displays operation response information (Yes in step S33), it proceeds to step S34; if it does not display operation response information (No in step S33), it bypasses step S34 and proceeds to step S35.

[0029] In S34, the control unit 123 changes the display mode of the operation response information. The control unit 123 changes the display mode of the operation response information displayed on the output unit 126's display to a mode different from the display mode of the dialogue response information. For example, the control unit 123 displays the operation response information in a color or brightness that is less visible than the speech response information, or hides it. Examples of changes in the display mode of the operation response information are shown in Figures 5A and 5B.

[0030] Figure 5A shows an example of changing the display mode of notifications 41 and 42 corresponding to operation response information in the screen 40a shown in Figure 4A. For example, the control unit 123 displays notifications 41 on the display of the output unit 126 in a manner that reduces visibility by masking, reducing brightness, etc., or hides notifications 42, thereby displaying them in a different manner from objects 43 and 44 corresponding to speech response information. The control unit 123 may reduce the visibility of notifications 41 and 42 corresponding to operation response information, while displaying objects 44 corresponding to dialogue response information in a more conspicuous manner, such as by increasing their brightness. Alternatively, the control unit 123 may make new objects corresponding to dialogue response information appear.

[0031] Figure 5B shows an example of changing the display mode of object 46 corresponding to operation-related information in the screen 40b shown in Figure 4B. For example, the control unit 123 displays object 46 on the output unit 126's display in a manner that reduces visibility by masking, reducing brightness, etc., or hides it, thereby displaying it in a different manner from objects 45 and 47 corresponding to speech-related information. The control unit 123 may reduce the visibility of object 46 corresponding to operation-related information, while displaying object 47 corresponding to dialogue-related information in a more conspicuous manner, such as by increasing its brightness. Alternatively, the control unit 123 may make a new object corresponding to dialogue-related information appear.

[0032] Figures 5A and 5B show examples where some of the operation-related information is hidden to reduce the visibility of others, but it is also possible to hide all of the operation-related information or reduce its visibility.

[0033] Returning to Figure 3, in step S35, the control unit 123 switches to speech response mode. In speech response mode, the control unit 123 converts the spoken audio acquired by the input unit 125 into text using speech recognition processing, analyzes the syntax of the converted spoken content, and extracts the user's instructions. Hereinafter, the user's instructions also include the user's requests.

[0034] On the other hand, in step S36, the control unit 123 determines whether the transition instruction was made by a switch operation. In this case, the control unit 123 is operating in speech-responsive mode. The control unit 123 refers to the movement instruction information obtained from the storage unit 122 and determines whether the movement instruction was made by a switch operation or by uttering a wake-up word. If the transition instruction was made by a switch operation (Yes in step S36), the control unit 123 proceeds to step S37. If the transition instruction was not made by a switch operation, that is, by uttering a wake-up word (No in step S36), the control unit 123 proceeds to step S38.

[0035] In S37, the control unit 123 decides to maintain the display mode of the operation response information. In this case, the control unit 123 has already transitioned to the voice response mode, and the display mode of the operation response information was changed in step S34 of the processing cycle when transitioning to the voice response mode, so it maintains the display mode of the operation response information after the change. The control unit 123 then continues to accept instructions by voice in the voice response mode.

[0036] On the other hand, in S38, the control unit 123 processes the wake-up word as an utterance. In this case, since the control unit 123 has already entered utterance-responsive mode, it processes the wake-up word as part of the utterance and continues to accept further utterance-based instructions.

[0037] After S35, S37, or S38, the control unit 123 terminates the procedure shown in Figure 3 and continues operation in speech-responsive mode.

[0038] In speech-responsive mode, the control unit 123 provides services corresponding to the user's speech. The control unit 123 transcribes the user's speech into text and performs syntactic analysis of the text to extract instructions from the user's speech. The control unit 123 may also use a Large Language Model (LLM) to extract user instructions from the speech text. The control unit 123 then provides various information or controls the operation of various equipment in the vehicle 13 according to the user's instructions. For example, if the user requests traffic information, weather information, or local information, the control unit 123 acquires the necessary information through vehicle-to-vehicle communication, information distribution from the server device 10, etc., and presents it to the user through voice output or image display. Also, if the user instructs the activation of multimedia functions such as telephone or audio, the control unit 123 activates the application or function corresponding to the instruction. Furthermore, if the user instructs the adjustment of volume, air conditioning, opening or closing windows or sunroofs, or seat displacement, the control unit 123 sends information to control the equipment or actuators that perform the action corresponding to the instruction.

[0039] In speech-responsive mode, some of the operations of the control unit 123 may be performed by the server device 10. For example, the control unit 123 may send spoken audio or text data generated from spoken audio to the server device 10, and when the server device 10 extracts the user's instructions using LLM, it may receive information indicating the user's instructions from the server device 10. The control unit 123 then provides information and performs control according to the user's instructions.

[0040] In this way, when the in-vehicle device 12 receives a transition command while in standby mode, it displays operation response information in a different manner from the voice response information, making it easier for the user to understand that they should speak according to the voice response mode, thus avoiding unnecessary confusion for the user. Therefore, user convenience can be improved.

[0041] As described above, embodiments have been explained based on various drawings and examples, but it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each means, each step, etc., can be rearranged in a logically consistent manner, and multiple means, steps, etc., can be combined into one or divided.

[0042] Some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Note 1] An input unit that accepts instructions from the user via speech, A display unit that displays information to the user, When in a first operating mode in which the acceptance of user utterances is stopped, if a predetermined instruction is received to switch to a second operating mode in which user utterances are accepted, the control unit causes the display unit to display first information prompting the user to utter and second information other than the first information in different manners. A control device having [Note 2] In Appendix 1, When the control unit receives the predetermined instruction, it further transitions to the second operating mode. Control device. [Note 3] In Appendix 1 or 2, In the second operating mode, the control unit performs actions in accordance with the user's instructions extracted from the utterance. Control device. [Note 4] In any of the appendices 1 to 3, The aforementioned predetermined instruction is an instruction given by uttering a predetermined phrase or by performing a predetermined operation. Control device. [Note 5] In any of the appendices 1 to 4, The control unit causes the display unit to display the second information as an object superimposed on other information. Control device. [Note 6] In any of the appendices 1 to 5, The manner in which the second information is displayed includes reducing brightness and not displaying it. Control device. [Note 7] A method for operating a control device having an input unit that receives instructions from a user through speech, and a display unit that displays information to the user, When in a first operating mode in which the acceptance of user utterances is stopped, if a predetermined instruction is received to switch to a second operating mode in which user utterances are accepted, the system includes displaying first information prompting the user to utter and second information other than the first information on the display unit in different manners. The method of operation of the control device. [Note 8] In Appendix 7, Upon receiving the aforementioned predetermined instruction, the process further includes transitioning to the second operating mode, The method of operation of the control device. [Note 9] In Appendix 7 or 8, The second operating mode includes performing actions in response to user instructions extracted from the utterance, The method of operation of the control device. [Note 10] In any of the appendices 7 to 9, The aforementioned predetermined instruction is an instruction given by uttering a predetermined phrase or by performing a predetermined operation. The method of operation of the control device. [Note 11] In any of the appendices 7 to 10, The display unit includes displaying the second information as an object superimposed on other information, The method of operation of the control device. [Note 12] In any of the appendices 7 to 11, The manner in which the second information is displayed includes reducing brightness and not displaying it. The method of operation of the control device. [Explanation of Symbols]

[0043] 1. Information Provision System 10 Server devices 11 Network 12 Onboard equipment 13 vehicles 121 Communications Department 122 Storage section 123 Control Unit 124 Positioning Unit 125 Input section 126 Output section

Claims

1. An input unit that receives instructions via the user's spoken voice or operation of a physical switch, A display unit that displays information to the user, When in the first operating mode, which has stopped accepting instructions from the user's utterance, the control unit causes the display unit to display or hide second information other than the first information prompting the user to speak, at a lower brightness than the first information, when it receives an instruction to switch to the second operating mode, which accepts instructions from the user's utterance, by uttering a predetermined word or performing a predetermined operation. A control device having

2. An input unit that accepts instructions from the user's voice, A display unit that displays information to the user, When in a first operating mode in which the acceptance of user utterances is stopped, if a predetermined instruction is received to switch to a second operating mode in which user utterances are accepted, the control unit causes the display unit to display first information prompting the user to utter and second information other than the first information in different manners. A control device having

3. In claim 2, Upon receiving the predetermined instruction, the control unit further transitions to the second operating mode. Control device.

4. In claim 3, In the second operating mode, the control unit performs actions in accordance with the user's instructions extracted from the utterance. Control device.

5. In claim 2, The aforementioned prescribed instructions are instructions given by uttering a prescribed phrase or by performing a prescribed operation. Control device.

6. In claim 2, The control unit causes the display unit to display the second information as an object superimposed on other information. Control device.

7. In claim 2, The modes of displaying the second information include reducing brightness and not displaying it. Control device.

8. A method for operating a control device having an input unit that receives instructions from a user's speech and a display unit that displays information to the user, When in a first operating mode in which the acceptance of user utterances is stopped, if a predetermined instruction is received to switch to a second operating mode in which user utterances are accepted, the system includes displaying first information prompting the user to utter and second information other than the first information on the display unit in different manners. The method of operation of the control device.

9. In claim 8, Upon receiving the aforementioned predetermined instruction, the process further includes transitioning to the second operating mode, The method of operation of the control device.

10. In claim 9, The second operating mode includes performing actions in response to user instructions extracted from the utterance, The method of operation of the control device.

11. In claim 8, The aforementioned prescribed instructions are instructions given by uttering a prescribed phrase or by performing a prescribed operation. The method of operation of the control device.

12. In claim 8, The display unit includes displaying the second information as an object superimposed on other information, The method of operation of the control device.

13. In claim 8, The modes of displaying the second information include reducing brightness and not displaying it. The method of operation of the control device.

Images