Control device and method of operation of a control device

Abstract

The present disclosure relates to a control device and a method of operation of a control device, which improves the convenience of a user. The control device has an input section that receives an instruction by speech from a user, a display section that displays information to the user, and a control section that, when a first operation mode that receives an instruction by speech from the user is stopped, causes first information that prompts the user to make speech and second information other than the first information to be displayed in different styles from each other on the display section if a predetermined instruction for shifting to a second operation mode is accepted, the second operation mode being a mode that receives an instruction by speech from the user.

Description

Technical Field

[0001] This disclosure relates to a control device and a method of operating the control device. Background Technology

[0002] Vehicle-mounted devices are known to have the function of recognizing requests contained in the speech of vehicle occupants and providing responses or services corresponding to those requests. For example, Patent Document 1 discloses a vehicle-mounted device that recognizes instructions, including a so-called wake-up word used to activate the agent function.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2022 / 254669 Summary of the Invention

[0006] There is room to improve user convenience in the control devices of vehicles that provide services corresponding to speech.

[0007] The following describes control devices and other devices that can improve user convenience.

[0008] This disclosure provides a control device having:

[0009] The input section is responsible for receiving instructions from the user via speech.

[0010] The display unit displays information to the user; and

[0011] When the control unit stops the first action mode based on the user's verbal instruction, if it receives a predetermined instruction to switch to the second action mode, it displays a first message urging the user to speak and a second message other than the first message on the display unit in different styles. The second action mode is the mode based on the user's verbal instruction.

[0012] Another aspect of this disclosure provides a method for operating a control device, the control device having: an input unit for receiving instructions given by a user through speech; and a display unit for displaying information to the user, wherein the method of operating the control device includes: when stopping a first operation mode given by the user through speech, if a predetermined instruction for switching to a second operation mode is received, then displaying a first piece of information urging the user to speak and a second piece of information other than the first piece of information in different styles on the display unit, the second operation mode being the mode given by the user through speech.

[0013] The control device and the like disclosed herein can improve user convenience. Attached Figure Description

[0014] Figure 1 This is a diagram illustrating an example of the structure of an information provision system.

[0015] Figure 2 This is a diagram showing an example of the structure of a control device.

[0016] Figure 3 This is a flowchart illustrating an example of the operation process of a control device.

[0017] Figure 4A This is an example of information display in a control device.

[0018] Figure 4B This is an example of information display in a control device.

[0019] Figure 5A This is an example of information display in a control device.

[0020] Figure 5B This is an example of information display in a control device. Detailed Implementation

[0021] The embodiments are described below with reference to the accompanying drawings.

[0022] Figure 1 This diagram illustrates a structural example of a vehicle control system in one embodiment. The vehicle control system 1 includes one or more server devices 10 interconnected via a network 11, and onboard devices 12 mounted on a vehicle 13. The server devices 10 are, for example, cloud computing systems or other computing systems, and function as servers implementing various functions. The vehicle 13 is a passenger car, commercial vehicle, etc., including internal combustion engine vehicles, hybrid electric vehicles (HEVs), plug-in hybrid electric vehicles (PHEVs), etc. The onboard devices 12 have computers with communication and information processing functions, providing various information or controlling the actions of the vehicle 13 based on the operations of the user of the vehicle 13. The user can be either the driver or a passenger of the vehicle 13. The network 11 includes, for example, mobile communication networks, the Internet, ad hoc networks, LANs (Local Area Networks), MANs (Metropolitan Area Networks), or other networks or any combination thereof.

[0023] In this embodiment, the vehicle-mounted device 12, acting as a "control device," receives instructions from the user of the vehicle 13 via speech or operation. While appropriately communicating with the server device 10, it provides various information to the user through sound output or image display, or controls the vehicle 13's audio system, air conditioning, windows, sunroof, seats, and other equipment, thereby providing convenience to the user. When the vehicle-mounted device 12 stops operating in the first mode (hereinafter referred to as the waiting mode) based on the user's speech, if it receives a predetermined instruction (hereinafter referred to as the transfer instruction) to transfer to the second mode (hereinafter referred to as the speech response mode) based on the user's speech, it displays a first message urging the user to perform a speech (hereinafter referred to as the speech response information) and a second message other than the speech response information (hereinafter referred to as the operation response information) in different styles. Here, the transfer instruction is a predetermined wake-up word spoken by the user, or a predetermined operation for a physical switch, etc. Switches are provided, for example, on the steering wheel, dashboard, etc., and accept operations such as pressing or pushing. Furthermore, the waiting mode is an operation mode that, in addition to transfer instructions, also provides and controls information corresponding to user instructions made through operations such as touch panels or physical switches. If the vehicle device 12 receives a transfer instruction while in waiting mode, it displays the operation-related information in a different style than the speech-related information—for example, using a color or brightness that is harder to visually discern than the speech-related information, or not displaying it at all—and then transfers to the speech-related mode. Therefore, it is easier for the user to understand when to use the speech corresponding to the speech-related mode, avoiding unnecessary confusion for the user. Moreover, in the speech-related mode, the vehicle device 12, in addition to user-made operations, also accepts instructions made through speech, providing and controlling information corresponding to those instructions. Thus, according to this embodiment, user convenience is improved.

[0024] Figure 2 An example of the structure of the vehicle-mounted device 12 is shown. The vehicle-mounted device 12 includes a communication unit 121, a storage unit 122, a control unit 123, a positioning unit 124, an input unit 125, and an output unit 126. These can be configured as a single control device, or as two or more control devices, or as a control device and other devices such as a communication unit. Control devices include, for example, ECUs (Electronic Control Units). Communication units include, for example, DCMs (Data Communication Modules). Each component is interconnected via an in-vehicle network according to standards such as CAN (Controller Area Network) or can communicate with the equipment of the vehicle 13. Furthermore, the vehicle-mounted device 12 can also be configured to include an information processing device such as a smartphone or tablet in one part.

[0025] The communication unit 121 includes modules corresponding to mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), and modules corresponding to in-vehicle LANs such as CAN. The vehicle-mounted device 12 uses the communication unit 121 to connect to the network 11 via a nearby router or mobile communication base station and communicates with other devices via the network 11, or communicates with various parts of the vehicle 13 via the in-vehicle LAN.

[0026] The storage unit 122 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them. Semiconductor memories are, for example, RAM (Random Access Memory) or ROM (Read Only Memory). RAM is, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). ROM is, for example, EEPROM (Electrically Erasable Programmable ROM). The storage unit 122 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 122 stores information used in the operation of the control unit 123 and information obtained through the operation of the control unit 123.

[0027] The control unit 123 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a dedicated processor such as a GPU (Graphics Processing Unit) specifically designed for particular processing. Dedicated circuits include, for example, FPGAs (Field-Programmable Gate Arrays) and ASICs (Application-Specific Integrated Circuits). While controlling the various parts of the vehicle-mounted device 12, the control unit 123 performs information processing related to the operation of the vehicle-mounted device 12.

[0028] The functions of the control unit 123 are implemented by a processor included in the control unit 123 executing a control / processing program. The control / processing program is a program that enables the computer to perform the functions corresponding to the steps in the operation of the control unit 123. In other words, the control / processing program is a program that enables the computer to function as the control unit 123. Furthermore, some or all of the functions of the control unit 123 may also be implemented using dedicated circuitry included in the control unit 123.

[0029] The positioning unit 124 includes one or more GNSS (Global Navigation Satellite System) receivers. Among GNSS receivers, this includes, for example, at least one of GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System), BeiDou, GLONASS (Global Navigation Satellite System), and Galileo. The positioning unit 124 sends the positioning results to the control unit 123, where the control unit 123 calculates the position information of the vehicle-mounted device 12.

[0030] The input unit 125 includes one or more input interfaces. These interfaces may include, for example, a microphone for voice input by the user, physical keys, electrostatic capacitive keys, indicator devices, or a touchscreen integrated with the display. Additionally, the input interfaces may include interfaces with various input devices located within the vehicle. These input devices may include a camera for capturing images of the vehicle interior and physical switches for navigation instructions. These navigation instructions are located on the console panel, steering wheel, etc. The input unit 125 accepts input of various information including the user's voice, and sends the input information to the control unit 123, or sends a camera image to the control unit 123.

[0031] The output unit 126 includes one or more output interfaces. These output interfaces include, for example, a speaker and a display. The display of the output unit 126 corresponds to the "display unit" in this embodiment. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, and is installed in a location easily visible to the driver or front passenger, such as an instrument panel, or in any location easily visible to the rear passenger. Furthermore, the output interfaces include interfaces for sending control signals to various equipment of the vehicle 13, such as the air conditioner, windows, sunroof, and seats. The output unit 126 outputs information obtained from the actions of the control unit 123 via sound or displays information obtained from the actions of the control unit 123 via images.

[0032] Figure 3 This is a flowchart used to explain the operation process of the vehicle-mounted device 12. Figure 3 Each step in the process is an information processing step performed by the control unit 123 of the vehicle-mounted device 12.

[0033] Figure 3 The process is executed when the control unit 123 receives a transfer instruction. The transfer instruction is given via a pre-set phrase, such as a wake-up word issued by the user, for example, "Dialogue mode activated," "Hi," or "00" (where 00 is the manufacturer's name or system name of vehicle 13). Alternatively, the transfer instruction is given by the user's operation of a switch for the transfer instruction. When the input unit 125 sends the voice data of the wake-up word acquired using the microphone to the control unit 123, the control unit 123 obtains the transfer instruction by detecting the wake-up word from the voice using voice recognition processing. Furthermore, when the input unit 125 receives input made through operation of the switch for the transfer instruction and sends the received information to the control unit 123, the control unit 123 obtains the transfer instruction by receiving information from the input unit 125. Upon receiving the transfer instruction, the control unit 123 stores the transfer instruction information corresponding to the transfer instruction in the storage unit 122, thus commencing the transfer instruction process. Figure 3 The process.

[0034] In step S31, the control unit 123 acquires transfer instruction information. This transfer instruction information includes information used to determine whether the transfer instruction was given using a wake-up word or by operating a switch for transfer instruction. The control unit 123 reads and acquires the transfer instruction information stored in the storage unit 122.

[0035] In step S32, the control unit 123 determines whether it is currently in a waiting mode. The control unit 123 refers to the information indicating the action mode stored in the storage unit 122 to determine whether it is in a waiting mode. If it is in a waiting mode ("Yes" in step S32), the control unit 123 proceeds to step S33; if it is not in a waiting mode, that is, in a speech response mode ("No" in step S32), it proceeds to step S36.

[0036] In step S33, the control unit 123 determines whether operation-corresponding information is displayed. The control unit 123 determines whether operation-corresponding information is displayed on the display of the output unit 126. The control unit 123 displays information corresponding to operations performed by the user on the touch panel integrated with the display or various physical switches on the display. For example, the display of the output unit 126 displays... Figure 4A The image shown is 40a, or as shown in the image. Figure 4B The image shown is 40b.

[0037] Figure 4A Screen 40a includes a map for providing navigation. Screen 40a includes notifications 41 and 42, corresponding to operation-related information, and objects 43 and 44, corresponding to speech-related information, respectively overlaid on the map. Notifications 41 and 42 are notifications corresponding to various events occurring during navigation and are notifications that urge the user to perform actions such as tapping to initiate actions associated with each notification. For example, notification 41 is a banner notifying the communication unit 121 of an incoming phone call; when the user taps notification 41, the control unit 123 initiates a hands-free call. Notification 42 is a push notification displayed by the control unit 123 in response to a traffic congestion message received by the communication unit 121 based on vehicle-to-vehicle communication or road traffic information; when the user taps notification 42, the control unit 123 also displays details of the congestion information. Furthermore, object 43 is a pop-up notification used by the control unit 123 to recommend alternative routes to the user in response to the congestion information notification 42, and is an object that urges the user to give instructions verbally. Furthermore, object 44 is an object that prompts the user to give instructions verbally by displaying an icon that can accept the user's speech.

[0038] Figure 4BScreen 40b is a menu screen used to provide users with various services, including multimedia. Screen 40b includes object 46 corresponding to operation information and objects 45 and 47 corresponding to speech information. Object 46 is an object that prompts the user to perform a tap operation to launch interface screens such as maps, messages, and function settings. In addition, object 45 is an object that prompts the user to give instructions via speech for launching telephone functions and various audio apps. Furthermore, object 47 is an object that displays an icon that can accept the user's speech to prompt the user to give instructions via speech.

[0039] Return to Figure 3 In step S33, if the operation-corresponding information is displayed ("Yes" in step S33), the control unit 123 proceeds to step S34; if the operation-corresponding information is not displayed ("No" in step S33), it bypasses step S34 and proceeds to step S35.

[0040] In step S34, the control unit 123 changes the display style of the operation correspondence information. The control unit 123 changes the display style of the operation correspondence information displayed on the output unit 126 to a style different from the display style of the speech correspondence information. For example, the control unit 123 may display the operation correspondence information using a color or brightness that is more visually difficult to discern than the speech correspondence information, or it may not display it at all. Figure 5A , 5B This shows an example of how the display style of the information corresponding to the operation has changed.

[0041] Figure 5A Shown in Figure 4A Examples of changes to the display style of notifications 41 and 42 corresponding to operation-related information in screen 40a shown. For example, the control unit 123 may display notification 41 on the display of the output unit 126 in a style that reduces visual visibility through masking, low brightness, etc., or may hide notification 42, thereby displaying it in a different style from the objects 43 and 44 corresponding to speech-related information. Alternatively, the control unit 123 may reduce the visual visibility of notifications 41 and 42 corresponding to operation-related information, while displaying, for example, objects 44 corresponding to speech-related information, in a more prominent display style, such as increased brightness. Or, the control unit 123 may cause a new object corresponding to speech-related information to appear.

[0042] Figure 5B Shown in Figure 4BExamples of changes to the display style of object 46 corresponding to operation correspondence information in screen 40b shown. For example, the control unit 123 may display object 46 on the display of the output unit 126 in a style that reduces visual discernibility through masking, low brightness, etc., thereby displaying it in a different style from objects 45 and 47 corresponding to speech correspondence information. Alternatively, the control unit 123 may reduce the visual discernibility of object 46 corresponding to operation correspondence information, while displaying, for example, object 47 corresponding to speech correspondence information in a more prominent display style, such as increased brightness. Or, the control unit 123 may cause a new object corresponding to speech correspondence information to appear.

[0043] exist Figure 5A , 5B The example shown is that disabling part of the operation-related information reduces the visual recognizability of other parts, but it is also possible to disable all operation-related information or reduce the visual recognizability of all operation-related information.

[0044] Return to Figure 3 In step S35, the control unit 123 switches to speech response mode. In speech response mode, the control unit 123 performs voice recognition processing to textualize the speech sounds obtained by the input unit 125, parses the syntax of the textualized speech content, and extracts the user's instructions. Hereinafter, the user's instructions also include the user's expectations.

[0045] On the other hand, in step S36, the control unit 123 determines whether a transfer instruction was given via a switch operation. In this case, the control unit 123 operates in speech-responsive mode. The control unit 123 refers to the transfer instruction information obtained from the storage unit 122 to determine whether the transfer instruction was given via a switch operation or via a wake-up word. If the transfer instruction was given via a switch operation ("Yes" in step S36), the control unit 123 proceeds to step S37; if the transfer instruction was not given via a switch operation, i.e., the transfer instruction was given via a wake-up word ("No" in step S36), the control unit 123 proceeds to step S38.

[0046] In step S37, the control unit 123 decides to maintain the display style of the operation response information. In this case, since the control unit 123 has already switched to speech response mode, and the display style of the operation response information was changed in step S34 of the processing loop during the switch to speech response mode, it maintains the changed display style of the operation response information. Furthermore, the control unit 123 continues to process instructions given via speech in speech response mode.

[0047] On the other hand, in S38, the control unit 123 processes the wake word as speech. In this case, since the control unit 123 has switched to speech-response mode, it processes the wake word as part of speech and continues to process more instructions made through speech.

[0048] After S35, S37, or S38, control unit 123 ends. Figure 3 The process continues with actions in the verbal correspondence pattern.

[0049] In speech-response mode, the control unit 123 provides services corresponding to the user's speech content. The control unit 123 performs textualization and syntactic parsing of the user's speech, extracting instructions from the user's speech. The control unit 123 can also use a Large Language Model (LLM) to extract user instructions from the spoken text. Furthermore, the control unit 123 provides various information or controls the actions of various equipment in the vehicle 13 according to the user's instructions. For example, when the user requests traffic information, weather information, or information related to a specific region, the control unit 123 obtains the necessary information through vehicle-to-vehicle communication, information delivery from the server device 10, etc., and provides prompts to the user through sound output or image display. Additionally, when the user instructs the activation of multimedia functions such as telephone or audio, the control unit 123 activates the application or function corresponding to the instruction. Furthermore, when the user instructs the adjustment of volume, air conditioning, opening or closing of windows or sunroof, or seat repositioning, the control unit 123 sends information to control the machines and actuators that perform the actions corresponding to the instruction.

[0050] Part of the operation of the control unit 123 in the speech-response mode can also be performed in the server device 10. For example, the control unit 123 can send speech data or text generated based on speech data to the server device 10, and after the server device 10 extracts the user's instruction through the LLM, it can receive information representing the user's instruction from the server device 10. Furthermore, the control unit 123 provides and controls information corresponding to the user's instruction.

[0051] In this way, if the vehicle-mounted device 12 receives a transfer instruction while in waiting mode, it displays the operation-related information in a different style than the speech-related information. This makes it easy for the user to understand the appropriate speech to be given, avoiding unnecessary confusion. Therefore, it improves user convenience.

[0052] In the foregoing, embodiments have been described with reference to the accompanying drawings. However, it is important to note that those skilled in the art can readily make various modifications and alterations based on this disclosure. Therefore, it is intended that such modifications and alterations are included within the scope of this disclosure. For example, the functions of each unit, step, etc., can be logically reconfigured without contradiction, and multiple units, steps, etc., can be combined into one or divided.

[0053] The following examples illustrate a portion of the embodiments of this disclosure. However, it is to be noted that the embodiments of this disclosure are not limited to these points.

[0054] [Postscript 1]

[0055] A control device, comprising:

[0056] The input section is responsible for receiving instructions from the user via speech.

[0057] The display unit displays information to the user; and

[0058] When the control unit stops the first action mode based on the user's verbal instruction, if it receives a predetermined instruction to switch to the second action mode, it displays a first message urging the user to speak and a second message other than the first message on the display unit in different styles. The second action mode is the mode based on the user's verbal instruction.

[0059] [Postscript 2]

[0060] According to the control device described in Appendix 1, wherein...

[0061] Upon receiving the predetermined instruction, the control unit then switches to the second operating mode.

[0062] [Postscript 3]

[0063] According to the control device described in Appendix 1 or 2, wherein,

[0064] In the second action mode, the control unit performs an action corresponding to the user's instruction extracted from the speech.

[0065] [Postscript 4]

[0066] The control device according to any one of Appendices 1 to 3, wherein,

[0067] The predetermined instruction is an instruction made through predetermined statements or predetermined actions.

[0068] [Postscript 5]

[0069] The control device according to any one of Appendices 1 to 4, wherein,

[0070] The control unit causes the display unit to display the second information as an object superimposed on other information.

[0071] [Postscript 6]

[0072] The control device according to any one of Appendices 1 to 5, wherein,

[0073] The display style of the second information includes reduced brightness and no display.

[0074] [Postscript 7]

[0075] A method for operating a control device, the control device comprising: an input unit, receiving instructions from a user via speech; and a display unit, displaying information to the user, wherein the method for operating the control device includes:

[0076] When the first action mode, which is based on the user's verbal instruction, is stopped, if a predetermined instruction to switch to the second action mode is received, a first message urging the user to speak and a second message other than the first message are displayed on the display unit in different styles. The second action mode is the mode based on the user's verbal instruction.

[0077] [Postscript 8]

[0078] The method of operating the control device according to Appendix 7 includes:

[0079] Upon receiving the predetermined instruction, the system then transitions to the second action mode.

[0080] [Postscript 9]

[0081] The method of operating the control device according to Appendix 7 or 8 includes:

[0082] In the second action mode, an action corresponding to the user's instruction extracted from the speech is performed.

[0083] [Postscript 10]

[0084] The method of operation of the control device according to any one of Appendices 7-9, wherein,

[0085] The predetermined instruction is an instruction made through predetermined statements or predetermined actions.

[0086] [Postscript 11]

[0087] The method of operating the control device according to any one of Appendices 7 to 10, wherein it includes:

[0088] The display unit displays the second information as an object superimposed on other information.

[0089] [Postscript 12]

[0090] The method of operation of the control device according to any one of Appendices 7-11, wherein,

[0091] The display style of the second information includes reduced brightness and no display.

[0092] (Symbol Explanation)

[0093] 1: Information providing system; 10: Server device; 11: Network; 12: Vehicle-mounted device; 13: Vehicle; 121: Communication unit; 122: Storage unit; 123: Control unit; 124: Positioning unit; 125: Input unit; 126: Output unit.

Claims

1. A control device comprising: The input section accepts instructions from users via voice or by operating physical switches; The display unit displays information to the user; and When the control unit stops the first operation mode based on the user's verbal instruction, if it receives an instruction to switch to the second operation mode via a predetermined word or a predetermined operation, it causes the display unit to prompt the user to display or not display second information other than the first information at a lower brightness than the first information. The second operation mode is the mode based on the user's verbal instruction.

2. A control device, comprising: The input section is responsible for receiving instructions from the user via speech. The display unit displays information to the user; and When the control unit stops the first action mode based on the user's verbal instruction, if it receives a predetermined instruction to switch to the second action mode, it displays a first message urging the user to speak and a second message other than the first message on the display unit in different styles. The second action mode is the mode based on the user's verbal instruction.

3. The control device according to claim 2, wherein, Upon receiving the predetermined instruction, the control unit then switches to the second operating mode.

4. The control device according to claim 3, wherein, In the second action mode, the control unit performs an action corresponding to the user's instruction extracted from the speech.

5. The control device according to claim 2, wherein, The predetermined instruction is an instruction made through predetermined statements or predetermined actions.

6. The control device according to claim 2, wherein, The control unit causes the display unit to display the second information as an object superimposed on other information.

7. The control device according to claim 2, wherein, The display style of the second information includes reduced brightness and no display.

8. A method for operating a control device, the control device comprising: an input unit, receiving instructions given by a user via speech; and a display unit, displaying information to the user, wherein, The operation method of the control device includes: When the first action mode, which is based on the user's verbal instruction, is stopped, if a predetermined instruction to switch to the second action mode is received, a first message urging the user to speak and a second message other than the first message are displayed on the display unit in different styles. The second action mode is the mode based on the user's verbal instruction.

9. The method of operating the control device according to claim 8, wherein, include: Upon receiving the predetermined instruction, the system then transitions to the second action mode.

10. The method of operating the control device according to claim 9, wherein, include: In the second action mode, an action corresponding to the user's instruction extracted from the speech is performed.

11. The method of operating the control device according to claim 8, wherein, The predetermined instruction is an instruction made through predetermined statements or predetermined actions.

12. The method of operating the control device according to claim 8, wherein, include: The display unit displays the second information as an object superimposed on other information.

13. The method of operating the control device according to claim 8, wherein, The display style of the second information includes reduced brightness and no display.

Images