Control device and method for operating a control device

Abstract

A control device includes an input interface configured to accept a user's language instruction, a display configured to show information to the user, and a control unit configured to, upon receiving a predetermined instruction to transition to a second operating mode that accepts a user's language instruction while in a first operating mode that terminates the acceptance of a user's language instruction, cause the display to show first information prompting the user to speak and second information different from the first, in different ways.

Description

TECHNICAL AREA

[0001] The present disclosure relates to a control device and a method for operating the control device. BACKGROUND

[0002] Devices in vehicles equipped with agent functions that recognize requests contained in the language of vehicle occupants and provide answers or services in response to these requests are known. For example, patent literature (PTL) 1 discloses a device in a vehicle that recognizes an instruction containing a so-called wake-up word to activate an agent function. CITATION LIST Patent literature

[0003] PTL 1: WO 2022 / 254669 A1 SUMMARY

[0004] The user-friendliness of in-vehicle controls that offer services depending on the language needs improvement.

[0005] The following discloses a control device and the like that can improve user comfort.

[0006] A control device according to the present disclosure comprises: an input interface configured to accept a command from a user's language; a display configured to show information to the user; and a control device configured to receive a predetermined instruction to transition to a second operating mode that accepts an instruction from the user's language while in a first operating mode that no longer accepts an instruction from the user's language, cause the display to show first information prompting the user to speak and second information that differs from the first information in different ways.

[0007] Another aspect of the present disclosure is a method for operating a control device with an input interface configured to accept an instruction from a user's language and a display configured to show information to the user, wherein, upon receiving a predetermined instruction to transition to a second operating mode that accepts an instruction from the user's language while in a first operating mode that stops accepting an instruction from the user's language, the method causes the display to show first information prompting the user to speak and second information different from the first information in different ways.

[0008] According to the control device and the like in the present disclosure, it is possible to improve the user-friendliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the attached drawings: Fig. Figure 1 is a diagram showing an example configuration of an information delivery system; Fig. 2 is a diagram showing an example configuration of a control device; Fig. Figure 3 is a flowchart showing an exemplary operating sequence of the control device; Fig. 4A is an example of the display of information in the control device; Fig. 4B is an example of the display of information in the control device; Fig. 5A is an example of displaying information in the control device; and Fig. 5B is an example of displaying information in the control device. DETAILED DESCRIPTION

[0010] An exemplary embodiment is described below with reference to the drawings.

[0011] Fig. Figure 1 is a diagram showing an example configuration of a vehicle control system according to the embodiment. A vehicle control system 1 comprises at least one server device 10 and at least one device in the vehicle 12, which is mounted in at least one vehicle 13, and which are communicatively connected to each other via a network 11. The server device 10 is, for example, one or more server computers belonging to a cloud computing system or another computer system and acting as a server that implements various functions. The vehicle 13 is a passenger car, a commercial vehicle, or the like, and is a vehicle with an internal combustion engine, a hybrid electric vehicle (HEV), a plug-in hybrid electric vehicle (PHEV), or the like.The device in vehicle 12 has a computer with communication and information processing functions and provides various information and controls the operation of vehicle 13 according to the operation by a user of vehicle 13. The user can be a driver or passenger of vehicle 13. The network 11 can be, for example, a mobile communication network, the Internet, an ad-hoc network, a local area network (LAN), a metropolitan area network (MAN), other networks, or any combination thereof.

[0012] The device in the vehicle 12, referred to as the "control device" in the present embodiment, receives instructions by voice input or operation by the user of the vehicle 13 and provides the user with various information by voice output or image display and controls devices such as audio, air conditioning, windows, sunroof or seats of the vehicle 13, while optionally communicating with the server device 10, thereby providing convenience to the user.Upon receiving a predetermined instruction (hereinafter referred to as the transition instruction) to transition to a second operating mode (hereinafter referred to as the voice response mode), which accepts instructions from the user's language while in a first operating mode (hereinafter referred to as the standby mode), which no longer accepts instructions from the user's language, the device in vehicle 12 displays first information (hereinafter referred to as voice response information), prompting the user to speak, and second information (hereinafter referred to as operating response information), which differs from the voice response information, in different ways. Here, the transition instruction is a predetermined operation by the user, such as speaking a specific wake-up word or pressing a physical switch, or the like. The switch is located, for example,on the steering wheel, dashboard, or similar, and is activated by pressing or sliding. Standby mode is an operating mode that, in addition to the transition instruction, receives user instructions via a touch panel, physical switch, or similar device and provides information and control accordingly. After receiving the transition instruction in standby mode, the device in the vehicle 12 displays the information about the operator response differently than the information about the voice output, for example, by displaying the information about the operator response in a color or brightness that is less visible than the information about the voice output, or by hiding the information about the operator response, and then switches to voice output mode.This makes it easier for the user to understand that they should speak according to the voice response mode, thus avoiding unnecessary confusion. In voice control mode, the device in the vehicle accepts instructions from the user's language, in addition to instructions from the user's operation, and provides information and controls accordingly. Therefore, user comfort can be improved according to the present embodiment.

[0013] Fig. Figure 2 shows an example configuration of the device in the vehicle 12. The device in the vehicle 12 comprises a communication interface 121, a storage device 122, a control device 123, a position determination device 124, an input interface 125, and an output interface 126. These components can be configured as a single control device, as two or more control devices, or with another device such as a control device and a communication device. The control device includes, for example, an electronic control unit (ECU) or similar. The communication device includes, for example, a data communication module (DCM) or similar. The components are communicatively connected to each other or to devices in the vehicle 13 via an in-vehicle network that conforms to a standard such as a Controller Area Network (CAN).The device in vehicle 12 can be configured to include, as a component, an information processing device such as a smartphone or a tablet terminal.

[0014] The communication interface 121 includes a module that complies with a mobile communication standard such as Long Term Evolution (LTE), the 4th generation (4G) standard, or the 5th generation (5G) standard, or a module that corresponds to an in-vehicle LAN such as CAN or the like. The device in the vehicle 12 uses the communication interface 121 to conduct information communication with other devices via the network 11, which is connected via a nearby router or mobile communication base station, or to conduct information communication with any component of the vehicle 13 via the on-board LAN.

[0015] The memory device 122 comprises one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these types. The semiconductor memories include, for example, random access memory (RAM) or read-only memory (ROM). The RAM includes, for example, static RAM (SRAM) or dynamic RAM (DRAM). The ROM is, for example, an electrically erasable programmable ROM (EEPROM). The memory device 122 functions, for example, as main memory, auxiliary memory, or cache memory. The memory 122 stores information to be used for operations of the control device 123 and information obtained through operations of the control device 123.

[0016] The control unit 123 comprises one or more processors, one or more dedicated circuits, or a combination thereof. The processors are general-purpose processors, such as central processing units (CPUs), or dedicated processors, such as graphics processing units (GPUs), which are specialized for a specific processing task. The dedicated circuits are, for example, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or similar devices. The control unit 123 performs the information processing related to the operation of the device in the vehicle 12 and simultaneously controls components of the device in the vehicle 12.

[0017] The functions of the control unit 123 are realized by the execution of a control / processing program by a processor contained within the control unit 123. The control / processing program is a program that instructs a computer to execute the steps contained in the operations of the control unit 123, thereby enabling the computer to perform the functions corresponding to the processing of these steps. In other words, the control / processing program is a program that causes a computer to act as the control unit 123. Some or all of the functions of the control unit 123 can be realized by a special circuit contained within the control unit 123.

[0018] The positioning device 124 comprises one or more GNSS (Global Navigation Satellite System) receivers. GNSS includes, for example, the Global Positioning System (GPS), the Quasi-Zenith Satellite System (QZSS), BeiDou, the Global Navigation Satellite System (GLONASS), and / or Galileo. The positioning device 124 transmits a position result to the control unit 123, and the control unit 123 calculates position information about the device in the vehicle 12.

[0019] The input interface 125 comprises one or more input interfaces. These input interfaces include, for example, a microphone that receives voice input from the user, physical buttons, capacitive buttons, a pointing device, a touchscreen integrated with a display, and the like. Furthermore, the input interfaces include interfaces to various devices located in the vehicle. These various input devices include a camera that captures images inside the vehicle and a physical switch for a transition instruction. The transition instruction switch is located on the console, steering wheel, or similar location. The input interface 125 accepts operations for inputting various types of information, including a user's voice, and transmits the input information to the control unit 123 or transmits the captured images to the control unit 123.

[0020] The output interface 126 comprises one or more interfaces for output. The output interfaces can include, for example, a speaker and a display. In this embodiment, the display of the output interface 126 corresponds to a "display." The display is, for example, a liquid crystal display (LCD) or an organic electroluminescent display (EL), which is mounted in a location clearly visible to a user in the driver's or front passenger's seat, e.g., on the dashboard, or in a location clearly visible to a user in the rear seat. Furthermore, the output interfaces include interfaces for transmitting control signals to various devices such as the air conditioning system, the windows, the sunroof, and the seats of the vehicle 13.The output interface 126 outputs information obtained from operations of the control unit 123 in the form of audio signals or displays the information in the form of images.

[0021] Fig. Figure 3 is a flowchart illustrating the operational sequence of the device in vehicle 12. Each step in Fig. 3 is a step of information processing that is carried out by the control unit 123 of the device in the vehicle 12.

[0022] The procedure in Fig. Step 3 is executed when the control unit 123 receives a transition instruction. The transition instruction is given by a user who speaks a predefined phrase, such as a wake-up word like "Start dialogue mode" or "Hey, XX" (where XX is the manufacturer's name of the vehicle 13 or the system name). Alternatively, the transition can also be triggered by the user pressing a transition instruction switch. When the input interface 125 transmits voice data of the wake-up word from the microphone to the control unit 123, the control unit 123 obtains a transition instruction by recognizing the wake-up word from the voice through speech recognition processing.When the input interface 125 accepts input from the user's actuation of the switch for a transition instruction and transmits the accepted information to the control unit 123, the control unit 123 obtains the transition instruction by receiving the information from the input interface 125. After the control unit 123 has obtained the transition instruction, it stores the transition instruction information corresponding to the transition instruction in the storage device 122 and begins the process. Fig. 3 described procedures.

[0023] In S31, the control unit 123 retrieves the transition instruction information. This information determines whether the transition instruction was issued using the wake-up word or by pressing the transition instruction switch. The control unit 123 reads and retrieves the transition instruction information stored in the memory unit 122.

[0024] In step S32, the control unit 123 determines whether it is currently in standby mode. The control unit 123 determines whether it is in standby mode by referring to information indicating an operating mode stored in the memory unit 122. If the control unit 123 is in standby mode (Yes in step S32), it proceeds to step S33. If the control unit 123 is not in standby mode, i.e., in a voice response mode (No in step S32), it proceeds to step S36.

[0025] In S33, the control unit 123 determines the presence or absence of operator response information. The control unit 123 determines whether the operator response information is displayed on a display of the output interface 126. The control unit 123 causes the display to show information according to the user's operations on a touch panel integrated into the display or on various physical switches. The display of the output interface 126 shows, for example, a screen 40a, as in Fig. 4A shown, or a screen 40b, as in Fig. 4B is shown.

[0026] The screen 40a in Fig. 4A contains a map for navigation. Screen 40a contains notifications 41 and 42, which correspond to operator response information, and objects 43 and 44, which correspond to voice response information, superimposed on the map. Notifications 41 and 42 are alerts that correspond to various events during navigation and prompt the user to perform a tap or similar action to initiate operations related to the respective notification. For example, notification 41 is a banner indicating an incoming call on the communication interface 121. When the user taps notification 41, the control unit 123 initiates a hands-free call.Notification 42 is a push notification displayed by the control unit 123 when the communication interface 121 receives information indicating a traffic congestion via vehicle-to-vehicle communication or the distribution of road traffic information. When the user taps Notification 42, the control unit 123 displays further details about the traffic information. Object 43 is a pop-up notification object that allows the control unit 123 to suggest an alternative route to the user in response to Notification 42 regarding traffic information, and it is also an object that prompts the user for a voice command. Object 44 is a symbol object that prompts the user for a voice command by indicating that the user's language can be accepted.

[0027] The screen 40b in Fig. 4B is a menu screen for providing various services, including multimedia, to the user. Screen 40b contains objects 46, which correspond to operator response information, and objects 45 and 47, which correspond to voice response information. Object 46 prompts the user to perform a tap to launch an interface screen for a map, messages, function settings, or similar. Object 45 prompts the user to give instructions in their voice to launch a phone function or various audio applications. Object 47 prompts for instructions in the user's voice by indicating that the user's voice is acceptable.

[0028] Back to Fig. 3: If the operator response information is displayed in step S33 (Yes in step S33), the control unit 123 proceeds to step S34. If the operator response information is not displayed (No in step S33), the control unit 123 skips step S34 and proceeds to step S35.

[0029] In step S34, the control unit 123 changes the display mode of the operating response information. The control unit 123 changes the display mode of the operating response information shown on the output interface 126 display to a different display mode than that of the voice response information. For example, the control unit 123 can display the operator response information in a color or brightness that is less visible than the voice response information, or it can hide the operator response information. Fig. 5A and Fig. Section 5B illustrates examples of changes in how operator response information is displayed.

[0030] Fig. Figure 5A illustrates an example of changes in the display of messages 41 and 42, which correspond to the operational response information, on the [document / system / etc.]. Fig. The screen 40a shown in Figure 4A. For example, the control unit 123 displays message 41 on the output interface 126 display in a manner with reduced visibility, such as by masking or decreasing the brightness, or it hides message 42, causing messages 41 and 42 to be displayed differently than objects 43 and 44, which correspond to the speech response information. For example, the control unit 123 can display object 44, which corresponds to the speech output information, more prominently by increasing the brightness or similar measures, while reducing the visibility of messages 41 and 42, which correspond to the operating output information. Alternatively, the control unit 123 can introduce a new object that corresponds to the speech response information.

[0031] Fig. 5B shows an example of changes in the display of objects 46 that correspond to the operational response information, on which in Fig. The screen 40b shown in Figure 4B. For example, the control unit 123 displays the objects 46 on the output interface 126 display in a manner with reduced visibility, such as by masking or reducing the brightness, or it hides the objects 46, causing the objects 46 to be displayed differently than the objects 45 and 47 corresponding to the speech response information. The control unit 123 can, for example, display the object 47 corresponding to the speech information in a more conspicuous manner by increasing the brightness or similar means, while reducing the visibility of the objects 46 corresponding to the operation information. Alternatively, the control unit 123 can introduce a new object corresponding to the speech response information.

[0032] Fig. 5A and Fig. 5B shows examples where some of the operator response information is hidden and the visibility of the other operator response information is reduced, but all operator response information can be hidden or have its visibility reduced.

[0033] Returning to Fig. In step S35, the control unit 123 switches to a speech response mode. In speech response mode, the control unit 123 converts the speech received from the input interface 125 into text using speech recognition processing, analyzes the syntax of the transcribed speech content, and extracts the user's instructions. These instructions also include the user's requests.

[0034] Secondly, in step S36, the control unit 123 determines whether the transition instruction was given by actuating the switch. In this case, the control unit 123 operates in voice response mode. Based on the transition instruction information obtained from the memory unit 122, the control unit 123 determines whether the transition instruction was given by actuating the switch or by speaking the wake-up word. If the transition instruction was given by actuating the switch (Yes in step S36), the control unit 123 proceeds to step S37. If the transition instruction was not issued by actuating the switch, i.e., if the transition instruction was issued by speaking the wake-up word (No in step S36), the control unit 123 continues with step S38.

[0035] In step S37, control unit 123 determines whether to maintain the display mode of the operating response information. Since control unit 123 has already transitioned to voice response mode in this case, and the display mode of the operating response information was changed in step S34 of the processing cycle during the transition to voice response mode, control unit 123 retains the display mode of the operating response information after the change. Control unit 123 then continues to accept instructions by voice in voice response mode.

[0036] On the other hand, the control unit 123 in S38 processes the wake-up word as speech. Since the control unit 123 has already switched to speech response mode in this case, it processes the wake-up word as part of the speech and continues to accept further instructions via speech.

[0037] According to S35, S37 or S38, the control unit 123 terminates the procedure in Fig. 3 and continues working in voice response mode.

[0038] In voice response mode, the control unit 123 provides services according to the content of the user's speech. The control unit 123 converts the user's speech into text, analyzes the text syntax, and extracts instructions from the user's speech. The control unit 123 can extract the user's instructions from the speech text using a large language model (LLM). The control unit 123 then provides various pieces of information or controls the operation of vehicle 13 devices according to the user's instructions. For example, if the user requests traffic information, weather information, or information about the surroundings, the control unit 123 obtains the necessary information through vehicle-to-vehicle communication, information distribution from the server device 10, or similar means, and presents the information to the user via voice output or image display.When the user instructs the user to start a multimedia function such as telephone or audio, the control unit 123 starts the corresponding application or function. When the user instructs the user to adjust the volume, set the air conditioning, open or close the windows or sunroof, adjust the seat, or similar actions, the control unit 123 transmits information to control the device or actuator that performs the operation according to the instruction.

[0039] Some of the operations of the control device 123 in voice response mode can be performed in the server device 10. For example, the control device 123 can transmit data to the server device 10 via speech or text generated from speech. When the user's instruction is extracted by LLM in the server device 10, the control device 123 can receive information from the server device 10 indicating the user's instruction. The control device 123 then provides information and control according to the user's instruction.

[0040] As described above, when receiving a transition instruction in vehicle 12, the device displays operational response information differently than voice response information while in standby mode. This makes it easier for the user to understand that they should speak according to a voice response mode, thus avoiding unnecessary confusion. Therefore, it is possible to improve usability.

[0041] While the embodiment has been described with reference to the drawings and examples, it should be noted that various modifications and revisions can be made by those skilled in the art based on the present disclosure. Accordingly, such changes and revisions are included within the scope of this disclosure. For example, functions or the like contained in each means, step, or the like can be rearranged without logical inconsistency, and a plurality of means, steps, or the like can be combined or divided into a single one.

[0042] Examples of some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these examples. [Annex 1] A control device comprising: an input interface configured to accept a command from a user's language; a display configured to show information to the user; and a control device configured to receive a predetermined instruction to transition to a second operating mode that accepts an instruction from the user's language while in a first operating mode that no longer accepts an instruction from the user's language, cause the display to show first information prompting the user to speak and second information that differs from the first information in different ways. (Annex 2) The control device according to Annex 1, wherein the control device is further configured to switch to the second operating mode upon receipt of the predetermined instruction. [Annex 3] The control device according to Annex 1 or 2, wherein in the second operating mode the control device is configured to perform an operation in response to the user instruction to be extracted from the language. (Annex 4) The control device according to any one of Annexes 1 to 3, wherein the predetermined instruction is a voice instruction using a predetermined phrase or a predetermined operating instruction. (Annex 5) The control device according to any one of Annexes 1 to 4, wherein the control device is configured to cause the display to show the second piece of information as an object to be superimposed on other information. (Annex 6) The control device according to any one of Annexes 1 to 5, wherein the manner of displaying the second information includes a reduction in brightness and concealment. (Annex 7) Method for operating a control device with an input interface configured to accept an instruction from a user's language and a display configured to show information to the user, the method comprising that, upon receiving a predetermined instruction to transition to a second operating mode that accepts an instruction from the user's language while in a first operating mode that no longer accepts an instruction from the user's language, the display is caused to show first information prompting the user to speak and second information different from the first information in different ways. (Annex 8) The procedure according to Annex 7 further includes the transition to the second operating mode after the predetermined instruction has been received. (Annex 9) The method according to Annex 7 or 8, which features performing an operation in the second operating mode in response to the user's instruction to be extracted from the language. (Annex 10) The procedure according to one of Annexes 7 to 9, wherein the predetermined instruction is a voice instruction using a predetermined phrase or a predetermined operating instruction. (Annex 11) The procedure according to any of Annexes 7 to 10, which shows that the display is caused to show the second piece of information as an object to be overlaid on other information. (Annex 12) The method according to any of Annexes 7 to 11, wherein the manner of displaying the second information involves a reduction in brightness and concealment. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] WO 2022 / 254669 A1

[0003]

Claims

[1] Control device with an input interface designed to accept an instruction from a user voice or a physical switch operation, a display device set up to display information to the user, and a control device which, upon receiving a voice instruction using a predetermined word or a predetermined operating instruction, is configured to switch to a second operating mode which accepts an instruction from the user's language while in a first operating mode which stops accepting an instruction from the user's language, to cause the display device to display second information at a lower brightness than first information, or to hide the second information which is different from the first information which prompts the user to speak. [2] Control device with an input interface designed to accept a user instruction via speech, a display device set up to display information to the user, and a control device which, upon receiving a predetermined instruction to switch to a second operating mode that accepts an instruction from the user via speech, while in a first operating mode that stops accepting an instruction from the user via speech, is configured to cause the display device to display first information prompting the user to speak, and second information different from the first information, in different ways. [3] Control device according to claim 2, wherein the control device is further configured to switch to the second operating mode upon receiving the predetermined instruction. [4] Control device according to claim 3, wherein the control device is configured in the second operating mode to perform an operation in response to the user instruction to be extracted from the language. [5] Control device according to claim 2, wherein the predetermined instruction is a voice instruction using a predetermined phrase or a predetermined operating instruction. [6] Control device according to claim 2, wherein the control device is configured to cause the display device to display the second information as an object to be superimposed on other information. [7] Control device according to claim 2, wherein the manner of displaying the second information includes a reduction in brightness and a hiding. [8] A method of operating a control device comprising an input interface configured to accept an instruction from a user via speech and a display device configured to display information to the user, wherein the method, upon receiving a predetermined instruction to transition to a second operating mode accepting an instruction from the user via speech, in a first operating mode which stops accepting an instruction from the user via speech, comprises causing the display device to display first information prompting the user to speak and second information different from the first information in different ways. [9] Method according to claim 8, further comprising a transition to the second operating mode upon receipt of the predetermined instruction. [10] Method according to claim 9, comprising performing an operation in the second operating mode in response to the user instruction to be extracted from the language.

Images