Control of a household appliance

Abstract

The invention relates to a method (200) for controlling a household appliance (105), having the steps of detecting (210) a verbally formulated support request in the region of the household appliance (105); converting (215) the support request into a textual representation; determining (220) a response in text form by means of a speech model (155); converting (225) the response into a verbal representation; and outputting (230) the verbal representation on the household appliance (105).

Description

[0001] Controlling a household appliance

[0002] The present invention relates to the control of a household appliance. In particular, the invention relates to improved user guidance when controlling the household appliance.

[0003] A household appliance, such as a washing machine, includes controls that allow a user to select a predefined program, set an option, or access information about the appliance. Although user guidance is intuitive in most cases, there are cases where the user needs assistance. For example, the user may have a question about the correct use of an option or program, or the appliance may signal an error or warning that is not self-explanatory.

[0004] Typically, the user can consult a user manual that may have been included with the device. Alternatively, the user can contact a help center or a specialist. Another source of information includes a forum or documentation center, which may be accessible online. These sources of information have in common that they are generally not trustworthy or do not address the user's specific problem or question. Furthermore, some methods involve a longer wait time for a response. Using some sources is subject to a fee. An answer is not always relevant or safe. In some cases, the user may follow advice and thereby worsen the problem or cause a new one.

[0005] One object underlying the present invention is to provide an improved technology for controlling a household appliance. The invention achieves this object by means of the subject matter of the independent claims. Subclaims specify preferred embodiments.

[0006] According to a first aspect of the present invention, a method for controlling a household appliance comprises steps of detecting a linguistically formulated support request in the area of ​​the household appliance; converting the support request into a textual representation; determining a response in text form using a language model; converting the response into a linguistic representation; and outputting the linguistic representation to the household appliance.

[0007] According to the method, expertise and language understanding, which can be achieved using a language model, can be used to provide a specific response to a request formulated by a user of the household appliance. This allows linguistic interaction between the household appliance and the user to solve an existing problem. The process can be controlled or triggered intuitively by the user. A response can be provided with little or no noticeable delay. The language model can be capable of learning, so that increasingly better answers can be found over time based on the support requests of one or more users.

[0008] The language model preferably comprises a Large Language Model (LLM) implemented on an artificial neural network (ANN) with a large number of parameters. In a typical embodiment, a general LLM can determine several billion weights between neurons. The network can be trained on a large number of unlabeled text using unsupervised learning. Examples of known language models that may be suitable for the specified purpose include GPT-2, GPT-3, GPT-4, Minerva, PaLM, or LaMDA.

[0009] The language model can be operated by an external service provider and made available to respond to the support request. The quality of the provided responses can be improved by applying one or more measures for specific training or adaptation of the language model.

[0010] In one embodiment, the language model is trained on vocabulary related to the household appliance. For example, with a washing machine, an improved technical understanding of a component such as a water pump, an eco-closure, or an aquastop can be achieved. The vocabulary can be incorporated into the language model so that the vocabulary can be meaningfully assembled into sentences in a given context, not only grammatically but also in terms of meaning.

[0011] In another embodiment, the language model is trained with respect to the operation of the household appliance. For example, the contents of an instruction manual or a general operating guide for household appliances of the same type or class can be used as training data. Other data sources can include, for example, a discussion in a forum or background information, such as from a scientific publication or a test report. For example, the user can learn how to gently remove a specific type of soiling from a given fabric with regard to a washing machine.

[0012] In a further embodiment, the language model is trained on a function of the household appliance. For example, information from a service manual or a general work regarding the function of the given class of household appliances can be used as training data.

[0013] In yet another embodiment, the language model is trained to handle errors in the household appliance. The error handling can include errors reported by the household appliance itself or errors noticed by the user due to a malfunction. Functional or repair documentation or forum discussions, for example, can also be used as training material for this purpose. In general, the language model's achievable responses can improve in quality if the additional training data mentioned is as accurate and comprehensive as possible.

[0014] An operator of a language model can offer an interface through which specialized knowledge can be contributed or made accessible to the language model. This allows training data related to the household appliance to be modularly combined with general training data of the language model. The provided training data can be separable from the rest of the language model complex, so that mixing or combining only occurs under predetermined conditions. For example, it can be ensured that the language model only accesses the additional information when a specific interface designated by the operator is used. A user who uses a different, particularly a public, interface of the language model may receive answers that are not based on the specifically contributed training data.

[0015] It is generally preferred that a predetermined keyword or key phrase is first recognized, which precedes the assistance request. This allows for the initial assessment to determine whether the keyword can be recognized in acoustic data within the range of the household appliance, and only if this is the case can further processing of the audio data be initiated. This can provide improved protection for the privacy of a person within the range of the household appliance. Furthermore, analysis of speech that does not include an assistance request can be prevented.

[0016] In a further preferred embodiment, the linguistically formulated support request is converted into a textual representation using a predetermined vocabulary that is particularly related to the household appliance. In this way, the conversion of spoken language into written word can be improved, more accurate, and less error-prone. A recognition model for converting spoken language can be trained in addition to recognizing technical terms and proper names in connection with the household appliance. A modular approach, in which the trained technical terms are separable from a general vocabulary, can be supported. The approach for this can correspond to that regarding the modularity of learning content in the language model.

[0017] According to a further aspect of the present invention, a device for controlling a household appliance comprises at least one microphone in the area of ​​the household appliance for detecting a voice-formulated support request; an acoustic output device in the area of ​​the household appliance; a communication device for communicating with a service external to the household appliance; and a processing device.

[0018] The processing device is configured to transmit an indication of the support request to the external service; to receive a response to the support request; and to output a linguistic representation of the response.

[0019] The device, and in particular the processing device it comprises, are preferably configured to at least partially execute a method described herein. The method can be in the form of a computer program product with program code means. The computer program product can be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the device, or vice versa.

[0020] The device can be retrofitted cost-effectively to a household appliance if it already contains one or more of the aforementioned components. The communication device is preferably configured to provide a wired or wireless connection to the external service via a data or communication network. For example, the communication device can be configured to communicate with a mobile network or the Internet.

[0021] In various embodiments, the device may include additional components used to execute the method. In one embodiment, the device further comprises a device for detecting a predetermined keyword that precedes the support request. The processing device is configured to forward the reference to the support request to the external service only after recognizing the keyword.

[0022] The device can also convert the support request from an acoustic to a textual representation. In one embodiment, the device comprises a device for converting the support request into a textual representation; the indication includes the textual representation. In another embodiment, the conversion can be performed by an external service contacted via the communication device. The external service can be implemented separately from the language model or combined with it.

[0023] In a further embodiment, the response is received in text form, wherein the device further comprises a device for converting the response into a linguistic representation. Thus, the response in text form can be short and can be transmitted quickly to the household appliance. Conversion into a linguistic representation then only takes place in the household appliance.

[0024] In a further embodiment, an interface is provided for controlling a function of the household appliance depending on the response. For example, a visual display device can be used to output the response in text form. In another embodiment, a visual indication of a control element of the household appliance to be used can be highlighted in accordance with the response.

[0025] A household appliance comprises a device described herein. The household appliance can be intended, in particular, for laundry care and can comprise a washing machine or a dryer. Other exemplary household appliances that can be equipped with the device include an extractor hood or a coffee machine.

[0026] A system for controlling a household appliance comprises at least one microphone in the area of ​​the household appliance for detecting a linguistically formulated support request; a device for determining a textual representation of the support request; a language model configured to provide a textual response to a textually represented support request; a device for providing a linguistic representation of a textually represented response; and an acoustic output device in the area of ​​the household appliance for acoustically outputting a provided linguistic representation.

[0027] The system can, in particular, comprise a household appliance described herein and a service external to the household appliance. Further services can implement additional functionalities. It is preferred that the system be configured for use by a plurality of household appliances. The household appliances can, for example, belong to a common appliance class or be manufactured by the same manufacturer.

[0028] The invention will now be described in more detail with reference to the accompanying figures, in which:

[0029] Figure 1 a system; and

[0030] Figure 2 shows a flow diagram of a process.

[0031] Figure 1 shows a system 100 for controlling a household appliance 105. The household appliance 105 is configured to serve a predetermined purpose in a household. This purpose may, in particular, include laundry care or a kitchen task. A user 110 can use the household appliance 105. In the following, the male gender is used for user 110 purely by way of example and without any specific intention or restriction.

[0032] The household appliance 105 comprises a device 112 for controlling an interaction with the user 110. The device 112 comprises at least one microphone 115, which is attached to the household appliance 105 in such a way that it can sample a recording of spoken words in the area of ​​the household appliance 105. The microphones 115 can be attached to different areas or with different orientations on the household appliance 105. Furthermore, a preprocessing unit 120 is provided, which is configured to condition and enhance a detected acoustic signal. For this purpose, for example, noise suppression can be applied, a frequency response can be adjusted, background or interfering noise can be suppressed, or a speaker can be isolated from a complex audio signal with multiple speakers. The preprocessing unit 120 can comprise a digital or analog signal processor.

[0033] A keyword recognition system 125 is configured to detect the occurrence of a predetermined keyword in an audio data stream. The keyword is preferably selected such that it is conspicuous in a typical household audio data stream and that a user 110 can easily remember it.

[0034] A processing device 130 is configured to control at least one of the illustrated components of the household appliance 105. Thus, when the keyword recognition 125 signals the recognition of the predetermined keyword, the processing device 130 can provide audio data provided by the preprocessing 120 to an external service 140 via a communication device 135. A response from the external service 140 can also be received via the communication device 135 and provided acoustically to the user 110 via an output device 145 in the area of ​​the household appliance 105. The output device 145 preferably comprises a loudspeaker and more preferably a matching amplifier.

[0035] Acoustic data provided externally by the communication device 135 can be converted by a service 150. This is also referred to as speech-to-text (STT). In another embodiment, the conversion of speech data into text form can also be performed using a component 150 that is part of the household appliance 105 or the device 112. In this case, the component 150 is connected upstream of the communication device 135 in the direction of the external service 140.

[0036] The provided text data, which represents the support request of user 110, can be provided to a language model 155. The language model 155 is preferably a large language model trained to process general language texts using a wide variety of data. Furthermore, additional training data 160 can be provided, particularly by a manufacturer of the household appliance 105, which can be combined or interwoven with the general data. It is preferred that the use of the additional training data 160 be limited to use cases related to the household appliance 105.

[0037] The language model 155 can provide a response based on the support request, which is typically also in text form. The response can be converted into acoustic data by a component 165, corresponding to a spoken word of the textual response. This technology is also referred to as text-to-speech (TTS). In the illustrated embodiment, the component 165 is represented as a service external to the household appliance 105. An output of the component 165 can be transmitted to the household appliance 105 via the communication device 135. In another embodiment, a component 165 with the same functionality can also be included in the household appliance 105 or the device 112, so that the text data of the language model 155 first passes through the communication device 135 and only then reaches the component 165.

[0038] The acoustic data thus provided can be output to the user via the acoustic output device 145 in the area of ​​the household appliance 105. Particularly if the text data is available from the household appliance 105, the processing device 130 can provide part of the data to the user 110 via a device of the household appliance 105. For this purpose, information can be exchanged via an interface 170. For example, an output device can be provided in the form of a text display, and the response can be provided not only in acoustic but also in text form on the household appliance 105.

[0039] If the user 110 makes a voice support request to the household appliance 105, their spoken words can be converted into text, converted into a response using the language model 155, and the response can be output to the user 110 on the household appliance 105. This can give the user 110 the impression that the household appliance 105 is capable of dialogue.

[0040] In a first variant, an assistance request may include advice on operating the household appliance 105 to achieve a predetermined goal. For example, if the household appliance 105 comprises a washing machine, the user 110 may request assistance with an existing cleaning or operating problem as follows:

[0041] "Hey washing machine! How do I remove grass stains from a silk shirt?"

[0042] The first part of this statement may correspond to the predetermined keyword to trigger processing of the following sentence. An example response provided to user 110 by language model 155 to this sentence could be:

[0043] "For a silk shirt with grass stains, it's best to select the delicate cycle on your Siemens washing machine and use a mild detergent. You may also want to consider an extra rinse cycle to ensure all the detergent is thoroughly rinsed out of the shirt. If your washing machine has a 'hand wash' option, choose this, as it treats the fabric more gently. Avoid using bleach or fabric softeners, as these can damage the silk fabric. It's also recommended to wash the shirt separately from other garments and use a mesh laundry bag to protect the shirt from damage."

[0044] The user 110 can also request advice if a problem occurs with the household appliance 105. In a first example, the user can ask a question related to general operation of the household appliance 105:

[0045] "Hey washing machine! Why can't I open the door?"

[0046] For example, this response could then be output as voice data on the household appliance 105:

[0047] "There may be various reasons why you cannot unlock the door of your Siemens washing machine. In your case, the washing machine is still in the middle of a wash cycle. Make sure the washing machine has finished its cycle and the 'door locked' indicator light has gone out." In this example, additional information representing a device status of household appliance 105 can also be considered. The device status can be transmitted along with the request to the language model 155. Preferably, the device status is also provided in text form so that it can be read by a human.

[0048] In a second variant, the user 110 can ask a question regarding operation or maintenance of the household appliance 105:

[0049] "Hey washing machine! Should I clean the lint filter with water?"

[0050] A possible answer to this could be:

[0051] "It is not recommended to clean the lint filter of your Siemens dryer with water, as this could damage the filter and pose a fire hazard."

[0052] In a third example, the user 110 may inquire about the meaning of an existing error or an error code that the household appliance 105 is displaying:

[0053] "Hey washing machine! What does this error code mean?"

[0054] In this example, a device status of the household appliance 105, here, for example, the provided error code, can be transmitted and provided to the language model 155 along with the linguistic request. A possible response could be:

[0055] "The error code E18 on your Siemens washing machine indicates a water drainage problem. This code appears when the washing machine is unable to drain the water from the drum."

[0056] Figure 2 shows a flowchart of an exemplary method 200 for controlling a household appliance 105. The method 200 preferably begins in a step 205, in which the pronunciation of a predetermined keyword in the area of ​​the household appliance 105 is detected. In a step 210, a voice request following the keyword for assistance in using the household appliance 105 can be detected. The detected request can be converted into text form in a step 215. For this purpose, the STT component 150 in the household appliance 105 or an STT service 150 outside the appliance 105 can be used. Speech recognition can be based on machine learning, for example, using an artificial neural network trained thereon. Available STT components can support different speakers and / or different languages ​​or dialects.When converting speech to text, a specific vocabulary can be used that is related to the household appliance 105 in question or the task to be performed with it. The specific vocabulary can be incorporated into the STT component for general recognition or used modularly only for queries in the current context. It is preferred that a manufacturer of the household appliance 105 provide information about an expected vocabulary to a provider of the STT component.

[0057] The support request in text form can be fed to the language model 155 in a step 220. Optionally, additional information can also be provided to the language model 155, which in particular relates to an identification, a device type, a device model, and / or a status of the household appliance 105. This information is preferably also expressed in text form. For example, a status of the household appliance 105 can comprise a number of structured parameters, which can be expressed, for example, in an XML structure or in JSON format. The language model 155 determines a text response with respect to the support request in text form and, if applicable, the additional information, based on the training data provided to it.

[0058] In a step 225, the response can be converted into a linguistic representation. For this purpose, a TTS service 165 outside of the household appliance 105 can be used, or a TTS component 165 can be included in the household appliance 105 for this purpose. In a step 230, the determined linguistic representation can be output in the area of ​​the household appliance 105 so that the user 110 can hear it.

[0059] Should the user 110 require further support, the dialogue with the language model 155—or the perceived dialogue with the household appliance 105—can be continued by repeating the method 200. In various embodiments, this may require re-capturing the keyword in step 205, or a further linguistic request following the output of the linguistic representation can be captured immediately in step 210. The method 200 can be executed as often as necessary to enable the user 110 to use their household appliance 105 in the planned manner or in the best possible way.

[0060] Reference symbol

[0061] 100 systems

[0062] 105 household appliances

[0063] 110 users

[0064] 112 Device

[0065] 115 Microphone

[0066] 120 Preprocessing

[0067] 125 Keyword Detection

[0068] 130 processing facility

[0069] 135 Communication device

[0070] 140 external service

[0071] 145 acoustic output device

[0072] 150 speech-to-text

[0073] 155 Language model, especially LLM

[0074] 160 additional training data

[0075] 165 Text-to-Speech

[0076] 170 Interface

[0077] 200 procedures

[0078] 205 Capture keyword

[0079] 210 language requirements

[0080] 215 Convert request into text form

[0081] 220 Determine answer in text form

[0082] 225 Convert answer into linguistic representation

[0083] 230 output linguistic representation

Claims

PATENT CLAIMS 1. A method (200) for controlling a household appliance (105), the method (200) comprising the following steps: - recording (210) a linguistically formulated support request in the area of the household appliance (105); - converting (215) the support request into a textual representation; - determining (220) a response in text form using a language model (155); - Converting (225) the answer into a linguistic representation; and - Output (230) of the linguistic representation on the household appliance (105).

2. The method (200) according to claim 1, wherein the language model (155) is trained with respect to vocabulary related to the household appliance (105).

3. The method (200) according to claim 1 or 2, wherein the language model (155) is trained with respect to an operation of the household appliance (105).

4. The method (200) according to any one of the preceding claims, wherein the language model (155) is trained with respect to a function of the household appliance (105).

5. The method (200) according to any one of the preceding claims, wherein the language model (155) is trained with respect to error handling of the household appliance (105).

6. The method (200) according to any one of claims 2 to 5, wherein training data relating to the household appliance (105) are combined modularly with general training data of the language model (155).

7. The method (200) of any preceding claim, further comprising recognizing (205) a predetermined keyword preceding the support request.

8. The method (200) according to any one of the preceding claims, wherein the conversion into a textual representation is performed on the basis of a predetermined vocabulary related to the household appliance (105).

9. Device (112) for controlling a household appliance (105), the device (112) comprising: - at least one microphone (115) in the area of the household appliance (105) for detecting a linguistically formulated support request; - an acoustic output device (145) in the area of the household appliance (105); - a communication device (135) for communicating with a service (240) external to the household appliance (105); - a processing device (130) configured to transmit an indication of the support request to the external service (140); to receive a response to the support request; and to output a linguistic representation of the response.

10. The apparatus (112) of claim 9, further comprising means (125) for detecting a predetermined keyword preceding the support request.

11. The apparatus (112) of claim 9 or 10, further comprising means for (150) converting the support request into a textual representation; wherein the indication comprises the textual representation.

12. The apparatus (112) of any one of claims 9 to 11, wherein the response is received in text form; further comprising means (165) for converting the response into a linguistic representation.

13. Device (112) according to one of claims 9 to 12, further comprising an interface (170) for controlling a function of the household appliance (105) in dependence on the response.

14. Household appliance (105) comprising a device (112) according to one of claims 9 to 13.

15. System (100) for controlling a household appliance (105), the system (100) comprising: - at least one microphone (115) in the area of the household appliance (105) for detecting a linguistically formulated support request; - means (150) for determining a textual representation of the support request; - a language model (155) configured to provide a textual response to a textually represented support request; - a device (165) for providing a linguistic representation of a response represented in text form; and - an acoustic output device (145) in the region of the household appliance (105) for acoustically outputting a provided linguistic representation.

Images