Controlling a household robot

Abstract

A method (200) for controlling a household robot (110) comprises steps of detecting a spoken instruction from a user, wherein the instruction comprises a description of a position in the household; using a classifier (135) to determine a position intended by the user on the basis of the description of the position included in the instruction; wherein the classifier (135) comprises a first encoder (315) for providing a first encoding (325) for text and a second encoder (320) for providing a second encoding (330) for a view; wherein the encoders (315, 320) are trained such that first and second encodings (325, 330), which are intended for text and a view of the household assigned to the text, are as similar as possible to one another; and controlling the household robot (110) to the intended position.

Description

[0001] Controlling a household robot

[0002] The present invention relates to the control of a household robot. In particular, the invention relates to the intelligent control of the household robot in response to a user's voice command.

[0003] A household robot is configured to perform a predetermined function in a household. In particular, the household robot can be configured to work on a floor and, preferably, to clean it, for example, by sweeping, mopping, or vacuuming. To control the household robot, the household is typically mapped, and a predetermined strategy is implemented based on the created map data. Typically, various strategies are predefined, for example, to take into account the type of floor, to avoid an area that is dangerous for the household robot, or to work on areas of the floor subject to different levels of wear and tear at different times or in different ways.

[0004] To trigger a household robot to spontaneously clean a predetermined location, the robot typically has to be transported to that location or manually guided there. It has been proposed to use voice recognition to recognize a spoken command and control a household robot based on the command.

[0005] US 2021 / 0401255 A1 concerns a robot that can be trained using machine learning. US 9 983 592 B2 proposes a robot and a control method.

[0006] However, the repertoire of instructions that can be understood in this way is limited, and the accuracy of such control is often unsatisfactory. Different users tend to refer to the same object differently, so that even with careful analysis, there are always terms that are not understood. For example, the same object can be referred to as a sofa, couch, chaise longue, play area, sofa, bed bench, divan, ottoman, or recamiere. Another term that is used rarely, colloquially, only regionally, or figuratively cannot be understood in this way.

[0007] One object underlying the present invention is to provide an improved technology for controlling a household robot by voice. The invention achieves this object by means of the subject matter of the independent claims. Subclaims specify preferred embodiments.

[0008] According to a first aspect of the present invention, a method for controlling a household robot comprises the steps of capturing a verbal instruction from a user, which includes a description of a position in the household; determining a position intended by the user based on the description of the position included in the instruction using a classifier; and controlling the household robot to the determined position. The classifier comprises a first encoder for providing a first encoding for a text and a second encoder for providing a second encoding for a view; wherein the encoders are trained such that first and second encodings, which are determined for a text and a view of the household associated with the text, are as similar to one another as possible. The description of the position in the instruction can, for example, be formulated in natural language.

[0009] For the first encoder, a BART encoder such as the one described in "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer (see https: / / doi.org / 10.48550 / arXiv.1910.13461) can be used. For the second encoder, a VQGAN encoder (VQGAN = Vector Quantized Generative Adversarial Network) can be used, for example, as described in "Taming Transformers for High-Resolution Image Synthesis" by Patrick Esser, Robin Rombach, and Björn Ommer (see https: / / doi.org / 10.48550 / arXiv.2012.09841). The disclosures of both documents are hereby incorporated by reference into the present disclosure.

[0010] The classifier is initially configured to use the encoders to encode similar texts and images as similarly as possible and to encode different texts or images as differently as possible. Encoding is not necessarily limited to a predetermined number of classes. In another embodiment, the classifier is configured to handle the determination of an encoder in the manner of a regression problem. Thus, an encoder can be formed that encodes a predetermined area of ​​a view into an encoding, so that the encoding describes a position in the household.

[0011] The classifier can, in particular, comprise a zero-shot classifier, which can assign a description or a term contained therein to an image even if this assignment is not included in the training data with which the classifier is trained. The classifier can predict an assignment, allowing for significantly better recognition of the description. The verbal instruction can include a freer or more precise description of the position. A user can control the household robot more effectively using a verbal instruction. The hit rate in finding the intended position by the household robot can be improved compared to conventional techniques. The user can formulate the instruction or description freely and does not have to adhere to a predetermined vocabulary or specific machine-understandable semantics.The position can be determined more precisely than with existing techniques, allowing the user to gain improved control over the household robot. This can also enable the use of the household robot in complex or large households.

[0012] Generally, an encoder is configured to compress an input into a vector, where the vector has significantly fewer dimensions than the input. An encoder may include an autoencoder. The encoders are preferably implemented as an artificial neural network. In another embodiment, an encoder may be configured as a large language model (LLM), allowing it to create encodings based on speech or text.

[0013] The encodings can each comprise a vector, with the encoders trained to maximize a vector product between corresponding encodings and minimize a vector product between non-corresponding encodings. This type of training can also be called "contrastive learning." The similarity between two vectors is also called cosine similarity and can allow an analogous measure of similarity to be determined based on the distance between the vectors in space.

[0014] Preferably, the encoders are trained on views of objects typically found in a household. The trained classifier may already possess a certain degree of "world knowledge," which can be used to improve location detection in a current household.

[0015] Such an object may comprise a structural element of a household, for example, a door, a window, a staircase, a threshold, or a passageway to another room. The object may also comprise a piece of furniture, in particular a piece of seating, a piece of bedroom furniture, a piece of work furniture, an piece of entertainment furniture, or a piece of decorative furniture. Furthermore, the object may comprise a home textile, for example, a carpet, a runner, a cushion, a tablecloth, a curtain, or a textile wall hanging. The home textile may also be combined with the furniture, for example, in the form of a corduroy sofa or a fabric armchair.

[0016] A text assigned to a view preferably refers to an object displayed in the view. The text preferably does not include a label in the sense of a "label" used in machine learning, but is unstructured. The text can be in the form of a caption or a short description. The text can be in natural language and provide additional information that goes beyond the description of the object. For example, if a furniture manufacturer's catalog is used as a source of learning data for the classifier, a text can include a possible use, a reference to an advantageous combination with another piece of furniture, or a suggested use. Using such information, the classifier can learn connections between objects that later allow it to make a statement about a text-image combination that was not directly covered by the learning data.

[0017] By using free descriptive text for the views, it is possible to transition from code-oriented processing to data-oriented processing, which can allow for automated inferences to be drawn by leveraging cross-connections. Unlike conventional speech recognition models, recognition can be based on a human's body of knowledge and / or way of describing things, thus achieving significantly better recognition.

[0018] Using the classifier and based on views of the household, an object displayed on a view can be determined whose second encoding is as similar as possible to a first encoding of the linguistic instruction.

[0019] The method can be based on a composite, comprehensive view of the household. Second encodings can be determined for objects depicted in the view. A captured linguistic instruction or a description encompassed by it, or more specifically, a concept encompassed by it, can be converted into a first encoding. By examining the similarity between the first encoding and the second encodings, objects—more precisely, image content of the view—can be identified that have maximum similarity to the linguistic instruction. In other words, for different areas of the view, it can be determined how likely it is that an object outlined by the instruction or description is depicted.

[0020] If an object is found whose similarity to the description exceeds a predetermined threshold, the position of this object can be used to control the household robot. If multiple objects are found whose second encoding is more similar to the first encoding than the predetermined threshold, the object with the greatest similarity can be selected, a query can be posed to a user, or the objects can be used sequentially to control the household robot. The latter option can be used in particular if the description includes a vocabulary that represents an all-quantifier, for example, "all" or "every." Such a vocabulary can also be recognized using conventional means.

[0021] The described position or the position intended by the user can be determined based on a position assigned to the specific view. Preferably, a number of views of the household are available, more preferably such that the views partially overlap, so that a seamless composite view can be formed. For each view, it can be known from which position in the household the view was taken. The position of an object depicted in a view can be determined with respect to a position from which the view was taken. In a simple embodiment, the described position can be determined to be the position from which the view was taken.

[0022] In one embodiment, the described or user-intended position is determined based on a location-relative statement regarding an object mentioned in the verbal instruction. The location-relative statement can include a local preposition such as "in front of," "behind," "on," "below," etc. The statement can be resolved depending on an orientation, whereby an orientation is typically assumed from the position from which the view was taken to the position where the object is located. In this way, statements such as "left of" or "right of" can be correctly recognized.

[0023] The information can refer to the geometry of a room in which the object is located. This can be the case if the object refers to a feature or element of the room, such as a door or a window. But information such as "in the middle" can also be resolved in this way.

[0024] In a further embodiment, composite information can be recognized that includes multiple positions that are related to a specific context. For example, a position "between" a first and a second position can be described in this way. A non-point-like location, such as a walking route, can also be described in this way.

[0025] In one embodiment, the classifier comprises a CLIP (Contrastive Language-Image Pretraining) classifier. Such a classifier was proposed by OpenAI; a description of the approach can be found in "Learning Transferable Visual Models From Natural Language Supervision" by Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, published on February 26, 2021. The entire disclosure of this article by Radford et al. is hereby incorporated by reference into the present disclosure. The classifier described in the present disclosure can be easily trained for use in controlling a household robot in a household using the specified procedure of said article by Radford et al.To this end, it is proposed to train the classifier with information concerning a typical or assumed household, as described in more detail herein.

[0026] Further preferably, the household appliance is controlled to perform a predetermined function of the household robot at the specific location. If the household robot is a floor-cleaning robot, a predetermined cleaning function can be performed. For example, the cleaning function can include vacuuming, wiping, mopping, or sweeping.

[0027] In one embodiment, the function is determined based on the instruction. For this purpose, a verb can be determined from the instruction, which is contained in a list of predetermined, known verbs. The list can include all functions that can be performed by the household robot and can be relatively short if the household robot is specialized in solving a specific problem, such as cleaning a floor.

[0028] It should be noted that the household robot can also perform other functions. For example, the household robot can help with tidying up the house. To do this, the robot can grasp and / or manipulate an object, in particular, bring it to a predetermined location. In addition to the location, the instruction can also specify an object that the robot should manipulate. Furthermore, the instruction can include further details about the manipulation, such as the type (e.g., grasping, moving) or a goal, such as where the object should be moved.

[0029] The instruction may be given acoustically, so that a user or generally a person in the range of the household robot can simply provide the instruction by spoken language, whereby the acoustic instruction is converted into a textual representation. For this purpose, a known technique of speech-to-text conversion may be used. The conversion may be performed locally, in particular by the household appliance or by another household appliance in the same household, or a detected acoustic instruction may be sent to an external location for recognition or conversion, and a result may be detected. The external location may comprise a server or a service in a cloud. In some embodiments, a keyword beginning an instruction may be recognized locally, and a remaining instruction may alternatively be recognized locally or remotely.Speech-to-text conversion can also be performed in a distributed manner using multiple processing devices. This can be achieved by using processing devices from several household appliances present in the household.

[0030] The household robot can be controlled to move through the household and capture views of the household. This process can occur before a spoken instruction is evaluated in the manner described herein. It is preferred that the household robot moves through the household at least once and creates views in the process. As described, it is preferred that the views partially overlap one another so that all parts of the household are captured in one view. The views can be restricted to a predetermined perspective and / or a predetermined scanning angle. For example, a vertical scanning angle can be specified so that no view of an object that lies above or below the scanning angle can be created. The household robot can already have a suitable scanning device, preferably a camera; alternatively, a radar sensor or a LiDAR sensor.

[0031] The creation of the views can be performed in conjunction with mapping the household. A map of the household can be created or updated using a SLAM (Simultaneous Localization and Mapping) technique. If information is collected for mapping, information for the views can be collected simultaneously. In one embodiment, the same scans can be used for mapping and views.

[0032] An existing scan can be updated regularly. This can involve creating just one new scan or multiple scans. In another embodiment, a view of the household is captured while mapping or editing the household.

[0033] According to one aspect, a method for training a household robot is disclosed, which can be used to make the household robot more controllable using the method described here. According to this training method, the household robot is controlled so that it moves through the household and captures views of the household. All or selected captured views are converted into a second encoding using the second encoder described above. Both encoders (i.e. the first and second encoders) were previously trained with images and associated image descriptions in text form, such that the encodings of text and image (i.e. the first and second encoding) are very similar when they describe the same object. The data pairs required for this were, for example, either manually labeled or collected by browsing the internet. The above-mentioned article by Radford et al.The CLIP classifier relies primarily on browsing the internet to collect training data and also refers to this encoding as embedding. The captured views can be selected, for example, using object recognition. If the method is able to detect a relevant object that has not yet been sufficiently investigated or understood, particularly meaningful views of the object could be used for learning. For example, the household robot could ask the user: "Whenever I drive through the living room, there's always this big blue thing. Please take a look at the attached image. What do you call that?"

[0034] Is that your sofa?" A list of suggested descriptions can of course also be displayed, from which the user can then select one or more descriptions, e.g. "sofa, couch, table, bed, bar." A query can be made, particularly if very different descriptions are not completely unlikely. It is also conceivable to make such queries more frequently based on reported errors. With the training procedure described here, the classifier can be further improved or fine-tuned in order to better recognize the semantic meaning of the images and to be able to integrate it more effectively into the encodings. This can be done in particular with field data (i.e. views of real households at the end user) that are recorded by household robots that are already in use. The labels or keywords belonging to the viewsText descriptions can be created either by manual labelers or by the robot owners (i.e. the users) using feedback functions in the associated app. Initial encodings are created based on this human feedback on the selected views. The classifier can then be adapted based on related first and second encodings. In this way, the classifier is specifically optimized for the specific site of use, which can be quite typical for its region. This includes the type of data to be expected and the objects depicted therein: e.g. the perspective (usually from below), focal length, distortion, noise level, viewing direction, the general recording quality of the robot sensors as well as the expected spatial conditions and the objects to be seen therein.Of course, this approach can also provide classifiers optimized for a specific region. For example, an average household in China or Iran will look different than a typical household in Germany. It is particularly preferred that pairs of views and user-entered or at least confirmed descriptions and / or the corresponding first and second encodings are collected in a central data lake of the home appliance manufacturer. Based on these pairs, classifiers can be provided that become increasingly better as the number of household robots in the field increases.

[0035] According to one aspect, the invention thus comprises a method for training a household robot to improve its controllability using the method described above. The training method may, for example, comprise the following steps:

[0036] • Controlling the household robot so that it moves through the household and captures views of the household,

[0037] • Providing a second encoding for selected views,

[0038] • Requesting human feedback on the selected views,

[0039] • Providing initial encodings based on human feedback and

[0040] • Adjusting the classifier based on the first and second encodings. Views can be selected, for example, using object recognition and / or relevance determination. The relevance determination can be performed, for example, mechanically (e.g., if the classifier is unsure of the object) and / or through user feedback. The classifier can be adjusted locally on the household robot and / or centrally at the home appliance manufacturer. For this purpose, collected data pairs of selected views and human feedback, or the associated first and second encodings, can be transferred to a server or cloud service of the home appliance manufacturer.

[0041] In the inference phase, a query can be processed as follows, for example. If, for example, a user gives a verbal instruction to search for a specific location or object, this verbal instruction is converted into a first encoding using the first encoder and then (in simple terms) compared with all previously recorded image embeddings, i.e., the second encodings. This can be done, for example, using the matrix described below. Alternatively or additionally, an implementation using an artificial neural network and / or a function approximator would be conceivable. The image embedding, i.e., the second encoding, with the shortest distance to the text embedding being sought, i.e., the first encoding, is selected, and the corresponding recording location is passed on to the robot navigation as the target.

[0042] According to a further aspect of the present invention, a control device for a household robot comprises a detection device for detecting a verbal instruction from a user, wherein the instruction comprises a description, in particular a natural language description, of a position in the household; a classifier; and a processing device for determining a position intended by the user based on the description of the position included in the instruction by means of the classifier; and for controlling the household robot to the determined position. The classifier comprises a first encoder for providing a first encoding for a text and a second encoder for providing a second encoding for a view; wherein the encoders are trained such that first and second encodings, which are determined for a text and a view of the household associated with the text, are as similar to one another as possible.

[0043] The classifier can be formed by or comprised in the processing device. The processing device is preferably configured to partially or completely carry out a method described herein. For this purpose, the processing device can be implemented electronically and, for example, comprise an integrated circuit, a programmable logic module, or a programmable microcomputer. The method can be implemented in the form of a configuration or as a computer program product with program code means for the processing device. The configuration or the computer program product can be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the device and / or vice versa. Additionally, it should be noted that some method steps relate to the inference phase and others to the training phase.These steps can also be freely combined, allowing for a continuous improvement process, such as pre-training and further improving the classifier based on field experience. Thus, features and advantages of the training method can be transferred to the inference method and / or vice versa.

[0044] A household robot comprises a control device described herein. The household robot can be configured to perform a predetermined task in a household, for example, cleaning or tidying up. The household robot can move through the household. In a preferred embodiment, the household robot comprises a floor-cleaning robot configured to clean a floor surface. The floor-cleaning robot preferably comprises a camera for optically scanning an environment and a microphone for detecting an acoustic or spoken instruction.

[0045] Non-limiting embodiments of the invention will now be described in more detail with reference to the accompanying figures, in which:

[0046] Figure 1 a system;

[0047] Figure 2 shows a flow diagram of a method;

[0048] Figure 3 illustrates the pre-training of an exemplary classifier; and Figure 4 illustrates the determination of a position using a classifier.

[0049] Figure 1 shows an exemplary system 100. A household robot 110 and an object 115 are located in a household 105. A user 120 of the household robot 105 is typically also present in the household 105. The household robot 105 includes a control device 125 configured to detect a verbal instruction from the user 120, extract a description of a position from the instruction, wherein the position may relate to the object 115, locate the position in the household 105, control the household robot 105 to the position, and control a predetermined function of the household robot 105 there.

[0050] The control device 125 comprises a processing device 130 and a classifier 135, which can also be part of the processing device 130 or be formed by the processing device 130. Furthermore, a preferably optical scanning device 140 and an acoustic scanning device 145 are provided.

[0051] The optical scanning device 140 preferably comprises a camera, particularly preferably a color camera. The camera 140 can capture images in a color spectrum visible to humans or in another, in particular expanded, color spectrum. Optionally, the camera 140 can also provide depth information, for example, if it is a depth camera, a stereo camera, a ToF camera, or a combination of an optical camera and a LiDAR sensor. An optics of the camera 140 has predetermined properties such as a focal length or an aperture angle of a scanning area. The camera 140 is generally immovably mounted on the household robot 110. In other embodiments, the camera 140 can be horizontally and / or vertically pivotable, or a focal length of the camera 140 can be controllable.

[0052] The acoustic scanning device 145 preferably comprises a microphone 145 or an arrangement of multiple microphones. The microphone 145 is configured to detect an acoustic command in the area of ​​the household robot 110, which was typically uttered by the user 120. The user 120 can also be located outside the household 105; in this case, their voice can be transmitted to the household robot 105, for example, by telephone or via a communications network. For example, the detection of the vocal command of the user 120 can take place at a base station of the household robot 110 or at a communications device at the user 120, for example, a smartphone. The vocal utterance can be transmitted to the household robot 110, preferably wirelessly.

[0053] The processing device 130 can be connected to a local memory 150 configured to store graphic information. In particular, one or more views of the household 105 captured by the camera 140 can be stored in the local memory 150. The views can be in uncoded or coded form. An interface 155 can be used to access a drive device configured to move the household robot 110 within the household. For this purpose, the household robot 110 typically comprises at least one drive wheel that can roll along a surface. An optional wireless communication device 160 can be configured to communicate with a device usable for voice input.

[0054] A location 165 external to the household robot 110 is typically located outside the household 105 and is configured to perform predetermined data processing. In one embodiment, the external location 165 can perform text-to-speech conversion. For this purpose, an acoustic instruction can be transmitted to the external location 165; the external location 165 can use speech recognition to create a written text that represents the spoken text; and the written text can be transmitted to the household robot 105.

[0055] It should be noted that the acoustic instruction can also be sent to the external location 165 from a device other than the household robot 105. For example, the user 120 can use a mobile device (smartphone) to capture speech, and acoustic data of the captured speech can also be transmitted directly from there to the external location 165. In yet another embodiment, the conversion of the acoustic data into textual data can occur in the mobile device. In this case, the mobile device can assume the function of the external location 165.

[0056] Another function that must be performed to create or use the system 100 can also be performed at a location 165 external to the household robot 110. This location can be the same or a different external location 165 than the one that performs text-to-speech conversion. For example, training or pre-training of the classifier 135 can be performed by an external location 165. The classifier 135 preferably comprises an artificial neural network (ANN) and is trained using contrastive learning to determine, based on linguistic input data, a section in a view of the household 105 in which an object 115 referenced in the linguistic input data is located with the highest possible probability. The function of the classifier 135 is described in more detail herein, for example, with reference to Figures 3 and 4.

[0057] Figure 2 shows a flowchart of a method 200 for controlling a household robot 110. The method 200 can be executed by a system 100. Optionally, a part of the method 200 can be executed by an external location 165. Such a part can, in particular, relate to preparatory processing to create or train a classifier 135. It should be noted that creating a classifier 135 can be complex, but a created classifier 135 can easily be used within a variety of control devices 125.

[0058] In a step 205, images with text can be provided. An image preferably shows an object 115 that is typically found in a household 105. An exemplary object 115 comprises a structural element of a typical household 105, i.e., a structural feature that physically determines the household 105, such as a wall, a window, or a door. The object 115 can also comprise, for example, a piece of furniture, such as an armchair, a sofa, a chair, a cupboard, or a table. Furthermore, the object 115 can comprise a home textile, such as a carpet, a floor covering, a cushion, or a curtain.

[0059] Text associated with an image typically does not have a fixed format. The text may name or describe an object 115 depicted in an image. The text may include additional information about the object 115, such as an explanation of a non-visible part or feature, a possible use, a categorization, or a reference to another object 115. The text may describe the image or object 115, for example, in the manner of a caption in a photo album or an explanation in a catalog or reference work. It is generally preferred that the texts do not exceed a predetermined length, such as approximately ten words or approximately 200 characters.

[0060] In a step 215, the images and texts can be used as training material for a classifier 135. The classifier 135 can be created using contrast-based learning, so that it assigns similar texts or terms to the same class and different texts or terms to different classes. In this case, a first encoding can be assigned to a text or term, with texts or terms with similar content receiving first encodings that are as similar as possible and different texts or terms receiving first encodings that are as different as possible. In the same way, different images can be assigned second encodings, with similar images or images of similar objects 115 receiving second encodings that are as similar as possible and different images or images of different objects 115 receiving second encodings that are as different as possible.A first encoder, which creates the first encodings, and a second encoder, which creates the second encodings, are trained in such a way that, for an image of an object 115 to which a text or term is assigned, the first encoding is as close as possible to the second. For unassigned pairs of images and texts, however, the first and second encodings are as different as possible. This method of creating the encoders is also called contrast-based learning.

[0061] In a step 215, a classifier 135 can be provided based on the encoders. The classifier 135 is configured to determine a distance between a first encoding and a second encoding, wherein the distance represents a similarity.

[0062] Steps 205 to 215 can be run once to create a classifier 135, which can subsequently be used as often as desired and in as many environments as desired.

[0063] In a step 220, the household robot 110 can be controlled to travel through a household 105 in which it is subsequently to be used. To this end, the household robot 110 can move within the household 105 in such a way that the boundaries of the household 105 can be detected on all sides. Images can be created in the area of ​​the boundaries and within the household 105, for example, using the camera 140. The position at which an image was detected and the orientation in which the household robot 110 was located when the image was scanned can be recorded for an image. This information can later make it possible to determine the position of an object 115 depicted in an image in the household 105. Such a determination can evaluate depth information contained in the image.

[0064] In a step 225, one or more views of the household can be created based on one or more images. A view can be based on multiple images and thus cover a larger area than would be possible with one image alone. For example, a view can optically cover a room, whereby the view can include optical information that, in the manner of a panoramic shot, can originate from a large horizontal angular range, in the extreme case from a full 360° circle. A view does not have to be length- or angle-conform, but should allow the position of a depicted object 115 to be determined from its position in the image. Optionally, in a step 230, multiple views can be combined to provide a combined view. Such a combined view can make it easier to locate an object 115 in the household 105.

[0065] Steps 220 to 230 can be performed in a specific household 105 to provide images of the household 105 that are as comprehensive as possible, on which an object 115 can be found that can be part of a verbal description of a position. Since not all objects 115 in the household 105 are usually immovable, steps 220 to 230 can be repeated at regular intervals to provide updated views. Optionally, a view can also be partially updated by combining it with information from a current image. It should be noted that scans of the household 105 can also originate from another source. For example, the user 120 can also scan the household 105 with a separate camera, or the scan can be performed by a first household robot 110 and made available to a second household robot 110.

[0066] Another embodiment can utilize the fact that household robot 110 is networked with another household appliance in household 105. Detection or notification from the other household appliance can then be used to determine further information regarding household 105. For example, if a dishwasher reports that it is being emptied, household robot 105 can know where the dishwasher is and that tiles have been laid in front of it. Thus, household robot 105 can automatically navigate to the specific location and, for example, wipe in front of it.

[0067] After pre-training the classifier 135 and providing scans of the household 105, the household robot 105 can be controlled based on a verbal position indication.

[0068] In step 235, a verbal instruction can be captured for the household robot 105. The instruction can include a keyword followed by a description of a location in the household and, optionally, a reference to an activity to be performed there. For example, the instruction can be: "Robi, vacuum in front of the blue sofa."

[0069] The keyword ("Robi") can be evaluated locally and used to trigger a speech-to-text conversion of a subsequent text. This conversion can occur in a step 240. The conversion can be performed locally by the control device 125 or by the external location 165. The result of the conversion should then be available on the household robot 110.

[0070] In a step 245, the instruction can be analyzed based on its textual representation. To do so, the instruction can be broken down into components, and certain words can be recognized directly. For example, in the present example, it can be determined that the part regarding the "blue sofa" refers to an object that can be used to determine position. Likewise, the local preposition "vor" can be recognized.

[0071] In a step 250, one or more words can be directly recognized that describe a position in more detail or refer to another position. For this purpose, a list of predetermined local prepositions can be available, with which words from the textual description are compared. Similarly, a verb can be determined that can refer to an activity to be performed by the household robot 110 at the described location. In the present example, the instruction "vacuum" refers, for example, to a floor-cleaning robot 110 with a suction unit. In a step 255, a description of an object 115 can be extracted from the instruction. In the given example, the description refers to a blue sofa 115.

[0072] In a step 260, the position of the object 115 on a view can be determined with respect to the position description, based on the scans, and using the trained classifier 135. For this purpose, the similarity of the representation to the location description can be determined for a plurality of representations on one or more views. The similarity can be determined in the manner of a distance, more precisely in the manner of a vector product between two vectors comprising a first encoding of the location description and a second encoding of the representation.

[0073] In a graphical illustration, a view of household 105 can be displayed in false color, with the color of each pixel representing a similarity of the representation in its region to the description. Thus, a region of the view can be determined in which a predetermined spatial cluster of similarities above a predetermined threshold exists.

[0074] In a step 265, the described position or the position intended by the user can be determined. For this purpose, a position of an object 115 displayed in the cluster area can first be determined based on the view. For example, a position of the object 115 in the household 105 can then be determined based on a position and / or orientation assumed by the household robot 110 when scanning an image for the view. Optionally, the described position can be determined more precisely with respect to the position of the object 115 based on a local preposition included in the description.

[0075] In the selected example, a position of the "blue sofa" 115 in the household could therefore first be determined. Assuming a predetermined directional system, a position or area in the household 105 that lies "in front" of the object 115 can then be determined. The directional system can be based on a current position and / or orientation of the user 120 or the household robot 110. A directional system can also be predetermined, for example, with respect to a compass direction or a predetermined direction and / or position in the household 105.

[0076] In a step 270, the function to be performed by the household robot 105 can be determined. This determination can be made based on a verb in the instruction recognized in step 250. If the household robot 105 is configured to perform only a single function, this function can be assumed in any case. It may also be sufficient to control the household robot 105 to the specific position, for example, to await further instructions there.

[0077] Optionally, a recognized element of the instruction can be provided to the user 120 to ensure that the instruction was processed correctly. If more than one position was determined based on the instruction, the user 120 can be asked to select one or more of the positions. The method 200 can continue once the user 120 has confirmed the recognized element(s).

[0078] In a step 275, the household robot 110 can be controlled to the specific position. This control can be based on a map of the environment of the household 105, which can be available from the control device 125. Optionally, the movement of the household robot 110 can be used to capture an image of the household 105, which can be used to update or generate a view. In a step 280, the household robot 110 can be controlled to perform the function determined in step 270 when it has reached the specific position. Figure 3 shows an illustration of a pre-training of an exemplary classifier 135 (cf. steps 210, 215 of the method 200 in Figure 2). Training data includes a plurality of texts 305 and a plurality of images 310, wherein each text 305 is bijectively assigned to an image 310.A first encoder 315 creates first encodings 325 (T_i) for the texts 305, and a second encoder 320 creates second encodings 330 (l_i) for the images 310. Each encoding 325, 330 comprises a vector, so that by forming a vector product, a similarity or a distance between the vectors or encodings 325, 330 can be determined. Such a metric is known as cosine similarity.

[0079] Pre-training is performed as contrast-based learning, in which the encoders 315, 320 are successively modified to produce encodings 325, 330 that meet predetermined conditions. More specifically, the encoders 315, 320 are preferably defined as AINNs, with a learning objective when processing the texts 305 and images 310 being to determine first encodings 325 and second encodings 330 that are as similar to each other as possible when a text 305 is associated with an image 310, and otherwise as different as possible.

[0080] The distances (l_i • T_i) to be determined between the resulting first encodings 325 (T_1 ... T_N) and second encodings 330 (l_1 ... I_N) are shown in a matrix 335 in Figure 3. The main diagonal of the matrix 335 comprises combinations of first and second encodings 325, 330 that are assigned to one another; the remaining elements comprise combinations that cannot be traced back to an assignment of the initial data 305, 310. Pre-training can be considered complete when the described learning objective has been sufficiently well achieved. It should be noted that pre-training can comprise a large number of texts 305 and associated images 310, and that the texts 305 and images 310 usually have to be coded very often until the encoders 315 and 320 exhibit the required properties. The encoders 315 and 320 can together form a classifier 135.

[0081] Figure 4 shows the determination of a text 305 for an image 310 using a classifier 135, which includes a first encoder 315 and a second encoder 320. A reverse procedure can be followed to determine an object 115 for a description.

[0082] A second encoding 330 is created for image 310, which is then compared with all known first encodings 325 of the texts in the training data. The comparison again involves forming a vector product and corresponds to determining a distance between the encodings 325, 330. Due to the way in which the encoders 315 and 320 are created, similarities can be inferred between the content displayed in image 310 and the content of the descriptions 305. The smaller the distance between two encodings 325, 330, the greater the similarity or correspondence between the respective assigned contents.

[0083] The smallest of the formed vector products (l_i • T_i) can be determined. Optionally, it can be checked whether the vector product is below a predetermined threshold or whether the similarity of the included first and second encodings 325, 330 is above a corresponding threshold. The text 305 associated with the first encoding 325 can be determined as the most probable text 305 that can be associated with the image 310.

[0084] Reference symbol

[0085] 100 systems

[0086] 105 household

[0087] 110 household robots

[0088] 115 objects

[0089] 120 users

[0090] 125 Control device

[0091] 130 processing facility

[0092] 135 Classifier

[0093] 140 optical scanning device, camera

[0094] 145 acoustic scanning device, microphone

[0095] 150 storage

[0096] 155 Interface

[0097] 160 communication device

[0098] 165 external positions

[0099] 200 procedures

[0100] Provide 205 images with text

[0101] 210 contrast-forming learning

[0102] 215 Create classifier

[0103] 220 households depart

[0104] Create 225 views

[0105] 230 create combined view

[0106] 235 capture verbal instruction

[0107] 240 Convert speech to text

[0108] 245 Analyze instruction

[0109] 250 Recognize verbs, prepositions, and conjunctions

[0110] 255 Extract location description

[0111] 260 Determine position

[0112] 265 Determine location

[0113] 270 Determine function

[0114] Control 275 robots

[0115] 280 Control function 305 Text

[0116] 310 Image

[0117] 315 first encoder, text encoder

[0118] 320 second encoder, image encoder 325 first encoding

[0119] 330 second encoding

[0120] 335 Matrix

Claims

PATENT CLAIMS 1. A method (200) for controlling a household robot (110), wherein the method (200) comprises the following steps: capturing (235) a linguistic instruction from a user (120), wherein the instruction comprises a description, in particular a natural language description, of a position in the household (105); Determining (260) a position intended by the user (120) based on the description of the position included in the instruction by means of a classifier (135); - wherein the classifier (135) comprises a first encoder (315) for providing a first encoding (325) for a text and a second encoder (320) for providing a second encoding (330) for a view; wherein the encoders (315, 320) are trained such that first and second encodings (325, 330) determined for a text and a view of the household (105) associated with the text are as similar to one another as possible; and controlling (275) the household robot (110) to the determined position.

2. The method (200) according to claim 1, wherein the encodings (325, 330) each comprise a vector and the encoders (315, 320) are trained to maximize a vector product between corresponding encodings (325, 330) and to minimize a vector product between non-corresponding encodings (325, 330).

3. The method (200) of claim 1 or 2, wherein the encoders (315, 320) are trained with respect to views of objects (115) commonly found in a household (105).

4. The method (200) of claim 3, wherein the objects (115) comprise a structural element of a household (105), a piece of furniture and / or a home textile.

5. The method (200) of claim 3 or 4, wherein text associated with a view is unstructured and related to an object (115) displayed on the view.

6. The method (200) according to any one of the preceding claims, wherein, by means of the classifier (135) and on the basis of views of the household (105), an object (115) shown on a view is determined, the second encoding (330) of which is as similar as possible to a first encoding (325) of the description of the position included in the instruction.

7. The method (200) of claim 6, wherein the position intended by the user (120) is determined based on a position associated with the particular view.

8. The method (200) according to claim 6 or 7, wherein the position intended by the user (120) is determined on the basis of a location-relative indication regarding an object (115) mentioned in the verbal instruction.

9. The method (200) of claim 8, wherein the information relates to a geometry of a room in which the object (115) is located.

10. The method (200) according to any one of the preceding claims, wherein the classifier (135) comprises a CLIP classifier (135).

11. Method (200) according to one of the preceding claims, wherein the household appliance is controlled to perform a predetermined function of the household robot (110) at the specific position.

12. The method (200) of claim 11, wherein the function is determined based on the instruction.

13. The method (200) according to any one of the preceding claims, wherein the instruction is given acoustically; wherein the acoustic instruction is converted into a textual representation.

14. A method (200) for training a household robot (110) to make it more controllable by means of a method according to one of the preceding claims, wherein the method for training comprises the following steps - controlling (220, 225) the household robot (110) so that it moves through the household (105) and captures views of the household (105), - Providing a second encoding for selected views, - Requesting human feedback on the selected views, - Providing initial encodings based on human feedback and - Adapting the classifier based on the initial and second encodings.

15. The method (200) of claim 14, wherein a view of the household (105) is captured (275, 280) during mapping or editing of the household (105).

16. A control device (125) for a household robot (110), the control device (125) comprising: a detection device (145) for detecting a verbal instruction from a user (120), the instruction comprising a description, in particular a natural language description, of a position in the household; a classifier (135) comprising a first encoder (315) for providing a first encoding (325) for a text and a second encoder (320) for providing a second encoding (330) for a view; the encoders (315, 320) being trained such that first and second encodings (325, 330), which are determined for a text and a view of the household associated with the text, are as similar to one another as possible; and a processing device (130) for determining a position intended by the user (120) based on the description of the position included in the instruction by means of the classifier (135);and for controlling the household robot (110) to the position.; 17. A household robot (110) comprising a control device (125) according to claim 16.

Images