Method for selecting data for transfer for further use in case of device

Abstract

The invention relates to a method for selecting data for further use in the case of a device, in particular a robot, comprising: providing (200) operating data (202), which have been obtained during operation of the device, at least part of the operating data (202) being used for operating the device; checking (204) whether there is an anomaly (206); and if there is an anomaly, selecting (210) data (212) from the operating data (202) according to at least one selection criterion (214) and providing (216) the selected data for transfer for further use, in particular for storage.

Description

Technical Field

[0001] The invention relates to a method for selecting data for transmission for further use in a device, in particular a robot, and to a system, a device and a computer program for data processing for carrying out the method. Background Art

[0002] Devices, in particular mobile devices such as robots, typically move in an environment, in particular an environment to be processed or a work area, such as a home or a garden. In this case, sensor data can be detected by means of sensors. Summary of the invention

[0003] According to the invention, a method for selecting data and a system, a device and a computer program for data processing for carrying out the method are proposed having the features of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims and the following description.

[0004] The present invention generally discusses equipment and particularly discusses mobile devices, which move or at least can move in an environment or there, for example in a work area. Examples of such (mobile) equipment (or mobile working equipment) are, for example, robots and / or drones and / or partially automated or (fully) automated (on the road, in the water or in the air) moving means of transport. As robots, for example, household robots, such as cleaning robots (for example in the form of suction and / or wiping robots), ground or street cleaning equipment, construction robots or mowing robots are considered, and other so-called service robots are also considered, as at least partially automated moving means of transport, such as passenger vehicles or freight vehicles (also so-called ground transport vehicles, for example in warehouses), but also aircraft, such as so-called drones or ships are considered. Although the present invention is mainly explained below with respect to mobile devices such as household robots, the present invention can also be applied to other, especially non-mobile devices, such as smart home cameras or other smart home devices or household appliances.

[0005] In particular in the case of mobile devices, such devices have, in particular, a control or regulating unit and a drive unit for moving the mobile device, so that the mobile device can be moved in the environment, for example, also along a movement path or trajectory. In addition, such devices preferably have one or more sensors, by means of which, for example, the environment or information in the environment can be detected. In this way, data or sensor data that have been detected using one or more sensors of the device can thus be obtained. However, other types of data can also be generated when the device is operated.

[0006] The data are typically firstly raw data, for example numbers in the case of encoders, vectors in the case of odometers or inertial measurement units (so-called IMUs), point clouds in the case of lidar sensors, video or image data in the case of cameras, etc. Combinations of different types of data are also possible, for example point clouds and time stamps. Typically, such data are acquired continuously or continuously. At least part of the data is used for operating the device, i.e., for example, to detect obstacles in the environment, to plan the movements of the (mobile) device (e.g., autonomous movement), etc., and in particular after appropriate processing.

[0007] Such devices typically also have a communication module for data transmission, and in particular for wireless data transmission. In this case, we can talk about so-called connectivity solutions. In this way, data or sensor data that have been detected by the device or its sensors can be transmitted and, for example, transmitted to an external computing system, such as a central server, or to a so-called cloud. Data can also be transmitted to the device.

[0008] In this way, it is also possible to control the device or update the software on the device, for example, by an application. In particular, the cloud or an external computing system can be used as a data storage device (for log files, usage behavior, image data, etc.). In particular, a so-called data loop (AIoT loop) can be established. In this case, robots and other devices can send their data to the cloud or an external computing system, and these data are stored there, for example, in a so-called data lake (or a universal data storage device). These data can then be used to support development, marketing, etc. For example, navigation and mapping software can be improved from a home map uploaded by a robot (these robots move in the map). Improved navigation and mapping software can then be loaded onto the device as an update, for example.

[0009] Thus, such data, for example from the customer's home and directly from the usage phase of the device, can be very relevant for the future development of new functions. Other benefits such as marketing, lifespan estimation, etc. are likewise possible. An example is the observation of the environment and, if necessary, also the observation of people and their interactions for the creation of a semantic model of the environment (e.g. the layout of the kitchen, which objects are located where, when people do something in the kitchen). Another example is the wear behavior changing typical values.

[0010] However, as mentioned, sending all the data of a device up all the time requires too many resources on the device, in the network and in the cloud. And just using a ring buffer and uploading the entire ring buffer of all or each sensor is usually not enough.

[0011] In order to make this data circulation less critical, it is proposed, as mentioned, that the device itself carries out the data selection and reduction of the relevant data, for example.

[0012] As has been shown, a particular challenge in such data loops is to decide which data must or should be sent or transmitted. This is not currently concerned with data protection aspects, but rather with technical aspects of data transmission, in particular the upload bandwidth and the storage of data on the device and in the cloud (or on an external computing system). Depending on the communication connection, the upload bandwidth or bandwidth in general may be different, but independently of this it is desirable to keep the amount of data to be transmitted as low as possible.

[0013] In this context, it is desirable that not all conceivable data of a device are simply uploaded to the cloud and used for data circulation. Instead, the data should be selected (selected) (as intelligently as possible) during the operation of the device or robot.

[0014] Within the scope of the invention, it is now proposed to select data for transmission or transfer depending on the occurrence of anomalies. To this end, the following steps are performed, for example, on a computing unit or a system for data processing of the device.

[0015] Data are provided which have been acquired during operation of the device and which have preferably been detected at least partially using one or more sensors of the device. At least part of the usage data is used to operate the device. This can be done, for example, continuously (or quasi-continuously). It should be mentioned here that, for example, various sensors only detect data from time to time.

[0016] The one or more sensors can include one or more of the following sensors: switches, encoders, inertial measurement units, lidar sensors. Depending on the type of sensor, the detected data can be different. Consider, for example, the presence of a bumper or a dust box of a household robot as a switch. Here, the data can include individual numerical values. Encoders can be used for odometers and measure or determine the number of wheel revolutions. Here, the data can include vectors. Likewise, the data of the inertial measurement unit can include vectors or matrices. In the case of a lidar sensor or other ranging sensors, the data can include a point cloud. In the case of a camera, such as a stereo camera or a 3D camera, the data can include images, depth images or videos.

[0017] Each sensor can generate a data stream, i.e., continuously detected data. Each data stream can also include metadata. Static data such as serial numbers, semi-static data such as used parameters or maps can also be present as data. Machine learning models or other virtual sensors can also provide data.

[0018] Now check whether there is an anomaly. The anomaly can be understood in particular as a disturbance in the normal operation or operation process of the device, that is, for example, there is an irregularity or something unexpected or unknown happens. For example, the anomaly can include at least one of the following situations: a disturbance of the device and / or in the environment of the device occurs. Or, the device is in an error state or an anomaly defined by, for example, a developer exists (for example, a specific internal system state or internal error state predefined by the developer). Or the movement of a device constructed as a mobile device stops or is stopped. Or an unknown object is identified based on data (for example, with the help of a sensor). Or an unknown action is performed at the device. For example, also through an application and, for example, at a time point where this should not normally be the case, in particular, the device is moved (for example, pushed), lifted or turned off.

[0019] If, and in particular only if, an anomaly is present, selected data are determined from the data according to at least one selection criterion and these selected data are then provided for transferring the selected data for further use, in particular for storage, for example in the cloud. In this way, therefore, all data are no longer simply transferred, but can be specifically adapted to data that are particularly relevant for the further operation and development of the device.

[0020] In one embodiment, the check for the presence of an anomaly comprises at least one of the following actions. For example, at least a portion of the data can be compared with a model, such as a world model or a motion model. For this purpose, for example, changes in the current situation relative to an environment map, a semantic map or a topology can be taken into account. If the map initially always remains the same, for example, but a door suddenly opens, the sensor will be able to detect a larger environment. The data then show significant deviations.

[0021] For example, it is also possible to check information about at least part of the data itself. Thus, internal metadata such as motion vectors of the compression method, bandwidth or a number of points in the point cloud can be checked in the MPEG stream. If there are strong changes here (e.g. deviations exceeding a predefined threshold value), an anomaly can be assumed. This also applies to the rate of images or laser scans per second, if this rate, for example, decreases.

[0022] For example, it is also possible to check whether at least part of the data changes over time. For this purpose, for example, a comparison can be made as to whether the same situation or the same object is detected at different times (e.g. t, tx, t-2x). If this is the case, a disturbance of the camera can be assumed even if the device should actually be moving. For this purpose, for example, changes in the video stream, such as average brightness, movement, etc. can also be taken into account.

[0023] For example, it is also possible to check whether there is an object recognition with an uncertainty exceeding a predefined value (e.g. by means of classification). Thus, for example, the confidence level for an object in an image at time t may be less than a certain value, or the object may be classified into different classes at time t, t-1, t-2; this also indicates an anomaly.

[0024] For example, it is also possible to check whether a new object has been identified, for example an object that does not match a database of known objects, ie does not exist there. This indicates that a previously unknown situation has occurred - or, for example, an error has occurred in the classification.

[0025] For example, it is also possible to check whether there are errors in the device. Thus, for example, there may be abnormal measurements in the CPU load, in the memory usage or in the execution duration of individual processes, which also indicate an abnormality. Another type of error is, for example, if the (mobile) device remains plugged in, is lifted (which can be determined, for example, by data from an inertial measurement unit) or if its battery is empty or below a certain charge level. It is also possible to check, for example, whether the device is in a predefined state for which data should be transmitted.

[0026] For example, it can also be checked whether the user interaction with the device occurs repeatedly at a specific location in the environment or at a specific location or situation of the device. This may indicate, for example, that the device has a problem at this location. This applies not only to user interactions with the device, but also to other situations that occur repeatedly at a specific location in the environment (or, for example, also in a map used by the device) or at a specific location or situation of the device, for example always at the docking station of the (mobile) device. This may indicate, for example, a problem at the docking station or in the interaction between the device and the docking station.

[0027] In this way, it is possible to quickly and efficiently identify whether an anomaly, ie an abnormal situation, exists and whether data should be selected and provided for transfer.

[0028] As mentioned, the data selected and provided for transmission are selected according to at least one selection criterion. Such a selection criterion can be: select data based on which the presence of an anomaly has been determined. This can be based, for example, on one of the above-mentioned possibilities for identifying an anomaly based on operating data. In this case, the data selected in this way can, for example, include only a part of the operating data for identifying an anomaly or all operating data required for this purpose. It is also possible to select other operating data, that is to say, for example, the entire data stream of a specific sensor.

[0029] Such a further selection criterion may be the selection of data which are temporally and / or functionally and / or spatially associated with the anomaly. For example, operating data may thus be selected within a certain duration before and / or after the identification or occurrence of the anomaly, for example always the last two or five minutes or the last 10 or 100 measurement data (or measurement data sets). Likewise, in the sense of a functional association, data of such sensors may be selected which are associated with the anomaly, i.e., have for example caused or identified the anomaly.

[0030] Another such selection criterion may be the selection of data comprising information about the device. This may be, for example, sensor data, such as images, videos or IMU data, or may be an environment model such as a map. This may also be the pose (position, orientation) of the device in the environment or in a map, or the current system state of the device, for example, which is relevant to the program flow.

[0031] In particular, the aforementioned selection criteria can also be combined, ie, for example, sensor data of a camera and the gestures of the last two or five minutes can be selected.

[0032] It is also conceivable to use a different ring buffer for each sensor. The data in the ring buffer can then be uploaded (i.e. selected and provided for transfer), for example, only partially, completely or not at all. The ring buffer typically has a defined size and records, for example, 100 data packets. As soon as the ring buffer is full, the oldest data packet is overwritten. In this way, for example, the above-mentioned possibilities for transmitting the data of the last 5 minutes or the last 100 data sets can be realized.

[0033] This proposed data selection or data selection brings various advantages. On the one hand, less bandwidth is used for data transmission or data upload, which leads to higher acceptance at the user. On the other hand, less storage space is required in the cloud, which brings cost advantages to the operator. In addition, such data selection is also user-friendly, especially in terms of data protection. Thus, for example, for selected situations, a review function can be implemented in an application (for example, the device can be controlled by the application). To this end, in the case of an identified abnormality, information can be output to the user, for example, via an application or an input or operating device commonly used for the device. For example, for this purpose, for example, in the manner of "The following problem has been identified: The robot cannot classify the image. Do you want to agree to upload?" The user can also be requested to agree to provide the selected data for transmission. Therefore, in this sense, it can also be stipulated that the selected data is only provided or the selected data is only transmitted after the transmission confirmation is obtained.

[0034] After processing in the cloud or in an external computing system, the parameterization or programming of a device or robot can then, for example, be updated directly, so that the data is included in the further development of the software and is later sent back to the device (system), for example in the form of a software update, or is submitted directly to a support team, for example as an error report.

[0035] Generally speaking, processed data may thus be received, for example, by a device or a computing unit thereof, wherein the processed data has been obtained or determined based on the selected data and at least a part of the transferred data. The processed data may thus be, for example, updated software.

[0036] Furthermore, it can be provided that the selected data are not transmitted immediately after being provided, but are transmitted only at a later point in time if necessary, for example when a communication link is available again or sufficient bandwidth is available for this purpose.

[0037] The system for data processing according to the invention comprises means for carrying out the method according to the invention or its method steps. The system can be, for example, a computer or a server in a so-called cloud or cloud environment. Likewise and particularly preferably, the system can be a device or a computing or processor unit or a computer or a control device in such a device.

[0038] The invention further relates to a device, in particular a mobile device, which has a system according to the invention for data processing and in particular also has one or more sensors. The device preferably also has a communication module for data transmission, in particular for wireless data transmission, and is designed to transmit selected data via the communication module.

[0039] The device is preferably configured as an at least partially automated mobile vehicle, in particular a passenger vehicle or a freight vehicle, and / or as a robot, in particular a household robot, for example a suction and / or wiping robot, a floor or street cleaning device or a lawn mowing robot and / or as a drone, as already explained in detail above.

[0040] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product with a program code for executing all method steps, because this causes particularly low costs, especially if the execution control device is also used for other tasks and therefore originally exists. Finally, a machine-readable storage medium is provided, with a computer program as described above stored thereon. Suitable storage media or data carriers for providing computer programs are especially magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. It is also possible to download programs via computer networks (the Internet, intranets, etc.). Such downloads can be carried out in a wired or cabled or wireless manner (e.g., via a WLAN network, 3G, 4G, 5G or 6G connection, etc.).

[0041] Further advantages and embodiments of the invention are apparent from the description and the accompanying drawings.

[0042] The invention is schematically illustrated in the drawings based on exemplary embodiments and is described below with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1a A mobile device is shown schematically for illustrating the invention.

[0044] Figure 1b Another view schematically shows Figure 1a Mobile devices in .

[0045] Figure 2 The sequence of the method according to the invention in a preferred embodiment is schematically shown. DETAILED DESCRIPTION

[0046] exist Figure 1a In FIG. 1 , a device 100, more precisely a mobile device, is shown schematically and exemplarily for explaining the present invention. Figure 1b In the embodiment, the device 100 is shown in different views and in different aspects. Figure 1a and 1b .

[0047] The device 100 is exemplarily a cleaning robot having a control or regulating unit 102 and a drive unit 104 (with wheels) for navigating or moving the cleaning robot 100 in an environment 120 and there, in particular in a working area 122, such as a residence. The cleaning robot 100 can, for example, move or be moved along a movement path or trajectory 130. In addition, the cleaning robot 100 exemplarily has a sensor 106 configured as a camera. With the aid of the camera 106, an image of the environment can be detected, which can be used again for navigation. In addition, a docking station 110 is exemplarily provided, at which the cleaning robot can, for example, be charged.

[0048] As already mentioned, such a device may also have other and / or additional sensors, such as lidar sensors, inertial measurement units and other sensors. Depending on the type of device, the type and number of sensors may vary.

[0049] exist Figure 1b In the environment, an object 124 , a further object 126 and a person 128 are shown by way of example, which can be taken into account as objects or obstacles when navigating the cleaning robot.

[0050] In addition, the cleaning robot 100 has a system 108 for data processing, such as a control device, by means of which data can be received and / or sent (transmitted), for example, via the indicated radio connection. Here, for example, a communication module for wireless data transmission can be integrated in the system 108, but such a communication module can also be provided separately and connected to the system 108 accordingly. Similarly, different communication modules can be provided for different types of communication. For example, the method according to the invention can also be performed using the system 108.

[0051] In addition, Figure 1a 1 shows by way of example a mobile input device 140, for example a smartphone, a central computing system 150 with a data memory 154 (or a server which can represent a so-called cloud), and a WLAN router 160. For example, various types of wireless communication connections 170, 174, i.e., for example, a WLAN connection 174 or a mobile radio connection 170, can be provided between the cleaning robot 100 or its system 108, the mobile input device 140 (here indirectly), the computing system 150 and the WLAN router 160.

[0052] It should be noted that a mobile radio connection 170 is established via a mobile radio transmitter 152, which in turn is connected to the computing system 150. It goes without saying that the WLAN router 160 has a suitable radio module for this. The WLAN router 160 can in turn be connected to the Internet via, for example, a wired connection and via this to the computing system 150. At this point it should be mentioned that the device or cleaning robot 100 can also be connected, for example, only via WLAN, but this also enables data to be transmitted to the computing system 150.

[0053] exist Figure 2 The flow chart of the method according to the invention in a preferred embodiment is schematically shown in FIG. Figure 1a and 1b The method is described.

[0054] In step 200, operating data 202 are provided. The operating data 202 are obtained when the device is operated or have already been obtained in this case. In this case, at least part of the operating data 202 is used for operating the device, that is, for example, for the mentioned navigation or for autonomously moving the device in an environment, for example, within the scope of a cleaning process. In particular, one or more sensors of the device are used, that is, for example, based on the data of the device. Figure 1a , 1b The operating data 202 has at least partially been detected by the camera 106. Thus, the operating data may include images, for example.

[0055] In this case, step 200 can be carried out in particular continuously or quasi-continuously, ie new operating data can be continuously acquired and provided.

[0056] In step 204, it is now checked whether an anomaly 206 is present. This step 204 can also be carried out continuously or quasi-continuously or also only at predefined time intervals and / or, for example, also in a situation-dependent manner.

[0057] For example, when a disturbance of the device and / or in the environment of the device occurs, or when the movement of the device is stopped, an anomaly 206 exists. For example, it is possible that the cleaning robot hits a vehicle according to the Figure 1b 128 and stops there. However, if an unknown object is detected based on the operating data 202, an anomaly 206 may also exist, for example. Figure 1b The object 126 can be identified for the first time and is therefore (still) unknown. In addition, an exception may also exist, for example, if the program of the system is in an error state, for example.

[0058] Checking 204 whether an anomaly is present can be performed, for example, by comparing operating data 202 with model 208. In addition, as explained in detail above, other possibilities for identifying anomalies are conceivable. Thus, for example, it can be checked whether a new object has been identified. If, for example, Figure 1b If object 126 is recognized for the first time and was previously unknown, ie, for example, not (yet) stored in an object database, a new object has been recognized.

[0059] If an anomaly is present, ie if the check in step 204 is positive, then in step 210 data 212 are selected from operating data 202 according to at least one selection criterion 214 and in step 216 are provided for transfer for further use.

[0060] These selected data 212 can then be transferred or transmitted to the (external) computing system 150 (ie, to the cloud), for example, via the aforementioned wireless communication connection (eg, via WLAN or mobile radio) in step 218. There, the selected data 212 can then be stored, for example, in the data memory 154.

[0061] In step 220, these selected data 212 can then be analyzed (for example by means of machine learning or artificial intelligence, or within the scope of so-called data mining). As mentioned, based on such an analysis, for example step 222, updated software 224 or the like (usually as processed data) can then be determined, step 226, which can then be loaded onto the device or received there, for example again via the mentioned wireless communication connection.

[0062] As a result, the so-called data loop can be closed, ie operating data can be detected and provided again with the updated software, step 200 .

Claims

1. A method for selecting data for further use in a device (100), in particular a robot, comprising: providing (200) operating data (202) which has been obtained when operating the device (100), wherein at least a portion of the operating data (202) is used for operating the device (100); Check (204) whether there is an anomaly (206); and If an anomaly is present, data (212) are selected (210) from the operating data (202) according to at least one selection criterion (214) and the selected data are provided (216) for transmission for further use, in particular for storage.

2. The method according to claim 1, wherein the anomaly (206) comprises at least one of the following situations: - the occurrence of interference in the device and / or in the environment of the device, - the movement of a device configured as a mobile device stops or is stopped, - identifying an unknown object based on the operating data, - unknown actions are performed on the device, in particular the device is moved, raised or switched off. -The system is in an error state, -The system is in a predefined system state that has been defined as abnormal.

3. The method according to claim 1 or 2, wherein checking (204) whether an anomaly (206) is present comprises at least one of the following actions: - comparing at least a portion of the operating data (202) to a model (208), - information on checking at least a portion of the operating data, - checking whether there is a change over time of at least a part of the operating data, - checking whether there is an object recognition with an uncertainty exceeding a predefined value, - check whether a new object has been recognized, - check if there are any errors in the device, - checking whether the device is in a predefined state for which data should be transferred, - Verify whether user interactions with the device or specific situations occur repeatedly at specific locations in the environment or in specific locations or situations of the device.

4. The method according to any one of the preceding claims, wherein the at least one selection criterion (214) according to which the data (212) is selected from the operational data (202) comprises at least one of the following selection criteria: - select the data on which the anomaly has been determined, - selecting data temporally and / or functionally and / or spatially associated with the anomaly, -Select data including information about the device. 5 . The method according to claim 1 , wherein the operating data are at least partially detected using one or more sensors of the device.

6. The method of claim 5, wherein the one or more sensors include one or more of the following sensors: a switch, an encoder, an inertial measurement unit, a lidar sensor.

7. The method according to any one of the preceding claims, further comprising: Processed data (224) is received (226), wherein the processed data has been obtained or determined based on the selected data and at least a portion of the communicated data.

8. A system (108) for data processing, the system comprising means for executing the method according to any one of the preceding claims.

9. A device (100), in particular a mobile device, having a system (108) according to claim 8, in particular having one or more sensors, the device in particular having a communication module for data transmission, in particular for wireless data transmission, and being configured to transmit selected data via the communication module.

10. The device (100) according to claim 9 is constructed as a robot, in particular as a household robot, such as a cleaning robot, a floor or street cleaning device, a lawn mowing robot, a service robot or a construction robot, an at least partially automated mobile transport vehicle, in particular a passenger vehicle or a freight vehicle and / or as a drone.

11. A computer program comprising instructions which, if executed on a computer, cause the computer to perform the method steps of the method according to any one of claims 1 to 7 when the program is executed by the computer.

12. A computer-readable storage medium having stored thereon the computer program according to claim 11.

Images