Automatic production of a digital twin

Abstract

The present invention relates to a method (10) for automatically producing a digital twin of at least one section of a building interior using at least one robot that is arranged in the building interior, wherein the at least one robot has at least one sensor for detecting a robot environment in the building interior, wherein at least one object is arranged in the robot environment, wherein the object is assigned at least one object type from a range of predetermined object types, wherein the method (10) has the following features: detecting (11) the robot environment using the at least one sensor of the robot to obtain detected environment data; assigning (13) at least one object type to the object in the robot environment using a neural network on the basis of the environment data, wherein the neural network is designed to recognise object types, wherein the neural network is implemented by a processor; and linking (15) the at least one associated object type to position information that specifies a position of the object in order to produce the digital twin.

Description

[0001] Description

[0002] Automatic creation of a digital twin

[0003] The invention relates to a method for automatically producing a digital twin of at least one section of a building interior using a robot, a robot, and a data processing device for automatically producing a digital twin of at least one section of a building interior using a robot.

[0004] In intralogistics, for example in internal material flow or in a warehouse, automated robots and partially automated industrial trucks, such as forklifts and the like, are being used more and more frequently. However, for automated or partially automated tasks, 3D maps of the robot's environment must be created. Conventional methods for mapping a robot's environment may require manual processing steps and repeated data recording for a map update, which means that automatic updates and thus an up-to-date 3D map are not possible. In addition, the robot's environment to be mapped must be blocked during map creation. For example, dynamic objects such as other robots or people are considered disruptive during map creation using conventional methods and must not be in the robot's environment to be mapped during map creation.Furthermore, mapping often requires specialized sensor hardware, which makes it impossible to use existing sensors. This can be due, on the one hand, to the lack of options for merging different data sources or the lack of localization of individual sensors, and, on the other hand, to the lack of distributability of the map production processes, for example, due to large data volumes and limited data transmission options. The use of 3D maps is also limited with conventional methods, as semantic information and its localization cannot be included in the 3D maps.

[0005] The present invention is based on the object of providing an efficient generation of a digital twin of a robot's robot environment. This object is achieved by the features of the independent claims.

[0006] Advantageous embodiments of the invention are the subject of the dependent claims, the description and the accompanying figures.

[0007] According to a first aspect, the object of the invention is achieved by a method for automatically producing a digital twin of at least one section of a building interior using at least one robot which is arranged in the building interior, wherein the at least one robot has at least one sensor for detecting a robot environment in the building interior, wherein at least one object is arranged in the robot environment, wherein at least one object type from a set of predetermined object types is assigned to the object, wherein the method has the following features: detecting the robot environment by the at least one sensor of the robot in order to obtain detected environment data;Assigning the at least one object type to the object in the robot environment using a neural network based on the environmental data, wherein the neural network is configured to recognize object types, wherein the neural network is executed by a processor; and linking the assigned object type with position information indicating a position of the object to create the digital twin.

[0008] The building interior can, in particular, be a warehouse, an assembly hall, an industrial area within a building, a sales area, or an industrially used space. An industrially used space can, in particular, be understood to mean a space with an interior surface that can, in particular, have the characteristics of a warehouse, an assembly hall, and / or an industrial area within a building or a sales area.

[0009] Preferably, at least one robot can be understood as a single robot, two robots or several robots.

[0010] In particular, "at least one sensor" can be understood as a single sensor, two sensors, multiple sensors, or a sensor array. In particular, "at least one object type" can be understood as a single object type, two object types, or multiple object types.

[0011] The above and following statements apply analogously to demarcated outdoor areas, such as outdoor storage areas, outdoor assembly areas or factory facilities.

[0012] The robot can be an industrial robot that is configured to perform additional tasks of an industrial robot, such as autonomous goods or parts transport, load handling, and / or forklift functions. However, the robot can be stationary, so the sensor monitors the same environment to detect any changes.

[0013] In one embodiment, the robot is capable of translational and / or rotational movement.

[0014] In one embodiment, the robot is a device for generating data, for example an autonomous vehicle, in particular an industrial truck, or a part of a vehicle, in particular an industrial truck, or a sensor arrangement which can be attached to vehicles, conveyor belts or frames, for example, or can be carried by persons.

[0015] In one embodiment, the processor is implemented in the robot, wherein the at least one associated object type is sent to a data processing device via a communication network, and wherein the data processing device links the at least one associated object type to the position information.

[0016] In one embodiment, the robot transmits the acquired environmental data to a data processing device via a communication network, wherein the processor is implemented in the data processing device, and wherein the data processing device links the at least one associated object type to the position information.

[0017] In one embodiment, the data processing device is a server, for example, a cloud server. In one embodiment, an output of the neural network is a digital feature set representing the associated object type.

[0018] In one embodiment, the position information is information about a position of the robot obtained from a global localization system, or the position information is a position of the object, or the processor derives the position information from the acquired environmental data.

[0019] Position information can be understood, in particular, as information regarding the x-axis, the y-axis, and the z-axis, or one or more spatial axes of a coordinate system. Particularly preferably, this can be understood as pose information, which preferably includes, in addition to the information regarding the x-axis, the y-axis, and the z-axis, information regarding a spatial angular position, in particular information regarding a rotation about the x-axis, the y-axis, and / or the z-axis.

[0020] For example, the position information can be a GPS position of a robot.

[0021] In one embodiment, the object has an attribute that is detected by the sensor in the environmental data, or the robot has another sensor for attribute detection, in particular a laser scanner, which outputs attribute data, wherein the processor is designed to detect the attribute in the environmental data or the attribute data.

[0022] In one embodiment, the attribute is a geometric characteristic of the object, in particular a geometric shape or a geometric extent, or an object label, or an object orientation.

[0023] In one embodiment, the processor is configured to recognize the attribute based on pattern recognition or on the basis of the neural network or on the basis of another neural network configured to recognize attributes.

[0024] In one embodiment, the attribute is linked to the associated object type with position information. In one embodiment, linking the associated object type with the position information is performed by assigning the position information to the object type or by entering the position information in a digital map.

[0025] In one embodiment, at least one further object is arranged in the robot environment, wherein at least one further object type of the at least one further object is detected by executing the neural network on the processor, and wherein the localization information or further localization information is linked to the at least one further object type.

[0026] Preferably, the object type can in particular be an object type that is common in the application case, preferably a pallet, a robot, a wall, an aisle, a shelf, a door, or an information sign.

[0027] In one embodiment, the object type is a pallet, a robot, a wall, an aisle, a shelf, a door, or an information sign.

[0028] According to a second aspect, the object of the invention is achieved by a robot which is arranged in the interior of a building, comprising: at least one sensor for detecting a robot environment in the interior of a building, wherein at least one object is arranged in the robot environment, wherein at least one object type from a set of predetermined object types is assigned to the object; a processor which is designed to assign at least one object type to the object in the robot environment by means of a neural network on the basis of the environment data, wherein the neural network is configured to recognize object types; and wherein the processor is further designed to link the at least one assigned object type to position information which indicates a position of the object in order to create the digital twin.

[0029] In one embodiment, the object has an attribute that is detected by the sensor in the environmental data, or the robot has another sensor for attribute detection, in particular a laser scanner, which outputs attribute data, wherein the processor is designed to detect the attribute in the environmental data or the attribute data.

[0030] According to a third aspect, the object of the invention is achieved by a data processing device for automatically producing a digital twin of at least one section of a building interior using at least one robot which is arranged in the building interior, wherein the robot has at least one sensor for detecting a robot environment in the building interior, wherein at least one object is arranged in the robot environment, wherein at least one object type from a set of predetermined object types is assigned to the object, wherein the data processing device has the following features: a communication interface for receiving environment data from the at least one robot via a communication network, wherein the detected environment data represent the detected robot environment;a processor configured to assign at least one object type to the object in the robot environment by executing a neural network based on the environment data, wherein the neural network is configured to recognize object types, and wherein the processor is further configured to link the at least one assigned object type with position information indicating a position of the object in order to produce the digital twin;

[0031] According to a fourth aspect, the object of the invention is achieved by a computer program for carrying out the method according to the first aspect when the computer program is executed by means of a processor.

[0032] The embodiments mentioned for the subject matter of the first aspect are also embodiments for the subject matter of the second aspect, the subject matter of the third aspect and the subject matter of the fourth aspect.

[0033] The embodiments mentioned for the subject matter of the second aspect are also embodiments for the subject matter of the first aspect, the subject matter of the third aspect and the subject matter of the fourth aspect.

[0034] The invention is described in more detail below using exemplary embodiments and the figures. The figures show: Fig. 1a is a flowchart illustrating steps of a method for automatically creating a digital twin of at least a portion of a building interior using a robot according to one embodiment;

[0035] Fig. 1b is a schematic view of a robot according to an embodiment and a data processing device according to an embodiment in a communication network;

[0036] Fig. 1c-e are schematic views of robots according to embodiments;

[0037] Fig. 2 is a schematic view of implementation steps by robots according to an embodiment and by a data processing device according to an embodiment; and

[0038] Fig. 3 is a schematic view of a system architecture of robots according to an embodiment, a data processing device according to an embodiment, and another data processing device according to an embodiment.

[0039] In the following description, reference is made to the accompanying figures, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present invention. The following detailed description, therefore, is not to be taken in a limiting sense. Further, it is to be understood that the features of the various embodiments described herein may be combined with one another unless specifically indicated otherwise.

[0040] The aspects and embodiments of the present invention are described with reference to the figures, wherein like reference numerals generally refer to like elements. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects of the present invention.

[0041] Figure 1a shows a flowchart illustrating steps of a method 10 for automatically creating a digital twin of at least a portion of a building interior using a robot according to one embodiment. The robot 101, 101', which is arranged in the building interior, has a sensor 105, 105' for detecting a robot environment in the building interior. At least one object is arranged in the robot environment, wherein the object is assigned an object type from a set of predetermined object types.The method 10 has the following features: detecting 11 the robot environment by the sensor of the robot 101, 101' to obtain detected environment data; assigning 12 an object type to the object in the robot environment by means of a neural network based on the environment data, wherein the neural network is configured to recognize object types, wherein the neural network is executed by a processor 103, 103', 113; and linking 15 the assigned object type with position information indicating a position of the object to create the digital twin.

[0042] Figure 1b shows a schematic view of a robot 101, 101' according to an embodiment and a data processing device 111 according to an embodiment in a communication network 100.

[0043] As described in more detail below, a plurality of robots 101, 101' may be configured to communicate with a data processing device 111 via a communication network 100 in order to produce the digital twin (hereinafter also referred to as a map).

[0044] The robots 101, 101' may each comprise corresponding first communication interfaces 109, 109', and the data processing device 111 may comprise a second communication interface 117 to transmit sensor data from the sensors 105, 105' of the robot 101, 101' via the communication network 100, for example, wirelessly or wired. As described in more detail below, the sensor data may be processed by corresponding first processors 103, 103' of the robots 101, 101 1preprocessed. To store the sensor data, the preprocessed data, and the digital twin, the robots 101, 101' can have corresponding first non-volatile memories 107, 107', and the data processing device 111 can have a second non-volatile memory 115. The data processing device 111 can comprise a second processor 113 for further processing the preprocessed data and / or the sensor data 105, 105'. The non-volatile memories 107, 107', 115 can be configured to store data and executable program code which, when executed by the corresponding processor 103, 103', 113, causes at least one of the processors 103, 103', 113 to perform the functions, operations, and methods described below.

[0045] The robots 101, 101' may comprise more than one sensor 105, 105'. The sensor 105, 105' may be a 3D sensor, for example a 3D lidar sensor, a 3D ToF camera, a radar sensor, an ultrasonic sensor, or a stereo camera, and generate distance information, in particular angle-related distance information, of the objects. Furthermore, the sensors 105, 105' may comprise a sensor for attribute detection, for example a laser scanner configured to output attribute data. Furthermore, the robots 101, 101' may comprise an internal measurement unit (IMU) or an odometry sensor system for detecting a movement and / or position of the sensors 105, 105'.

[0046] As described in more detail below, the implementation steps for producing the digital twin or the map can be distributed between the first processors 103, 103' of the robots 101, 101' and a second processor 113 of the data processing unit 111.

[0047] Figures 1c-e show schematic views of robots according to embodiments.

[0048] As shown in Figures 1c and 1d, the robot 101, 101' may be an autonomous vehicle (AMR) and the robot 101, 101' may comprise a plurality of sensors 105a-c, which may span a common field of view by overlapping the fields of view of the individual sensors 105, 105'. As shown in Figure 1c, the robot 101, 101' may comprise 2D color cameras 105a as sensors 105, 105' and the robot 101, 101 1a 3D lidar sensor 105b as a further sensor 105, 105'. As shown in Figure 1d, the sensors 105, 105' for generating depth images can be preferably TOF cameras and / or stereo cameras 105c, each of which is arranged on one side of the robot 101, 10T. In alternative embodiments (not shown here), at least one sensor, preferably the sensors 105a-c, can also be arranged on a load handling device, preferably a mast and / or a fork carrier and / or a fork tine. At least one sensor, preferably the sensors 105a-c, can be arranged, for example, such that they have a viewing direction towards the hall floor. As shown in Figure 1e, the robot 101, 10T may be a sensor arrangement with a 3D lidar sensor 105b as sensor 105, 105' and 2D color cameras 105a as further sensors 105, 105'.The communication interface 109, 109' can be designed as a radio interface, as illustrated in Figures 1c-e.

[0049] In the communication network 100, a plurality of existing sensors can be used by the automated robots 101, 10T to create the digital twin, such as fixed sensors or vehicle sensors from multiple vehicles and from multiple locations.

[0050] The functioning of the robots 101, 101' and the data processing device 111 is described in more detail below with reference to Figures 2 and 3. Figure 2 shows a schematic view of implementation steps by robots 101, 10T according to an embodiment and by a data processing device 111 according to an embodiment, and Figure 3 shows a schematic view of a system architecture of robots 101, 10T according to an embodiment, a data processing device 111 according to an embodiment, and a further data processing device 310 according to an embodiment. As shown in Figure 3, in one embodiment, the steps implemented by the second processor 113 can be distributed between the data processing device 111 and the further data processing device 310.

[0051] Figure 2 shows, by way of example, the processing steps of the first processors 103, 103' of two robots 101, 10T shown in Figure 1b. However, for the automatic production of the digital twin, more than two robots 101, 101' or just one robot 101, 10T are also possible. The robots 101, 10T can be configured differently with regard to the number and / or type of sensors 105, 105'.

[0052] The respective first processors 103, 103' can be configured to read information about update rates 203, which are described in more detail below. The update rates can be stored in a predefined manner in the corresponding first memories 107, 107' shown in Figure 1b. Furthermore, the update rates 203 can be determined and / or modified by the first processors 103, 103' based on an area of ​​the map 211 in which the respective robots 101, 101' are located, and / or depending on movement data, for example, depending on speed data from the movement sensors 213, 213'.

[0053] Based on the read-out update rates 203, the first processors 103, 103' can be configured, if the corresponding robot 101, 101' comprises more than one sensor 105, 105', to carry out a data fusion 201, 201' of the respective sensor data in a processing step, for example as shown in Figure 3, to form a point cloud (also known as a "point cloud") by a point cloud generator 305. In this case, the sensors can be configured and calibrated based on information for sensor calibration 205, 205' such that a common or at least partially common field of view is spanned by the corresponding sensors 105, 105' of the respective robot 101, 101'. If one of the sensors 105, 105' is a color sensor, for example a 2D color camera, a colored point cloud can be generated by the corresponding first processors 103, 103'.

[0054] Furthermore, the respective first processors 103, 103' can be configured to implement a processing step 207, 207' for classifying and segmenting the sensor data. In this case, the sensor data can be classified with regard to dynamics. A distinction can be made between dynamic data, which can change in a first time interval, semi-dynamic data, which can change in a second time interval, and static data, which can change neither in the first nor in the second time interval. Dynamic data can be assigned, for example, to vehicles or persons; semi-dynamic data can be assigned, for example, to goods; and static data can be assigned, for example, to shelves or walls. The second time interval can be longer than the first time interval.

[0055] Furthermore, module 207 can recognize attributes or semantic information in the sensor data, for example through pattern recognition. Attributes or semantic information are understood to be information that can indicate types of an object, such as a pallet, vehicle, person, or shelf, for example based on a geometric shape or extent of the object, and / or a numbering and / or labeling of the object. Furthermore, the semantic information or attributes can include an object orientation. The respective first processor 103, 103' can be configured to determine and / or modify the corresponding update rates 203 of the sensors based on the attributes or semantic information.

[0056] For the classification and segmentation described above and / or for recognizing the semantic information, the respective first processor 103, 103' can be configured to implement a corresponding first neural network 307, 307', which is configured for machine learning and / or computer vision. As shown in Figure 3, the respective neural network 307, 307' can comprise subnetworks 301, 301', which can be specifically assigned to individual sensors 105, 105' of the corresponding robot 101, 101'. The respective first neural network 307, 307' and the subnetworks 301, 301' can be trained using objects that are typically found in warehouses, for example, trained sensor-specifically, and thus configured to recognize objects in a warehouse.Module 207 can be configured to discard sensor data that does not correspond to a specific classification based on dynamics, which is specified, for example, in a class-specific manner using update rates 203. Thus, the sensor data can be reduced by module 207.

[0057] As shown in Figure 3, the subnetworks 301, 301' may further be configured prior to the data fusion 201 for localization 303, localization information based on the movement data of the movement sensors 213, 213 of the respective robot 101, 101 1 Thus, by localizing 303 the corresponding robot 101, 101 1 a position on the map can be assigned to the corresponding sensor data 105, 105'. In one embodiment, the localization 303 can alternatively and / or additionally be based on the attributes or semantic information.

[0058] The corresponding first processor 103, 103' can further be designed to implement a processing step 209 in which the corresponding first processor 103, 103' is designed to determine a deviation of the sensor data from the map 211 of the environment and to further process only those areas of the map 211 which have changed compared to the map 211.

[0059] The corresponding first processor 103, 103' can further be configured to implement a further processing step 215. The further processing step 215 can add the information about the class and the attributes or semantic information from processing step 207, as well as data from motion sensors 213, 213', to the sensor data. Via the respective first communication interfaces 109, 109' of the corresponding robots 101, 101', the corresponding first processor 103, 103' can be configured to send the thus supplemented sensor data to the data processing device 111 using a predefined transmission protocol, such as MQTT.

[0060] The second processor 113 of the data processing device 111 can be configured, in a processing step 217, to collect the individual data by receiving the sensor data preprocessed by the respective first processors 103, 103' via the second communication interface 117 using the predefined transmission protocol of the data processing device 111. The data of the robots 101, 101', which are received at time intervals according to the current update rates 203 via the second communication interface 117 and, depending on the type of the corresponding sensors 105, 105', are preferably images and / or point clouds and / or semantic information and / or attributes, in particular from processing step 215, can be individually stored in the second memory 115 of the data processing device 111.In the second memory 115 of the data processing device 111, the data can be collected and stored with a time stamp in a data storage facility, for example in a database 309. The collected data can be regularly forwarded to the modules described below for evaluation based on a metric, for example on the amount of data, the distance moved by the robots 101, 101' after a time interval determined by the second processor 113, depending on the update data 203, the environment, a change assessment of the environment determined based on the collected data and / or the map areas of the map 211. In one embodiment, the collected forwarding can be limited to a maximum of once per meter of movement of the robots 101, 101' and / or to a time interval range of 30 seconds to 5 minutes. These values ​​can depend on the environment or areas of the map 211. In environments orAreas of map 211 with relatively many changes can be evaluated more frequently than areas that change little or not at all.

[0061] The processor 113 of the data processing device 211 can be designed to implement an online calibration module 315 for self-monitoring of the respective robots 101, 101'. By means of the online calibration module 315, an online calibration of the corresponding sensors 105, 105' of the respective robots 101, 101' can be achieved in a processing step 219. The individual data of the sensors can be correlated with one another based on the route traveled. The online calibration module 315 can implement a suitable method, for example a simultaneous localization and mapping (SLAM) method, for each sensor 105, 105' for this purpose. From the trajectories of the sensors 105, 105' and the information that the corresponding sensors of the respective robots 101, 101 1are firmly coupled to one another, their movement must also be identical. Calibration, for example, extrinsic calibration, can be used to determine how the sensors 105, 105' are aligned to one another. For this purpose, the second processor 113 can perform corresponding difference calculations from the individual data of the relevant sensors 105, 105'. The more precisely the time stamp of the individual sensors is known, the more accurate the calibration can be. The calibration can be transmitted via the first and second communication interfaces 109, 109', 117 to the respective robot 101, 101' as a corresponding further sensor calibration 205, 205'. The further sensor calibration 205, 205' can then be compared with the previous, for example initial, sensor calibration 205, 205' by the respective first processors 103, 103'.If the deviation is significantly too large, the corresponding robot 101, 101' can calibrate itself for further sensor calibration 205, 205' or can detect the presence of a possible error and respond with an appropriate system reaction, for example, by stopping the service.

[0062] The second processor 113 can further be configured to perform data reduction in a processing step 221. In this case, the second processor 113 can implement a point cloud mapping module 313. The individual data from one of the robots 101, 101' forwarded by the point cloud mapping module 313 can be combined to form an entire point cloud map. This can thus result in a unique data set. The individual data can represent sections of this entire point cloud map. The second processor 113 can be configured to perform a change assessment of the entire point cloud map in comparison to the currently current map 211 and, based on the change assessment, to decide whether the entire point cloud will be further processed. In addition, dynamic objects, such as vehicles and people, can be removed from the map or transferred to a separate map.The finished point cloud map can be written back to database 309, and the individual data accumulated up to the point cloud map creation can be deleted. New individual data can then be combined with the point cloud map to create a new point cloud map in subsequent passes, based on the time-interval-controlled forwarding of the collected data.

[0063] The second processor 113 can further implement a multi-point cloud matcher module 317, which operates similarly to the point cloud mapping module 313. Here, completed point clouds from different robots 101, 101' are combined, whereas the point cloud mapping module 313 uses the data from one robot 101, 101'. In one embodiment, the data from multiple robots 101, 101' can be combined directly in the point cloud mapping module 313.

[0064] For further data reduction, the second processor 113 can implement a point cloud reducer module 311. The task of the point cloud reducer module 311 is to reduce the data volume of a point cloud as optimally as possible. For this purpose, a further check can be performed regarding the dynamics of the objects, for example, to filter out dynamic objects or static objects. If, for example, dynamic objects such as vehicles and / or people are removed, a further reduction in the data volume can be achieved. Another reduction option is the use of AI algorithms using an encoder-decoder principle. The AI ​​algorithms can be trained so that objects are detected in their entirety.In the encoder, the data volumes can be essentially reduced to features that are actually needed by subsequent neural networks, such as deep neural networks (also known as "deep neural networks"), to locate and recognize objects such as vehicles, pallets, etc. These features would be sufficient to create the map. Due to a possible loss of information due to the features when converting from a "dense" map to a "sparse" map, feature reduction can also be applied in the opposite processing direction, and the point cloud can be extracted from the features, for example, using the neural network and a decoder.

[0065] As shown in Figure 3, the data processing device 111 can be connected to the further data processing device 310 via the second communication interface 117 for synchronization 323. The data-reduced point clouds can be stored in a database 325 of the data processing device 111. Furthermore, the map 211 generated by the data processing device 111 can be stored in a map memory 321 of the data processing device 111 and synchronized with a map memory 319 of the further data processing device 310. The maps can be fed back to the Point Cloud Reducer module 311 via the map memory 319 of the further data processing device 310 in order to enable merging into a new point cloud, as described above.

[0066] In one embodiment, the functionalities and components of the further data processing device 310 can also be included in the data processing device 111, as illustrated in Figure 2. By splitting the data between the data processing device 111 and the further data processing device 310, the individual data from the sensors 105, 105' can remain on the further data processing device 310, thus reducing the data traffic between the data processing device 111 and the further data processing device 310, for example, especially when there are a large number of further data processing devices 310.

[0067] The second processor or data processing device 111 can further be configured, in a processing step 223, to merge the attributes or semantic information. Instance designations taken from the individual data can be cleaned up. For example, a numbering of 1 to 5 pallets of first individual data and a numbering of 1 to 5 pallets of second individual data can be cleaned up to a numbering of 1 to 10 in the merged data. Furthermore, segmentation instances can be merged. For example, a left half of a shelf in the first individual data and a right half of a shelf in the second individual data can be referred to as an entire shelf in the merged data.Furthermore, based on a difference model between the map of the map 211 and the merged point cloud, already segmented and recognized objects can be calculated out of the point cloud, and only the unknown objects can be passed on to a segmentation engine 327, thus achieving a separation of the data set into a segmented data set and a non-segmented data set.

[0068] The segmentation engine 327 can be configured to load the point cloud maps for segmentation. Segmentation is not only performed by objects, but individual objects can also be differentiated from one another. For example, if ten adjacent palettes are used for segmentation, all objects can be collectively labeled "palette." However, the individual palettes can still be differentiated; for example, each palette can be labeled "palette" but also assigned a sequence number from 1 to 10. This information is used for further processing. The segmentation engine 327 can use a machine learning approach for segmentation and implement another neural network.

[0069] The segmentation engine 327 can thus, in a processing step 225, re-segment the non-segmented data set in which the class or object type is not yet known. Thus, an independent object type can be assigned; for example, the floor can have the same object type everywhere or depending on the area in the environment, or goods can be segmented into a point set. By means of a connection to a database 231, the objects can be specifically categorized, for example manually, based on the current re-segmentation and / or a re-segmentation from a previous run. In this case, an object type can be assigned to the re-segmentation data and a CAD model can be stored. Furthermore, the segmentation engine 327 can detect tags on goods and, based on this, update information from a goods management system (WMS) 227 or use other data sources, such as the Internet 229.Via the internet 229, a search on search engines such as Google, for example, and a corresponding comparison of the information can be performed based on RGB image data, for example based on the colored point cloud. Furthermore, the segmentation engine 327 can be configured to query the positions of goods via the WMS 227 in order to identify which goods are involved and to query existing map information, such as a building plan with a dedicated coordinate system, via the database 231 in order to optimize the map 211 for existing coordinate systems. In one embodiment, the second processor 113 assigned to the further neural network of the segmentation engine 327 is more powerful than the respective first processors 103, 103', which allows for finer segmentation and thus a higher number of classes.

[0070] To train the additional neural network, previous, already labeled data can be used. Furthermore, the segmentation engine 327 can include a data labeler configured to label specific load carriers or goods to be segmented in order to specifically train the additional neural network on this data. Furthermore, the segmentation engine 327 can be linked to a customer's CAD system for object recognition, so that the data for objects can be directly imported into the map. Furthermore, additional information about goods can be queried via the WMS 227, for example, system information about their properties. Thus, for example, the product type can be searched for directly on the internet 229 in order to obtain additional training data, which can then be used to quickly train the additional neural network on new objects.The segmentation engine 327 can further be configured for segmentation using temporal information in a processing step 233. The position of all objects or points that could not be assigned to any object can be tracked over time. Points / objects that do not move over a defined period of time can then be classified as static. This also allows points caused by data artifacts to be removed. Data that cannot be summarized by the segmentation engine 327, for example, because noise from the sensors 105, 105' has resulted in isolated points still being present, can then be removed in a processing step 235.

[0071] The second processor 113 can further be configured to implement a method step 237 for replacing known segments with 3D models. As shown in Figure 3, the data processing device 111 can comprise an object replacement module 331 for this purpose. In this case, objects of a warehouse, for example pallets, shelves, or wire mesh boxes, can be replaced with a 3D representation, for example a CAD file. This allows mapping from the point cloud to the 3D map 211. The map 211 can be stored in the map memory 321 of the data processing device 111 and, as described, also reflected back to the further data processing device 310. In addition, the position and contents of goods are known from the WMS 227, whereby this information can be simplified into models, for example CAD models, and can also be incorporated into the map 211.

[0072] The second processor 113 can further be configured to implement a processing step 239 for replacing unknown segments with geometric models. As shown in Figure 3, the data processing device 111 can comprise a point cloud meshing module 329 for this purpose. Large areas of the point cloud can be combined as a polygon mesh (also referred to as a "mesh"), for example, to replace the ground with this polygon mesh. This can reduce the number of points, since otherwise a very large number of points would be required to describe and represent the entire ground. The polygon mesh itself can be formed from only a few vertices, for example, as is known from gaming engines. Furthermore, other large areas, such as walls, ceilings, but also pallets, shelves, and goods, provided they have not yet been replaced by the object replacement module 331, can be displayed on the map 211.

[0073] The map 211 can be distributed back to the robot 101, 101' via a Map API 333, and / or the map 211 can be used in a simulation via a card reader 335. Furthermore, the map 211 can be made available to other programs, such as CAD programs or goods management systems 227, via a corresponding interface. In one embodiment, the Map API 333 can also be integrated into the further data processing device 310 in order to provide the robots 101, 101' with the map update of the map 211. The Map API 333 can have interfaces and / or standardized protocols, such as VDA5050, for distributing the map 211. The robots 101, 101' can be configured to locate themselves using the map 211, for example, via SLAM.

[0074] Visualization interfaces 337, 341 of the data processing device 111, 310 can be used to display the generated map 211, i.e., the 3D twin of the department store, via a defined interface 339, thus enabling the user to move virtually within their warehouse. The user can be connected to the defined interface 339 via an internet connection and user-specific access data. The view can be displayed on a device, for example, a tablet, cell phone, PC, or via virtual reality (VR) and augmented reality (AR), for example, in the form of an AR / VR device. The advantage of an AR device is that it can be worn directly in the warehouse, allowing the map 211 and the real environment to be overlaid.Furthermore, card 211 can be used to carry out a live inventory of the warehouse or a manual inventory, preferably supported by AR devices.

[0075] To display dynamic objects on map 211, the method steps described above for Figures 2 and 3 can be based on filtering static and semi-dynamic objects. The dynamic objects can then be shown or hidden in the updateable map 211.

[0076] This eliminates the need for special hardware that interrupts production operations to generate a digital image of the environment. The entire work process can be automated, requiring no human intervention or data processing. The system can thus be self-contained, allowing multiple functions to be implemented simultaneously, such as live mapping of the environment and related digital services, such as live inventory, localization, and simulation of the real environment. This reduces costs and processing times, while also collecting data that is essential for advancing digitalization using artificial intelligence.

Claims

Claims: Method (10) for automatically producing a digital twin of at least one section of a building interior using at least one robot (101, 101') arranged in the building interior, wherein the at least one robot (101, 101') has at least one sensor (105, 105') for detecting a robot environment in the building interior, wherein at least one object is arranged in the robot environment, wherein the object is assigned at least one object type from a set of predetermined object types, wherein the method (10) has the following features: Acquisition (11) of the robot environment by the at least one sensor (105, 105') of the at least one robot (101, 101') in order to obtain acquired environment data; Assigning (13) at least one object type to the object in the robot environment using a neural network based on the environment data, wherein the neural network is configured to recognize object types, and wherein the neural network is executed by a processor (103, 103', 113); and Linking (15) the at least one assigned object type with position information indicating the object's position to create the digital twin. Method (10) according to claim 1, wherein the processor (103, 103') is implemented in the robot (101, 101') and wherein the at least one assigned object type is transmitted to a data processing unit (111) via a communication network (100), and wherein the data processing unit (111) links the at least one assigned object type with the position information. Method (10) according to claim 1, wherein the robot (101, 101') transmits the acquired environmental data to a data processing unit (111) via a communication network (100), wherein the processor (113) is implemented in the data processing unit (111), and wherein the Data processing device (111) that links at least one assigned object type with the position information. Method (10) according to any one of the preceding claims, wherein an output of the neural network is a digital feature set representing the at least one assigned object type. Method (10) according to any one of the preceding claims, wherein the position information is information about a position of the robot (101, 101') obtained from a global localizing system, or wherein the position information is a position of the object, or wherein the processor (103, 103', 113) derives the position information from the acquired environmental data.Method (10) according to any one of the preceding claims, wherein the object has an attribute that is detected by the sensor (105, 105') in the environmental data, or wherein the robot (101, 101') has a further sensor (105, 105') for attribute detection, in particular a laser scanner, which outputs attribute data, wherein the processor (103, 103', 113) is configured to detect the attribute in the environmental data or the attribute data. Method (10) according to claim 6, wherein the attribute is a geometric characteristic of the object, in particular a geometric shape or a geometric extent, or an object label, or an object orientation. Method (10) according to claim 6 or 7, wherein the processor (103, 103', 113) is configured to recognize the attribute on the basis of pattern recognition or on the basis of the neural network (307, 307') or on the basis of another neural network configured for attribute recognition.Method (10) according to claim 6, 7 or 8, wherein the attribute is linked to the associated object type with position information. Method (10) according to claim 9, wherein the linking of the assigned object type with the position information is carried out by assigning the position information to the object type or by entering the position information in a digital map (211). Method (10) according to any one of the preceding claims, wherein at least one further object is arranged in the robot environment, wherein at least one further object type of the at least one further object is detected by executing the neural network on the processor (103, 103', 113), and wherein the localization information or further localization information is linked to the at least one further object type. Method (10) according to any one of the preceding claims, wherein the object type is a pallet, a robot (101, 101'), a wall, a corridor, a shelf, a door, or an information sign.A robot (101, 10T) located in a building interior, comprising: at least one sensor (105, 105') for detecting the robot's environment in the building interior, wherein at least one object is located in the robot's environment, and wherein the object is assigned at least one object type from a set of predetermined object types; a processor (103, 103') configured to assign at least one object type to the object in the robot's environment using a neural network based on the environment data, the neural network being configured to recognize object types; and wherein the processor (103, 103*) is further configured to link the at least one assigned object type with position information indicating the object's position in order to create the digital twin.Robot (101, 10T) according to claim 13, wherein the object has an attribute that is detected in the environment data by the sensor (105, 105'), or wherein the robot (101, 10T) has a further sensor (105, 105') for attribute detection. in particular a laser scanner, which outputs attribute data, wherein the processor (103, 103') is configured to detect the attribute in the environment data or the attribute data.

15. Data processing device (111) for automatically creating a digital twin of at least one section of a building interior using at least one robot (101, 101') located in the building interior, wherein the robot (101, 101') has at least one sensor (105, 105') for detecting a robot environment in the building interior, wherein at least one object is located in the robot environment, and wherein the object is assigned at least one object type from a set of predetermined object types, and wherein the data processing device (111) has the following features: a communication interface (117) for receiving environment data from the at least one robot (101, 101') via a communication network (100), wherein the detected environment data represents the detected robot environment;a processor (113) configured to assign at least one object type to the object in the robot environment by executing a neural network based on the environment data, wherein the neural network is configured to recognize object types, and wherein the processor (113) is further configured to link the at least one assigned object type with position information indicating a position of the object in order to create the digital twin.

6. Computer program for executing the method (10) according to any one of claims 1 to 12, when the computer program is executed by means of a processor.

Images