Device and method for automatically detecting danger situations in a warehouse

Abstract

The invention relates to a data processing device (110) for automatically detecting a potential danger situation in an industrial environment, in particular in a warehouse, having a plurality of objects, including a plurality of dynamic objects, for example industrial trucks (120a, b), and a plurality of static objects, for example load carriers (140a, b). The data processing device (110) comprises a communication interface (113) which is designed to receive a plurality of sensor data items captured in the industrial environment. The data processing device (110) furthermore comprises a processor unit (111) which is designed to determine an object type from a set of predefined object types and an object state from a set of predefined object states for each object of the plurality of objects on the basis of the sensor data and a digital map of the industrial environment. The processor unit (111) is furthermore designed to implement an evaluation logic, wherein the evaluation logic is designed to determine whether there is a potential danger situation for each object of the plurality of objects on the basis of the object type and the object state. The invention also relates to a corresponding method for automatically detecting a potential danger situation in an industrial environment, in particular a warehouse.

Description

[0001] Description

[0002] Device and method for the automated detection of hazardous situations in a warehouse

[0003] The invention relates to a data processing device and a method for the automated detection of hazardous situations in an industrial environment, in particular a warehouse.

[0004] For the storage and transport of products, goods, and materials in an industrial environment, such as a warehouse, load carriers such as wire mesh boxes or pallets, especially Euro pallets, are often used. Industrial trucks such as forklifts, transport robots, and the like, which can be at least partially automated, are used to handle such load carriers, e.g., in intralogistics, i.e., the internal material flow.

[0005] In a dynamically changing environment, such as a warehouse, where the control of dynamic industrial trucks for transporting goods is usually either manual or at least partially automated, it is a challenge to ensure the safety of people and smooth operations in the warehouse on the one hand, while at the same time ensuring smooth operations in the warehouse. Traditionally, attempts are made to increase the safety of people and materials through infrastructural measures that are regularly checked by a person. However, such "manual" checks are associated with a great deal of effort and usually occur at long intervals. In the meantime, the dynamic environment can lead to situations that could endanger the safety of people and the smooth operations in the warehouse.

[0006] The present invention is therefore based on the object of providing a device and a method for the automated detection of hazardous situations in an industrial environment, in particular a warehouse. According to a first aspect, the object of the invention is achieved by a data processing device for the automated detection of a potential hazardous situation in an industrial environment, in particular a warehouse, with a large number of objects, including a large number of dynamic, i.e. moving objects, for example industrial trucks, people and the like, as well as a large number of static objects, for example load carriers. The data processing device comprises a communication interface which is designed to receive a large number of sensor data, in particular image data, recorded in the industrial environment.The data processing device further comprises a processor device configured to determine, based on the sensor data and a digital map of the industrial environment, an object type from a set of predetermined object types and an object state from a set of predetermined object states for each object of the plurality of objects. The processor device is further configured to implement evaluation logic, wherein the evaluation logic is configured to determine, based on the respective object type and the respective object state, for each object of the plurality of objects whether a potentially hazardous situation exists. The data processing device according to the invention can thus advantageously increase safety in an industrial environment, in particular a warehouse, through the automated detection of potential hazardous situations.

[0007] In one embodiment, the communication interface is configured to continuously receive a plurality of sensor data acquired in the industrial environment.

[0008] According to one embodiment, the processor device can be designed to determine not only the object type and the object state but also the object identity of a respective object on the basis of the sensor data and the digital map of the industrial environment. The object identity thus defines an individualization of objects of an object type. The object identity of an object can be determined, for example, based on unique features, for example based on the appearance of an object, i.e. the shape and color of the object, or based on a barcode sticker attached to the object. According to one embodiment, the object state can comprise position information data of an object. For example, an object state can be that a load carrier (object type) is located in a specific section of the warehouse defined in the digital map.In addition, the object status can contain information about when the load carrier has been in this section of the warehouse.

[0009] In one embodiment, the processor device is configured to implement a neural network configured to determine, based on the sensor data and the digital map of the industrial environment, the object type from the set of predetermined object types and the object state from the set of predetermined object states for each object of the plurality of objects. In one embodiment, the neural network is configured as a neural network for object recognition.

[0010] According to one embodiment, the evaluation logic can comprise a set of rules with a plurality of rules, wherein the plurality of rules define whether a potentially hazardous situation exists for a plurality of combinations of object type and object state. Thus, a set of rules can be stored in the evaluation logic using the set of rules, which define whether a potentially hazardous situation exists for different combinations of object type and object state. In other words, the evaluation logic can define a type of "object policy" using the set of rules. Thus, a plurality of hazardous situations orAnomalies are defined, such as damaged shelves, blocked emergency exits, missing or blocked fire extinguishers, missing or blocked first aid kits, parking of industrial trucks in areas where it is not permitted, storage of material outside designated areas, and the like, which can be recognized as a potential dangerous situation by means of the data processing device according to the invention.

[0011] In one embodiment, one or more of the plurality of rules in the rule set can "positively" define that no potentially hazardous situation exists for one or more combinations of object type and object state. In this embodiment, analogous to a "white list," it can be positively defined for each object which object states are permitted for the object with the object identity or object type and thus do not represent an anomaly or potentially hazardous situation.

[0012] Alternatively or additionally, one or more of the numerous rules in the rule set can "negatively" define that a potentially hazardous situation exists for one or more combinations of object type and object state. In this embodiment, analogous to a "blacklist," it is possible to define for each object identity and / or object type which object states of an object of a specific object identity or object type are not permitted, thus representing a potentially hazardous situation.

[0013] According to one embodiment, the processor device is designed to implement a further neural network, which is designed to implement the evaluation logic in order to determine, for each object of the plurality of objects, whether a potential hazardous situation exists based on the object type and the object state. In one embodiment, the processor device can advantageously be designed to continuously retrain the neural network for implementing the evaluation logic based on user feedback as to whether a detected potential hazardous situation was actually a hazardous situation. In a further embodiment, the neural network for implementing the evaluation logic can be designed to output that a potential hazardous situation exists with a probability or confidence, for example with a probability of more than 80%.

[0014] In one embodiment, if a potentially hazardous situation has been automatically detected, the communication interface can output or transmit a warning. According to one embodiment, this warning can be embodied as a visual, acoustic, and / or haptic warning, and in one embodiment, can contain the position information data of the object that caused the potentially hazardous situation. The warning can also be a message to a user of the data processing device.

[0015] According to an embodiment in which the plurality of dynamic objects comprise a plurality of industrial trucks, if a potential hazardous situation has been determined automatically with a high degree of probability, the processor device can generate control signals and send them to the plurality of industrial trucks via the communication interface in order to control the plurality of industrial trucks to avoid the potential hazardous situation, for example in order to change a respective movement route of the industrial trucks.

[0016] According to a second aspect, the above object is achieved by a method for the automated detection of a potentially hazardous situation in an industrial environment, in particular a warehouse, with a plurality of objects, including a plurality of dynamic, i.e., moving objects, as well as a plurality of static objects. The method comprises the following steps: receiving a plurality of sensor data, in particular image data, acquired in the industrial environment;

[0017] Determining for each object of the plurality of objects an object type from a set of predetermined object types and an object state from a set of predetermined object states on the basis of the sensor data and a digital map of the industrial environment; and

[0018] Determine for each of the plurality of objects whether a potential hazardous situation exists based on the respective object type and the respective object condition.

[0019] Further advantages and details of the invention are explained in more detail using the exemplary embodiments illustrated in the schematic figures. Herein:

[0020] Figure 1 is a schematic representation of a system with a data processing device according to the invention for the automated detection of hazardous situations in an industrial environment, in particular a warehouse;

[0021] Figure 2 shows a schematic representation of data processing modules of a data processing device according to the invention for the automated detection of hazardous situations in an industrial environment, in particular a warehouse; and Figure 3 shows a flowchart with steps of a method for the automated detection of hazardous situations in an industrial environment, in particular a warehouse.

[0022] Figure 1 shows a schematic representation of a system 100 with a data processing device 110 according to the invention for the automated detection of potentially hazardous situations in an industrial environment, in particular a warehouse. The industrial environment contains a plurality of objects, including a plurality of mobile, i.e., movable or movable objects, for example, industrial trucks 120a, b, people, and the like, as well as a plurality of static objects, for example, load carriers 140a, b or objects of the infrastructure of the industrial environment, such as shelves for storing the load carriers 140a, b.

[0023] Although a preferred embodiment of the invention is described here in connection with a warehouse, those skilled in the art will recognize that the data processing device 110 according to the invention can also be advantageously used in other areas for the automated detection of potentially hazardous situations, such as a warehouse, an assembly hall, an industrial area within a building, a sales area, or an industrially used space. An industrially used space can be understood, in particular, as a space with an interior surface, which can, in particular, have characteristics of a warehouse, an assembly hall, and / or an industrial area within a building or a sales area. This applies analogously to demarcated outdoor areas, such as outdoor storage areas, outdoor assembly areas, or factory facilities.

[0024] The data processing device 110 shown in Figure 1 can be, for example, an industrial PC 110 or a cloud server, in particular an edge cloud server 110. As shown in Figure 1, the data processing device 110 comprises a processor device 111, e.g. one or more processors 111, a communication interface 113 and a, in particular non-volatile, memory 115. The memory 115 can be designed to store data and executable program code which, when executed by the processor device 111 of the data processing device 110, causes the processor device 111 to perform the functions, operations and methods described below.

[0025] As shown in Figure 1, the system 100 further comprises a plurality of sensors 130a, b, which are configured to continuously acquire sensor data 201 (see Figure 2) in the warehouse and transmit it to the data processing device 110. The plurality of sensors 130a, b may comprise a plurality of imaging sensors, in particular cameras 130a, b, which are mounted in the industrial environment, in particular the warehouse and / or on the plurality of mobile industrial trucks 120a. For example, the cameras 130a, b may be 2D cameras, 3D cameras, color cameras, TOF cameras, stereo cameras, and the like. Alternatively or additionally, the plurality of sensors 130a, b may comprise, for example, radar and / or lidar sensors 130a, b mounted on the industrial trucks. In one embodiment, the plurality of sensors 130a,b can span a common field of view by overlapping the fields of view of the individual sensors 130a,b.

[0026] In one embodiment, the communication interface 113 of the data processing device 110 is configured to wirelessly and / or wiredly receive the plurality of sensor data 201 from the sensors 130a, b and / or to communicate with the industrial trucks 120a, b. For example, the communication interface 113 of the data processing device 110 can be configured to receive the plurality of sensor data 201 from the sensors 130a, b and / or to communicate with the industrial trucks 120a, b via a communication network, e.g., a WLAN, 5G network, and / or the Internet 150.

[0027] As will be described in detail below with further reference to Figure 2, the processor device 111 of the data processing apparatus 110 according to the invention is designed to determine, on the basis of the preferably continuously recorded sensor data 201 and a digital map 203 of the warehouse (also known as a digital twin 203 of the warehouse), for each object of the plurality of objects, an object type 205 from a set of predetermined object types and an object state 207 from a set of predetermined object states, specifically with the processing module 111a shown in Figure 2. The object type 205 can, for example, be the object type “load carrier”, “industrial truck”, “person”, and the object state 207 can, for example, be that the object is located at a specific location and / or in a defined area of ​​the digital map 203.

[0028] As further schematically illustrated in Figure 2, the processor device 111 of the data processing device 110 according to the invention is further configured to implement an evaluation logic 111b via a processing module 111b, wherein the evaluation logic 111b is configured to determine for each object of the plurality of objects, based on the respective object type 205 and the respective object state 207 and, if applicable, the digital map 203, whether a potentially hazardous situation exists (see 209). The data processing device 110 according to the invention can thus advantageously increase safety in an industrial environment, in particular a warehouse, through the automated detection of potentially hazardous situations.

[0029] In one embodiment, the digital map 203 of the warehouse is stored in the memory 105 of the data processing device 110. The data processing device 110 can be configured to create the digital map 203 of the warehouse itself or receive it from another entity. Further details on possible embodiments of the digital map 203 are described in German patent applications DE102022105079 and DE102021133614, the further disclosure of which is hereby incorporated by reference.

[0030] According to one embodiment, the processor device 111 can be configured to determine, in addition to the object type 205 and the object state 207, the object identity of a respective object based on the sensor data 201 and the digital map 203 of the warehouse. The object identity can thus define an individualization of objects of an object type. The object identity of an object can be determined by the processor device 111, for example, based on unique features, for example, the appearance of an object, i.e., the shape and color of the object, or based on an identification code attached to the object, for example, a barcode sticker.

[0031] According to one embodiment, the object state 207

[0032] Position information data, such as GPS coordinates, of an object can be included. Additionally, the object status 207 can contain information about when the object, such as the load carrier 140a, has been located in this section of the warehouse.

[0033] In one embodiment, the processor device 111 is configured to implement a neural network 111a, which is configured to determine, for each object of the plurality of objects, the object type 205 from the set of predetermined object types and the object state 207 from the set of predetermined object states based on the sensor data 201 and the digital map 203 of the warehouse. In other words: in one embodiment, the processing module 111a shown in Figure 2 is implemented by a neural network 111a, preferably a neural network for object recognition. In one embodiment, an output of the neural network 111a for object recognition is a digital feature set representing the associated object type 205 and / or the associated object status 207.

[0034] According to one embodiment, the evaluation logic 111b of the processing module 111b shown in Figure 2 can comprise a set of rules with a plurality of rules, wherein the plurality of rules define whether a potential dangerous situation exists for a plurality of combinations of object type 205 and object state 207. Thus, by means of the set of rules, a set of rules can be stored in the evaluation logic 111b which defines, for different combinations of object type 205 and object state 207, whether such a combination represents a potential dangerous situation. In other words: the evaluation logic 111b can define a type of “object policy” by means of the set of rules. Thus, by means of this set of rules, a plurality of dangerous situations orAnomalies in the warehouse are defined, such as damaged load carriers or damaged shelves for storing load carriers, blocked emergency exits, missing or blocked fire extinguishers, missing or blocked first aid kits, parking of industrial trucks in areas where it is not permitted, storage of material outside of designated areas of the digital map 207, load carriers 140a, b that are standing in an aisle of the warehouse for a configurable period of time, e.g. 5 minutes (when removal normally takes place during this time), industrial trucks 120a, b or people in areas of the warehouse in which they are not allowed to be, and the like. In one embodiment, one or more of the plurality of rules of the rule set can define "positively" that for one or more combinations of object type 205 and object state 207, no potentially hazardous situation exists.In this embodiment, analogous to a "white list," it is possible to positively define for each object which object states 207 are permitted for the object with the object identity or object type 205 and thus do not represent an anomaly or a potentially dangerous situation. For example, the rule set can define that objects of the object type 205 "load carrier" may be located at certain locations and / or in certain areas of the digital map 203.

[0035] Alternatively or additionally, one or more of the plurality of rules in the rule set can "negatively" define that a potentially dangerous situation exists for one or more combinations of object type 205 and object state 207. In this embodiment, analogous to a "blacklist," it is thus possible to define for each object identity and / or object type 205 which object states 207 of an object of a specific object identity or object type 205 are not permitted, thus representing a potentially dangerous situation. For example, the rule set can define that objects of the object type 205 "load carrier" may not be located at certain restricted locations and / or in certain restricted areas of the digital map 203.

[0036] According to one embodiment, the processor device 111 is configured to implement a further neural network 111b, which is configured to implement the evaluation logic 111b in order to determine, for each object of the plurality of objects, whether a potentially dangerous situation exists based on the object type 205 and the object state 207. In other words, in one embodiment, the processing module 111b shown in Figure 2 is implemented by a further neural network 111b. In one embodiment, the neural network 111a for object recognition and the further neural network 111b for implementing the evaluation logic can be subnetworks of a neural network. The neural network 111a for object recognition and / or the neural network 111b for implementing the evaluation logic can first be trained in a training phase using training data sets.The training data sets can, for example, be generated in various warehouses and labeled so that selected anomalies can be identified as potentially hazardous situations. In one embodiment, the processor device 111 can advantageously be configured to continuously retrain the neural network 111b for implementing the evaluation logic based on user feedback as to whether a detected potentially hazardous situation was actually a hazardous situation. In a further embodiment, the neural network 111b for implementing the evaluation logic can be configured to output that a potentially hazardous situation exists with a probability or confidence, for example, with a probability of more than 80%.

[0037] In one embodiment, if a potentially hazardous situation has been automatically detected by the data processing device 110 according to the invention, the communication interface 113 can transmit a warning. According to one embodiment, this warning can be embodied as a visual, acoustic, and / or haptic warning, and in one embodiment, can contain the position information of the object that caused the potentially hazardous situation. The warning can also be a message, for example, a text message or email, to a user of the data processing device 110. For example, in the event of a potentially hazardous situation, a warning can contain the following information: "Shelf aisle 5, shelf location 10, level 3 - defective horizontal shelf strut" or "Exit 15 is blocked."As already described above, the assignment can be made using the digital map 203, since the warehouse is represented as a digital twin, thus providing a localization of the potential hazardous situation. This allows for quick and efficient action to eliminate the potentially hazardous situation, thus increasing safety in a warehouse.

[0038] According to one embodiment, if a potential hazardous situation has been automatically determined with a high degree of probability, the processor device 111 can generate control signals and send them to the plurality of industrial trucks 120a, b via the communication interface 113 in order to suitably control the plurality of industrial trucks 120a, b to avoid the potential hazardous situation, for example, to adapt a respective movement route of the industrial trucks 120a, b. Figure 3 shows a flowchart with steps of a method 300 for the automated detection of a potential hazardous situation in an industrial environment, in particular a warehouse, with a plurality of objects, including a plurality of dynamic objects 120a, b, for example industrial trucks 120a, b, persons or the like, and a plurality of static objects 140a, b, for example load carriers 140a, b.The method 300 includes receiving 301 a plurality of sensor data 201 acquired in the industrial environment. The method further includes determining 303, for each object of the plurality of objects, an object type 205 from a set of predetermined object types and an object state 207 from a set of predetermined object states based on the sensor data 201 and the digital map 203 of the industrial environment. The method 300 further includes determining 303, for each object of the plurality of objects, whether a potentially hazardous situation exists based on the respective object type 205 and the respective object state 207.

[0039] The inventive method 300 can be carried out using the inventive data processing device 110. Therefore, further embodiments of the inventive method 300 result from the above-described embodiments of the inventive data processing device 110.

Claims

Patent claims 1. A data processing device (110) for the automated detection of a potential hazardous situation in an industrial environment having a plurality of objects, including a plurality of dynamic objects (120a, b) and a plurality of static objects (140a, b), the data processing device (110) comprising: a communication interface (113) configured to receive a plurality of sensor data (201) acquired in the industrial environment;and a processor device (111) which is designed to determine, on the basis of the sensor data (201) and a digital map (203) of the industrial environment, for each object of the plurality of objects, an object type (205) from a set of predetermined object types and an object state (207) from a set of predetermined object states, wherein the processor device (111) is further designed to implement an evaluation logic (111b), wherein the evaluation logic (111b) is designed to determine for each object of the plurality of objects, on the basis of the object type (205) and the object state (207), whether a potential hazardous situation exists.

2. Data processing device (110) according to claim 1, wherein the processor device (111) is designed to implement a neural network (111a) which is designed to determine, on the basis of the sensor data (201) and the digital map (203) of the industrial environment, for each object of the plurality of objects, the object type (205) from the set of predetermined object types and the object state (207) from the set of predetermined object states.

3. Data processing device (110) according to claim 2, wherein the neural network (111a) comprises a neural network for object recognition.

4. Data processing device (110) according to one of the preceding claims, wherein the evaluation logic (111b) comprises a set of rules with a plurality of rules, wherein the plurality of rules define for a plurality of combinations of object type and object state whether a potential dangerous situation exists.

5. The data processing device (110) of claim 4, wherein one or more rules of the plurality of rules of the rule set positively define that no potential hazardous situation exists for one or more combinations of object type and object state.

6. Data processing device (110) according to claim 4 or 5, wherein one or more rules of the plurality of rules of the rule set negatively define that a potential hazardous situation exists for one or more combinations of object type and object state.

7. Data processing device (110) according to one of the preceding claims 4 to 6, wherein the processor device (111) is designed to implement a further neural network (111b) which is designed to implement the evaluation logic (111b) in order to determine for each object of the plurality of objects on the basis of the object type (205) and the object state (207) whether a potential dangerous situation exists.

8. Data processing device (110) according to one of the preceding claims, wherein, if a potential dangerous situation has been determined, the communication interface is designed to send out a warning message.

9. Data processing device (110) according to one of the preceding claims, wherein the plurality of dynamic objects (120a, b) comprise a plurality of industrial trucks (120a, b) and wherein, if a potential dangerous situation has been determined, the processor device (111) is designed to generate control signals and to transmit them to the plurality of industrial trucks (120a, b) via the communication interface (113). to send in order to control the plurality of industrial trucks (120a, b) in order to avoid the potential dangerous situation.

10. A method (300) for the automated detection of a potential hazardous situation in an industrial environment with a plurality of objects, including a plurality of dynamic objects (120a, b) and a plurality of static objects (140a, b), the method (300) comprising: Receiving (301) a plurality of sensor data (201) acquired in the industrial environment; Determining for each object of the plurality of objects an object type (205) from a set of predetermined object types and an object state (207) from a set of predetermined object states on the basis of the sensor data (201) and a digital map (203) of the industrial environment; and Determining for each object of the plurality of objects whether a potential hazardous situation exists based on the object type (205) and the object state (207).

Images