OPERATING PROCEDURES FOR A LOADING SYSTEM AND AUTOMATION SYSTEM FOR CONTROLLING A LOADING SYSTEM

DE502022002017D1Active Publication Date: 2024-11-14SIEMENS AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502022002017
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-09-19
Publication Date
2024-11-14
Estimated Expiration
2042-09-19

AI Technical Summary

Technical Problem

Current loading systems, including cranes, face challenges in collision avoidance due to high data volume and computing power requirements, which hinder certification for functional safety, leading to potential damage and safety risks.

Method used

The method involves creating a digital load image file and an environmental image file using an imaging sensor system, adding a protective distance to generate a collision model image file, and then forming a difference image file by subtracting reduced depth images, allowing collision detection on a certified control unit for timely intervention.

Benefits of technology

This approach enables efficient collision avoidance by relocating collision calculations to a safety-certified control unit, reducing computational effort and enhancing reaction time, thus ensuring functional safety and compliance with safety standards.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to an operating method for a loading system designed for loading or moving a load along a path, in which an environmental image file is cyclically recorded by means of an imaging sensor system during the loading process in order to avoid a collision between the load and objects in the surrounding area.

[0002] The invention also relates to an automation system designed to control a loading system for loading or moving a load along a path, comprising an imaging sensor system arranged to scan an environment, a sensor data computing unit designed to cyclically record sensor data of the sensor system and to create an environment image file and a control unit designed for fail-safe operation designed to control the loading system.

[0003] For the purposes of this invention, functional safety refers to the concepts and specifications described in the ISO 13849-1 standard. The DIN EN ISO 13849-1 standard "Safety of machinery - Safety-related parts of control systems" specifies requirements for the design of safety-related parts of control systems.

[0004] Collisions between a crane load (especially ISO containers) and the surroundings (especially stacks of ISO containers) or other obstacles pose a problem for both manually controlled and automatically controlled cranes. Collisions can cause damage to the crane itself, the load, or objects in the surroundings, and can also lead to injury or death. Crane operators therefore require a system to avoid and prevent collisions.

[0005] WO 2020 / 221490 A1 already describes a method for collision-free movement of a load with a crane.

[0006] A solution for controlling loading systems is also available at https: / / new.siemens.com / global / en / markets / cranes / harborcranes / load-collision-prevention-system.html. A sensor node records the crane's surroundings using sensors (radar, LIDAR, ultrasound, etc.). A control node is responsible for the actual movement control of the crane. The sensor node creates two- or three-dimensional maps of the crane's surroundings from the sensor data. The sensor node determines the position of the load either through its own measurements using its sensors or receives information from another source. For collision detection, the sensor node continuously calculates whether objects in the crane's surroundings are violating a protection zone around the load. As soon as the protection zone is violated, the sensor node sends a stop signal to the control node.The control node has control over the movement of the crane and can now stop the crane in case of an impending collision, depending on the stop signal.

[0007] The control node is typically implemented as a PLC (programmable logic controller). Due to the high memory and computing power requirements for processing the sensor data and storing the map, the sensor node is typically an industrial PC with correspondingly powerful software. The hardware currently used in the industrial PC is not certified according to applicable safety criteria, and TÜV would not approve such a system as a safety controller. The high memory and computing power requirements arise from the high-frequency, sometimes three-dimensional, environmental detection.

[0008] The object of the present invention is to provide an operating method for a loading system which makes it possible to implement collision avoidance despite the increased data volume on a specially certified control system designed for functional fault safety.

[0009] The problem is solved for an operating method for a loading system in that a digital load image file describing a total spatial extent of the load is provided, an environmental image file is cyclically recorded by means of an imaging sensor system, a protective distance is added to the data of the load image file and thus a collision model image file is provided and a section file is provided as a virtually recorded image file with a viewing angle of a position of the load depending on the path, wherein the section file is obtained as a reduced representation from the environmental image file, a difference image file is provided from the collision model image file and the section file by subtraction, an evaluation step is carried out, wherein the difference image file is examined for a possible collision.

[0010] For the purposes of the invention, "from a position" of the load is understood as follows. This position could be an origin or zero point in the load, or any chosen point in or on the load. This position does not necessarily have to be the center, but it must be ensured that all components or calculation methods involved have the same understanding of this position or reference point.

[0011] The method presented in the invention avoids the problem that the sensor node requires a large amount of information from the control node and may even require project-specific adaptation. As a solution, it is proposed to shift the collision calculation to the control node. The sensor node then cyclically transmits its view of the crane's environment to the control node. From this reduced representation of the environmental information, the control node is able to perform a collision calculation. The calculations required for this could thus also be performed on a safety controller, in particular an automation controller designed for functional fail-safety, and would thus be certifiable by a TÜV or other institution.

[0012] A continuation of the method involves using the imaging sensor system to record a point cloud as an environmental image file and to convert the section file from the point cloud into a data-reduced first depth image from the viewpoint of a virtual camera from a position of the load as a function of the path, whereby the collision model image file is available as a second depth image or is converted into this.

[0013] Accordingly, a reduced depth image of the crane's surroundings has been obtained from an environmental image file, which requires a very large amount of data, using a "virtual camera" (cropping algorithm). Furthermore, a depth image of the load, including a safety zone for the load, is now available. The existing depth images can be subtracted from one another without increased computational effort. In the resulting difference image, it is possible to determine whether a safety zone or protection zone has been violated. In the depth images, each point describes a distance value. In the simplest case, this distance value is the distance between an object's surface and the sensor. The first depth image shows the crane's surroundings as the distance to a position of the load or a load-handling device and is dynamically updated. The second depth image can, for example, represent the area of ​​the protection zone and thus maps the protection zone around the load from the same position.It is possible to dynamically adjust the protection area based on the speed driven.

[0014] For this purpose, an enveloping surface in the second depth image is dynamically adjusted by the protective distance.

[0015] With a view to improving the response time, the method provides that the difference formation for the provision of the difference image file from the collision model image file and the section file is not completed or the subtraction of the depth images starts in a focus area, because as soon as a first negative result is available, a collision statement can be made.

[0016] In order to achieve the best possible reaction time in predicting a possible collision, a difference formation can also begin in a focus area.

[0017] If the result is negative, the protection zone is violated and the crane reduces its speed or stops. Based on the reaction time, a collision avoidance reaction can be initiated at the first negative result. With regard to safety certification according to SIL or PL levels, collision detection using this method is very well suited. Because it involves a simple arithmetic operation, namely subtraction, it can be provided safely and quickly as a certified function in a safety controller designed for functional safety.

[0018] With regard to the nodes mentioned above, such as the sensor node and the control node, the method is advantageously executed on a sensor data processing unit and a control unit. The load image file is provided when the loading system is commissioned; the sensor data, which is determined by the imaging sensor system, is cyclically recorded on a sensor data processing unit and stored as the environment image file.

[0019] The section file is calculated as a reduced representation of the environment image file on the sensor data processing unit or on another processing unit, the conversion of the section file into the first depth image is carried out on the sensor data processing unit or on the another processing unit and the difference formation between the collision model image file and the section file is carried out on a control unit designed for fault tolerance.

[0020] It is considered advantageous here that the collision calculation is shifted to the control node or control unit. Collision calculation based on two-dimensional depth images can now be performed on the control node with minimal effort.

[0021] For the purposes of the invention, there are two common forms of representation for the results of stereoscopy or the recording of image files. One is point clouds, and the other is depth images. A depth image is essentially the representation of depth in a two-dimensional image using color coding. A grayscale representation is usually chosen, but coding similar to that used in a thermal imaging camera is also occasionally used.

[0022] Preferably, the depth images have Cartesian coordinates that are sampled equidistantly. A known neighborhood in depth images makes many evaluations easier to perform and allows for the application of 2D processing operations.

[0023] Advantageously, a container crane or a container bridge is used as the loading system and a container or a container including a load handling device is used as the load.

[0024] For the automation system mentioned at the outset, the object is achieved in that a digital load image file describing a total spatial extent of the load is present in the control unit designed for fail-safe operation, the control unit designed for fail-safe operation has a collision algorithm which is designed to add a protective distance to the data of the load image file in order to thereby provide a collision model image file, the sensor data computing unit has a projection means which is designed to provide a section file as a virtually recorded image file with a viewing angle of a position of the load depending on the path, wherein the projection means is designed to obtain the section file as a reduced representation from the environment image file, the collision algorithm is further designed,to create a difference image file from the collision model image file and the section file by subtraction, and there is an evaluation tool designed to examine the difference image file for a possible collision.

[0025] In order to minimize the computational effort on a control unit designed for fail-safe operation, the imaging sensor system and the sensor data unit are designed to record a point cloud as an environmental image file and, furthermore, the projection means is designed to convert the section file from the point cloud as a data-reduced first depth image from the viewpoint of a virtual camera of a position of the load as a function of the path, wherein the collision model image file is present in the control unit designed for fail-safe operation as a second depth image.

[0026] Furthermore, the control unit, which is designed for fail-safe operation, has an adaptation means which is configured to dynamically adapt an enveloping surface generated in the second depth image by the protective distance. Continuous adaptation of the safety zones is often necessary during operation of a loading system, since the design of the add-on components can vary from system to system. This enables a rapid response to a new system being put into operation. The dynamic adaptation of the protective zone also depends on the speed at which the load is being moved. For example, a large protective zone can be selected for a high speed and a smaller protective zone for a lower speed. External disturbances, such as a known uncertainty or sensor noise, can also be taken into account.

[0027] With regard to a fast reaction time, the collision algorithm has a focusing means which is designed to start the subtraction of the depth images in a focus area in order to obtain the best possible reaction time with regard to the collision.

[0028] For example, subtraction for providing the difference image file from the collision model image file and the detail file cannot necessarily be completed. If the subtraction of the depth images starts in a particularly relevant focus area, a stop signal can be issued as soon as the subtraction produces a negative result.

[0029] The drawing shows an embodiment of the invention, showing FIG 1 shows a block diagram of an automation system, FIG 2 shows a side view of a loading system (crane), FIG 3 shows a simplified representation of the environment with container stacks and simplified depth images from a bird's eye view, FIG 4 shows a side view of the crane with a load and a protective area, and FIG 5 shows the same side view of the crane with a load and a protective area that is violated.

[0030] According to FIG 1 An automation system 100 is shown configured to control a loading system for loading or moving a load L along a path x. The automation system 100 is essentially divided into a sensor data processing unit 12 and a control unit 14, wherein the control unit 14 is a certified programmable logic controller designed for functional safety. According to the prior art, the problem was that extensive data volumes from projection images could only be evaluated on a powerful industrial PC. The invention now advantageously proposes that, for a possible collision calculation on the control unit 14, only a section file is provided to the control unit as a reduced representation from an environmental image file UBD.

[0031] A point cloud 20 is obtained from sensor data 11 in the real world U using an imaging sensor system C, for example, LIDAR. This sensor data 11 is cyclically stored as the point cloud 20 in a memory 15 of the sensor data processing unit 12. This results in the environmental image file UBD.

[0032] The control unit 14, designed for fail-safe operation, in particular functional fail-safe operation, is fundamentally configured for the complete control of the loading system. According to the invention, a collision calculation is now also performed on the control unit 14 using a collision algorithm 30. The collision algorithm 30 implemented on the control unit 14 is configured to add a safety distance SA to the data of the load image file LBD and thus provide a collision model image file KMD. For different systems, the control unit 14 can also have additional collision model image files KMD2. The collision algorithm 30 is further configured to create a difference image file DBD from the collision model image file KMD and the detail file VAD from the sensor data processing unit 12.The difference image file DBD obtained by the difference formation is examined by means of an evaluation means 32 for a possible collision of the load L with the environment U.

[0033] The imaging sensor system 10 and the sensor data processing unit 12 are designed to record the point cloud 20 as an environmental image file UBD and furthermore the projection means 31 is designed to create the detail file VAD from the point cloud 20 as a data-reduced first depth image 22 from the viewpoint of a virtual camera 24 (see FIG 2 ) depending on the path x. The collision model image file KMD is available in the fail-safe control unit 14 as a second depth image 23.

[0034] The conversion of the sensor data 11 or the recorded point cloud 20 into a depth image 22 does not necessarily have to take place in the sensor data computing unit 12, but can also take place in another outsourced computing unit 13.

[0035] The control unit 14, which is designed for fail-safe operation, further comprises an adaptation means 33, which is designed to form an enveloping surface 26 generated in the second depth image 23 by the protective distance SA (see FIG 2 ) dynamically.

[0036] The control unit 14, which is designed for fail-safe operation, also has a focusing means 34, which is embedded, for example, in the collision algorithm 30 and is configured to begin the subtraction of the first depth image 22 and the second depth image 23 in a focus area 28 in order to obtain the best possible reaction time with regard to the collision or a prediction of a future collision. As soon as the subtraction results in a negative result, this is an indication that the protection zone SA has been or will be violated, and the control unit 14 can send a speed reduction or a stop signal.

[0037] The FIG 2 shows a container crane CC with a container C as load L, wherein the container C is held by a load handling device LM. The container crane CC can move a load L or the container C to a height h along a path x. The imaging sensor system 10 is installed on the container crane CC and on the load handling device LM and ensures that the environment U is recorded using the environment image file UBD. The sensor data 11, which the imaging sensor system 10 supplies, is forwarded to the sensor data processing unit 12. According to the invention, only a reduced image of the environment is forwarded in the sensor data processing unit 12 to the control unit 14 for collision calculation. This is because, using a virtual camera 24, which forms a view from the origin of the container C, only the section along the path over which the container C is located is required from the environment image file UBD.

[0038] The FIG 3 The explanation is based on simplified depth images from a bird's eye view of the surroundings. In the upper half of FIG 3 A first container stack 41, a second container stack 42 and a third container stack 43 are shown as examples. In the lower half of FIG 3 The simplified depth images are represented as a collision image KB, an environment image UB, a detail image AB, and a difference image DB. The representation in the drawing is highly simplified. In practice, spherical depth images will likely be used.

[0039] The surrounding image UB represents the container stacks 41, 42, 43 in a top view as black and gray areas, respectively. The collision image KB was generated from the collision model image file KMB as a second depth image 23. A generated enveloping surface 26 was added to the second depth image 23 using the safety distance SA. This enveloping surface 26 shows the protection zone that must not be violated. The difference image DB indicates a negative result when subtracting the section image AB from the surrounding image UB and the collision image KB, so that the control unit 14 can evaluate that a collision is imminent and can therefore reduce the speed or even send a stop signal.

[0040] Considering that the edge areas of the container stacks 41, 42, 43 in particular can be dangerous for a collision, a focus area 28 in which the subtraction is first started could be located in an edge area of ​​the container stacks 41, 42, 43.

[0041] The FIG 4 shows, by way of example, the container crane CC, how it intends to place the container C with its load-handling device LM between the second container stack 42 and the third container stack 43. Since, when forming the difference in the collision algorithm 30, it is calculated based on the evaluation of the data images that the envelope surface 26 is not touched, the container C can be placed completely between the second container stack 42 and the third container stack 43.

[0042] On the other hand, the FIG 5a placement of the container C between the second container stack 42 and the third container stack 43, but the container C is aligned too far towards the third container stack 43 and if the container C were to be moved further with the loading system, the collision algorithm 30 would indicate that the envelope surface 26 would be violated and the control unit 14 would then send out a stop signal.

Claims

1. Operating method for a loading system designed to load or move a load (L) along a route (x), with which during the loading process for avoiding a collision between the load (L) and objects in the environment (U), wherein an environment image file (UBD) is recorded cyclically by means of an imaging sensor system (10), characterised in that - a digital load image file (LBD) describing a spatial overall extent of the load (L) is provided, - a protective distance (SA) is added to the data of the load image file (LBD) and a collision model image file (KMB) is thus provided and - a cutout file (VAD) is provided as a virtually recorded image file with a viewing angle from one position of the load (L) as a function of the route (x), wherein the cutout file (VAD) is obtained as a reduced representation from the environment image file (UBD), - a differential image file (DBD) is provided from the collision model image file (KMD) and the cutout file (VAD) by means of differentiation, - an evaluation step is carried out, wherein the differential image file (DBD) is examined for a possible collision.

2. Operating method according to claim 1, wherein with the imaging sensor system (10) a point cloud (20) is recorded as an environment image file (UBD) and from the point cloud (20) the cutout file (VAD) is converted as a data-reduced first depth image (22) from the perspective of a virtual camera (24) from a position of the load (L) as a function of the route (x), wherein the collision model image file (KMD) is present as a second depth image (24) or is converted herein.

3. Operating method according to claim 2, wherein an enveloping surface (26) generated in the second depth image (23) by means of the protective distance (SA) can be adjusted dynamically.

4. Operating method according to one of claims 1 to 3, wherein the differentiation for the provisioning of the differential image file (DBD) from the collision model image file (KMD) and the cutout file (VAD) is not completed or the subtraction of the depth images (22,24) is started in a focus area (28), since as soon as a first negative result exists, a collision statement can be made.

5. Operating method according to one of claims 2 to 4, wherein - the load image file (LBD) is provided during commissioning of the loading system, - sensor data (11), which is determined by means of the imaging sensor system (10), is captured cyclically on a sensor data computing unit (12) and stored as the environmental image file (UBD), - the cutout file (VAD) is calculated as a reduced representation of the environment image file (UBD) on the sensor data computing unit (12) or on a further computing unit (13), - the conversion of the cutout file (VAD) into the first depth image (22) is carried out on the sensor data computing unit (12) or on a further computing unit (13), - the differentiation from the collision model image file (KMD) and the cutout file (VAD) is carried out on a control unit (14) designed for fault safety.

6. Operating method according to one of claims 1 to 5, wherein a container crane (CC) or a container bridge is used as a loading system and a container (C) or a container (L) including a load receiving means (LM) is used as a load (L).

7. Automation system (100) designed to control a loading system for loading or moving a load (L) along a route (x) comprising - an imaging sensor system (10) arranged to scan an environment, - a sensor data computing unit (12) designed to cyclically capture sensor data (11) of the sensor system (10) and to create an environment image file (UBD), characterised in that - a control unit (14) configured for fault safety and designed to control the loading system, - a digital load image file (LBD) describing a spatial overall extent of the load (L) is available in the control unit (14) designed for fault safety, - the control unit (14) designed for faulty safety has a collision algorithm (30), which is designed to add a protective distance (SA) to the data of the load image file (LBD) and thus to provide a collision model image file (KMD), - the sensor data computing unit (12) has a projection means (31), which is designed to provide a cutout file (BAD) as a virtually recorded image file with a viewing angle from one position of the load (L) as a function of the route (x), wherein the projection means (31) is designed to obtain the cutout file (VAD) as a reduced representation from the environment image file (UBD), - the collision algorithm (30) is further designed to create a differential image file (DBD) from the collision model image file (KMD) and the cutout file (VAD) by means of differentiation and an evaluation means (32) is available, which is designed to examine the differential image file (DBD) for a possible collision.

8. Automation system (100) according to claim 7, wherein the imaging sensor system (10) and the sensor data computing unit (12) are designed to record a point cloud (20) as an environment image file (UBD), furthermore the projection means (31) is designed, from the point cloud (20), to convert the cutout file (VAD) as a data-reduced first depth image (22) from the perspective of a virtual camera (24) from a position of the load (L) as a function of the route (x), wherein the collision model image file (KMD) exists in the control unit (14) designed for fault safety as a second depth image (24).

9. Automation system (100) according to claim 8, wherein the control unit (14) designed for fault safety has an adjustment means (33), which is designed to dynamically adjust an enveloping surface (26) generated by the protective distance (SA) in the second depth image (23).

10. Automation system (100) according to one of claims 7 or 9, wherein the collision algorithm (30) has a focusing means (34), which is designed to begin the subtraction of the depth images (22,23) in a focus area in order to obtain a best possible reaction time with respect to the collision.