Method of operating a loading system and automation system for controlling a loading system

Abstract

The present invention relates to a method of operating a loading system and an automation system for controlling a loading system, and more particularly to a method of operating a loading system designed to load or move a load along a path. During the loading process, in order to avoid collisions between the load and objects in the environment, a digital load image file describing the overall spatial extent of the load is provided by means of the loading system, an environmental image file is periodically recorded by means of an imaging sensor system, a protection distance is added to the data of the load image file, thereby providing a collision model image file, and a cut file is provided, which has a perspective from a position related to the path of the load as a virtual recorded image file. The cut file is obtained as a reduced representation from the environmental image file, a difference image file is provided from the collision model image file and the cut file by means of difference acquisition, and an evaluation step is performed to check the difference image file for possible collisions.

Description

Technical Field

[0001] The present invention relates to an operating method for a loading system, which is designed to load or move a load along a path. During the loading process, in order to avoid collisions between the load and objects in the environment, the loading system is used to periodically record environmental image files with the aid of an imaging sensor system.

[0002] The present invention also relates to an automation system designed to control a loading system for loading or moving a load along a path. The automation system includes: an imaging sensor system configured to scan the environment; a sensor data calculation unit designed to periodically capture the sensor data of the sensor system and create an environmental image file; and a control unit configured for fail-safe operation and designed to control the loading system. Background Art

[0003] In the context of the present invention, functional safety is understood as the concepts and specifications described in the standard ISO 13849-1. The "Machine Safety - Safety-related parts of control systems" in the standard DIN EN ISO 13849-1 specifies the safety-related parts of the control system.

[0004] Whether it is a manually controlled crane or an automatically controlled crane, collisions between the crane load (especially an ISO (International Organization for Standardization) type container regarded as a load) and the environment (especially an ISO type container stack regarded as the environment) or other obstacles pose problems. The collisions can damage the crane itself, the load, or objects in the environment or the crane environment, and can also cause casualties. Therefore, crane operators need equipment for avoiding and preventing collisions.

[0005] A method for moving a load collision-free using a crane has been described in the document WO 2020 / 221490 A1.

[0006] The link to https: / / new.siemens.com / global / en / markets / cranes / harbor-cranes / load-collision-prevention-system.html also discloses a method for controlling a loading system. Among them, the sensor node detects the crane environment with the help of sensors (radar, lidar, ultrasonic, etc.). The control node is responsible for the actual movement guidance of the crane. The sensor node creates a two-dimensional or three-dimensional map of the crane environment from the sensor data. The sensor node determines the load position by means of its own measurements with the sensor, or the load position is notified to the sensor node in other ways. To identify collisions, the sensor node continuously calculates whether the objects in the crane environment violate the protection area around the load. Once the protection area is violated, the sensor node sends a stop signal to the control node. The control node has a crane motion control system and can stop the crane according to the stop signal in the face of a collision.

[0007] The control node is usually implemented as a PLC (programmable logic controller). Due to the high requirements for the memory and computing power for processing sensor data and storing maps, the sensor node is usually an industrial computer with corresponding high-performance software. The hardware currently used by industrial computers is not certified according to effective safety standards, and (Technical Supervision Association) will not approve such a device as a safety control device. The high requirements for memory and computing power stem from the high-frequency and sometimes three-dimensional detection of the environment. Summary of the Invention

[0008] The object of the present invention is to provide an operating method for a loading system, which realizes collision avoidance on a control unit designed for special certification and functional fail-safe even in the case of increased data traffic.

[0009] The object of the invention is achieved for the operating method for a loading system by providing a digital load image file describing the overall spatial range of the load, periodically recording an environmental image file with the help of an imaging sensor system, adding a protection distance to the data of the load image file, and thus providing a collision model image file, and also providing a cut file, which has a perspective from the path-related position of the load as a virtual recorded image file, wherein the cut file is obtained from the environmental image file as a reduced representation, and a difference image file is provided from the collision model image file and the cut file by means of difference acquisition, and an evaluation step is performed, in which the difference image file is checked for possible collisions.

[0010] In the context of the present invention, the expression "from a position" is understood to mean that the position can be the origin or zero point in the load, or any arbitrarily selected point in or on the load. The position does not necessarily have to be located at the center of the load, but it must be ensured that all components or calculation methods involved have the same understanding of this position or reference point.

[0011] The method proposed by the present invention avoids the following problems: Sensor nodes require a large amount of information from the control node and may even have to be adjusted for specific projects. As a solution, it is proposed to transfer the collision calculation to the control node, so that the sensor nodes can periodically transmit their view of the crane environment to the control node. The control node can perform the collision calculation based on this reduced representation of the environmental information. The calculations required for this purpose can thus also be transferred to a safety controller, especially an automation controller designed for functional failure safety, and can thus be certified by other institutions.

[0012] An improvement of the method proposes to record the point cloud as an environmental image file using an imaging sensor system, and the clipped file is converted from the point cloud to a first depth image with reduced data, the first depth image being from the view of a virtual camera that is from a path-related position of the load, where the collision model image file exists or is converted to a second depth image.

[0013] For this purpose, a reduced depth image of the crane environment is obtained from the environmental image file, which requires a very high data volume, by means of a "virtual camera" (clipping algorithm). In addition, there is a load depth image including the load safety area. These depth images can be subtracted from each other without increasing the computational effort. In the difference image file thus formed, it can be recognized whether the safety area or the protection area is violated. For each point of the depth image, a distance value is described. In the simplest case, this distance value is the distance between the object surface and the above-mentioned sensor. The first depth image shows the crane environment as the distance to the load position or to the load handling device position, and the first depth image is updated dynamically. The second depth image can, for example, represent the area of the protection area and thus depicts the protection area around the load from the same position. The protection area can be adjusted dynamically based on the speed during travel.

[0014] For this purpose, the envelope surface in the second depth image can be adjusted dynamically according to the protection distance.

[0015] In terms of improved reaction time, the method proposes that the difference acquisition for providing the difference image file from the collision model image file and the clipped file is not completed, or the subtraction of the depth images is started within the focus area, because once the first negative result appears, a collision conclusion can be drawn.

[0016] In order to achieve the best possible reaction time when predicting a possible collision, differential acquisition can also be started within the focus area.

[0017] In the case of a negative result, the protection area is violated and the crane reduces its speed or stops. For the reaction time, the reaction to avoid a collision can be initiated when the first negative result occurs. In terms of safety certification according to SIL or PL levels, the collision recognition according to this method is very suitable. Since it involves simple arithmetic operations, namely subtraction, it can be provided as a certified function in a safety control designed for functional safety safely and quickly.

[0018] For the nodes mentioned at the beginning, such as sensor nodes and control nodes, it is advantageous to execute this method on the sensor data calculation unit and the control unit. During the commissioning of the loading system, a load image file is provided, and the sensor data determined by means of the imaging sensor system is periodically captured on the sensor data calculation unit and stored as an environmental image file. The clipping file is calculated as a reduced representation of the environmental image file on the sensor data calculation unit or other calculation units, the conversion of the clipping file to a first depth image is performed on the sensor data calculation unit or other calculation units, and the differential acquisition from the collision model image file and the clipping file is performed on the control unit designed for fail-safe operation.

[0019] It is considered advantageous here that the collision calculation is transferred to the control node or the control unit. With the aid of a two-dimensional depth image, the collision calculation can be performed on the control node at low cost.

[0020] In the context of the present invention, there are two common presentation forms for the results of stereoscopic photography or image file recording. One is a point cloud, and the other is a depth image. The depth image is essentially a presentation of the image depth in a two-dimensional image by means of color coding. Here, mostly a gray-scale presentation is selected, but occasionally a coding similar to that of a thermal imaging camera is also used.

[0021] Preferably, the depth image has equidistantly scanned Cartesian coordinates. Through the known adjacent regions in the depth image, many evaluations can be performed more easily, and operations of two-dimensional processing can be applied.

[0022] Advantageously, a container crane or a container bridge is used as the loading system, and a container or a container including a load receiving mechanism is used as the load.

[0023] For the automated system mentioned at the beginning, the above object of the invention is achieved as follows: a digital load image file describing the overall spatial extent of the load is provided in the control unit designed for fail-safe operation, the control unit designed for fail-safe operation has a collision algorithm which is designed to add a protective distance to the data of the load image file, thereby providing a collision model image file, the sensor data calculation unit has a projection mechanism which is designed to provide a cut file, the cut file being a virtual recorded image file having a perspective of a position related to the path from the load, wherein the projection mechanism is designed to obtain the cut file as a reduced representation from an environmental image file, the collision algorithm is also designed to create a difference image file from the collision model image file and the cut file by means of difference acquisition, and an evaluation mechanism is provided which is designed to check the difference image file for possible collisions.

[0024] To minimize the computational effort of the control unit designed for fail-safe operation, the imaging sensor system and the sensor data calculation unit are designed to record a point cloud as the environmental image file, the projection mechanism is also designed to convert the cut file from the point cloud into a first depth image with reduced data, the first depth image being from the view of a virtual camera which is from a position related to the path of the load, wherein the collision model image file exists in the control unit designed for fail-safe operation as a second depth image.

[0025] Furthermore, the control unit designed for fail-safe operation has an adjustment mechanism which is designed to dynamically adjust the envelope surface generated via the protective distance in the second depth image. Since the designs of the components of different devices may be different, it is usually necessary to continuously adjust the safety area during the operation of the loading system, and thus the system newly put into operation can be reacted to quickly. The dynamic adjustment of the above-mentioned protective area also depends on the speed at which the load is moving, so for example the protective area can be selected to be larger at high speeds and smaller at lower speeds, and external interference variables such as the known uncertainties or noise of the sensors can also be taken into account.

[0026] For a fast response time, the collision algorithm also has a focusing mechanism which is designed to start subtracting the depth images within a focused area in order to obtain the best possible response time for a collision.

[0027] The difference acquisition for providing the difference image file from the collision model image file and the cut file does not have to be completed. Because, if the subtraction of the depth images is started in a particularly relevant focused area, a stop signal is output as soon as the result of the difference acquisition is negative. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings illustrate embodiments of the present invention.

[0029] Figure 1 A block diagrammatic schematic diagram showing an automated system

[0030] Figure 2 A side view showing a loading system (crane)

[0031] Figure 3 A simplified environmental schematic diagram with a container stack and a bird's-eye view of a simplified depth image

[0032] Figure 4 A side view of a crane with a load and a protection area, and

[0033] Figure 5 A side view of the same side of the crane with a load and a violated protection area Detailed implementation mode

[0034] According to Figure 1 , the automated system 100 is designed to control a loading system for loading or moving a load along a path x. The automated system 100 is basically divided into a sensor data calculation unit 12 and a control unit 14, where the control unit 14 is a certified programmable logic controller designed for functional safety. The problem in the prior art is that a large amount of data of a projection image can only be evaluated on a powerful industrial computer. Now the advantage of the present invention is that, for possible collision calculations on the control unit 14, only a cut file, which is a reduced representation from an environmental image file UBD, needs to be provided to the control unit

[0035] With the help of an imaging sensor system, such as LIDAR (Light Detection and Ranging), a point cloud 20 composed of sensor data 11 is obtained from the real world. The sensor data 11 is periodically stored as the point cloud 20 in the memory 15 of the sensor data calculation unit 12. An environmental image file UBD is generated therefrom

[0036] The control unit 14, which is designed for fail-safe, especially functionally fail-safe, is basically used to fully control the loading system. According to the present invention, collision calculations are now also performed on the control unit 14 with the help of a collision algorithm 30. The collision algorithm 30 implemented on the control unit 14 is designed to add a protection distance SA to the data of a load image file LBD, and thus provide a collision model image file KMD. For different systems, the control unit 14 can also have another collision model image file KMD2. The collision algorithm 30 is also designed to create a difference image file DBD between a cut file VAD from the sensor data calculation unit 12 and the collision model image file KMD. With the help of an evaluation mechanism 32, the difference image file DBD obtained by means of difference acquisition is checked for possible collisions between the load and the environment U

[0037] The imaging sensor system 10 and the sensor data processing unit 12 are designed to record the point cloud as an environment image file UBD. In addition, the projection device 31 is designed to convert the clipping file VAD from the point cloud 20 into a data-reduced first depth image 22, which originates from the view of the virtual camera 24 related to the path (see Figure 2 ). The collision model image file KMD exists as a second depth image 23 in the control unit 14 designed for fail-safe operation.

[0038] The conversion of the sensor data 11 or the recorded point cloud 20 into the first depth image 22 does not necessarily have to take place in the sensor data computing unit 12 , but can also take place in an outsourced further computing unit 13 .

[0039] The control unit 14 designed for fail-safe operation further comprises an adjustment mechanism 33, which is designed to dynamically adjust the envelope surface 26 generated in the second depth image 23 via the protection distance SA (see Figure 3 ).

[0040] The control unit 14 designed for fail-safe operation also has a focusing mechanism 34, which is embedded in the collision algorithm 30, for example, and is designed to start subtracting the first depth image 22 from the second depth image 23 within the focus area 28 in order to obtain the best possible reaction time to a collision or a description of a future collision. Because, as soon as the subtraction results in a negative result, this indicates that the protective zone has been violated or will be violated, and the control unit 14 can issue a deceleration or stop signal.

[0041] Figure 2 A container crane CC is shown with a container C as a load, wherein the container C is held by a load receiving mechanism LM. The container crane CC can move the load or container C along a path x to a height h. An imaging sensor system 10 is mounted on the container crane CC and the load receiving mechanism LM and ensures the recording of the environment U by means of an environment image. The sensor data 11 provided by the imaging sensor system 10 are forwarded to a sensor data calculation unit 12. According to the invention, the sensor data calculation unit 12 now only forwards a reduced environment image to the control unit 14 for collision calculations. Since a virtual camera 24 is used which forms a view originating from the container C, only the sections along the path, in which the container is also fixed, need to be acquired from the environment image file UBD.

[0042] to this end, Figure 3 An illustration of a simplified depth image resulting from a bird's-eye view of an environment is shown. Figure 3 The upper part of FIG. 4 shows, by way of example, a first container stack 41, a second container stack 42 and a third container stack 43. Figure 3In the lower part, the simplified depth image is presented as a collision image KB, an environment image UB, a shear image AB, and a difference image file DB. The illustration in the drawing is significantly simplified. In practice, a spherical depth image is more likely to be used.

[0043] The environment image UB is presented as black or gray areas in the top view of the container stacks 41, 42, 43. The collision image KB is generated from the collision model image file KMB that is the second depth image 23. The envelope surface 26 generated via the protection distance SA is added to the second depth image 23. The envelope surface 26 shows the protection area where infringement is not allowed. The difference image file DB gives a negative result obtained by subtracting the shear image AB from the environment image UB and the collision image KB, such that the control unit 14 can evaluate that a collision occurs first, and thus can reduce the speed or even send a stop signal.

[0044] Considering that especially the edge areas of the container stacks 41, 42, 43 may be dangerous for collisions, the focusing area 28 where the subtraction starts first can be located in the edge areas of the container stacks 41, 42, 43.

[0045] Figure 4 Exemplarily shown is how the container crane CC places the container C between the second container stack 42 and the third container stack 43 with the load receiving mechanism LM of the container. Since it is calculated that the envelope surface 26 is not touched when obtaining the difference using the data image evaluation in the collision algorithm 30, the container C can be placed between the second container stack 42 and the third container stack 43 until the execution is completed.

[0046] On the contrary, Figure 5 Shows the situation where the container C is positioned between the second container stack 42 and the third container stack 43, where the container C is too far from the third container stack 43. If the container C moves further with the loading system, according to the collision algorithm 30, it will be shown that the envelope surface 26 is violated, and the control unit 14 will thus send a stop signal.

Claims

1. An operating method for a loading system, which is designed to load or move a load along a path (x), and during the loading process, in order to avoid a collision of the load with an object in the environment (U), an environmental image file (UBD) is periodically recorded by means of the loading system with the aid of an imaging sensor system (10), characterized in that a digital load image file (LBD) is provided, which describes the spatial overall extent of the load, a protection distance (SA) is added to the data of the load image file (LBD), thereby providing a collision model image file (KMD), and a cut file (VAD) is provided, which, as a virtual recorded image file, has a perspective from a position of the load related to the path (x), wherein the cut file (VAD) is obtained as a reduced representation from the environmental image file (UBD), a difference image file (DBD) is provided from the collision model image file (KMD) and the cut file (VAD) by means of difference acquisition, an evaluation step is performed, in which the difference image file (DBD) is checked for possible collisions.

2. The operating method according to claim 1, wherein, A point cloud (20) is recorded as the environmental image file (UBD) by means of the imaging sensor system (10), and the cut file (VAD) is converted from the point cloud (20) into a first depth image with reduced data, which first depth image is from the view of a virtual camera (24), the view being from a position of the load related to the path (x), wherein the collision model image file (KMD) exists as a second depth image or is converted into a second depth image.

3. The operating method according to claim 2, wherein, An envelope surface (26) generated by means of the protection distance (SA) in the second depth image can be dynamically adjusted.

4. The operating method according to claim 2 or 3, wherein The difference acquisition for providing the difference image file (DBD) from the collision model image file (KMD) and the cut file (VAD) is not completed, or the subtraction of the first depth image and the second depth image is started within a focus area (28), because a collision conclusion can be drawn as soon as the first negative result appears.

5. The operating method according to claim 2 or 3, wherein the load image file (LBD) is provided during commissioning of the loading system, sensor data (11) determined by means of the imaging sensor system (10) are periodically captured on a sensor data calculation unit (12) and stored as the environmental image file (UBD), the cut file (VAD) is calculated on the sensor data calculation unit (12) as a reduced representation of the environmental image file (UBD), the conversion of the cut file (VAD) into the first depth image is performed on the sensor data calculation unit (12), the difference acquisition from the collision model image file (KMD) and the cut file (VAD) is performed on a control unit (14) designed for fail-safe operation.

6. The operating method according to any one of claims 1 to 3, wherein Use a container crane (CC) or a container bridge as the loading system, and use a container (C) or a container (C) including a load receiving mechanism (LM) as the load.

7. An automated system (100) designed to control a loading system for loading or moving a load along a path (x), the automated system (100) comprising: An imaging sensor system (10) configured to scan the environment, A sensor data calculation unit (12) designed to periodically capture sensor data (11) of the imaging sensor system (10) and create an environmental image file (UBD), Characterized in that, A control unit (14) configured for fail-safe operation and designed to control the loading system, In the control unit (14) designed for fail-safe operation, there is a digital load image file (LBD) describing the overall spatial extent of the load, The control unit (14) designed for fail-safe operation has a collision algorithm (30), the collision algorithm (30) being designed to add a safety distance (SA) to the data of the load image file (LBD) to provide a collision model image file (KMD), The sensor data calculation unit (12) has a projection mechanism (31), the projection mechanism (31) being designed to provide a cut file (VAD), the cut file (VAD) having a perspective of a position related to the path (x) from the load as a virtual recorded image file, wherein the projection mechanism (31) is designed to obtain the cut file (VAD) as a reduced representation from the environmental image file (UBD), The collision algorithm (30) is also designed to create a difference image file (DBD) from the collision model image file (KMD) and the cut file (VAD) by means of difference acquisition, and there is an evaluation mechanism (32), the evaluation mechanism (32) being designed to check the difference image file (DBD) for possible collisions.

8. The automated system (100) according to claim 7, wherein, The imaging sensor system (10) and the sensor data calculation unit (12) are designed to record a point cloud (20) as an environmental image file (UBD), the projection mechanism (31) is also designed to convert the cut file (VAD) from the point cloud (20) into a first depth image with reduced data, the first depth image being from the view of a virtual camera (24), the view being from a position related to the path (x) of the load, wherein the collision model image file (KMD) exists in the control unit (14) designed for fail-safe operation as a second depth image.

9. The automated system (100) according to claim 8, wherein, The control unit (14) designed for fail-safe operation has an adjustment mechanism (33), the adjustment mechanism (33) being designed to dynamically adjust the envelope surface (26) generated by the safety distance (SA) in the second depth image.

10. The automation system (100) according to claim 8 or 9, wherein, The collision algorithm (30) has a focusing mechanism (34) designed to start subtracting the first depth image and the second depth image within a focusing area (28) in order to obtain an optimal possible reaction time regarding the collision.

Images