Crane to assist operator in handling container hoisting system

Abstract

Embodiments of the present disclosure relate to a crane that assists an operator in handling a container hoisting system. A method, computer program and computer program product for assisting an operator in handling a crane of a container hoisting system, a container hoisting management device and a container hoisting system. The container hoist management device displays to an operator a progress indicator (PI) showing a representation of a plurality of consecutive steps performed by the crane in an operation cycle, acquires sensor data of the crane at a current point in time during the operation cycle, a current step currently performed by the crane in the operating cycle and a progress of the crane having completed the current step are determined based on the sensor data, and the progress at the current point in time is indicated in a representation of the current step of a progress indicator (PI).

Description

Cranes that assist operators in handling container lifting systems Technical Field

[0001] The present invention relates to a method for assisting an operator in handling at least one crane in a container lifting system, a computer program and computer program product, a container lifting management device, and a container lifting system. Background Technology

[0002] As crane systems become increasingly intelligent, their expanded capabilities offer new opportunities for automating crane operations. For crane operators, this can mean less workload or less need for human intervention, but it also presents some dangerous side effects, such as tedious and meaningless supervisory tasks. Creating meaningful work with stimulating tasks is a challenge. One solution is to offload workloads from multiple cranes and assign (multiple) operators to supervise multiple cranes simultaneously.

[0003] For example, CN102491206 describes a tower crane monitoring system that includes a single crane monitoring unit and multiple centralized crane monitoring units. The single crane monitoring unit is used for on-site monitoring and management of a single tower crane, transmitting monitoring information to the multiple centralized crane monitoring units, and is connected to a GPRS (General Packet Radio Service) public network. The multiple centralized crane monitoring units are used to centrally monitor the information from all the individual crane monitoring units on-site and control the tower cranes on-site.

[0004] However, there is still room for improvement in the way cranes are monitored, especially when the monitoring scale is expanded to several cranes. Summary of the Invention

[0005] Therefore, one object of the present invention is to improve the method of monitoring cranes in container lifting systems.

[0006] According to the first aspect, this objective is achieved by a method for assisting an operator in handling a crane of a container lifting system, the method being performed by a container lifting management device and comprising: displaying a progress indicator to the operator, the progress indicator showing a representation of a plurality of consecutive steps performed by the crane in an operating cycle; acquiring sensor data of the crane at a current point in time during the operating cycle; determining, based on the sensor data, the current step currently being performed by the crane in the operating cycle and the progress of the crane in completing the current step; and indicating the progress at the current point in time in the representation of the current step in the progress indicator.

[0007] According to the second aspect, this objective is achieved by a container lifting management device, the device including a processor operable to: display a progress indicator to an operator, the progress indicator showing a representation of a plurality of consecutive steps performed by the crane in an operating cycle; acquire sensor data of the crane at a current point in time during the operating cycle; determine, based on the sensor data, the current step currently being performed by the crane in the operating cycle and the progress of the crane in completing the current step; and indicate the progress at the current point in time in the current step representation of the progress indicator.

[0008] According to the third aspect, this objective is achieved by a container lifting system comprising a crane, a crane control device for controlling the crane, and a container lifting management device according to the second aspect.

[0009] According to the fourth aspect, this objective is achieved by a computer program for assisting an operator in handling a container lifting system. The computer program includes computer program code that, when executed by a processor, causes the processor to: display a progress indicator to the operator, showing a representation of multiple consecutive steps performed by the crane in an operating cycle; acquire sensor data at the current point in time during the operating cycle; determine, based on the sensor data, the current step currently being performed by the crane in the operating cycle and the progress of the crane in completing the current step; and indicate the progress at the current point in time in the representation of the current step in the progress indicator.

[0010] According to the fifth aspect, this objective is achieved by a computer program product for assisting operators in handling container lifting systems. The computer program product includes a data carrier having the computer program described in the fourth aspect.

[0011] Determining whether the crane has completed the current step can be done by determining whether the crane has completed the current step at the current point in time.

[0012] Container lifting systems may also include video capture systems associated with the crane.

[0013] The video capture system may include multiple cameras, at least one of which is mounted on a crane.

[0014] The steps can be represented as multiple rectangular areas provided along the first line, and the progress at the current time point can be indicated using a second line perpendicular to the first line. The second line can be, for example, a vertical line that moves along the horizontal first line during successive steps in the loop. The movement of the second line can be from left to right.

[0015] Each step can be represented as having one of two heights or levels above the first line, wherein the first height or level above the first line indicates that the crane is above the dock, while the second height or level above the first line indicates that the crane is above the cargo ship, and vice versa.

[0016] The progress indicator may also include symbols representing at least some of the steps in the process. These symbols may be displayed in rectangular areas within those steps. These symbols may be implemented as icons, for example.

[0017] The crane may have a spreading device, and the symbol for that spreading device may be used to indicate progress at the current point in time. This symbol may be used as a supplement to or alternative to the second line.

[0018] The method may also include indicating the current step in a progress indicator, and the container lifting management device may also be operable to indicate the current step in a progress indicator.

[0019] A video acquisition system can provide multiple video streams, where at least some of the steps performed by the crane in its operation cycle are linked to the video streams. A single step can be linked to one video stream. More specifically, it can be linked to only one video stream.

[0020] In this case, the method may also include identifying and displaying the video stream linked to the current step.

[0021] In the above situation, the container lifting management device can also be operated to identify and display the video stream linked to the current step.

[0022] At least some of the steps in a loop can be of different types, and each step type can have a corresponding confidence score. The confidence score provides the probability of a problem for that step type.

[0023] In this case, the method may further include: for each step, investigating the confidence score of the corresponding step type, and indicating in the representation of the step when the confidence score of that step type is insufficient.

[0024] In this case, the container lifting management device can also be operated to: for each step, investigate the confidence score of the corresponding step type, and indicate in the representation of the step when the confidence score of that step type is insufficient.

[0025] The survey of confidence scores may include: comparing the confidence score of a step type with the corresponding confidence score threshold, and determining that the confidence score is insufficient if the confidence score is higher or lower than the confidence score threshold.

[0026] When a step with an insufficient confidence score becomes the current step, that step can be indicated.

[0027] Different step types can be linked to indications of the potential complexity and / or time consumption of problem handling.

[0028] Different problems that may occur in the step type can be categorized according to the problem type.

[0029] It can provide an indication as the probability of a problem being the corresponding problem type.

[0030] For each type of problem, the skill set required to solve the problem (e.g., defined as a role) and / or the time required to solve the problem can be indicated.

[0031] An indication of potential complexity can provide an indication of the probability of a problem being a specific problem type, as well as the complexity of that problem type and / or the time required to process it.

[0032] In addition, the crane can initially be operated in supervised mode, where the crane is controlled by a corresponding crane control device.

[0033] In this case, the method may further include: when a step with an insufficient confidence score of the step type becomes the current step, switching from the supervision mode to the manual control mode, in which the crane is controlled by the operator.

[0034] In this case, the container lifting management device can also be operated to switch from supervisory mode to manual control mode when a step with insufficient confidence score becomes the current step, in which the crane is controlled by the operator.

[0035] When the crane is initially operated in supervised mode, the method may further include: detecting a problem at the current point in time; switching from supervised mode to manual control mode based on the detected problem, in which the crane is controlled by the operator; and displaying the problem to the operator in the progress indicator in the representation of the current step.

[0036] In this scenario, the container lifting management device can also be operated to detect problems at the current point in time; based on the detected problem, it switches from supervisory mode to manual control mode, in which the crane is controlled by the operator; and displays the problem to the operator in the progress indicator showing the current step.

[0037] Problem detection may include acquiring data from crane control devices and / or video acquisition devices, analyzing the data, and identifying problems based on the analysis.

[0038] The method may also additionally include providing the operator with suggestions on how to handle the problem, and the container lifting management device may also be used to provide the operator with suggestions on how to handle the problem.

[0039] The crane can be configured to handle containers in the current target area within an operation cycle.

[0040] In this case, the method may also include displaying the target area, as well as the previous target area, the subsequent target area, and the safety zone.

[0041] In this context, the container lifting management device can also be used to display the target area, as well as the previous and subsequent target areas and the safety zone.

[0042] The problem may be that one or more people are located within a safe zone surrounding the current target area.

[0043] In this case, the method may also include identifying a person who will handle the problem in or around the current target area, and establishing a communication session between the operator and that person.

[0044] In this scenario, the container lifting management device can also be operated to identify the person who will handle the issue at or around the current target area and establish a communication session between the operator and that person.

[0045] It should be emphasized that the term "including / comprises" as used in this specification is interpreted as specifying the presence of the stated feature, whole, step, or component, but does not exclude the presence or addition of one or more other features, wholes, steps, components, or combinations thereof. Attached Figure Description

[0046] The invention will now be described in more detail with reference to the accompanying drawings, in which:

[0047] Figure 1 schematically illustrates a container lifting system, which includes a group of cranes (each crane connected to a corresponding crane control device), multiple video acquisition systems (each video acquisition system associated with a corresponding crane), and a container lifting management device.

[0048] Figure 2 schematically illustrates the first video acquisition system and the multiple video streams it provides;

[0049] Figure 3 illustrates one implementation of a container lifting management device;

[0050] Figure 4 shows a computer program product in CD-ROM form with computer program code, which is used to implement the container lifting management function of the container lifting management device;

[0051] Figure 5 schematically illustrates a container ship using a crane to lift containers into or out of it.

[0052] Figure 6 schematically shows the crane progress indicator presented to the operator by the container lifting management device in supervisory mode;

[0053] Figure 7 shows a flowchart of the first number of method steps in a method for assisting operators in handling a container lifting system;

[0054] Figure 8 schematically shows the crane progress indicator presented to the operator by the container lifting management device in manual intervention mode;

[0055] Figure 9 shows a flowchart of the second number of method steps in a method for assisting operators in handling a crane in a container lifting system;

[0056] Figure 10 shows a flowchart of the third method step in a method for assisting operators in handling a container lifting system. Detailed Implementation

[0057] In the following description, specific details such as architectures, interfaces, and techniques are set forth in order to explain, rather than limit, the invention, in order to provide a full understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of known devices, circuits, and methods are omitted to avoid obscuring the description of the invention with unnecessary detail.

[0058] This invention is generally intended to assist operators of container lifting systems with respect to cranes in container lifting systems.

[0059] Figure 1 schematically illustrates a container lifting system 10, which includes a container lifting management device (CLMD) 12, a group of cranes, a group of crane control devices, and a group of video acquisition systems. For example, there is a first crane (CRA) 18A connected to a first crane control device (CCDA) 16A, a second crane (CRB) 18B connected to a second crane control device (CCDB) 16B, a third crane (CRC) 18C connected to a third crane control device (CCDC) 16C, and a fourth crane (CRD) 18D connected to a fourth crane control device (CCDD) 16D. The first crane control device 16A controls the first crane 18A, the second crane control device 16B controls the second crane 18B, the third crane control device 16C controls the third crane 18C, and the fourth crane control device 16D controls the fourth crane 18D. Furthermore, there is a group of video acquisition systems 14A, 14B, 14C, and 14D, each associated with a corresponding crane 18A, 18B, 18C, or 18D within that group of cranes. Therefore, there exists a first video acquisition system VCSA 14A associated with the first crane 18A, a second video acquisition system VCSB 14B associated with the second crane 18B, a third video acquisition system VCSC 14C associated with the third crane 18C, and a fourth video acquisition system VCSD 14D associated with the fourth crane 18D. Each video acquisition system may include one or more cameras mounted on or at the corresponding crane, and at least one camera is mounted on the corresponding crane and provides a main video stream having a field of view covering at least a portion of the crane (such as the crane's spreader).

[0060] Therefore, the crane group includes a first crane 18A, a second crane 18B, a third crane 18C, and a fourth crane 18D. The crane control device group includes a first crane control device 16A, a second crane control device 16B, a third crane control device 16C, and a fourth crane control device 16D. The video acquisition system group includes a first video acquisition system 14A, a second video acquisition system 14B, a third video acquisition system 14C, and a fourth video acquisition system 14D.

[0061] It should be understood here that other video capture systems may exist, such as a terminal video system covering one or more terminals where the crane operates. Crane control units 16A, 16B, 16C, 16D and video capture systems 18A, 18B, 18C, 18D are all connected to the container lifting management unit 12. The number of cranes and crane control units can be more or less. The simplest container lifting system 10 may consist of only one crane, corresponding crane control units, and container lifting management unit.

[0062] Crane control devices can be controllers, such as programmable logic controllers (PLCs), which are used to control the operation of the corresponding crane, such as controlling the crane to lift or raise objects such as containers.

[0063] The container lifting management device 12 can also be connected to the procedure database SDB 20.

[0064] Each video acquisition system can provide multiple video streams associated with the operation of the corresponding crane. Figure 2 shows four different video streams provided by the first video acquisition system 14A. As an example, the first video capture system 14A can provide a first video stream V1, a second video stream VS2, a third video stream VS3, and a fourth video stream VS4 together with the previously mentioned dock video system.

[0065] Figure 3 schematically illustrates one implementation of the container lifting management device CLMD 12. The container lifting management device 12 includes a processor PR 22 and a data storage device 24 with a computer program 26, which includes computer program instructions that, when executed by the processor 22, implement container lifting management functions. Furthermore, a first communication interface CI 28, implemented here as an Ethernet interface, is provided for communication with crane control devices 16A, 16B, 16C, 16D and video capture systems 14A, 14, 14C, 14D.

[0066] The processor 22 is also connected to a user interface 30 or HMI (human-machine interface) through which the operator can input and be presented with information about the operation of the crane lifting system 10. Therefore, the user interface 30 may include a display D32, which can advantageously be implemented as a touchscreen. However, the user interface may also include other data input methods, such as a keyboard, numeric keypad, joystick, and / or trackball.

[0067] Therefore, the container lifting management device 12 may include a processor 22 and an associated program memory 24, which includes a computer program 26 with computer program code for implementing container lifting management functions.

[0068] Computer programs can also be provided via computer program products, such as in the form of non-transitory computer-readable storage media or data carriers, like CD-ROMs, memory sticks, or memory cards, thereby carrying a computer program containing computer program code that, when run by a processor, will implement container lifting management functions. Figure 4 schematically illustrates a computer program product in the form of a CD-ROM 34 having the aforementioned computer program 26 including computer program code.

[0069] Figure 5 schematically shows the first crane 18A for moving container 48 into or out of cargo ship 36.

[0070] As can be seen in Figure 5, the first crane 18A may be equipped with a spreader SP 42, which is used to move container 48 from quay 38 to cargo ship 36 or from cargo ship 36 to quay 38. Container 48 may be lifted from or onto a truck chassis at quay 38, with the lifting accomplished using the flaps of spreader 42. The container may be placed inside hatch 44 or on deck 46 of cargo ship 36. For placement inside hatch 44, the hatch cover may need to be opened. Furthermore, locking mechanisms such as twist locks may be used to secure the container to the cargo ship, other containers, and / or chassis.

[0071] According to this disclosure, the lifting of containers onto or off a cargo ship using cranes is typically performed automatically under the control of a crane control device, and one or more operators can supervise this operation. The container lifting management device 12 can provide a supervision mode in which the operator obtains an overview of the automated crane control of a group of cranes. It can also provide other modes, which the operator can select at least some of.

[0072] When an operator is supervising several cranes, quickly getting an overview of the status of all of them can be extremely challenging. Currently, to gather status information for multiple cranes, the operator must navigate to multiple screens / windows separately. There is currently no comprehensive overview that allows for quick interpretation of the status of multiple cranes. To facilitate a faster overview of the operation of multiple cranes, it would be highly beneficial to compress and visualize information in a new way. Such a representation does not currently exist. Aspects of this disclosure aim to improve such visualization.

[0073] The present disclosure aims to provide a progress indicator for displaying information about a crane. The proposed crane progress indicator has a structure that allows it to be easily combined with other crane progress indicators. Therefore, multiple progress indicators can be combined on a single display to show information about multiple cranes.

[0074] These aspects will now be described in more detail with reference to Figures 6 and 7, wherein Figure 6 schematically shows a crane progress indicator presented to the operator by the container lifting management device in supervisory mode; and Figure 7 shows a flowchart of a first number of method steps in a method to assist the operator in handling the first crane of the container lifting system.

[0075] The container lifting management function of the container lifting management device can provide multiple progress indicators, with one progress indicator corresponding to each crane in a group of cranes monitored by the operator.

[0076] Figure 6 illustrates the Current Movement Window (CMW) provided for a crane, for example, for a first crane 18A. The Current Movement Window (CMW) is provided for the current movement performed in the crane's current operating cycle (i.e., the current operating cycle). The Current Movement Window (CMW) includes a progress indicator (PI) and additional data, which may include data such as movement number (MN), movement type (MT), container CTR being processed, first target area of ​​the container (e.g., target placement area (TPA)), estimated duration of the movement (ED), estimated remaining time of the movement (ETR), crane / spreader speed, and the weight of the container processed during the movement. The movement number (MN) may be a set of movement numbers defined in the work order for the crane, while the movement type may be one of several movement types, such as under-deck unloading, under-deck unloading, under-deck loading, under-deck loading, hatch opening, and hatch closing. The estimated duration (ED) may be the estimated time required for the movement, while the estimated remaining time (ETR) may be the estimated remaining time from the current time point in the operating cycle to the completion of the movement.

[0077] A progress indicator (PI) is an indicator of the progress of the current cycle. It can show a representation of multiple consecutive or sequential steps performed by the crane in an operating cycle, where each representation can be shown, for example, along a first (possibly horizontal) progress line. The representation of a step can be shown, for example, as multiple rectangular areas provided along the first progress line. The progress indicator PI can also indicate the step currently being performed by the crane and how far the crane has progressed in that step. Progress can be indicated by a second line perpendicular to the first line, which moves along the first progress line as a step is being performed, and this movement can be from left to right. Thus, the second line can move along the first line during consecutive steps in the cycle.

[0078] Each step can be represented as having one of two heights or levels above the first line, where the first height or level above the first line indicates that the crane is above the dock, while the second height or level above the first line indicates that the crane is above the cargo ship, and vice versa.

[0079] The progress indicator may also include symbols representing at least some of the steps. These symbols may be displayed in rectangular areas within these steps. These symbols may be implemented as icons, for example. Therefore, at least some of the steps may also include symbols indicating the type of operation being performed in the given step. Symbols may also be omitted from the step, for example if the operation performed in the step is the same as in a previous or subsequent step.

[0080] These operations may include: moving the crane between the dock and the cargo ship to reach the target lifting area, which may be divided into two steps: one above the dock and the other above the cargo ship; lowering the crane's spreader to the target lifting area; locking the spreader onto the container at the target lifting area; marking the container at the target lifting area; raising the spreader and moving the crane, along with the container, between the cargo ship and the dock to reach the target placement area, which may also be divided into two steps: one above the dock and the other above the cargo ship. Similar movements and symbols may exist when opening and closing hatches.

[0081] Therefore, different steps can be of different types, such as the following: moving the crane above the dock, between the dock and the cargo ship to reach the first target processing area; moving the crane above the cargo ship, between the dock and the cargo ship to reach the first target processing area; lowering the crane spreader to the first target processing area; locking the spreader to the container in the first target processing area; marking the container in the first target processing area; raising the spreader with the container from the first target processing area; moving the crane with the container above the cargo ship, between the cargo ship and the dock to reach the second target processing area; moving the crane with the container above the dock, between the cargo ship and the dock to reach the second target processing area; opening the hatch in the first or second target processing area; and closing the hatch in the first or second target processing area.

[0082] The first target processing area can be the target lifting area, which can be located on a dock or on a cargo ship; while the second target processing area can be the target placement area, which can also be located on a cargo ship or on a dock. The target area on a cargo ship can also be located above or below deck. The first target area can also be the hatch area on a cargo ship.

[0083] Furthermore, at least some of the steps performed by the crane in its operating cycle are linked to the video stream of the corresponding video acquisition system. Therefore, a step can be linked to a single video stream. More specifically, it can be linked to only one video stream.

[0084] In the example of Figure 6, this movement refers to the movement of unloading the container from the cargo ship and placing it on the dock. Therefore, the first step is to move the crane without the container from above the dock; the second step is to continue moving the crane without the container above the cargo ship, indicated by the first symbol S1; the third step is to lower the crane's spreader to the target area on the cargo ship, indicated by the second symbol S2; the fourth step is to lock the container at the target area on the cargo ship onto the crane spreader, indicated by the third symbol S3; the fifth step is, for example, to identify the container by reading a barcode or QR code, indicated by the fourth symbol S4; and the sixth step is to raise the crane spreader with the container from the target area on the cargo ship, indicated by the fifth symbol S5. Furthermore, there is the movement of the crane with the container from the cargo ship to the dock, where moving it above the cargo ship is performed in the seventh step and indicated by the sixth symbol S6; and moving it above the dock is performed in the eighth step. It should be recognized here that there may be other steps, such as lowering the container to the target area on the dock and releasing the container in that target area.

[0085] In addition to the current moving window, a view of the current target area environment can also be displayed, i.e., the target area environment into which or from which containers being processed in the loop will be placed. In other words, the target area is the area to be processed, where processing involves placing containers in or lifting containers from the target area.

[0086] The environment view of the current target area can also show adjacent container areas, which can be shown as previously processed target areas and / or subsequent target areas to be processed. Previous target areas are those that precede the current target area (i.e., in previous loops), while subsequent target areas are those that will be processed in the future (i.e., those that will become target areas after the current target area). A safety zone SZ may also exist around the current target area, taking into account whether adjacent target areas are previous or subsequent target areas. Therefore, the target area can be displayed together with previous target areas, subsequent target areas, and safety zones.

[0087] In the example of Figure 6, the target area is the target lifting area on the cargo ship, that is, the area from which the container will be lifted. Therefore, the target lifting area environment TLAE is shown.

[0088] In addition to the current moving window CMW and the target lifting area environment TLAE, one of the video streams from the associated video acquisition system can also be shown. In the example of Figure 6, the fourth video stream VS4 of the first video acquisition system 14A is shown.

[0089] Furthermore, different video streams provided by the first video acquisition system can cover different areas. For example, the first video stream VS1 can cover the dock, the second video stream VS2 can cover the deck of the cargo ship, the third video stream VS3 can cover the loading target area at a first zoom level, and the fourth video stream VS4 can cover the loading target area at a second zoom level. Additionally, at least some of the steps performed by the crane in the operation cycle can be linked to the video streams respectively. For example, the first video stream VS1 can be linked to the first and eighth steps, the second video stream VS2 can be linked to the second and seventh steps, the third video stream VS3 can be linked to the third and sixth steps, and the fourth video stream VS4 can be linked to the fifth and sixth steps.

[0090] This operation can be initiated by starting an operation cycle (S100), which may involve the crane beginning to perform movements within the operation cycle. This may trigger the container lifting management function of the container lifting management device, displaying a progress indicator (PI) to the operator (S110), which may involve displaying different steps and symbols associated with one or more of these steps.

[0091] The crane can also operate in a supervised mode, in which the crane is controlled by the crane control unit, and the operator only monitors the operation of the crane.

[0092] The container lifting management function can understand the duration of previous cycles. This can be used to estimate the duration (ED) of the current cycle. It can also predict the duration of the current movement based on one or more movements from previous cycles.

[0093] The container lifting management function can also acquire sensor data associated with the crane at the current point in time during the operation cycle (S120).

[0094] Cranes can be equipped with multiple sensors, and the crane can supply signals from these sensors to the corresponding crane control unit. A video acquisition system associated with the crane can also provide different views of the crane and the target area. One or more of these views can also overlay the crane spreader. The crane control unit can control the corresponding crane based on such sensor signals. Operations are typically cyclical and specified in a work order. Therefore, the crane control unit can control the crane to move the crane spreader to the target lifting area, identify the container at the target lifting area, lock the spreader onto the container, move the container from the target lifting area to the target delivery area, place the container in the target delivery area, and unlock the spreader from the container. Another type of cycle involves moving the crane spreader to a hatch and opening or closing the hatch. One target area is typically an area on a cargo ship, such as an area on or below deck, while another target area is typically an area on a dock, such as a chassis located on the dock.

[0095] The container crane management function can use this sensor data to determine the extent to which the crane has operated in a cycle, which can be used to determine the estimated remaining time (ETR). The progress of a completed cycle can be determined based on the time elapsed since the start of the cycle. This data, along with video images, can also be used to determine which step the crane is currently performing and how far along its operation is in that current step.

[0096] Therefore, the container lifting management function determines the current step of the crane's current execution or operation in the operation cycle based on sensor data, as well as the progress of the crane in completing the current step (S130).

[0097] The progress at the current point in time is then indicated in the representation of the current step in the progress indicator PI (S140). This can be achieved by showing the extent of the crane's progress in the current step, for example, by a second vertical line that moves along the first horizontal progress line during the execution of different steps, the movement of which can be from left to right. The second vertical line can be supplemented by a symbol representing at least a portion of the crane, such as a symbol representing the crane spreader, which can move along the first horizontal progress line together with the second vertical line. Therefore, the symbol for the spreader can also be used to indicate the progress at the current point in time. This symbol can be used as a supplement to or alternative to the second line.

[0098] The container lifting management function can also indicate the current step in the progress indicator (PI). In the example of Figure 6, the current step is indicated by a third line, which is located above and parallel to the first horizontal progress line and borders the upper edge of the rectangular area of ​​the current step.

[0099] The displayed information may also include the movement number MN, movement type MT, container CTR, and container destination area environment TPAE. This data can be obtained from the aforementioned work order.

[0100] Completed steps and the completed portions of the current step can be indicated, for example, by a special color (such as green), while the remaining steps and the remaining portions of the current step can be indicated by other colors (such as gray).

[0101] Optionally, the container lifting management function can also identify and display the video stream linked to the current step (S150), which in this example is the fourth video stream VS4. This video stream can be displayed between the current moving window CMW and the target lifting area environment TLAE.

[0102] After the current status mentioned above has been displayed to the operator, the container lifting management function can investigate whether the cycle has been completed. If it has not been completed (S160), new sensor data is acquired (S120), the current step is determined (S130), indicated in the progress indicator (S140), and the video stream linked to the current step is identified and displayed (S150). If the cycle has been completed (S160), the operation ends (S170).

[0103] The above operations can be performed on all cycles of a work order.

[0104] This demonstrates that a good overview of the cranes can be provided in a concise manner. This can be combined with similar views of other cranes under operator supervision, thereby simplifying the monitoring of the progress of each crane.

[0105] This demonstrates that different types of industry-specific operational data are collected, processed, and represented in the HMI, providing the supervisory view needed for continuous monitoring and intervention. This information can be displayed on RCS (Remote Control Station) or other devices with display capabilities.

[0106] In other words, the operation of a crane can be broken down into movements comprising different steps. This breakdown can be based on a predetermined motion map. This map is shown in a progress indicator, such as a horizontal bar chart with the parts. The progress indicator is associated with the crane's movement, typically defined as, for example, moving a container from a dock to a cargo ship. The exact specification definition of the movement can be retrieved from the work order.

[0107] The progress indicator can be supplemented with additional information, which is visualized in a separate area located near the progress indicator. The progress indicator and the supplementary information area can be continuously linked and updated as the crane operates.

[0108] Therefore, it can provide instructions on the actions planned and executed by the crane, as well as related information.

[0109] Therefore, the movement operation can be broken down into multiple steps or parts, each consisting of one or more actions performed or planned to be performed by the crane. These actions are collected and processed based on various data obtained from crane control devices and video acquisition systems associated with the crane. Each part or step is then combined into a concise progress indicator, such as a horizontal bar (but not limited to this), to graphically represent the crane's progress in a simple and comprehensive way. Each part can be supplemented with symbols (e.g., in the form of icons) to facilitate viewing the type of operation / work the crane is performing. Symbols are not limited to icons. Any type of graphic means (text, animation, etc.) can be used.

[0110] The progress indicator is associated with crane movements, which can be defined in a work order retrieved from a work order database. For each movement, the container crane management function can estimate the time required for the cycle and the remaining time needed to complete the movement. It can also break down the movement into different sub-movements based on a predetermined mapping. This information is displayed in the progress indicator.

[0111] The progress indicator can also visualize the location of the spreader, such as whether it is above a dock or a cargo ship. This can be achieved by adding a progress indicator area in any dimension, thus plotting height differences.

[0112] In addition, supplementary information can be visualized near the operation progress indicator. The current movement window can display detailed information about the crane's current operation and updates in sync with the progress indicator.

[0113] Therefore, a progress indicator with relevant information can be included within a graphical element (in this case, the currently moving window) to make it easy to identify and understand. Any type of graphical element or visualization technique can be used to "include" a progress indicator and its related information.

[0114] Therefore, the current mobile window can use any graphical method to represent various data types.

[0115] The progress indicator can be refreshed / updated with new moves until all moves in the work order are completed.

[0116] Video streams can provide a good view of the current step.

[0117] The container crane management function can therefore read various data related to crane operations and continuously refresh / update the progress indicator as the operation progresses. The progress indicator can be continuously updated with each move and can stop updating when there are no more moves remaining. Therefore, the progress indicator can be refreshed / updated with each new move until all moves in the work order are completed.

[0118] Progress indicators have many advantages:

[0119] It is easy to understand and provides concise visualizations that facilitate a good overview of the crane's operation process. This allows operators to better understand the crane's status and processes.

[0120] The video stream provides further assistance to operators in monitoring the current process.

[0121] Problems can occur during the operation of automated cranes. For example, the wrong container may be lifted. Hatch covers may also malfunction. Hatch covers may close when they should be open, or vice versa. Hatch covers may also be locked when they should be unlocked, or vice versa. Twist locks or spreader flaps may also malfunction. Twist locks may lock when they shouldn't, and spreader flaps may be misaligned. Furthermore, personnel may be found in the secure area (such as the secure area around the target area where the container is to be placed or lifted).

[0122] The aspects of this disclosure are intended to assist operators in dealing with such problems.

[0123] This will now be described with reference to Figures 8 and 9, wherein Figure 8 schematically shows a crane progress indicator presented to the operator by the container lifting management device in manual intervention mode, and Figure 9 shows a flowchart of a second number of additional method steps in the method of assisting the operator in handling the crane of the container lifting system.

[0124] Therefore, problems may arise regarding the crane.

[0125] For example, the wrong container might be lifted. Furthermore, the container's orientation during movement might be incorrect, for example, due to wind. This could prevent it from being placed in its intended target area. As another example, the twist lock on a container to be lifted from its target lifting area might accidentally lock to a lower surface, such as a deck, another container, or the chassis. There might also be people within the safety zone surrounding the target area. The target area might be inside a hatch, and the hatch cover might not be open. It might even be locked. All these scenarios are examples of potential problems. The aspects of this disclosure are intended to assist crane operators in handling such problems.

[0126] The container lifting management function initially offers a supervisory mode, where each crane in the group is controlled by a corresponding crane control unit, and a supervisory view of the group of cranes is displayed to the operator. This supervisory view can show the progress indicators of all cranes in the group. In this mode, each crane in the group is controlled by a corresponding crane control unit from a set of crane control units 16A, 16B, 16C, and 16D.

[0127] It can also display a video stream associated with the current step and the target area environment.

[0128] Problem detection may include acquiring data from crane control devices and / or video acquisition devices, analyzing the data, and identifying problems based on the analysis.

[0129] Then, the container lifting management function can detect a problem with one of the cranes at the current point in time (S200), for example, it could be the first crane 18A. Therefore, a problem with the first crane 18A can be detected at the current point in time. The container lifting management function can acquire data from the crane control unit and / or video acquisition system, analyze this data, and determine the problem based on the analysis. The data may include signals from the crane control unit, such as detected wind speed, crane position, and crane movement speed, as well as images from different video acquisition systems, which can then be analyzed. The problem can then be identified via images and / or by investigating signals received from the crane control unit. The container lifting management device 12 can also receive the location of people around the target area, for example, location signals transmitted by GPS transmitters on personnel or wireless terminals of these personnel. Such locations can then be compared with the target location of the container currently being processed to determine whether the location is within the safe zone of the container currently being processed. Alternatively, the presence of personnel can also be detected in the video stream of the video acquisition system.

[0130] Furthermore, when a crane problem is detected, the container lifting management function can also determine at which step the problem was detected. This can be achieved by determining the crane's position and direction of movement and / or the spreader's position and direction of movement, as well as the location information of the terminal and container ship. The current time of the cycle and the estimated remaining time can also be considered. The step can be determined by analyzing the video stream and / or position and speed signals associated with the crane provided by the crane control unit.

[0131] When a problem is detected with the crane, the crane will switch from automatic control to manual control, S210. Therefore, the operation of the crane experiencing the problem will switch from supervisory mode to manual control mode, where it is controlled by the operator. Thus, the crane switches from being controlled by the corresponding crane control device to being manually controlled by the operator. For example, the container lifting management function can suspend the automatic control performed by the corresponding crane control device and transfer control to the operator. Therefore, in this case, the crane experiencing the problem can be controlled by the operator instead of the corresponding crane control device. Then, the container lifting management function notifies the operator of the problem in the crane's progress indicator, S220, which can be done in the current step of the progress indicator PI. Therefore, the container lifting management function can display the problem to the operator in the representation of the current step of the progress indicator PI.

[0132] For example, a problem might exist with the first crane 18A, which could be indicated by the progress indicator PI shown in Figure 8. At least a portion of the rectangle representing the current step could then indicate the problem, for example, by a different color (such as yellow or red). In this case, the portion of the current step that has already been performed might be affected in this way, and the horizontal line on the upper edge of the rectangle might also be affected. The vertical lines representing the progress of the crane / spreader can be the same or other colors. For example, if one line is yellow, another line could be red.

[0133] Here, the progress indicator (PI) can signal about problems in the current step and, possibly, in subsequent steps. This signaling can be achieved by indicating the covered square area in the current step with a special color or highlighting, and by providing lines with the same highlighting or color above the rest of the square area and above square areas in one or more subsequent steps.

[0134] Operators can be further informed of issues through the video stream displayed in relation to the current step.

[0135] In this scenario, the view of the current target area environment may change. If the spreader SP is close to the current target area, the position of the spreader SP relative to the current target area can be indicated. The current target area can be displayed along with previous and subsequent target areas. Additionally, a safety zone SZ around the current target area can be indicated, which may extend through subsequent target areas near the current target area. The safety zone may cover the current target area and adjacent previous target areas, as well as subsequent target areas bordering the current target area.

[0136] For example, a container might fail to be parked in the target area because someone is detected inside the safety zone SZ. Therefore, crane 18A should handle the container in the current target area. However, the container lifting management function has detected the location of people in or around the current target area, and these locations mean that one or more people are within the safety zone surrounding the current target area, and furthermore, the safety zone SZ crosses multiple adjacent target areas that will be handled in subsequent cycles.

[0137] The container lifting management function can also identify the person in or around the current target area who needs to handle the problem and establish a communication session between the operator and that person.

[0138] For example, the target area might be located on the deck of a container ship, and the proposal could be linked to the nearest person with management responsibility, in this case, the deckhand. The location of person P1 detected within the safety zone SZ, as well as the location of deckhand P2, could also be displayed. These locations could be displayed, for example, in relation to the target lifting area environment TLAE.

[0139] Personnel location can be obtained using location services such as GPS. Image identification can also be used to detect personnel location within one or more video streams. Deckhands can also be equipped with mobile phones, such as smartphones, to establish communication sessions between operators and deckhands via container hoisting management devices and deckhand terminals. Information about deckhands can be retrieved from databases.

[0140] When a problem is detected in one of the cranes (S200), the container lifting management device 12 switches the crane from automatic control to manual control (S210) and informs the crane of the problem in the crane's progress indicator (S220).

[0141] The container lifting management function can also suggest at least one action to handle the problem (S230). The container lifting management function can therefore provide the operator with a suggestion for handling the problem. For example, one such suggestion could be to contact the nearest deckhand, P2.

[0142] Here, the container lifting management device can optionally investigate whether the operator has selected the action (S240), and perform an appropriate activity (S250) if the operator has not selected the action. One appropriate activity might be to wait until the operator has completed the action. Another possibility is to suggest an alternative action.

[0143] After the operator has performed the original or alternative action, the container lifting management function will continue and investigate whether the problem has been resolved. This check will also be performed if no investigation is conducted on the operator's action selection. If the problem has not been resolved, at S260, the process can return and wait for the problem to be resolved. If the problem has been resolved, at S260, for example, if no one is in the safety zone SZ, the container lifting management function can return to the supervisory mode where the crane is automatically controlled (S270), which can be achieved based on the operator's confirmation that the problem has been resolved. The progress indicator will then indicate that the problem has been stopped (S280).

[0144] Furthermore, after each cycle is completed, the container lifting management function can store the data in the step database 20. For each step and step type, it can indicate whether a problem exists. If a problem exists, it can also indicate the type of problem.

[0145] Therefore, problems can be categorized based on their type. Each problem type may be complex, requiring a certain level of skill to handle. There may also be an associated time commitment in processing that type of problem.

[0146] This demonstrates that operators receive assistance in problem-solving. When issues arise, the switch from supervisory mode to manual intervention mode can be initiated more quickly, potentially preventing dangerous situations. Furthermore, by storing settings indicating the presence of problems, a step database can be established. This database provides statistics on the probability of problems occurring in different types of steps, as well as statistics on the different types of problems that may occur within a step, which may have varying degrees of severity and require different skill levels to handle.

[0147] In the example given above, the function is reactive. Therefore, it reacts to detected problems. It should be recognized that progress indicators can also be used to alert the operator to potential problems. Thus, the user can be alerted to a potential problem in a step, which the operator can then use to switch to manual control mode (if he or she deems it necessary). Alternatively, if a potential problem is anticipated in a step, the system can automatically switch to manual control mode and then automatically return to supervised mode if no problem is actually detected.

[0148] Now, referring to Figures 8 and 10, we will describe how to handle this situation, where Figure 10 shows a flowchart of a third additional method step in the method of assisting the operator in handling the crane of the container lifting system.

[0149] Different types of steps can have confidence scores. These scores can be calculated based on the probability that a step of the corresponding type is prone to problems, and this probability can be determined based on the number of problems stored in the step database for that step type.

[0150] When calculating the confidence score for a step, steps that may require human intervention can also be identified. If later steps require human intervention, they can be highlighted in the progress indicator, thus drawing the operator's attention early and raising awareness so that the operator is prepared to intervene in a timely manner. Detecting whether certain actions require human intervention can be based on the frequency of past failures or on the confidence score calculated by the system for successful completion of the task. The same progress bar type shown in Figure 8 can be used for this purpose.

[0151] Each step in the loop can be of a different type, and each step type can have a corresponding confidence score.

[0152] A confidence score can be calculated as the probability that a step type is prone to problems. Therefore, the confidence score provides the probability of a problem for that step type. This can be calculated by dividing the number of times a problem occurs for that step type by the total number of times that step type is executed. Alternatively, the probability can also be calculated by dividing the number of times the step type is executed correctly by the total number of times that step type is executed.

[0153] In this case, the method may further include: for each step, investigating the confidence score for the corresponding step type (S300). This investigation may involve: comparing the confidence score for the step type with a corresponding confidence score threshold, and determining that the confidence score is insufficient if the confidence score is higher or lower than the confidence score threshold. Typically, if the confidence score is lower than the corresponding confidence score threshold, the confidence score can be determined to be insufficient. Alternatively, if the confidence score is higher than the corresponding confidence score threshold, the confidence score can be determined to be insufficient.

[0154] After discovering that the confidence score for a step type is insufficient, this can be indicated in the representation of the step for that step type (S310), and when that step becomes the current step, this can be indicated in the representation of the step.

[0155] For steps with insufficient confidence scores, the possible complexity or time required to solve them can also be indicated.

[0156] As mentioned earlier, the different problems that may arise in the step types can be categorized according to the type, and for each type, the skill set required to solve the problem (e.g., defined as a role) and / or the time required to solve the problem can be indicated.

[0157] Different step types can be linked to indications of the potential complexity and / or time consumption of problem handling.

[0158] Different problems that may occur in the step type can be categorized according to the problem type.

[0159] It can provide an indication as the probability of a problem being the corresponding problem type.

[0160] For each type of problem, the skill set required to solve the problem (e.g., defined as a role) and / or the time required to solve the problem can be indicated.

[0161] An indication of potential complexity can provide an indication of the probability of a problem being a specific problem type, as well as the complexity of that problem type and / or the time required to process it.

[0162] The percentage of different categories can be determined and presented to the operator. When a step's confidence score is unsatisfactory, the probability of different possible problem type categories can be presented. This allows the operator to be presented with information about which category the problem is most likely to belong to and the skills and / or time required to handle it.

[0163] Therefore, operators can prepare in advance for the types of interventions that may be needed. This can be used to allow appropriate personnel to be on standby.

[0164] In addition, when a step with insufficient confidence score becomes the current step, the mode can be switched from supervised mode to manual control mode.

[0165] The advantage of doing this is that it allows for the rapid resolution of unsafe situations, as the operator can be notified about it in advance, giving him or her more time to prepare.

[0166] While the invention has been described in conjunction with embodiments currently considered to be the most practical and preferred, it should be understood that the invention is not limited to the disclosed embodiments; rather, it is intended to cover various modifications and equivalent arrangements. Therefore, the invention is limited only by the following claims.

Claims

1. A method for assisting an operator in handling a crane (18A) of a container lifting system (10), the method being performed by a container lifting management device (12) and comprising: The operator is shown (S110) a progress indicator (PI), which shows a representation of multiple consecutive steps performed by the crane (18A) in an operating cycle; (S120) Obtain sensor data of the crane at the current time point during the operation cycle; determine (S130) the current step currently being performed by the crane in the operation cycle and the progress of the crane in completing the current step based on the sensor data; And the progress at the current time point is indicated (S140) in the representation of the current step in the progress indicator (PI).

2. The method of claim 1, wherein the step is shown as a plurality of rectangular regions provided along a first line, and the progress at the current time point is indicated by a second line perpendicular to the first line.

3. The method according to any one of the preceding claims, wherein each step is shown as having one of two heights above the first line, wherein the first height above the first line indicates that the crane is located above the dock, and the second height above the first line indicates that the crane is located above the cargo ship.

4. The method according to claim 2 or 3, wherein the progress indicator (PI) further includes symbols (S1, S2, S3, S4, S5, S6) representing at least some of the steps.

5. The method according to any one of the preceding claims, wherein the crane has a lifting device, the symbol of which is used to indicate progress at the current point in time.

6. The method according to any one of the preceding claims, wherein each step in the loop is of a different type, wherein each step type has a corresponding confidence score, and further comprising: For each step, investigate (S300) the confidence score of the corresponding step type, and indicate (S310) in the representation of the step when the confidence score of the step type is insufficient.

7. The method of claim 6, wherein the step is indicated when the step with an insufficient confidence score of the step type becomes the current step.

8. The method according to claim 6 or 7, wherein the crane (18A) is initially operated in supervised mode, wherein the crane (18) is controlled by a corresponding crane control device (16A), and further comprising: When a step with an insufficient confidence score becomes the current step, the system switches from the supervisory mode to the manual control mode, in which the crane is controlled by the operator.

9. The method according to any one of the preceding claims, wherein the crane is initially operated in a supervised mode, wherein the crane is controlled by a corresponding crane control device (16A), and further comprising: The problem is detected (S200) when the crane (18A) is at the current time point; Based on the detection of the problem, the system switches from the supervision mode (S210) to the manual control mode, in which the crane is controlled by the operator; and displays to the operator in the progress indicator (PI) the existence of the problem in the representation of the current step (S220).

10. The method according to any one of the preceding claims, wherein the crane is configured to process containers in the current target area during the operation cycle, and further comprises: The target area, as well as the previous target area, the subsequent target area, and the safety zone are displayed.

11. The method of claim 10, which is dependent on claim 9, wherein the problem is that one or more persons are located within the security zone surrounding the current target area, and the method further comprises identifying persons who will handle the problem at or around the current target area, and establishing a communication session between the operator and the persons.

12. The method according to any one of the preceding claims, wherein the container lifting system (10) includes a video acquisition system (14A) associated with the crane (18a), the video acquisition system providing multiple video streams (VS1A, VS2A, VS3A, VS4A, VS5A), wherein at least some of the steps are each linked to a corresponding video stream, the method further comprising identifying and displaying (S150) the video stream (VS4) linked to the current step.

13. A container lifting management device (12) includes a processor (22) operable to: display a progress indicator (PI) to an operator, the progress indicator (PI) showing a representation of a plurality of consecutive steps performed by a crane (18A) in an operating cycle; Acquire sensor data of the crane at the current point in time during the operation cycle; Based on the sensor data, determine the current step that the crane is currently performing in the operation cycle and the progress that the crane has made in completing the current step; And the progress at the current time point is indicated in the representation of the current step in the progress indicator (PI).

14. A container lifting system (10) comprising the crane (18A), a crane control device (16A) for controlling the crane, and a container lifting management device (12) according to claim 13.

15. A computer program (26) for assisting an operator in handling a crane (18A) of a container lifting system (10), the computer program (26) comprising computer program code that, when run by a processor (22), causes the processor (22) to: display a progress indicator (PI) to the operator, the progress indicator (PI) showing a representation of a plurality of consecutive steps performed by the crane (18A) in an operating cycle; Acquire sensor data of the crane at the current point in time during the operation cycle; Based on the sensor data, determine the current step that the crane is currently performing in the operation cycle and the progress that the crane has made in completing the current step; And the progress at the current time point is indicated in the representation of the current step in the progress indicator (PI).

16. A computer program product for assisting an operator in handling a crane (18A) of a container lifting system (10), the computer program product comprising a data carrier (34) having the computer program (26) according to claim 15.

Images