Real-time monitoring system and method for container gantry crane yard operation status
Through the real-time monitoring system for operating status of container gantry cranes on port cranes, the distributed processing platform and multi-view cameras are used to realize automated monitoring and alarms, which solves the problems of low utilization rate and low intelligence of remote cameras, and improves the Work efficiency and safety protection level.
Patent Information
- Application Number
- CN202111477953.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-06
AI Technical Summary
The existing remote control cameras have low utilization rates in port crane operation status monitoring, rely on artificial visual observation, and have low intelligence, resulting in insufficient operating efficiency and safety protection levels.
Design a real-time monitoring system for operating status of container gantry cranes, using existing remote control cameras and distributed processing platforms, and conducts anti-collision monitoring of large trucks, confirmation of door orientation, confirmation of yards and container status through core algorithm modules. Box deformation detection realizes automatic switching and alarm information output, improving operational safety and abnormal protection.
It effectively improves the operating efficiency and safety protection level, reduces the economic losses and efficiency reduction caused by large truck collision accidents and abnormal box door orientation, promptly detects safety hazards in the yard and container, and improves the timeliness of safety accident warnings.
Smart Images

Figure CN114132842B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of port remote safety monitoring, and in particular to a real-time monitoring system and method for container gantry crane yard operation status based on visual recognition. Background Art
[0002] Remote control system is one of the latest technologies in the port industry. The remote control driver operates in the remote control room (central control room) instead of in the driver's cab of the port machinery equipment at the terminal. This brings many benefits to terminal operators, such as remote control drivers can stay away from noise, have a more comfortable working environment, more free working space, higher operating efficiency, and lower operating costs. In the remote operation station (ROS), the remote control driver can see the operation status and working area of the port machinery equipment through the video sent back from the port machinery equipment by the camera; the remote control driver can control the port machinery equipment through the control handle or the buttons on the touch screen; the remote control driver can understand the operation status of the port machinery equipment in real time through the remote operation interface displayed on the main screen; the cameras installed at various positions on the port machinery equipment provide the remote control driver with necessary visual assistance. In fact, the remote operation station can achieve the same functions in the remote control room as in the driver's cab of the port machinery equipment.
[0003] After the local operation is switched to remote operation, higher requirements are put forward for operation safety and abnormal protection; at the same time, in order to meet the operation requirements of the driver in the remote control room and the traditional manual inspection requirements, multiple cameras are installed on the current crane, which basically covers all the operation angles of the crane. However, the utilization rate of the existing remote control cameras is low, and the monitoring of the operation status still relies more on manual visual observation, with a low degree of intelligence. To this end, the applicant has researched and developed a state monitoring system that integrates the operation status of the crane and multi-view cameras. It efficiently utilizes the existing remote equipment's own cameras, can assist or gradually replace traditional manual inspections, has a high degree of intelligence, and can effectively improve the operation efficiency and safety protection level. Summary of the invention
[0004] In view of the defects and shortcomings in the prior art, the present invention proposes a real-time monitoring system and method for the operating status of a container gantry crane yard, which fully utilizes the existing remote-control cameras, distributed front-end computing processing units, and efficiently integrates the crane operating status and multi-view cameras. The multiple functions can be automatically switched according to the operating status, with a high degree of intelligence, which effectively improves the operating efficiency and safety protection level.
[0005] The above-mentioned purpose of the present invention is achieved through the following technical solutions:
[0006] The real-time monitoring system for the operation status of container gantry crane yard includes a stand-alone control system installed on the container gantry crane and a remote operation console installed in the central control room. The stand-alone control system is connected to the remote operation console in communication and includes a stand-alone PLC, several network cameras, and a distributed processing platform; wherein,
[0007] The stand-alone PLC controls the operation of the crane and can send the current operation status information of the crane to the distributed processing platform;
[0008] The plurality of network cameras are arranged at different positions of the crane, and can capture the operation dynamics or working area of the crane from different working angles in real time;
[0009] The remote control console is provided with a main screen which can display the remote operation interface, so as to facilitate the remote control driver to perform remote control operations;
[0010] The distributed processing platform includes:
[0011] The video stream access module is responsible for accessing and forwarding the video data captured by the network camera;
[0012] The core algorithm module is responsible for completing large vehicle anti-collision monitoring, container door orientation confirmation, yard and container status confirmation, container deformation detection based on the real-time images captured by the network camera, and obtaining corresponding operation safety and abnormal protection monitoring results;
[0013] The PLC interaction module is responsible for interacting with the stand-alone PLC, obtaining the current operation status information of the crane, and transmitting action instructions;
[0014] The alarm information output module sends the corresponding alarm information to the remote operation console according to the operation safety and abnormal protection monitoring results, and gives safety hazard prompts in time;
[0015] The WEB server provides a remote access portal to facilitate historical record query, network camera calibration parameter setting and system preset configuration.
[0016] Preferably, the plurality of network cameras include:
[0017] There are four network cameras for trolley anti-collision monitoring, which are installed on the left trolley on the reel side, the right trolley on the reel side, the left trolley on the non-reel side, and the right trolley on the non-reel side of the crane;
[0018] The network camera used to confirm the direction of the container door is shared by two reel-side container cameras and two non-reel-side container cameras used to capture the top number of the container truck;
[0019] There are six network cameras used to confirm the status of the yard and containers, two of which are installed in front and behind the crane trolley frame, and the remaining four are installed on the inside of the crane spreader lock pin;
[0020] There are six network cameras used for box deformation detection, four of which are installed on the inside of the crane sling lock pin, and the remaining two are installed on the front and rear of the crane trolley frame.
[0021] Preferably, the distributed processing platform uses an 8-core ARM V8.2 64-bit CPU, a 512-core Volta GPU, and 32GB of memory.
[0022] Preferably, the distributed processing platform is connected to the plurality of network cameras and the remote operation console via TCP communication via a convergence switch in the crane electrical room.
[0023] Preferably, the remote operating console is also provided with a control handle, and the main screen is a touch display screen, and the remote control driver can perform remote operation through the control handle or buttons on the touch display screen.
[0024] Preferably, the central control room is deployed with multiple remote operating consoles, and according to actual operation requirements, the distributed processing platform on each crane can be switched and connected to a certain remote operating console.
[0025] Preferably, the core algorithm module runs a program including multiple deep neural networks, wherein the multiple deep neural networks include multiple detection networks for implementing different detection tasks and multiple segmentation networks for implementing different segmentation tasks.
[0026] The present invention also proposes a method for real-time monitoring of the operating status of a container gantry crane yard, using the above-mentioned monitoring system, comprising the following steps:
[0027] S1: The distributed processing platform on the crane scans the single-machine PLC point at a preset frequency to obtain the current operation status information of the crane, and selects the enablement and shutdown of the corresponding functional module according to the current operation status to achieve automatic switching. Different PLC point values correspond to different current operation states of the crane;
[0028] S2: Call the video stream access module, connect to the corresponding network camera video stream, pull the original code stream information of the picture, obtain the real-time picture of the crane's current operation dynamics or working area through the hardware decoding module of the distributed processing platform, and push the picture to the core algorithm module;
[0029] S3: The corresponding detection network and segmentation network in the core algorithm module process, calculate and compare the current real-time image to obtain the operation safety and abnormal protection monitoring results under the operation state. When the monitoring results indicate safety hazards, the distributed processing platform will link the stand-alone PLC to perform corresponding processing, send alarm information to the remote operation console, and push the current monitoring video pop-up window to prompt the remote control driver to confirm. After the remote control driver makes a judgment, the distributed processing platform sends the corresponding action instructions to the stand-alone PLC.
[0030] Furthermore, when the distributed processing platform obtains the trolley travel signal and direction according to the scanned PLC points in step S1, the algorithm operation mode of the core algorithm module is set to trolley travel, and the trolley anti-collision function is enabled, then in step S3, the drivable area segmentation network performs pixel-level segmentation on the drivable area, and the obstacle detection network calculates the specific location information of the obstacle, compares the distance between the specific location of the obstacle and the segmented drivable area with the preset alarm threshold, obtains the current crane travel danger situation assessment, and performs corresponding processing according to the three different levels of normal, deceleration, and stop;
[0031] When the distributed processing platform obtains the spreader locking signal & spreader lifting signal according to the scanned PLC point in step S1, the algorithm operation mode of the core algorithm module is set to the container has been grabbed, and the container door direction confirmation function is enabled, then in step S3, the container and container door detection network processes the current real-time image, obtains the current container position and container door position information, comprehensively considers the spatial position relationship between the two, and compares with the container door allowed to appear on the pre-specified side of the image, and performs corresponding processing according to normal or abnormal;
[0032] When the distributed processing platform obtains the signal of spreader unlocking & spreader lifting according to the scanned PLC point in step S1, the algorithm operation mode of the core algorithm module is set to container placement, and the yard and container status confirmation function is enabled, then in step S3, the container detection network detects the gap between the two bays of the container, and determines whether there is an uneven hidden danger in the yard ground according to the size of the gap, and determines whether there is a safety hazard in the gaps between adjacent bays and adjacent columns of containers in the yard; the container position in the current picture is obtained through the container detection network, and then the container segmentation network is cascaded to segment multiple faces of the container, and the segmented single plane of the container is sent to the classification network to determine whether the current container is enlarged, damaged, or deformed; the yard picture is pushed to the smoke and flame detection network, and when smoke and flame appear in the detection results, it is determined that there is a fire hazard;
[0033] When the distributed processing platform obtains the sling descending & sling and keyhole alignment signals according to the scanned PLC points in step S1, the algorithm operation mode of the core algorithm module is set to grab the box, and the box deformation detection function is enabled, then in step S3, the container detection network performs lock and beam detection on the current images taken by each corresponding network camera, and compares the detected lock hole and beam area with the preset threshold. When it is less than the preset threshold, the current sling position is not judged; when the lock hole and beam area is greater than the preset threshold, the lock hole and beam images are cut and sent to the classification network to determine whether there are debris in the lock of the current container and whether the beam is deformed.
[0034] Furthermore, in step S3, when the remote control driver determines that a false alarm occurs, the distributed processing platform sends a corresponding action instruction to the stand-alone PLC, and at the same time, the core algorithm module stores the current detection image in the training update path, and the corresponding detection network weight value based on deep learning is updated, wherein the algorithm includes:
[0035] ① Random data mixing enhancement: Mix abnormal pictures with less data and randomly selected normal pictures, and scale and splice them according to two, three, or four pictures respectively to obtain a new picture containing abnormal information, which is used as new data to update the algorithm weights to increase the number of abnormal data training sets and improve the accuracy;
[0036] ② Data random noise enhancement: Random noise is added to the manually annotated images collected in the early stage and the images manually corrected by the system during operation to generate a new data set for training;
[0037] ③ Multi-scale detection: obtain the real-time video image of the camera, perform pyramid scale transformation on the original resolution image to obtain new samples of different sizes to be detected, send them to the corresponding detection network to obtain the detection output results at multiple scales, and fuse and judge the target detection result labels to obtain more accurate result output.
[0038] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0039] The present invention makes full use of existing remote-controlled cameras, does not add new camera installation points, saves hardware quantity and cost; adds a distributed processing platform to the crane, places the computing processing unit in front, does not increase the external bandwidth demand of the crane, and forms a distributed processing architecture; efficiently integrates the crane operation status and multi-view cameras, and the multi-functions can be automatically switched according to the operation status; adopts a video fusion processing strategy, which can effectively reduce large vehicle collision accidents; confirms the direction of the container door in real time, and reduces the economic losses and the decline in operation efficiency caused by the abnormal direction of the container door; uses the operation perspective to automatically and intelligently analyze the status of the yard and container, completes the early detection and early treatment of container damage and yard damage, and reduces production safety accidents; detects smoke and flames in the yard, improves the timeliness of safety accident warning, and reduces the occurrence of accidents; the system has a high degree of intelligence, effectively improves the operation efficiency and safety protection level, and can also be linked with multiple systems such as ECS and TOS to expand more application functions.
[0040] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 Schematic diagram of the real-time monitoring system for container gantry crane yard operation status of the present invention;
[0042] Figure 2 A schematic flow chart of a real-time monitoring system for container gantry crane yard operation status according to the present invention;
[0043] In the figure: central control room 1, remote operation console 11, stand-alone control system 2, stand-alone PLC 21, network camera 22, distributed processing platform 23. DETAILED DESCRIPTION
[0044] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0045] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;
[0046] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings;
[0047] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "set" and "connection" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. It can be said that the internal connection of two elements is connected. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The orientation or position relationship indicated by the terms "upper", "lower", "left", "right", "front", "back", etc. is based on the orientation or position relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0048] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0049] like Figure 1 As shown, the real-time monitoring system for the container gantry crane yard operation status of this embodiment includes a stand-alone control system 2 arranged on the container gantry crane and a remote operating console 11 arranged in the central control room 1. The stand-alone control system is communicatively connected to the remote operating console and includes a stand-alone PLC 21, a plurality of network cameras 22, and a distributed processing platform 23. Among them, the single-machine PLC controls the operation of the crane and can send the current operation status information of the crane to the distributed processing platform; the several network cameras are arranged at different positions of the crane, which can shoot the operation dynamics or working area of the crane under different operation angles in real time; the remote operation console is provided with a main screen which can display the remote operation interface, so as to facilitate the remote control driver to perform remote control operation; the distributed processing platform 23 includes a video stream access module, a core algorithm module, a PLC interaction module, an alarm information output module and a WEB server, among which the video stream access module is responsible for accessing and forwarding the video data captured by the network camera 22, the core algorithm module is responsible for completing the large vehicle anti-collision monitoring, the box door direction confirmation, the yard and container status confirmation, the box deformation detection and obtaining the corresponding operation safety and abnormal protection monitoring results according to the real-time picture captured by the network camera, and the PLC interaction module is responsible for interacting with the single-machine PLC 21 to interact, obtain the current operating status information of the crane and transmit action instructions, etc. The alarm information output module sends the corresponding alarm information to the remote operating console 11 according to the operating safety and abnormal protection monitoring results, and gives safety hazard prompts in time. The WEB server provides a remote access portal for historical record query, network camera calibration parameter setting and system preset configuration.
[0050] In this embodiment, the several network cameras utilize the remote control cameras currently installed on the crane. Specifically, four cameras are respectively installed on the left side of the crane's reel-side trolley (this position can observe whether there are people or other obstacles on the reel-side lane when the trolley moves to the left), the right side of the reel-side trolley (this position can observe whether there are people or other obstacles on the reel-side lane when the trolley moves to the right), the left side of the non-reel-side trolley (this position can observe whether there are people or other obstacles on the non-reel-side lane when the trolley moves to the left), and the right side of the non-reel-side trolley (this position can observe whether there are people or other obstacles on the non-reel-side lane when the trolley moves to the right) as the trolley protection cameras in this system. The network camera for collision monitoring; the two reel-side container cameras and two non-reel-side container cameras used to shoot the container top number are used as network cameras for confirming the direction of the container door in this system; the two cameras installed in front and behind the crane trolley frame (the situation of the long side guide plate area of the spreader container can be observed from the trolley frame) and the four cameras installed on the inside of the crane spreader lock pin (the alignment of the spreader and the container lock hole can be observed) are used as network cameras for confirming the yard and container status in this system; the four cameras installed on the inside of the crane spreader lock pin are also used as network cameras for box deformation detection in this system, and the two cameras installed in front and behind the crane trolley frame are used for assistance. In this way, the existing remote control cameras are fully utilized, and no new camera installation points are added, saving the number of hardware and costs.
[0051] Among them, the distributed processing platform 23 preferably has an 8-core ARM V8.2 64-bit CPU, a 512-core Volta GPU, and a 32GB memory, and is connected to the several network cameras and the remote operation console TCP communication through the convergence switch in the crane electrical room. In order to meet the high-bandwidth and long-distance transmission requirements, a single-mode optical fiber connection is adopted between the convergence switch and the remote operation console in the central control room. It can be understood that there are multiple container gantry cranes operating at the port terminal, and there are also multiple remote operation consoles 11 deployed in the central control room 1 (only one is shown in the figure). According to actual operation requirements, the distributed processing platform 23 on each crane can be switched to a certain remote operation console 11, and each remote operation console is operated by a remote control driver. For the convenience of operation, the remote operation console is also provided with a control handle, and the main screen is a touch display screen. The remote control driver can perform remote operation through the control handle or the buttons on the touch display screen.
[0052] Among them, the core algorithm module runs a program including multiple deep neural networks, and the multiple deep neural networks are specifically such as: obstacle detection, container and door detection and other detection networks, drivable area segmentation, container segmentation and other segmentation networks, to achieve different detection tasks or different segmentation tasks. It is well known that a deep neural network is a network structure composed of some basic image operations, including operations such as image convolution and pooling, and achieves the functional effects of detection and segmentation through different combinations. The operation of a deep neural network mainly includes two stages: training and reasoning. During the training process, the input is the image and the corresponding label data (for the detection network, it is the image coordinates of each target; for the segmentation network, it is the attribute label of the pixel point). By continuously iterating the network output results and comparing them with the real label data, some parameters in the network are adjusted, and finally a network model containing parameters is obtained; during the reasoning process, the input is the image, and the output is the corresponding result (target position, pixel point attribute). It can be understood that the detection network is a deep neural network whose output is the target category and position; the segmentation network is a deep neural network whose output is the attribute of each pixel point.
[0053] The method for real-time monitoring of the container gantry crane yard operation status using the above monitoring system comprises the following steps:
[0054] S1: The distributed processing platform on the crane continuously scans the single-machine PLC points at a preset frequency to obtain the current operating status information of the crane, and selects the enablement and shutdown of the corresponding functional modules according to the current operating status to achieve automatic switching, where different PLC point values correspond to different current operating states of the crane. It should be noted that: according to the current operating status, there are related signal enablements in the single-machine PLC. For example, when the crane is in walking mode, the corresponding PLC point position is set; the current operating status changes, and accordingly, the PLC point value changes. The distributed processing platform continuously scans the single-machine PLC points to capture point changes so as to timely analyze the current operating status.
[0055] S2: Call the video stream access module, connect the corresponding network camera video stream according to the current operation status of the crane, pull the original code stream information of the picture, obtain the real-time picture of the current operation dynamics or working area of the crane through the hardware decoding module of the distributed processing platform, and push the picture to the core algorithm module. It should be noted that the hardware decoding module in this system is based on the Nvidia Jetson Xavier series of edge computing devices, which mainly include CPUs for logic processing (based on the Arm architecture) and GPUs for AI computing.
[0056] S3: Figure 2As shown in the figure, the corresponding detection network and segmentation network in the core algorithm module process, calculate and compare the current real-time image to obtain the operation safety and abnormal protection monitoring results under the operation state. When the monitoring results indicate safety hazards, the distributed processing platform will link the single-machine PLC to perform corresponding processing, send alarm information to the remote operation console, and push the current monitoring video pop-up window to prompt the remote control driver to confirm. After the remote control driver makes a judgment, the distributed processing platform sends the corresponding action instruction to the single-machine PLC.
[0057] Furthermore, in a specific implementation, the monitoring system is applied to the following aspects:
[0058] (1) Large vehicle collision avoidance. The present invention uses four network cameras that can fully cover the walking direction, uses deep learning to train the crane's traversable area and obstacle detection model, and links with the crane's stand-alone PLC. When the large vehicle is moving, the video image of the forward direction is calculated and processed in real time. When an abnormal situation occurs, the stand-alone PLC is immediately linked to slow down or stop processing. At the same time, the alarm information and the current monitoring video pop-up window are pushed to the remote control driver to remind him to confirm the abnormal situation in time. Specifically, when the distributed processing platform in step S1 obtains the large vehicle walking signal and direction according to the scanned PLC point, the algorithm operation mode of the core algorithm module is set to large vehicle walking, and the large vehicle collision avoidance function is enabled, then in step S3, the drivable area segmentation network performs pixel-level segmentation on the drivable area, and the obstacle detection network calculates the specific location information of the obstacle. According to the distance between the specific location of the obstacle and the segmented drivable area, it is compared with the pre-set alarm threshold to obtain the current crane walking danger situation assessment, and corresponding processing is performed according to the three different levels of normal, deceleration, and stop. When the monitoring results indicate a safety hazard, the distributed processing platform will link the stand-alone PLC to take corresponding actions (slow down or stop), and at the same time send an alarm message to the remote control console and push the current monitoring video pop-up window to prompt the remote control driver to confirm. If the remote control driver confirms the safety hazard, the distributed processing platform will send the corresponding action instructions to the stand-alone PLC (maintain deceleration or stop); if the remote control driver determines that the safety hazard is a false alarm, the distributed processing platform will send the corresponding action instructions to the stand-alone PLC (cancel deceleration or stop), and at the same time the core algorithm module will store the current detection image in the training update path, and the corresponding deep learning-based detection network weight value will be updated. The algorithm includes: ① Random data mixing enhancement: Mix abnormal images with less data volume and randomly selected normal images, and scale and splice them according to two, three, and four images respectively to obtain a new image containing abnormal information as the new data. 1. Update the algorithm weights to increase the number of abnormal data training sets and improve the accuracy; 2. Data random noise enhancement: add random noise to the manually annotated images collected in the early stage and the images corrected by the manual system during the system operation (for example, add noise points to the images, randomly erase part of the content or randomly rotate them) to generate new data sets for training; 3. Multi-scale detection: obtain the real-time video image of the camera, perform pyramid scale changes on the original resolution image to obtain new samples of different sizes to be detected, send them to the corresponding detection network, and obtain the detection output results at multiple scales. The target detection result labels are fused and judged to obtain more accurate result output and improve the monitoring accuracy of the system.
[0059] (2) Confirmation of the door orientation. According to transportation specifications, the container door needs to face the rear of the truck. Abnormal door orientation will bring operating costs and safety hazards. Generally, confirmation of the door orientation requires the installation of separate tallying software and separate camera detection and analysis, which increases system redundancy and hardware costs. The four truck cameras installed on the crane for shooting the top number of the truck are used to automatically determine the door orientation information during operation. When an abnormality is found, an alarm message is promptly given to the remote driver operation interface, and the abnormal operation process is interrupted in time, reducing the time for subsequent modification operations and improving production efficiency. Specifically, when the distributed processing platform obtains the spreader locking signal & spreader lifting signal according to the scanned PLC point in step S1, the algorithm operation mode of the core algorithm module is set to the container has been grabbed, and the door orientation confirmation function is enabled, then in step S3, the container and door detection network processes the current real-time image, obtains the current container position and door position information, comprehensively considers the spatial position relationship between the two, and compares it with the pre-specified side (for example, the specified left side) of the image that the door is allowed to appear on, and performs corresponding processing according to normal and abnormal conditions. When the monitoring results indicate that the direction of the container door is abnormal (inconsistent with the specified side), the distributed processing platform will link the stand-alone PLC to take corresponding actions (stop lifting the hoist), and send an alarm message to the remote control console and push the current monitoring video pop-up window to prompt the remote control driver to confirm. If the remote control driver confirms that the direction of the container door is abnormal, the distributed processing platform will send the corresponding action instructions to the stand-alone PLC (according to the terminal yard management rules, exception handling); if the remote control driver determines that the abnormal direction of the container door is a false alarm, the distributed processing platform will send the corresponding action instructions to the stand-alone PLC (resume lifting the hoist), and the core algorithm module will store the current detection image in the training update path, and the corresponding detection network weight value based on deep learning will be updated. The algorithm processing is the same as above, and will not be repeated here.
[0060] (3) Confirmation of yard and container status. Containers stacked in the yard may cause damage to the yard due to overweight and other reasons, causing the yard to sink and be damaged; container damage and other reasons may cause leakage of container items; the yard is in a high temperature state in summer, which will cause fire safety hazards for flammable goods. Traditional methods mostly rely on manual inspections, which are inefficient and cannot detect hidden dangers in time. Two cameras installed in front and behind the crane trolley frame and four cameras installed on the inside of the crane sling lock pin are used to comprehensively monitor the containers within the operating range, and safety hazards are discovered in time, which greatly improves the degree of intelligence and efficiency. Specifically, when the distributed processing platform obtains the signal of sling unlocking & sling lifting according to the scanned PLC points in step S1, the algorithm operation mode of the core algorithm module is set to container placement, and the yard and container status confirmation function is enabled, then in step S3, the container detection network detects the gap between the container and the adjacent container (detects the edge position of the container->calculates the gap according to the camera calibration information, such as height, resolution, field of view, etc.), and judges whether there is an uneven hidden danger in the yard ground according to the size of the gap, and judges whether there is a safety hazard in the gaps between adjacent containers and adjacent columns of the yard containers; obtains the position of the container in the current picture through the container detection network, and then cascades the container segmentation network to segment multiple surfaces (top surface and four side surfaces) of the container, and sends the segmented single plane of the container to the classification network to judge whether the current container has abnormalities such as belly, stain, deformation, etc.; the yard picture is pushed to the smoke and flame detection network, and when smoke and flame appear in the detection results, it is judged that there is a fire hazard. When the monitoring results indicate a safety hazard, the distributed processing platform will link the stand-alone PLC to perform corresponding processing (send an alarm signal), and at the same time send an alarm message to the remote control console and push the current monitoring video pop-up window to prompt the remote control driver to confirm. If the remote control driver confirms the safety hazard, the distributed processing platform will send the corresponding action instruction to the stand-alone PLC (continue to send an alarm signal); if the remote control driver determines that the safety hazard is a false alarm, the distributed processing platform will send the corresponding action instruction to the stand-alone PLC (cancel the alarm signal), and at the same time the core algorithm module will store the current detection image in the training update path, and the corresponding deep learning-based detection network weight value will be updated. The algorithm processing is the same as before, and will not be repeated here.
[0061] (4) Detection of box deformation. When the crane is operating, the outline of the lock hole is required to be intact. The deformation of the box corners, beams, and box columns or the blockage of the lock hole by debris may cause the sling to be locked, resulting in an operation accident. In the traditional operation process, the judgment of deformation can only rely on the driver's manual judgment, which is easy to cause missed judgments and misjudgments. Using four cameras installed on the inside of the crane sling lock pin (commonly referred to as sling cameras) and two cameras installed in front and behind the crane trolley frame (these six cameras are also used for the aforementioned yard and container status confirmation), deformation detection is performed on each container in the operation, thereby improving operation safety. Specifically, when the distributed processing platform obtains the signal of the spreader lowering and the spreader and the lock hole alignment according to the scanned PLC point in step S1, the algorithm operation mode of the core algorithm module is set to grab the box, and the box deformation detection function is enabled, then in step S3, the container detection network performs lock and beam detection on the current picture taken by each corresponding network camera, and compares the detected lock hole and beam area with the preset threshold. When it is less than the preset threshold, the current spreader position is not judged; when the lock hole and beam area is greater than the preset threshold, the lock hole and beam images are cut and sent to the classification network to determine whether there are debris in the lock of the current container and whether the beam is deformed. When the monitoring results indicate a safety hazard, the distributed processing platform will link the stand-alone PLC to perform corresponding processing (stop grabbing the box), and at the same time send an alarm message to the remote control console and push the current monitoring video pop-up window to prompt the remote control driver to confirm. If the remote-control driver confirms the safety hazard, the distributed processing platform sends the corresponding action instructions to the single-machine PLC (continue to stop grabbing boxes); if the remote-control driver determines that the safety hazard is a false alarm, the distributed processing platform sends the corresponding action instructions to the single-machine PLC (resume grabbing boxes). At the same time, the core algorithm module stores the current detection image into the training update path, and the corresponding deep learning-based detection network weight value is updated. The algorithm processing is the same as before and will not be repeated here.
[0062] In addition to the above applications, the monitoring system in the present invention can also be linked with multiple systems such as ECS and TOS to expand more application functions.
[0063] The above description is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and concepts of the present invention within the technical scope disclosed by the present invention, and they should be covered by the protection scope of the present invention.
Claims
1. A method for real-time monitoring of container gantry crane yard operation status, characterized in that: The monitoring system used in the method includes a stand-alone control system installed on the container gantry crane and a remote operation console installed in the central control room. The stand-alone control system is connected to the remote operation console in communication and includes a stand-alone PLC, a number of network cameras, and a distributed processing platform; wherein, The stand-alone PLC controls the operation of the crane and sends the current operation status information of the crane to the distributed processing platform; The plurality of network cameras are arranged at different positions of the crane, and can capture the operation dynamics or working area of the crane from different working angles in real time; The remote control console is provided with a main screen which can display the remote operation interface, so as to facilitate the remote control driver to perform remote control operations; The distributed processing platform includes: The video stream access module is responsible for accessing and forwarding the video data captured by the network camera; The core algorithm module is responsible for completing large vehicle anti-collision monitoring, container door orientation confirmation, yard and container status confirmation, container deformation detection based on the real-time images captured by the network camera, and obtaining corresponding operation safety and abnormal protection monitoring results; The PLC interaction module is responsible for interacting with the stand-alone PLC, obtaining the current operation status information of the crane, and transmitting action instructions; The alarm information output module sends the corresponding alarm information to the remote operation console according to the operation safety and abnormal protection monitoring results, and gives safety hazard prompts in time; WEB server provides remote access to facilitate historical record query, network camera calibration parameter setting and system preset configuration; The method comprises the following steps: S1: The distributed processing platform on the crane scans the single-machine PLC point at a preset frequency to obtain the current operation status information of the crane, and selects the enablement and shutdown of the corresponding functional module according to the current operation status to achieve automatic switching. Different PLC point values correspond to different current operation states of the crane; S2: Call the video stream access module, connect to the corresponding network camera video stream, pull the original code stream information of the picture, obtain the real-time picture of the crane's current operation dynamics or working area through the hardware decoding module of the distributed processing platform, and push the picture to the core algorithm module; S3: The corresponding detection network and segmentation network in the core algorithm module process, calculate and compare the current real-time image to obtain the operation safety and abnormal protection monitoring results under the operation state. When the monitoring results indicate safety hazards, the distributed processing platform will link the stand-alone PLC to make corresponding processing, send alarm information to the remote operation console and push the current monitoring video pop-up window to prompt the remote control driver to confirm. After the remote control driver makes a judgment, the distributed processing platform sends the corresponding action instruction to the stand-alone PLC; in: When the distributed processing platform obtains the trolley travel signal and direction according to the scanned PLC points in step S1, the algorithm operation mode of the core algorithm module is set to trolley travel, and the trolley anti-collision function is enabled, then in step S3, the drivable area segmentation network performs pixel-level segmentation on the drivable area, and the obstacle detection network calculates the specific location information of the obstacle, compares the distance between the specific location of the obstacle and the segmented drivable area with the preset alarm threshold, obtains the current crane travel danger situation assessment, and performs corresponding processing according to the three different levels of normal, deceleration, and stop; When the distributed processing platform obtains the spreader locking signal and spreader lifting signal according to the scanned PLC points in step S1, the algorithm operation mode of the core algorithm module is set to the container has been grabbed, and the container door direction confirmation function is enabled, then in step S3, the container and container door detection network processes the current real-time image, obtains the current container position and container door position information, comprehensively considers the spatial position relationship between the two, and compares with the container door allowed to appear on the pre-specified side of the image, and performs corresponding processing according to normal or abnormal; When the distributed processing platform obtains the signal of the spreader unlocking and the spreader lifting according to the scanned PLC points in step S1, the algorithm operation mode of the core algorithm module is set to the container has been placed, and the yard and container status confirmation function is enabled, then in step S3, the container detection network detects the gap between the two bays of the container, and judges whether there is a hidden danger of unevenness on the yard ground according to the size of the gap, and judges whether there is a safety hazard between the gaps between adjacent bays and adjacent columns of containers in the yard; the container position in the current picture is obtained through the container detection network, and then the container segmentation network is cascaded to segment multiple faces of the container, and the single plane of the segmented container is sent to the classification network to determine whether the current container is enlarged, damaged, or deformed; the yard picture is pushed to the smoke and flame detection network, and when smoke and flame appear in the detection results, it is determined that there is a fire hazard; When the distributed processing platform obtains the signals of the sling descending and the sling and the lock hole alignment according to the scanned PLC points in step S1, the algorithm operation mode of the core algorithm module is set to grab the box, and the box deformation detection function is enabled, then in step S3, the container detection network performs lock and beam detection on the current picture taken by each corresponding network camera, and compares the detected lock hole and beam area with the preset threshold. When it is less than the preset threshold, the current sling position is not judged; when the lock hole and beam area is larger than the preset threshold, the lock hole and beam images are cut and sent to the classification network to determine whether there are debris in the lock of the current container and whether the beam is deformed.
2. The method for real-time monitoring of container gantry crane yard operation status according to claim 1 is characterized in that: The plurality of network cameras include: There are four network cameras for trolley anti-collision monitoring, which are installed on the left trolley on the reel side, the right trolley on the reel side, the left trolley on the non-reel side, and the right trolley on the non-reel side of the crane; The network camera used to confirm the direction of the container door is shared by two reel-side container cameras and two non-reel-side container cameras used to capture the top number of the container truck; There are six network cameras used to confirm the status of the yard and containers, two of which are installed in front and behind the crane trolley frame, and the remaining four are installed on the inside of the crane spreader lock pin; There are six network cameras used for box deformation detection, four of which are installed on the inside of the crane sling lock pin, and the remaining two are installed on the front and rear of the crane trolley frame.
3. The method for real-time monitoring of container gantry crane yard operation status according to claim 1 is characterized in that: The distributed processing platform uses an 8-core ARM V8.2 64-bit CPU, a 512-core Volta GPU and 32GB of memory.
4. The method for real-time monitoring of container gantry crane yard operation status according to claim 1 is characterized in that: The distributed processing platform is connected to the plurality of network cameras and the remote operation console via TCP communication through a convergence switch in the crane electrical room.
5. The method for real-time monitoring of container gantry crane yard operation status according to claim 1 is characterized in that: The remote control console is also provided with a control handle, and the main screen is a touch display screen. The remote control driver can perform remote operation through the control handle or buttons on the touch display screen.
6. The method for real-time monitoring of container gantry crane yard operation status according to claim 1 is characterized in that: The central control room is equipped with multiple remote operating consoles. According to actual operation requirements, the distributed processing platform on each crane can be switched and connected to a remote operating console.
7. The method for real-time monitoring of container gantry crane yard operation status according to any one of claims 1 to 6, characterized in that: The core algorithm module runs a program including multiple deep neural networks, wherein the multiple deep neural networks include multiple detection networks for implementing different detection tasks and multiple segmentation networks for implementing different segmentation tasks.
8. The method for real-time monitoring of container gantry crane yard operation status according to claim 7 is characterized in that: In step S3, when the remote control driver determines that a false alarm occurs, the distributed processing platform sends a corresponding action instruction to the stand-alone PLC, and the core algorithm module stores the current detection image in the training update path, and the corresponding detection network weight value based on deep learning is updated. The algorithm includes: ① Random data mixing enhancement: Mix abnormal pictures with less data and randomly selected normal pictures, and scale and splice them according to two, three, or four pictures respectively to obtain a new picture containing abnormal information, which is used as new data to update the algorithm weights to increase the number of abnormal data training sets and improve the accuracy; ② Data random noise enhancement: Random noise is added to the manually annotated images collected in the early stage and the images manually corrected by the system during operation to generate a new data set for training; ③ Multi-scale detection: obtain the real-time video image of the camera, perform pyramid scale transformation on the original resolution image to obtain new samples of different sizes to be detected, send them to the corresponding detection network to obtain the detection output results at multiple scales, and fuse and judge the target detection result labels to obtain more accurate result output.
Citation Information
Cited By
Monitoring and early warning method and system for container portal crane
CN120097222A
A monitoring and early warning method and system for a container gantry crane
CN120097222B
