Vision-based Intelligent Electronic Fence Protection System and Method for the Safety of Ship-to-Shore Crane Operations

Abstract

The present application provides a vision-based safety intelligent electronic fence protection system and method for quay crane operations. The system according to the present application includes: monitoring equipment, a switch, a perception workstation, and an audible and visual alarm; the monitoring equipment is used to monitor the quay crane operation area to obtain monitoring images, integrate the monitoring images into monitoring data, and send the monitoring data to the perception workstation through the switch; the switch is used to transmit the monitoring data to the perception workstation; the perception workstation is used to configure the intelligent electronic fence protection function for the quay crane operation area, and identify the lane lines on the driving path of the quay crane in real time; perform real-time processing, identification and detection of abnormal intrusion in the quay crane operation area, and generate a warning signal; the audible and visual alarm is used to give an audible and visual alarm according to the warning signal. The technical solution provided by the present application can accurately detect obstacles in the operation area and can warn of abnormal situations within a specific range.

Description

Technical Field

[0001] This document relates to the technical field of intelligent electronic fences, and particularly to a vision-based intelligent electronic fence protection system and method for the safety of quay crane operations. Background Art

[0002] Quay cranes are the main loading and unloading equipment in container terminals, and have characteristics such as a complex operating environment and limited visibility. The current operating modes include the quay crane driver operating in the driver's cab and the quay crane driver operating in the remote control room. Among them, when operating in the driver's cab, the driver's line of sight is affected by the mechanical equipment structure and cannot monitor whether the running route of the trolley is safe throughout the process; when operating in the remote control room, the driver needs to observe multiple monitoring screens simultaneously, which distracts attention and may cause important information to be missed.

[0003] At the same time, since there are truck drivers and maintenance personnel under the quay crane, the operating environment is complex and there are a large number of cross-operation conditions. The existing operating modes are difficult to ensure the safety of the personnel and property in the terminal. Summary of the Invention

[0004] This application provides a vision-based intelligent electronic fence protection system and method for the safety of quay crane operations, aiming to solve the above problems.

[0005] An embodiment of the present invention provides a vision-based intelligent electronic fence protection system for the safety of quay crane operations, including: a monitoring device, a switch, a perception workstation, and an audible and visual alarm;

[0006] The monitoring device is used to monitor the quay crane operation area to obtain monitoring images, integrate the monitoring images into monitoring data, and send the monitoring data to the perception workstation through the switch;

[0007] The switch is used to transmit the monitoring data to the perception workstation;

[0008] The perception workstation is used to configure the intelligent electronic fence protection function for the quay crane operation area, identify the lane lines on the driving path of the quay crane in real time; perform real-time processing, identification, and detection of abnormal intrusions in the quay crane operation area on the monitoring data, and generate a warning signal;

[0009] The audible and visual alarm is used to perform audible and visual alarms according to the warning signal.

[0010] An embodiment of the present invention provides a vision-based intelligent electronic fence protection method for the safety of quay crane operations, including:

[0011] Monitoring the quay crane operation area through a monitoring device, and sending the monitoring data to the perception workstation through a switch;

[0012] The electronic fence of the field bridge operation area is configured through the perception workstation, and the lane lines on the field bridge's driving path are identified in real time based on the built-in lane line recognition algorithm to ensure the field bridge's safe driving within the predetermined route; the monitoring data is processed in real time based on the intrusion perception algorithm to identify and detect abnormal intrusions in the field bridge operation area, generate early warning signals, and,

[0013] The early warning signal is given an audible and visual alarm by an audible and visual alarm.

[0014] This system uses monitoring equipment installed in different areas of the Changqiao operation area to achieve comprehensive monitoring and data acquisition within the operation area. Sensing workstations configure electronic fences to accurately detect obstacles within the operation area. Deep learning computer vision technology is used to provide early warning of abnormal conditions within the monitoring range, thereby improving the safety of terminal operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 Schematic diagram of a vision-based intelligent electronic fence protection system for field bridge operation safety according to an embodiment of the present invention;

[0017] Figure 2 This is a topological diagram of a direct deployment method of a vision-based intelligent electronic fence protection system for field bridge operation safety according to an embodiment of the present invention;

[0018] Figure 3 This is a topological diagram of the linkage deployment mode of the vision-based intelligent electronic fence protection system for field crane operation safety and the field crane PLC according to an embodiment of the present invention;

[0019] Figure 4 Schematic diagram of the overhead detection coverage area of the vision-based intelligent electronic fence protection system for field bridge operation safety according to an embodiment of the present invention;

[0020] Figure 5 This is a front view of the camera arrangement in the traveling direction of the field bridge according to an embodiment of the present invention;

[0021] Figure 6 A rear view of the camera arrangement in the traveling direction of the field bridge according to an embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram of the camera arrangement on the electrical room side according to an embodiment of the present invention;

[0023] Figure 8 Schematic diagram of the camera arrangement on the power unit side according to an embodiment of the present invention;

[0024] Figure 9 Schematic diagram of whether to enable the intelligent electronic fence function according to an embodiment of the present invention;

[0025] Figure 10 Specific operation schematic diagram of the vision-based intelligent electronic fence protection method for quay crane operation safety according to an embodiment of the present invention;

[0026] Figure 11 Flow chart of the vision-based intelligent electronic fence protection method for quay crane operation safety according to an embodiment of the present invention. Specific embodiments

[0027] In order to enable those skilled in the art of the present technology to better understand the technical solutions in one or more embodiments of this specification, the following will clearly and completely describe the technical solutions in one or more embodiments of this specification with reference to the accompanying drawings in one or more embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this document.

[0028] System embodiments

[0029] According to an embodiment of the present invention, a vision-based intelligent electronic fence protection system for quay crane operation safety is provided. Figure 1 Vision-based intelligent electronic fence protection system for quay crane operation safety according to an embodiment of the present invention. According to Figure 1 As shown, the vision-based intelligent electronic fence protection system for quay crane operation safety according to an embodiment of the present invention specifically includes:

[0030] Monitoring device 15, switch 9, perception workstation 10, and audible and visual alarm 12;

[0031] The monitoring device 15 is used to monitor the quay crane operation area to obtain monitoring images, integrate the monitoring images into monitoring data, and send the monitoring data to the perception workstation 10 through the switch 9;

[0032] Figure 2 Topological schematic diagram of the direct deployment method of the vision-based intelligent electronic fence protection system for quay crane operation safety according to an embodiment of the present invention. According to Figure 2It can be known that the monitoring cameras include a total of four cameras, namely the left leg camera 1 on the electrical room side at the leg of the gantry crane, the right leg camera 2 on the electrical room side, the left leg camera 3 on the power room side, and the right leg camera 4 on the power room side. The field of view covers the road area in front of the gantry crane traveling, and is used to obtain environmental image data in front of the gantry crane during operation; there are a total of four cameras, namely the inner upper beam camera 5 on the electrical room side, the outer upper beam camera 6 on the electrical room side, the inner upper beam camera 7 on the power room side, and the outer upper beam camera 8 on the power room side, installed on the upper crossbeam of the gantry crane. The field of view covers the surrounding environment areas inside and outside the gantry crane, and is used to obtain environmental image data inside and outside the periphery of the gantry crane during operation, and the image data is transmitted through a switch. Figure 4 It is a schematic diagram of the detection coverage top view area of the vision-based intelligent electronic fence protection system for gantry crane operation in the embodiment of the present invention. Figures 5 - 8 They are respectively schematic diagrams of the arrangement positions of the monitoring cameras in the embodiments of the present invention, where the shaded part represents the monitoring area.

[0033] The switch 9 is used to transmit the monitoring data to the perception workstation 10;

[0034] The perception workstation 10 is used to configure the intelligent electronic fence protection function in the gantry crane operation area, and to identify the lane lines on the driving path of the gantry crane in real time; to perform real-time processing, identification and detection of abnormal intrusion in the gantry crane operation area on the monitoring data, and generate a warning signal;

[0035] The perception workstation 10 obtains the image data transmitted by the monitoring cameras through the switch 9, and analyzes and processes the obtained image data. Through computer vision and deep learning algorithms, the image data is processed in real time to identify and detect abnormal intrusion in the specified operation area, and generate a warning signal. The perception workstation 10 integrates an electronic fence custom configuration and debugging software, a lane line recognition algorithm software, and an intrusion perception algorithm software.

[0036] The perception workstation 10 specifically includes:

[0037] An electronic fence configuration sub-module, used to configure whether to enable the intelligent electronic fence protection function;

[0038] A lane line recognition sub-module, used to recognize the lane lines on the driving path in the gantry crane operation area based on the built-in lane line recognition algorithm.

[0039] An intrusion perception sub-module, used to perform intrusion monitoring on the gantry crane operation area based on the preset intrusion perception algorithm and the monitoring data when the intelligent electronic fence protection function is enabled, and generate a warning signal.

[0040] Figure 9Schematic diagram of the basic process of the electronic fence configuration sub-module according to an embodiment of the present invention; start the electronic fence custom configuration debugging software, select whether to enable the visual electronic fence function. If not enabled, record the function that the electronic fence is not triggered during the operation. If enabled, select the electronic fence area configuration, including two methods: the fixed electronic fence configuration mode for the custom area and the dynamic electronic fence configuration mode for the area calculated based on the lane line. The default is the lane line calculation area method, which is called; selecting the custom area method will generate a region box in the configuration image, and the region box can be adjusted to select the area range of the electronic fence; selecting the lane line calculation area method will fit the current operation lane line in the image in real time, and set the area inside the lane line as the electronic fence warning area. After saving the settings, it will be applied to the quay crane operation safety intelligent electronic fence protection system.

[0041] Figure 10 This is the basic working process of the quay crane operation safety intelligent electronic fence protection system based on vision according to an embodiment of the present invention. According to Figure 10 it can be seen that the system will first judge whether the electronic fence function is enabled. If not enabled, it will provide a regular monitoring screen; if enabled, it will provide a monitoring screen with virtual electronic fences and intrusion target detection results. Then, retrieve the configured electronic fence strategies, including the custom area mode and the lane line calculation mode, and perform real-time analysis according to relevant algorithms. If an intrusion target appears, it will be reminded through the monitoring screen and the alarm module. The above-mentioned intrusion targets include, but are not limited to, people, vehicles, engineering vehicles, cone barrels, boxes, large lumps, spiky objects, etc., all targets that affect the normal operation of the quay crane.

[0042] The sound and light alarm 12 is used to perform sound and light alarms according to the warning signal. When an intrusion situation is detected, the system will remind the quay crane driver and the remote control operator through the sound and light alarm to prevent accidents.

[0043] The quay crane operation safety intelligent electronic fence protection system based on vision in an embodiment of the present invention further includes: a network video recorder 11, which obtains the image data transmitted by the monitoring camera through a switch and stores it for subsequent playback and analysis. The network video recorder has a high-capacity storage function and can record the image data of the quay crane operation environment for a long time, which is convenient for subsequent review and safety accident investigation.

[0044] Figure 3 This is the topological schematic diagram of the linkage deployment method of the quay crane operation safety intelligent electronic fence protection system based on vision and the quay crane PLC according to an embodiment of the present invention; according to Figure 3 it can be seen that the quay crane operation safety intelligent electronic fence protection system based on vision in an embodiment of the present invention is further used to be linked with the quay crane PLC to realize remote alarm and the function of docking with the automation system.

[0045] In the embodiments of the present invention, the monitoring cameras are installed at the portal legs and the upper crossbeam of the quay crane. According to Figures 5 - 8 it can be known that the specific positions are as follows:

[0046] One camera is installed on the left portal leg of the quay crane electrical room, and the field of view covers the area in the advancing direction on the left side of the quay crane trolley electrical room;

[0047] One camera is installed on the right portal leg of the quay crane electrical room, and the field of view covers the area in the advancing direction on the right side of the quay crane trolley electrical room;

[0048] One camera is installed on the left portal leg of the quay crane power room, and the field of view covers the area in the advancing direction on the left side of the quay crane trolley power room;

[0049] One camera is installed on the right portal leg of the quay crane power room, and the field of view covers the area in the advancing direction on the right side of the quay crane trolley power room;

[0050] Two cameras are installed on the upper crossbeam on the side of the quay crane electrical room. One lens faces downward inside the quay crane, and the other lens faces downward outside the quay crane. The field of view covers the area near the vehicle body on the side of the quay crane trolley electrical room;

[0051] Two cameras are installed on the upper crossbeam on the side of the quay crane power room. One lens faces downward inside the quay crane, and the other lens faces downward outside the quay crane. The field of view covers the area near the vehicle body on the side of the quay crane trolley power room.

[0052] In a specific example of the present invention, the perception workstation includes the following capabilities:

[0053] Efficient data processing ability. Through a highly integrated parallel algorithm framework combined with a hardware acceleration strategy, it can process large-scale image data in real time.

[0054] Deep neural network model. It has a built-in deep neural network model, with high-precision recognition and classification capabilities. It can effectively identify various intrusion targets in the operation area and can identify the accurate lane information of the quay crane operation.

[0055] Electronic fence custom configuration and debugging software. It allows users to customize the configuration and debugging of the electronic fence according to specific operation requirements to ensure the accuracy and flexibility of the electronic fence.

[0056] Lane line recognition algorithm software, which is used to recognize the lane lines on the driving path of the quay crane in real time to ensure the safe driving of the quay crane within the predetermined route.

[0057] Intrusion perception algorithm software. Through deep learning algorithms, it can detect intrusion situations in the operation area in real time and generate corresponding warning signals.

[0058] The electronic fence custom configuration debugging software in the above-mentioned perception workstation has the function of customizing the electronic fence, and can monitor the specified operation area in real time through the virtual electronic fence setting. When an intrusion target that affects normal operation enters the range of the electronic fence, the system immediately issues a warning signal.

[0059] The electronic fence includes two configuration modes:

[0060] Fixed electronic fence configuration mode: By adjusting the bounding box, manually configure the specific position and area size of the electronic fence.

[0061] Dynamic electronic fence configuration mode: Based on the lane line detection algorithm, dynamically update the electronic fence area by real-time solving the driving trajectory of the gantry crane and the lane line.

[0062] The intrusion perception algorithm in the above-mentioned perception workstation is implemented through computer vision and deep neural network. As the number of network layers of the traditional neural network deepens, it becomes more and more difficult to retain the complete information about the prediction target. This makes the image information used in the neural network training process incomplete, resulting in unreliable gradients and poor convergence effects.

[0063] The transformation function of the neural network of this system is composed of invertible functions, and more reliable gradients can be obtained to update the model. When the function r has an inverse transformation function v, we call this function an invertible function, as shown in the following formula: X = v α ( β ()) where α and β are the parameter data of r and v respectively. Information will not be lost through the invertible function transformation. The system's deep neural network adopts an auxiliary invertible branch to generate reliable gradients and update network parameters. The deep features of the main branch will be able to receive reliable gradient information from the auxiliary invertible branch, thereby driving parameter learning and helping to extract correct and important information.

[0064] The lane line detection algorithm in the above-mentioned perception workstation is implemented through computer vision and image processing algorithms. Specifically:

[0065] First, a large amount of road surface image data, especially the road surface image data of the terminal yard, needs to be collected. These data contain scenes with different lighting conditions and weather conditions. And perform conventional preprocessing operations such as filtering and denoising on the images.

[0066] Use the method of row-column mixed reference points to define the gantry crane lane line. Specifically, use a series of coordinates of rows and columns in the image to represent the lane line of the gantry crane. Among them, the number of set marked row reference points is R, the number of set marked column reference points is C, and record the intersection coordinates of each reference point and the lane line. If there is no intersection, set its coordinates to -1. Count the number of lane lines of row reference points as R l , and the number of lane lines of column reference points is C l, the reference position of the lane line in the image coordinate system dimension can be obtained as

[0067] T = T r + c = × R l + × C l ;

[0068] Each element is either the lane coordinate or -1.

[0069] Then, it is necessary to predict the lane line through the designed deep neural network architecture, where the loss function is defined as

[0070]

[0071] where LCE is the cross-entropy loss, is the inference value of the i-th lane line at the j-th coordinate point under the row coordinate point, is the inference value of the m-th lane line at the n-th coordinate point under the column coordinate point, and T is the corresponding classification label.

[0072] Track the lane line between consecutive frames, and combine the vehicle motion model to predict the position of the lane line in the next frame.

[0073] Adopting the embodiment of the present invention has the following beneficial effects:

[0074] By setting the monitoring devices in different areas of the long bridge operation area, the present invention realizes all-round monitoring of the operation area to obtain monitoring data, configures the electronic fence through the perception workstation to accurately detect obstacles in the operation area, and uses deep learning computer vision technology to realize the early warning of abnormal situations within the monitoring range, improving the safety of terminal operations.

[0075] Method embodiment

[0076] According to the embodiment of the present invention, a vision-based safety intelligent electronic fence protection method for quay cranes is provided. Figure 11 For the vision-based safety intelligent electronic fence protection method of the embodiment of the present invention, according to Figure 11 As shown, the vision-based safety intelligent electronic fence protection method of the embodiment of the present invention specifically includes:

[0077] Monitor the quay crane operation area through the monitoring device, and send the monitoring data to the perception workstation through the switch;

[0078] Configure the electronic fence in the quay crane operation area through the perception workstation, and based on the built-in lane line recognition algorithm, real-time recognize the lane lines on the driving path of the quay crane to ensure the safe driving of the quay crane within the predetermined route; based on the intrusion perception algorithm, real-time process the monitoring data to recognize and detect abnormal intrusions in the quay crane operation area, generate warning signals, and,

[0079] Give an audible and visual alarm for the warning signal through an audible and visual alarm device.

[0080] The method for protecting the safety intelligent electronic fence for quay crane operation based on vision in the embodiment of the present invention is a method embodiment corresponding to the above system embodiment. For specific implementation steps, please refer to the above system embodiment and will not be elaborated here.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vision-based intelligent electronic fence protection system for the safety of quay crane operations, characterized in that Including: Monitoring device, switch, perception workstation and audible and visual alarm; The monitoring device is used to monitor the yard crane operation area to obtain monitoring images, integrate the monitoring images into monitoring data, and send the monitoring data to the perception workstation through the switch; The switch is used to transmit the monitoring data to the perception workstation; The perception workstation is used to configure the intelligent electronic fence protection function for the yard crane operation area and real-time identify the lane lines on the driving path of the yard crane; Perform real-time processing and identification on the monitoring data to detect abnormal intrusion in the yard crane operation area and generate a warning signal; The audible and visual alarm is used to give an audible and visual alarm according to the warning signal; The perception workstation specifically includes: An electronic fence configuration sub-module for configuring whether to enable the intelligent electronic fence protection function; A lane line recognition sub-module for recognizing the lane lines on the form path in the yard crane operation area based on the built-in lane line recognition algorithm; An intrusion perception sub-module for, when the intelligent electronic fence protection function is enabled, performing intrusion monitoring on the yard crane operation area based on the preset intrusion perception algorithm and monitoring data, and generating a warning signal; The electronic fence configuration sub-module is specifically used for: Configuring based on whether the user selects to enable the intelligent electronic fence protection function; If the user does not enable the intelligent electronic fence protection function, the function of the electronic fence is not triggered during the yard crane operation; If the user enables the intelligent electronic fence protection function, then select the electronic fence configuration mode and monitor the yard crane operation area according to the electronic fence configuration mode; The electronic fence configuration mode includes a custom electronic fence configuration mode and a lane-line-based dynamic electronic fence configuration mode; The custom electronic fence configuration mode specifically includes: generating a region box in the configuration image by selecting a custom region, and adjusting the region box to select the warning region range of the intelligent electronic fence protection function; The lane-line-based dynamic electronic fence configuration mode specifically includes: fitting the current operation lane line in real time in the monitoring image, and setting the region within the current operation lane line as the warning region range of the intelligent electronic fence protection function.

2. The system according to claim 1, wherein The installation positions of the monitoring device specifically include: the left gantry leg of the yard crane electrical room, the right gantry leg of the yard crane electrical room, the left gantry leg of the yard crane power room, the right gantry leg of the yard crane power room, the inner side of the upper crossbeam on the side of the yard crane electrical room, the outer side of the upper crossbeam on the side of the yard crane electrical room, the inner side of the upper crossbeam on the side of the yard crane power room, and the outer side of the upper crossbeam on the side of the yard crane power room.

3. The system according to claim 1, wherein The lane line recognition sub-module is specifically used for: Obtaining the monitoring image and preprocessing the monitoring image; Obtaining the reference position of the lane line in the image coordinate system through Formula 1: Formula 1; Among them, the number of row reference points set for annotation is R, the number of column reference points set for annotation is C, and the intersection coordinates of each reference point and the lane line are recorded. If there is no intersection, its coordinates are set to -1. The number of lanes of the row reference points is R l , and the number of lanes of the column reference points is C l , and each element is either the lane coordinate or -1; Predicting the lane line through a preset deep neural network architecture and obtaining the loss function through Formula 2: Formula 2; Among them, LCE is the cross-entropy loss, is the inference value of the i th lane line at the j th coordinate point, is the inference value of the m th lane line at the n th coordinate point, represents extracting the lane line classification label of the ( i, j )th row reference point from represents extracting the lane line classification label of the ( m, n )th column reference point from Tracking the lane line between consecutive frames of the monitoring image and predicting the position of the lane line in the next frame through a preset vehicle motion model.

4. The system according to claim 1, characterized in that, The intrusion perception sub-module is implemented through computer vision and a deep neural network, and the transformation function implemented by the deep neural network consists of reversible functions.

5. The system according to claim 1, characterized in that The vision-based intelligent electronic fence protection system for quay crane operation is further used for linkage with the quay crane PLC to achieve remote alarm and connection to the automation system function.

6. The system according to claim 1, wherein The vision-based intelligent electronic fence protection system for quay crane operation further includes: a network video recorder, which is used to store monitoring data and play back the monitoring data.

7. A vision-based safety intelligent electronic fence protection method for the vision-based quay crane operation safety intelligent electronic fence protection system according to any one of the above claims 1-6, characterized in that Including: Monitoring the quay crane operation area through a monitoring device, and sending the monitoring data to the perception workstation through a switch; Configuring the electronic fence of the quay crane operation area through the perception workstation, and real-time identifying the lane lines on the driving path of the quay crane based on the built-in lane line recognition algorithm to ensure the safe driving of the quay crane within the predetermined route; Real-time processing and identifying the monitoring data based on the intrusion perception algorithm to detect abnormal intrusion in the quay crane operation area and generate a warning signal, and Performing audible and visual alarm on the warning signal through an audible and visual alarm.

Images