Vision-based intelligent electronic fence protection system and method for field bridge operation safety

Abstract

The invention provides a vision-based intelligent electronic fence protection system and method for field bridge operation safety. The system comprises monitoring equipment, a switch, a sensing workstation and an audible and visual alarm, the monitoring equipment is used for monitoring a field bridge operation area to obtain monitoring images, integrating the monitoring images into monitoring data and sending the monitoring data to the sensing work station through the switch; the switch is used for transmitting the monitoring data to the sensing work station; the sensing work station is used for configuring an intelligent electronic fence protection function of a yard bridge operation area and identifying lane lines on a yard bridge driving path in real time; performing real-time processing and identification on the monitoring data, detecting abnormal invasion in the field bridge operation area, and generating an early warning signal; according to the technical scheme provided by the invention, obstacles in an operation area can be accurately detected, and abnormal conditions in a specific range can be early warned.

Description

Technical Field

[0001] This document relates to the field of intelligent electronic fence technology, and in particular to a vision-based intelligent electronic fence protection system and method for field bridge operation safety. Background Art

[0002] The yard crane is the main loading and unloading equipment at the container terminal, and has the characteristics of complex working environment and limited vision. The current operation mode includes the yard crane driver operating in the driver's cab and the yard crane driver operating in the remote control room. When operating in the driver's cab, the driver's vision is affected by the structure of the mechanical equipment and cannot monitor the safety of the truck's running route throughout the whole process; while operating in the remote control room, the driver needs to observe multiple monitoring screens at the same time, which distracts attention and may cause important information to be missed.

[0003] At the same time, due to the presence of truck drivers and maintenance personnel under the yard bridge, the working environment is complex and there are a large number of cross-operation conditions. The existing operating mode is difficult to ensure the safety of terminal personnel and property. Summary of the invention

[0004] The present application provides a vision-based intelligent electronic fence protection system and method for field bridge operation safety, aiming to solve the above-mentioned problems.

[0005] The embodiment of the present invention provides a vision-based intelligent electronic fence protection system for field bridge operation safety, including: monitoring equipment, switches, sensing workstations and sound and light alarms;

[0006] The monitoring device is used to monitor the field bridge operation area to obtain monitoring images, integrate the monitoring images into monitoring data, and send the monitoring data to the perception workstation through the switch;

[0007] The switch is used to transmit the monitoring data to the perception workstation;

[0008] The sensing workstation is used to configure the intelligent electronic fence protection function of the field bridge operation area, identify the lane lines on the field bridge driving path in real time; process and identify the monitoring data in real time and detect abnormal intrusions in the field bridge operation area, and generate early warning signals;

[0009] The sound and light alarm is used to make a sound and light alarm according to the early warning signal.

[0010] The embodiment of the present invention provides a vision-based intelligent electronic fence protection method for field bridge operation safety, including:

[0011] The field crane operation area is monitored by monitoring equipment, and the monitoring data is sent to the sensing workstation through a switch;

[0012] The electronic fence of the field bridge operation area is configured through the perception workstation, and the lane lines on the field bridge driving path are identified in real time based on the built-in lane line recognition algorithm to ensure the safe driving of the field bridge within the predetermined route; the monitoring data is processed and identified in real time based on the intrusion perception algorithm, and abnormal intrusions in the field bridge operation area are detected to generate early warning signals, and,

[0013] The early warning signal is given an audible and visual alarm by an audible and visual alarm.

[0014] The present invention realizes all-round monitoring of the operation area and obtains monitoring data by setting monitoring equipment in different areas of the long bridge operation area, configures the electronic fence through the sensing workstation, and achieves accurate detection of obstacles in the operation area; and uses deep learning computer vision technology to realize early warning of abnormal situations within the monitoring range, thereby improving the safety of terminal operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 A schematic diagram of a vision-based intelligent electronic fence protection system for field bridge operation safety according to an embodiment of the present invention;

[0017] Figure 2 A topological diagram of a direct deployment mode of a vision-based intelligent electronic fence protection system for field bridge operation safety according to an embodiment of the present invention;

[0018] Figure 3 A topological diagram of a linkage deployment mode between a vision-based intelligent electronic fence protection system for field bridge operation safety and a field bridge PLC according to an embodiment of the present invention;

[0019] Figure 4 It is a schematic diagram of the detection coverage overhead area of ​​the vision-based intelligent electronic fence protection system for field bridge operation safety according to an embodiment of the present invention;

[0020] Figure 5 A front view of the camera arrangement in the traveling direction of the field bridge according to an embodiment of the present invention;

[0021] Figure 6 A rear view of the camera arrangement in the traveling direction of the field bridge according to an embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram of the arrangement of cameras on the electrical room side of an embodiment of the present invention;

[0023] Figure 8 This is a schematic diagram of the arrangement of cameras on the power room side of an embodiment of the present invention;

[0024] Fig. 9 A schematic diagram of whether to enable the smart electronic fence function according to an embodiment of the present invention;

[0025] Fig.10 A schematic diagram of specific operations of a vision-based intelligent electronic fence protection method for field bridge operation safety according to an embodiment of the present invention;

[0026] Fig.11 The present invention is a flowchart of a vision-based intelligent electronic fence protection method for field bridge operation safety according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0028] System Example

[0029] According to an embodiment of the present invention, a vision-based intelligent electronic fence protection system for field bridge operation safety is provided. Figure 1 The vision-based intelligent electronic fence protection system for field bridge operation safety of the embodiment of the present invention is based on Figure 1 As shown, the vision-based intelligent electronic fence protection system for field bridge operation safety in an embodiment of the present invention specifically includes:

[0030] Monitoring equipment 15, switch 9, sensing workstation 10 and sound and light alarm 12;

[0031] The monitoring device 15 is used to monitor the field crane operation area to obtain monitoring images, integrate the monitoring images into monitoring data, and send the monitoring data to the perception workstation 10 through the switch 9;

[0032] Figure 2 This is a topological diagram of a direct deployment mode of a vision-based intelligent electronic fence protection system for field bridge operation safety according to an embodiment of the present invention. Figure 2It can be seen that the monitoring cameras include four cameras, namely, the left door leg camera 1 on the electrical room side, the right door leg camera 2 on the electrical room side, the left door leg camera 3 on the power room side, and the right door leg camera 4 on the power room side, whose field of view covers the road area in front of the field bridge trolley, and is used to obtain image data of the environment in front of the field bridge operation; the cameras 5 on the inner side of the upper beam on the electrical room side, the outer side of the upper beam on the electrical room side, the inner side of the upper beam on the power room side, and the outer side of the upper beam on the power room side are installed at the upper beam of the field bridge, and the field of view covers the surrounding environment areas inside and outside the field bridge, and is used to obtain environmental image data inside and outside the field bridge operation area, and transmit the image data through the switch. Figure 4 Schematic diagram of the detection coverage overhead area of ​​the vision-based intelligent electronic fence protection system for field bridge operation safety according to an embodiment of the present invention, Figure 5-Figure 8 They are schematic diagrams of the arrangement positions of monitoring cameras in the embodiments of the present invention, wherein the shaded part represents the monitoring area.

[0033] The switch 9 is used to transmit the monitoring data to the sensing workstation 10;

[0034] The sensing workstation 10 is used to configure the intelligent electronic fence protection function of the field bridge operation area, and identify the lane lines on the field bridge driving path in real time; process and identify the monitoring data in real time and detect abnormal intrusions in the field bridge operation area, and generate early warning signals;

[0035] The sensing workstation 10 acquires the image data transmitted by the monitoring camera through the switch 9, analyzes and processes the acquired image data, processes the image data in real time through computer vision and deep learning algorithms, identifies and detects abnormal intrusions in the designated operation area, and generates warning signals. The sensing workstation 10 integrates electronic fence custom configuration debugging software, lane line recognition algorithm software, and intrusion perception algorithm software.

[0036] The sensing workstation 10 specifically includes:

[0037] The electronic fence configuration submodule is used to configure whether to enable the intelligent electronic fence protection function;

[0038] The lane line recognition submodule is used to try to recognize the lane lines on the formal path in the field bridge operation area based on the built-in lane line recognition algorithm;

[0039] The intrusion sensing submodule is used to monitor the intrusion of the field crane operation area based on the preset intrusion sensing algorithm and monitoring data when the intelligent electronic fence protection function is enabled, and generate an early warning signal.

[0040] Fig. 9It is a basic flow chart of the electronic fence configuration submodule of an embodiment of the present invention; start the electronic fence custom configuration debugging software, select whether to enable the visual electronic fence function, if not enabled, record the configuration of not triggering the electronic fence function during the operation process, if enabled, you need to select the configuration of the electronic fence area, including a custom area fixed electronic fence configuration mode and a lane line solution area dynamic electronic fence configuration mode. The lane line solution area mode is used by default; selecting the custom area mode will generate an area frame in the configuration image, and the area frame can be adjusted to select the area range of the electronic fence; selecting the lane line solution area mode will fit the actual operation lane line in the image in real time, and set the lane line as the electronic fence warning area. After saving the settings, it will be applied to the field bridge operation safety intelligent electronic fence protection system.

[0041] Fig.10 The basic working process of the vision-based intelligent electronic fence protection system for field bridge operation safety in the embodiment of the present invention is as follows: Fig.10 It can be seen that the system will first determine whether the electronic fence function is enabled. If it is not enabled, it will provide a regular monitoring screen; if it is enabled, it will provide a monitoring screen with a virtual electronic fence and intrusion target detection results. Then it will retrieve the configured electronic fence strategy, including the custom area mode and the lane line solution mode, and perform real-time analysis based on the relevant algorithm. If an intrusion target appears, it will be reminded through the monitoring screen and alarm module. The above-mentioned intrusion targets include but are not limited to people, cars, engineering vehicles, cones, boxes, large blocks, spikes, and all other targets that affect the normal operation of the field bridge.

[0042] The sound and light alarm 12 is used to make a sound and light alarm according to the early warning signal. When an intrusion is detected, the system will remind the field crane driver and the remote control operator through the sound and light alarm to prevent accidents.

[0043] The vision-based intelligent electronic fence protection system for field bridge operation safety in the embodiment of the present invention further includes: a network hard disk recorder 11, which obtains image data transmitted by the monitoring camera through a switch and stores it for later playback and analysis. The network hard disk recorder has a high-capacity storage function and can record image data of the field bridge operation environment for a long time, which is convenient for subsequent review and safety accident investigation.

[0044] Figure 3 The topological diagram of the vision-based intelligent electronic fence protection system for field bridge operation safety and the field bridge PLC linkage deployment method according to the embodiment of the present invention; Figure 3 It can be seen that the vision-based intelligent electronic fence protection system for field crane operation safety in the embodiment of the present invention is further used to interact with the field crane PLC to achieve remote alarm and docking automation system functions.

[0045] The monitoring camera in the embodiment of the present invention is installed at the door leg and the upper crossbeam of the field bridge. Figure 5-Figure 8 The specific locations are as follows:

[0046] A camera is installed on the left door leg of the field crane electrical room, and its field of view covers the left side of the field crane electrical room in the direction of travel;

[0047] A camera is installed on the right door leg of the field crane electrical room, and its field of view covers the right side of the field crane electrical room in the direction of travel;

[0048] A camera is installed on the left door leg of the crane power house, and its field of view covers the left side of the crane power house in the direction of travel;

[0049] A camera is installed on the right side of the gate leg of the crane power house, and its field of view covers the right side of the crane power house in the direction of travel;

[0050] Two cameras are installed on the upper crossbeam on the electrical room side of the field bridge, with one camera facing the lower inside of the field bridge and the other facing the lower outside of the field bridge. The field of view covers the area near the vehicle body on the electrical room side of the field bridge trolley.

[0051] Two cameras are installed on the upper crossbeam on the power room side of the field crane, with one lens facing the inner and lower side of the field crane, and the other lens facing the outer and lower side of the field crane, covering the area near the vehicle body on the power room side of the field crane.

[0052] In a specific embodiment of the present invention, the sensing workstation includes the following capabilities:

[0053] Efficient data processing capabilities, through a highly integrated parallel algorithm framework combined with hardware acceleration strategies, can process large-scale image data in real time.

[0054] The deep neural network model has a built-in deep neural network model with high-precision recognition and classification capabilities. It can effectively identify various intrusion targets in the operating area and can identify accurate lane information of the field crane operation.

[0055] The electronic fence custom configuration and debugging software allows users to customize the configuration and debugging of the electronic fence according to specific operation requirements, ensuring the accuracy and flexibility of the electronic fence.

[0056] Lane line recognition algorithm software is used to identify lane lines on the field crane's driving path in real time to ensure that the field crane drives safely within the predetermined route.

[0057] The intrusion perception algorithm software uses deep learning algorithms to detect intrusions within the operating area in real time and generate corresponding warning signals.

[0058] The electronic fence custom configuration debugging software in the above-mentioned perception workstation has a custom electronic fence function, which can monitor the designated operation area in real time through the virtual electronic fence setting. When an intrusion target that affects normal operation enters the electronic fence range, the system immediately sends out an early warning signal.

[0059] The electronic fence includes two configuration modes:

[0060] Fixed electronic fence configuration mode: The specific location and area size of the electronic fence are manually configured by adjusting the boundary box.

[0061] Dynamic electronic fence configuration mode: Based on the lane line detection algorithm, the electronic fence area is dynamically updated by real-time solving the driving trajectory of the field bridge and the lane line.

[0062] The intrusion perception algorithm in the above-mentioned perception workstation is implemented through computer vision and deep neural network. As the number of layers of traditional neural networks increases, it becomes more difficult to retain complete information about the predicted target. This makes the image information used in the neural network training process incomplete, resulting in unreliable gradients and poor convergence effects.

[0063] The transformation function of the neural network in this system is composed of reversible functions, which can obtain more reliable gradients to update the model. When a function r has an inverse transformation function v, we call this function a reversible function, as shown in the following formula: X = v α ( β ()) where α and β are the parameter data of r and v respectively. The data can be converted through the reversible function without losing information. The system deep neural network adopts auxiliary reversible branches to generate reliable gradients and update network parameters. The main branch deep features will be able to receive reliable gradient information from the auxiliary reversible branches, thereby driving parameter learning and helping to extract correct and important information.

[0064] The lane detection algorithm in the above perception workstation is implemented through computer vision and image processing algorithms. Specifically:

[0065] First, it is necessary to collect a large amount of road surface image data, especially the road surface image data of the dock yard, which contains scenes with different lighting conditions and weather conditions. And perform conventional preprocessing operations such as filtering and denoising on the images.

[0066] The lane lines of the field bridge are defined by using a mixed row and column reference point method. Specifically, a series of row and column coordinates in the image are used to represent the lane lines of the field bridge, where the number of marked row reference points is set to R, the number of marked column reference points is set to C, and the coordinates of the intersection between each reference point and the lane line are recorded. If there is no intersection, its coordinates are set to -1, and the number of row reference point lanes is counted as R l , the number of reference point lanes is C l, then the reference position of the lane line in the image coordinate system dimension can be obtained as

[0067] T=T r + c = × R l +×C l ;

[0068] Each element is a lane coordinate or -1.

[0069] Then the lane line needs to be predicted through the designed deep neural network architecture, where the loss function is defined as

[0070]

[0071] Where LCE is the cross entropy loss, is the inferred value of the i-th lane line at the j-th coordinate point under the row coordinate point, is the inferred value of the mth lane line at the nth coordinate point under the column coordinate point, and T is the corresponding classification identifier.

[0072] The lane lines are tracked between consecutive frames, and the position of the lane lines in the next frame is predicted in combination with the vehicle motion model.

[0073] The embodiments of the present invention have the following beneficial effects:

[0074] The present invention realizes all-round monitoring of the operation area and obtains monitoring data by setting monitoring equipment in different areas of the long bridge operation area, configures the electronic fence through the sensing workstation, and achieves accurate detection of obstacles in the operation area; and uses deep learning computer vision technology to realize early warning of abnormal situations within the monitoring range, thereby improving the safety of terminal operations.

[0075] Method Embodiment

[0076] According to an embodiment of the present invention, a vision-based intelligent electronic fence protection method for field bridge operation safety is provided. Fig.11 The invention provides a vision-based intelligent electronic fence protection method for field bridge operation safety according to an embodiment of the present invention. Fig.11 As shown, the vision-based intelligent electronic fence protection method for field bridge operation safety in an embodiment of the present invention specifically includes:

[0077] The field crane operation area is monitored by monitoring equipment, and the monitoring data is sent to the sensing workstation through a switch;

[0078] The electronic fence of the field bridge operation area is configured through the perception workstation, and the lane lines on the field bridge driving path are identified in real time based on the built-in lane line recognition algorithm to ensure the safe driving of the field bridge within the predetermined route; the monitoring data is processed and identified in real time based on the intrusion perception algorithm, and abnormal intrusions in the field bridge operation area are detected to generate early warning signals, and,

[0079] The early warning signal is given an audible and visual alarm by an audible and visual alarm.

[0080] The vision-based intelligent electronic fence protection method for field bridge operation safety in an embodiment of the present invention is a method embodiment corresponding to the above-mentioned system embodiment. Please refer to the above-mentioned system embodiment for specific implementation steps, which will not be repeated here.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vision-based intelligent electronic fence protection system for field bridge operation safety, characterized by include: Monitoring equipment, switches, sensing workstations, and sound and light alarms; The monitoring device is used to monitor the field bridge operation area to obtain monitoring images, integrate the monitoring images into monitoring data, and send the monitoring data to the perception workstation through the switch; The switch is used to transmit the monitoring data to the perception workstation; The sensing workstation is used to configure the intelligent electronic fence protection function of the field bridge operation area and identify the lane lines on the field bridge driving path in real time; Process the monitoring data in real time to identify and detect abnormal intrusions in the field crane operation area and generate early warning signals; The sound and light alarm is used to make a sound and light alarm according to the early warning signal.

2. The system according to claim 1, characterized in that The installation locations of the monitoring equipment specifically include: the left door leg of the field bridge electrical room, the right door leg of the field bridge electrical room, the left door leg of the field bridge power room, the right door leg of the field bridge power room, the inner side of the upper beam on the field bridge electrical room side, the outer side of the upper beam on the field bridge electrical room side, the inner side of the upper beam on the field bridge power room side, and the outer side of the upper beam on the field bridge power room side.

3. The system according to claim 1, characterized in that The sensing workstation specifically includes: The electronic fence configuration submodule is used to configure whether to enable the intelligent electronic fence protection function; The lane line recognition submodule is used to try to recognize the lane lines on the formal path in the field bridge operation area based on the built-in lane line recognition algorithm; The intrusion sensing submodule is used to monitor the intrusion of the field crane operation area based on the preset intrusion sensing algorithm and monitoring data when the intelligent electronic fence protection function is enabled, and generate an early warning signal.

4. The system according to claim 3, characterized in that The electronic fence configuration submodule is specifically used for: Configure based on whether the user chooses to enable the smart electronic fence protection function; If the user does not enable the intelligent electronic fence protection function, the electronic fence function will not be triggered during the field bridge operation; If the user enables the intelligent electronic fence protection function, the electronic fence configuration mode is selected, and the field bridge operation area is monitored according to the electronic fence configuration mode.

5. The system according to claim 4, characterized in that The electronic fence configuration mode includes a custom electronic fence configuration mode and a lane line-based dynamic electronic fence configuration mode; The custom electronic fence configuration mode specifically includes: generating an area frame in the configuration image by selecting a custom area, and adjusting the area frame to select the warning area range of the intelligent electronic fence protection function; The lane-line-based dynamic electronic fence configuration mode specifically includes: fitting the current operating lane line in real time in the monitoring image, and setting the area within the current operating lane line as the warning area range of the intelligent electronic fence protection function.

6. The system according to claim 3, characterized in that The lane line recognition submodule is specifically used for: Acquire a surveillance image and preprocess the surveillance image; The reference position of the lane line in the image coordinate system dimension is obtained by formula 1: T=T r +T c =R×R l +C×C l Formula 1; The number of marked row reference points is set to R, the number of marked column reference points is set to C, and the coordinates of the intersection between each reference point and the lane line are recorded. If there is no intersection, its coordinates are set to -1. The number of row reference point lanes is R. l , the number of reference point lanes is C l , each element is the lane coordinate or -1; The lane line is predicted through the preset deep neural network architecture, and the loss function is obtained through formula 2: Among them, LCE is the cross entropy loss, is the inferred value of the i-th lane line at the j-th coordinate point under the row coordinate point, is the inferred value of the mth lane line at the nth coordinate point under the column coordinate point, and T is the corresponding classification identifier; The lane lines are tracked between consecutive frames of the monitoring image, and the position of the lane lines in the next frame is predicted through the preset vehicle motion model.

7. The system according to claim 3, characterized in that The intrusion perception submodule is implemented by computer vision and a deep neural network, and the transformation function implemented by the deep neural network consists of reversible functions.

8. The system according to claim 1, characterized in that The vision-based intelligent electronic fence protection system for field crane operation safety is further used to interact with the field crane PLC to achieve remote alarm and docking automation system functions.

9. The system according to claim 1, characterized in that The vision-based field bridge operation safety intelligent electronic fence protection system further includes: a network hard disk video recorder, which is used to store monitoring data and play back the monitoring data.

10. A vision-based intelligent electronic fence protection method for field bridge operation safety, characterized in that include: The field crane operation area is monitored by monitoring equipment, and the monitoring data is sent to the sensing workstation through a switch; The electronic fence of the field crane operation area is configured through the sensing workstation, and the lane lines on the field crane driving path are identified in real time based on the built-in lane line recognition algorithm to ensure that the field crane drives safely within the predetermined route; Based on the intrusion perception algorithm, the monitoring data is processed in real time to identify and detect abnormal intrusions in the field crane operation area, generate early warning signals, and, The early warning signal is given an audible and visual alarm by an audible and visual alarm.

Images