A shore crane ship side anti-punching person detection method and system based on machine vision

Abstract

The application provides a shore bridge ship side anti-punching person detection method and system based on machine vision, wherein the method comprises: obtaining a detection frame of a target by using machine vision technology, and transforming a personnel detection frame of a second camera into an image coordinate system of a first camera; calculating an estimated pixel coordinate of a spreader when the spreader is lowered to a personnel height in real time according to a position, a size of a current spreader detection frame and a distance from the spreader to the camera; determining a distance between the spreader and the personnel by using the estimated spreader and the personnel detection frame pixel coordinates; and generating an alarm information when the distance between the spreader and the personnel is less than a safety distance. The application calculates the distance from the personnel and the spreader to the camera by using a depth estimation network and a camera imaging principle, estimates the distance between the spreader and the personnel when the spreader is lowered to the personnel height according to the distance, and generates an alarm information according to the distance value, thereby solving the problem that the automatic spreader anti-punching person alarm judgment cannot be performed due to the change of the height from the ship to the camera.

Description

Technical Field

[0001] This invention relates to the field of safety assistance technology for port quay crane operations, and more specifically, to a machine vision-based method and system for detecting and preventing people from being hit by falling objects from the side of a quay crane. Background Technology

[0002] As a crucial carrier in modern logistics transportation, the efficiency of container loading and unloading operations directly impacts the overall port transportation efficiency. Quay cranes are bridge-type transshipment equipment used on the shore to unload containers from ships onto the dock or load them from the dock onto ships. During loading operations, the quay crane first lifts the container from the landside, transports it by air to above the container ship, and then lowers it to the designated location on the ship. Because containers are stacked on the ship, the upper and lower containers are directly connected by locking pads to prevent them from tipping over during ship rocking. Before lowering the container, workers need to install locking pads on the bottom container and direct the quay crane operator to lower the container. Due to the massive size of the quay crane equipment, the operator is often twenty meters in the air, and the spreader and the container it is lifting obstruct the view of personnel below, potentially leading to accidents where the spreader might fall and injure people. Such accidents not only cause injury or death but also severely impact operational efficiency. Detecting the distance between personnel and the spreader is a prerequisite for preventing accidents where spreader strikes people. However, container ships float on the water, and changes in the ship's load due to tides and loading operations cause changes in the height of the ship's working surface. Therefore, it is difficult to use automated methods to detect personnel positions, and using manual monitoring would increase costs. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a machine vision-based method and system for detecting and preventing people from being hit by falling objects on the side of a quay crane.

[0004] A machine vision-based method for detecting falling objects on the side of a quay crane, comprising:

[0005] Step 1: Use at least two cameras to acquire images of personnel and lifting equipment during the container placement operation on the side of the quay crane vessel from different angles;

[0006] Step 2: Input the images of personnel and lifting equipment into the lifting equipment and personnel detection network model to obtain personnel and lifting equipment detection boxes;

[0007] Step 3: Transform the person detection box of the second camera into the image coordinate system of the first camera to obtain the pixel coordinates of the fused person detection box;

[0008] Step 4: Determine the height value of the person's location based on the pixel coordinates of the person detection box;

[0009] Step 5: Calculate the estimated pixel coordinates of the hoist detection frame when the hoist descends to the height of the personnel's position based on the intrinsic parameter data of the first camera and the obtained pixel coordinates of the hoist detection frame;

[0010] Step 6: Determine the distance between the lifting device and the personnel based on the estimated pixel coordinates of the lifting device detection frame and the pixel coordinates of the personnel detection frame;

[0011] Step 7: When the distance between the lifting equipment and the personnel is less than the safe distance, a lifting equipment anti-collision alarm will be generated.

[0012] Preferably, before step 2, the method further includes: acquiring images of the lifting equipment and personnel, and inputting the images of the lifting equipment and personnel into a neural network for training to obtain a lifting equipment and personnel detection network model.

[0013] Preferably, step 4: determining the height value of the person's location based on the pixel coordinates of the person detection box includes:

[0014] Step 4.1: Acquire the work surface scene image captured by the first camera, establish a work surface scene depth estimation training sample database, and train the depth estimation network model based on the obtained work surface scene depth estimation training sample database;

[0015] Step 4.2: Input the images captured by the first camera during the container placement operation on the side of the quay crane into the depth estimation network model to obtain the depth data of the working surface;

[0016] Step 4.3: Calculate the height value of the personnel's location based on the pixel coordinates of the personnel detection box and the depth data of the work surface.

[0017] This invention also provides a machine vision-based anti-fall detection system for the side of a quay crane, comprising:

[0018] The multi-angle acquisition module is used to acquire images of personnel and lifting equipment from different angles during container placement operations on the side of the quay crane vessel using at least two cameras.

[0019] The personnel and lifting equipment detection module is used to input the images of the personnel and lifting equipment into the lifting equipment and personnel detection network model to obtain the personnel and lifting equipment detection boxes;

[0020] The personnel detection box fusion module is used to transform the personnel detection box of the second camera into the image coordinate system of the first camera to obtain the pixel coordinates of the fused personnel detection box;

[0021] The personnel height calculation module is used to calculate the height value of the personnel's location based on the pixel coordinates of the personnel detection box;

[0022] The lifting device detection frame pixel coordinate calculation module is used to calculate the estimated pixel coordinates of the lifting device detection frame when the lifting device is lowered to the height of the personnel's position based on the intrinsic parameter data of the first camera and the obtained pixel coordinates of the lifting device detection frame;

[0023] The distance determination module between the lifting equipment and the personnel is used to determine the distance between the lifting equipment and the personnel based on the estimated pixel coordinates of the lifting equipment detection frame and the pixel coordinates of the personnel detection frame.

[0024] The alarm module is used to generate a crane anti-collision alarm when the distance between the crane and personnel is less than the safe distance.

[0025] Preferred options also include:

[0026] The training module is used to acquire images of the lifting equipment and personnel, and input the images of the lifting equipment and personnel into the neural network for training to obtain a network model for detecting the lifting equipment and personnel.

[0027] Preferably, the personnel height calculation module includes:

[0028] The deep training unit is used to acquire images of the work surface scene captured by the first camera, establish a training sample database for depth estimation of the work surface scene, and train a depth estimation network model based on the acquired training sample database for depth estimation of the work surface scene.

[0029] The working face depth data calculation unit is used to input the images captured by the first camera during the container placement operation on the side of the quay crane into the depth estimation network model to obtain the working face depth data;

[0030] The personnel position calculation unit is used to calculate the height value of the personnel's position based on the pixel coordinates of the personnel detection box and the depth data of the work surface.

[0031] The present invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are connected via the bus. The computer program, when executed by the processor, implements the steps in the above-described machine vision-based method for preventing people from being hit on the side of a quay crane.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps in the above-described machine vision-based method for detecting pedestrian fall on the side of a quay crane.

[0033] The beneficial effects of the machine vision-based method and system for detecting personnel falling from quay cranes on the ship side provided by this invention are as follows: Compared with the prior art, this invention uses multiple cameras to capture images of quay crane operations on the ship side and obtains a fused personnel detection frame based on these images, reducing the missed detections caused by occlusion when using a single camera; it uses a depth estimation network and camera imaging principles to calculate the distance between personnel and the lifting equipment and the camera, thereby predicting the distance between the lifting equipment and the personnel when the lifting equipment descends to the personnel's height, and generating alarm information based on this distance value, thus solving the problem that the lifting equipment cannot automatically perform anti-fall alarm judgment due to changes in the height between the ship and the camera.

[0034] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 The steps of the shore bridge ship-side anti-fall-person detection method provided in the embodiment of the present invention are shown;

[0037] Figure 2 This diagram shows a view of the camera installation location provided in an embodiment of the present invention. Detailed Implementation

[0038] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0039] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0040] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0041] Please refer to the following: Figure 1-2 A machine vision-based method for detecting falling objects on the side of a quay crane, comprising:

[0042] Step 1: Use at least two cameras to acquire images of personnel and lifting equipment during the container placement operation on the side of the quay crane vessel from different angles;

[0043] Step 2: Input the images of personnel and lifting equipment into the lifting equipment and personnel detection network model to obtain personnel and lifting equipment detection boxes; wherein, the present invention can obtain the lifting equipment and personnel detection network model by acquiring images of lifting equipment and personnel, inputting the images of lifting equipment and personnel into a neural network for training.

[0044] Step 3: Transform the person detection box of the second camera into the image coordinate system of the first camera to obtain the pixel coordinates of the fused person detection box;

[0045] Step 4: Determine the height value of the person's location based on the pixel coordinates of the person detection box;

[0046] Furthermore, step 4 includes:

[0047] Step 4.1: Acquire the work surface scene image captured by the first camera, establish a work surface scene depth estimation training sample database, and train the depth estimation network model based on the obtained work surface scene depth estimation training sample database;

[0048] Step 4.2: Input the images captured by the first camera during the container placement operation on the side of the quay crane into the depth estimation network model to obtain the depth data of the working surface;

[0049] Step 4.3: Calculate the height value of the personnel's location based on the pixel coordinates of the personnel detection box and the depth data of the work surface.

[0050] Step 5: Calculate the estimated pixel coordinates of the hoist detection frame when the hoist descends to the height of the personnel's position based on the intrinsic parameter data of the first camera and the obtained pixel coordinates of the hoist detection frame;

[0051] Step 6: Determine the distance between the lifting device and the personnel based on the estimated pixel coordinates of the lifting device detection frame and the pixel coordinates of the personnel detection frame;

[0052] Step 7: When the distance between the lifting equipment and the personnel is less than the safe distance, a lifting equipment anti-collision alarm will be generated.

[0053] The following describes a machine vision-based method for detecting falling objects on the side of a quay crane, in further detail with reference to specific embodiments:

[0054] Please see Figure 2 This invention discloses a machine vision-based method for preventing personnel from being struck by falling objects on the side of a quay crane, comprising a real-time video image acquisition device for personnel and lifting equipment, a working surface depth acquisition device, and a real-time processing system for preventing personnel from being struck by falling objects. The real-time video image acquisition device for personnel and lifting equipment consists of a first camera and a second camera, used to acquire images of personnel and lifting equipment during operation. The working surface depth acquisition device includes a first camera and a depth estimation algorithm, estimating depth information in the images by analyzing the images from the first camera. The real-time processing system for preventing personnel from being struck by falling objects calculates the distance between personnel and lifting equipment based on the positions of personnel and lifting equipment, and the depth data of personnel, and generates alarm information. The camera installation positions are shown in the attached diagram; the first camera is used to capture images of the lifting equipment, personnel, and working surface; the second camera is used to capture images of personnel.

[0055] The implementation steps are as follows:

[0056] 1. Using images of lifting equipment and personnel acquired by a first camera and a second camera, establish a database of lifting equipment and personnel objects; construct a target training sample dataset and a test sample dataset based on the obtained database of lifting equipment and personnel objects; train a network model for detecting lifting equipment and personnel based on the target training sample dataset and the test sample dataset.

[0057] 2. Acquire the work surface scene images captured by the first camera, establish a depth estimation training sample database for the work surface scene, and train a depth estimation network model based on the obtained depth estimation training sample database.

[0058] 3. Perform intrinsic and extrinsic parameter calibration on the first and second cameras to obtain their intrinsic and extrinsic parameter data. Using the intrinsic and extrinsic parameter calibration data, the pixel position of the detection box in the second camera image and its position in the first camera image can be calculated.

[0059] 4. Input the image from the first camera into the depth estimation network to detect the depth data of the working surface in the image.

[0060] 5. Acquire images from the first and second cameras during the container placement operation on the ship's side. Input these images into the lifting equipment and personnel detection network for detection, generating detection boxes for personnel and lifting equipment. Transform the personnel detection box from the second camera into the image coordinate system of the first camera and perform target fusion to obtain the pixel coordinates of the fused personnel detection box. It should be noted that target fusion is used to compensate for the problem that personnel obscured by the lifting equipment cannot be captured due to the limited field of view of the first camera.

[0061] In this embodiment, the calculation method for target fusion is as follows:

[0062] C = A∪B

[0063] Where: A is the set of personnel detection results from the first camera.

[0064] B is the set of personnel detection results from the second camera (already transformed to the coordinate system of the first camera).

[0065] C is the union of the two sets, i.e., the merged personnel detection results.

[0066] When performing union calculations, the method for determining if two elements are the same is to calculate the Intersection over Union (IOU) value of the two detection result bounding boxes. If the IOU value is greater than 85%, they are considered the same element; otherwise, they are considered different elements. The calculation formula is as follows:

[0067] IOU(a,b)>85%

[0068] Where: a is an element of set A, the coordinate frame of the personnel detection results;

[0069] b is an element of set B, representing the coordinate frame of the personnel detection results.

[0070] 6. Based on the pixel position of the personnel inspection box and the depth data of the work surface, determine the depth value of the personnel's location;

[0071] 7. Calculate the estimated pixel coordinates of the sling detection frame when the sling descends to the height of the personnel position using the camera intrinsic data and the pixel coordinates of the current sling detection frame;

[0072] 8. Calculate the distance between the sling and the personnel based on the pixel coordinates of the personnel inspection frame and the estimated pixel coordinates of the sling detection frame when the sling reaches the personnel height; when the distance between the sling and the personnel is less than the safe distance, generate a sling anti-collision alarm and notify the bridge crane operator.

[0073] This invention captures images of the quay crane operation from the side of the ship using multiple cameras. It then uses a depth estimation network and camera imaging principles to calculate the distances from personnel and the lifting equipment to the cameras. Based on this, it estimates the distance between the lifting equipment and personnel when the equipment descends to the personnel's height. Simultaneously, it generates alarm information based on this distance value, thus solving the problem of not being able to automatically determine the lifting equipment anti-fall alarm due to changes in the height of the ship from the camera.

[0074] This invention also provides a machine vision-based anti-fall detection system for the side of a quay crane, comprising:

[0075] The multi-angle acquisition module is used to acquire images of personnel and lifting equipment from different angles during container placement operations on the side of the quay crane vessel using at least two cameras.

[0076] The personnel and lifting equipment detection module is used to input the images of the personnel and lifting equipment into the lifting equipment and personnel detection network model to obtain the personnel and lifting equipment detection boxes;

[0077] The personnel detection box fusion module is used to transform the personnel detection box of the second camera into the image coordinate system of the first camera to obtain the pixel coordinates of the fused personnel detection box;

[0078] The personnel height calculation module is used to calculate the height value of the personnel's location based on the pixel coordinates of the personnel detection box;

[0079] The lifting device detection frame pixel coordinate calculation module is used to calculate the estimated pixel coordinates of the lifting device detection frame when the lifting device is lowered to the height of the personnel's position based on the intrinsic parameter data of the first camera and the obtained pixel coordinates of the lifting device detection frame;

[0080] The distance determination module between the lifting equipment and the personnel is used to determine the distance between the lifting equipment and the personnel based on the estimated pixel coordinates of the lifting equipment detection frame and the pixel coordinates of the personnel detection frame.

[0081] The alarm module is used to generate a crane anti-collision alarm when the distance between the crane and personnel is less than the safe distance.

[0082] Preferred options also include:

[0083] The training module is used to acquire images of the lifting equipment and personnel, and input the images of the lifting equipment and personnel into the neural network for training to obtain a network model for detecting the lifting equipment and personnel.

[0084] Preferably, the personnel height calculation module includes:

[0085] The deep training unit is used to acquire images of the work surface scene captured by the first camera, establish a training sample database for depth estimation of the work surface scene, and train a depth estimation network model based on the acquired training sample database for depth estimation of the work surface scene.

[0086] The working face depth data calculation unit is used to input the images captured by the first camera during the container placement operation on the side of the quay crane into the depth estimation network model to obtain the working face depth data;

[0087] The personnel position calculation unit is used to calculate the height value of the personnel's position based on the pixel coordinates of the personnel detection box and the depth data of the work surface.

[0088] Compared with the prior art, the beneficial effects of the machine vision-based quay crane ship-side anti-pinch detection system provided by the present invention are the same as those of the machine vision-based quay crane ship-side anti-pinch detection method described in the above technical solution, and will not be repeated here.

[0089] This invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are connected via the bus. The computer program, when executed by the processor, implements the steps in the aforementioned machine vision-based method for preventing pedestrian falls on the side of a quay crane. Compared with the prior art, the beneficial effects of the electronic device provided by this invention are the same as those of the machine vision-based method for preventing pedestrian falls on the side of a quay crane described above, and will not be repeated here.

[0090] The present invention also provides a computer-readable storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps in the aforementioned machine vision-based method for detecting pedestrians falling on the side of a quay crane. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as those of the machine vision-based method for detecting pedestrians falling on the side of a quay crane described above, and will not be repeated here.

[0091] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting a ship side anti-punching person of a shore-to-ship crane based on machine vision, characterized in that, The method comprises the following steps: Step 1: acquiring personnel and spreader images during shipside container placing of a quay crane from different angles by using at least two cameras; Step 2: inputting the personnel and spreader images into a spreader and personnel detection network model to obtain personnel and spreader detection boxes; Step 3: transforming the personnel detection box of the second camera into the image coordinate system of the first camera to obtain a fused personnel detection box pixel coordinate; Step 4: obtaining a height value of a position where the personnel is located according to the personnel detection box pixel coordinate; Step 5: calculating an estimated pixel coordinate of the spreader detection box when the spreader is lowered to the height of the position where the personnel is located according to the pixel coordinate of the spreader detection box and the internal parameter data of the first camera; Step 6: determining the distance between the spreader and the personnel according to the estimated pixel coordinate of the spreader detection box and the personnel detection box pixel coordinate; Step 7: generating a spreader anti-punching alarm information when the distance between the spreader and the personnel is less than a safety distance. Before the step 2, the method further comprises the following steps: acquiring images of the spreader and the personnel, and inputting the images of the spreader and the personnel into a neural network for training to obtain the spreader and personnel detection network model; The step 4: obtaining a height value of a position where the personnel is located according to the personnel detection box pixel coordinate, comprises the following steps: Step 4.1: acquiring a work surface scene image collected by the first camera, establishing a work surface scene depth estimation training sample database, and training a depth estimation network model according to the obtained work surface scene depth estimation training sample database; Step 4.2: inputting the image collected by the first camera during the shipside container placing of the quay crane into the depth estimation network model to obtain work surface depth data; 2. A machine vision based anti-piercing detection system for a ship side of a shore crane, characterized in that, Step 4.3: obtaining a height value of a position where the personnel is located according to the personnel detection box pixel coordinate and the work surface depth data. The method comprises the following steps: a multi-angle acquisition module, configured to acquire personnel and spreader images during shipside container placing of a quay crane from different angles by using at least two cameras; a training module, configured to acquire images of the spreader and the personnel, and input the images of the spreader and the personnel into a neural network for training to obtain a spreader and personnel detection network model; a personnel and spreader detection module, configured to input the personnel and spreader images into the spreader and personnel detection network model to obtain personnel and spreader detection boxes; a personnel detection box fusion module, configured to transform the personnel detection box of the second camera into the image coordinate system of the first camera to obtain a fused personnel detection box pixel coordinate; a personnel height calculation module, configured to obtain a height value of a position where the personnel is located according to the personnel detection box pixel coordinate, and comprising the following steps: a depth training unit, configured to acquire a work surface scene image collected by the first camera, establish a work surface scene depth estimation training sample database, and train a depth estimation network model according to the obtained work surface scene depth estimation training sample database; a work surface depth data calculation unit, configured to input the image collected by the first camera during the shipside container placing of the quay crane into the depth estimation network model to obtain work surface depth data; a personnel position calculation unit, configured to obtain a height value of a position where the personnel is located according to the personnel detection box pixel coordinate and the work surface depth data; The lifting device detection frame pixel coordinate calculation module is used to calculate the estimated pixel coordinates of the lifting device detection frame when the lifting device is lowered to the height of the personnel's position based on the intrinsic parameter data of the first camera and the obtained pixel coordinates of the lifting device detection frame; The distance determination module between the lifting equipment and the personnel is used to determine the distance between the lifting equipment and the personnel based on the estimated pixel coordinates of the lifting equipment detection frame and the pixel coordinates of the personnel detection frame. The alarm module is used to generate a crane anti-collision alarm when the distance between the crane and personnel is less than the safe distance.

3. An electronic device comprising a bus, a transceiver, a memory, a processor, and a computer program stored on the memory and executable on the processor, the transceiver, the memory and the processor being connected by the bus, characterized in that, When the computer program is executed by the processor, it implements the steps in the machine vision-based method for preventing people from being hit on the side of a quay crane as described in claim 1.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the machine vision-based method for preventing people from being hit on the side of a quay crane as described in claim 1.

Images