Method and system for identifying and calculating swing of grab bucket of port portal crane based on target detection

Abstract

The invention relates to the field of portal cranes, in particular to a target-detection-based swing identification and calculation method and system for a grab bucket of a port portal crane, and the method comprises the steps: providing a real-time grab bucket image through a camera installed on a trunk beam of the portal crane, and obtaining the current swing of the grab bucket after target identification and calculation; by means of the target detection-based port portal crane grab bucket swing amplitude recognition and calculation method, the swing amplitude of the portal crane grab bucket can be calculated in real time, ship anti-collision warning can be provided during portal crane operation, and the safety of ship unloading and ship loading is guaranteed; a target detection algorithm is used as a basic framework, center coordinates of a static point of the grab bucket are calculated through a geometric method, pixel deviation of the real-time grab bucket is calculated according to an identification result, parameters such as height are obtained through communication with a PLC, and the actual swing amplitude is calculated according to a pixel centimeter conversion formula.

Description

Technical Field

[0001] The present invention relates to the technical field of gantry cranes, and particularly to a method and system for identifying and calculating the swing amplitude of a grab of a port gantry crane based on object detection. Background Art

[0002] With the rapid development of artificial intelligence, the application fields of object recognition are becoming more and more extensive. In recent years, the development of unmanned and intelligent terminals has made the application prospect of object recognition in port terminals very broad. In the unmanned operation of quay gantry cranes, the swing amplitude of the grab of the gantry crane is calculated through object recognition, and the safety during ship unloading and loading is ensured according to the swing amplitude.

[0003] To sum up, the method and system for identifying and calculating the swing amplitude of a grab of a port gantry crane based on object detection utilize the application of object detection technology in the operation of the grab of a port gantry crane. By real-time detecting the position of the grab and calculating its swing amplitude, the safety and efficiency in port operations are ensured. Especially in an unmanned operation environment, accurate dynamic monitoring and anti-collision warning functions are provided for the operation of the grab of the port gantry crane. Summary of the Invention

[0004] In view of the problems existing in the above-mentioned prior art, the present invention is proposed.

[0005] Therefore, the technical problem to be solved by the present invention is to determine whether there is a sufficient safety distance between the current operation position of the grab of the gantry crane and the position of the hatch by calculating the swing amplitude of the grab of the gantry crane, to ensure the safety of the gantry crane during unmanned operation, and to provide effective anti-collision warning and operation monitoring functions.

[0006] To solve the above technical problem, the present invention provides the following technical solution. A method for identifying and calculating the swing amplitude of a grab of a port gantry crane based on object detection includes: collecting the image data of the grab of the gantry crane collected by a camera; processing the image data to make a data set; training the data set to obtain an object detection model of the grab of the gantry crane; and performing real-time calculation in combination with the image coordinates of the static initial point to obtain the real-time swing amplitude of the grab of the gantry crane.

[0007] As a preferred solution of the method for identifying and calculating the swing amplitude of a grab of a port gantry crane based on object detection according to the present invention, wherein: the collecting the image data of the grab of the gantry crane collected by a camera includes obtaining the image of the grab of the gantry crane collected by a fixed-focus camera;

[0008] The fixed-focus camera is installed on the jib of the gantry crane.

[0009] As a preferred solution of the method for identifying and calculating the swing amplitude of a grab of a port gantry crane based on object detection according to the present invention, wherein: the data set includes the images of the grab of the gantry crane in various weather conditions and various working environments.

[0010] As a preferred solution of the swing recognition calculation method of the port portal crane grab based on object detection according to the present invention, wherein: the training data set includes dividing the data set into a training set, a validation set, and a test set for training.

[0011] As a preferred solution of the swing recognition calculation method of the port portal crane grab based on object detection according to the present invention, wherein: the portal crane grab object detection model includes re-collecting image data according to the actual situation on site and continuing to train.

[0012] As a preferred solution of the swing recognition calculation method of the port portal crane grab based on object detection according to the present invention, wherein: the real-time calculation by combining the image coordinates of the static initial point includes calculating and correcting the image coordinates of the static initial point of the portal crane grab by collecting the static image data of the grab, expressed as:

[0013] X′0 = X0 + H tanα, Y′0 = Y0 + Htanβ

[0014] Wherein, X'0 represents the X coordinate of the center point of the corrected grab, Y'0 represents the Y coordinate of the center point of the corrected grab, X0 represents the X coordinate of the center point of the grab in the target recognition result at rest, α represents the horizontal deviation angle of the camera, Y0 represents the Y coordinate of the center point of the grab in the target recognition result at rest, β represents the vertical deviation angle of the camera, and H represents the distance from the camera lens to the grab;

[0015] After calculation, the coordinates of the center point of the portal crane grab in the working state are obtained.

[0016] As a preferred solution of the swing recognition calculation method of the port portal crane grab based on object detection according to the present invention, wherein: obtaining the real-time swing of the portal crane grab converts the pixel size of the detected portal crane grab offset in the image into the actual swing size through the pixel density conversion formula;

[0017] Wherein, the pixel size conversion is expressed as:

[0018]

[0019] Wherein, h represents the pixel size of the deviation on the image, f represents the focal length of the lens, and the actual swing size L is calculated

[0020] Another object of the present invention is to provide a swing recognition and calculation system for a port gantry crane grab based on object detection. Through data acquisition, image processing, model training, and real-time calculation modules, it can monitor the swing of the port gantry crane grab in real time; the system collects grab images through a camera and uses an object detection model to process the images and calculate the real-time swing of the grab; after correction in combination with the coordinates of the static initial point, the system can accurately calculate the actual swing, provide anti-collision warnings, and ensure the safety of loading and unloading operations. This system not only adapts to complex weather and working environments, but also supports the unmanned operation of port gantry cranes, improves operation efficiency, reduces manual intervention, has good automation and safety guarantee functions, and is widely applicable to the unmanned operation environment of intelligent ports.

[0021] To solve the above technical problems, the present invention provides the following technical solutions: A swing recognition and calculation system for a port gantry crane grab based on object detection, including: a data acquisition module, a data processing module, a model training module, and a real-time calculation module;

[0022] The data acquisition module collects the image data of the gantry crane grab captured by the camera;

[0023] The data processing module processes the image data and makes it into a data set;

[0024] The model training module trains the data set to obtain an object detection model for the gantry crane grab;

[0025] The real-time calculation module performs real-time calculation in combination with the image coordinates of the static initial point to obtain the real-time swing of the gantry crane grab.

[0026] A computer device includes a memory and a processor. The memory stores a computer program. The processor, when executing the computer program, implements the steps of the above-mentioned swing recognition and calculation method for a port gantry crane grab based on object detection.

[0027] A computer-readable storage medium stores a computer program. The computer program, when executed by a processor, implements the steps of the above-mentioned swing recognition and calculation method for a port gantry crane grab based on object detection.

[0028] The beneficial effects of the present invention: By calculating the swing of the port gantry crane grab in real time, the operation safety is improved, abnormalities can be detected in time and anti-collision warnings can be provided to ensure the safety of the ship unloading and loading processes. At the same time, the use of object detection technology realizes automatic monitoring, supports the unmanned operation of port gantry cranes, reduces manual intervention, and improves operation efficiency. This system can operate stably in complex weather and working environments, ensuring the accuracy of grab position recognition and swing calculation. Through real-time monitoring and accurate calculation, the system can quickly respond to changes in operations, reduce risks and improve the overall operation efficiency. Brief Description of the Drawings

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0030] Figure 1 It is a schematic flow chart of a swing recognition calculation method for a port gantry crane grab based on object detection provided by an embodiment of the present invention.

[0031] Figure 2 It is a schematic diagram of camera angle correction for a swing recognition calculation method for a port gantry crane grab based on object detection provided by an embodiment of the present invention.

[0032] Figure 3 It is a schematic flow chart of a swing recognition calculation method for a port gantry crane based on object detection provided by an embodiment of the present invention. Detailed Embodiments

[0033] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will provide a detailed description of the specific embodiments of the present invention with reference to the accompanying drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0034] Embodiment 1, referring to Figures 1 - 3 , which is an embodiment of the present invention. This embodiment provides a swing recognition calculation method for a port gantry crane grab based on object detection, including:

[0035] S1: Collect the image data of the gantry crane grab captured by the camera.

[0036] It should be noted that, as shown in S1 in Figure 1 , collecting the image data of the gantry crane grab captured by the camera includes obtaining the image of the gantry crane grab captured by the bullet camera, where the bullet camera is installed on the elephant trunk beam of the gantry crane.

[0037] Furthermore, as shown in Figure 2As shown in the figure, the position in the image is corrected. Since the camera lens is installed on the boom nose, which is equivalent to being installed at the farthest end of a lever, during the operation of the gantry crane, due to the large amount of materials and heavy gravity in the grab bucket, plastic deformation occurs, resulting in an offset angle because the lens is not at a vertical angle relative to the grab bucket. Therefore, before calculating the swing amplitude of the gantry crane grab, it is necessary to first identify and calculate the image coordinates of the static initial point, and obtain the offset angle by calculating with the coordinates of the image center point. It should be noted that the offset angle of the camera is decomposed into different angles on the X-axis and Y-axis in the image; the correction formula is expressed as:

[0038]

[0039] Among them, X1 represents the X coordinate of the image center point, X0 represents the X coordinate of the grab center point in the target recognition result at rest, α represents the horizontal deviation angle of the camera, Y1 represents the Y coordinate of the image center point, Y0 represents the Y coordinate of the grab center point in the target recognition result at rest, β represents the vertical deviation angle of the camera, and H represents the distance from the camera lens to the grab bucket.

[0040] S2: Process the image data and make it into a data set.

[0041] It should be noted that, as shown in S2 Figure 1 the data set includes gantry crane grab images in various weather conditions and various working environments.

[0042] Furthermore, as shown in Figure 2 gantry crane grab images are obtained through a camera installed on the boom nose of the gantry crane, including different weather conditions such as day, night, sunny, cloudy, and rainy days. Images of different postures of the gantry crane grab opening and closing are collected, manually labeled, and made into a standard training set.

[0043] S3: Train the data set to obtain a gantry crane grab target detection model.

[0044] It should be noted that, as shown in S3 Figure 1 the training data set includes dividing the data set into a training set, a validation set, and a test set for training.

[0045] Furthermore, the collected image data is divided into a training set, a validation set, and a test set in a ratio of 5:3:2. When training the target detection model, data augmentation processing is performed on the image data to increase the diversity of the data, including geometric transformation methods such as translation, stretching, cropping, and flipping.

[0046] Furthermore, to meet the requirements of real-time detection, a one-stage object detection algorithm is adopted. The core is to convert the object detection problem into a regression problem, and obtain the position of the prediction box and the object category contained in the prediction box from the regression results. After multiple rounds of training, a door grab object detection model is obtained.

[0047] S4: Combine the image coordinates of the static initial point for real-time calculation to obtain the real-time swing amplitude of the door grab.

[0048] It should be noted that, as Figure 1 shown in S4, combining the image coordinates of the static initial point for real-time calculation includes collecting grab static image data and calculating and correcting the image coordinates of the static initial point of the door grab; obtaining the real-time swing amplitude of the door grab by converting the pixel size offset of the detected door grab in the image into the actual swing amplitude size through the pixel density conversion formula.

[0049] Further, as Figure 3 shown, it is the recognition of the door swing amplitude. First, the grab image during the operation of the door collected by the camera is input into the object detection model; to facilitate the use of the object detection model, the ONNX Runtime inference framework is selected for on-site deployment, the object detection model is converted into the ONNX format, and the GPU is used for acceleration. The output initial result is the upper left coordinate and the lower right coordinate of the door grab detection box, and the center point coordinate is obtained through calculation; then, the returned object recognition result is processed by non-maximum suppression to screen out the most suitable object box, and the center point coordinate is calculated as the returned result; the length of the grab suspension rope, that is, the distance from the lens to the grab, is obtained from the door system through PLC communication; finally, the recognized center point coordinate is calculated and corrected, and the pixel deviation is calculated from the corrected coordinate and the static center point coordinate, and the actual swing amplitude value is calculated through the pixel centimeter conversion formula.

[0050] Furthermore, during the operation of the door, the image position of the door grab is detected in real time and the center point coordinate of the grab at the current position is calculated; the recognized door grab coordinates are corrected by the angle α, expressed as:

[0051] X′0 = X0 + H tanα, Y′0 = Y0 + H tanβ

[0052] where, X'0 represents the X coordinate of the center point of the corrected grab, and Y'0 represents the Y coordinate of the center point of the corrected grab;

[0053] The center point coordinate of the door grab under the working state is obtained after calculation;

[0054] The deviation is calculated using the obtained corrected center point coordinate of the grab and the pixel center point coordinate of the static point, expressed as:

[0055] X′0 - X0 = X, Y′0 - Y0 = Y

[0056] Wherein, X represents the pixel size in the x-axis direction, Y represents the pixel size in the y-axis direction. The calculated pixel size of the swing amplitude, after pixel-centimeter conversion, is expressed as:

[0057]

[0058] Wherein, h represents the pixel size of the image upper deviation, f represents the focal length of the lens, and the actual size L of the swing amplitude is calculated.

[0059] The above is a schematic solution of a swing amplitude recognition and calculation method for a port gantry crane grab based on object detection in this embodiment. It should be noted that the technical solution of the system for the swing amplitude recognition and calculation method of the port gantry crane grab based on object detection belongs to the same concept as the technical solution of the above swing amplitude recognition and calculation method for the port gantry crane grab based on object detection. For the details not described in detail in the technical solution of the swing amplitude recognition and calculation system for the port gantry crane grab based on object detection in this embodiment, reference can be made to the description of the technical solution of the swing amplitude recognition and calculation method for the port gantry crane grab based on object detection.

[0060] Embodiment 2 is an embodiment of the present invention, which provides a swing amplitude recognition and calculation system for a port gantry crane grab based on object detection, including: a data acquisition module, a data processing module, a model training module, and a real-time calculation module;

[0061] The data acquisition module collects the image data of the gantry crane grab captured by the camera;

[0062] The data processing module processes the image data and makes it into a data set;

[0063] The model training module trains the data set to obtain a gantry crane grab object detection model;

[0064] The real-time calculation module performs real-time calculation in combination with the image coordinates of the static initial point to obtain the real-time swing amplitude of the gantry crane grab.

[0065] This embodiment also provides a computing device applicable to the case of the swing amplitude recognition and calculation method for a port gantry crane grab based on object detection, including:

[0066] A memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the swing amplitude recognition and calculation method for a port gantry crane grab based on object detection as proposed in the above embodiment.

[0067] This embodiment also provides a storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the swing recognition calculation method of the port gantry grab based on target detection proposed in the above embodiment.

[0068] The storage medium proposed in this embodiment and the swing recognition calculation method of the port gantry grab based on target detection proposed in the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0069] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical disks and other various media that can store program codes.

[0070] Logic and / or steps described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0071] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following techniques known in the art or a combination thereof can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0072] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A swing amplitude recognition and calculation method for grab bucket of port gantry crane based on target detection, characterized in that: include: Collect the image data of the gantry crane grab bucket captured by the camera; Process image data and make it into a data set; The training data set is used to obtain the gantry crane grab target detection model; Combined with the image coordinates of the static initial point, real-time calculation is performed to obtain the real-time swing amplitude of the gantry crane grab.

2. The swing amplitude recognition and calculation method of the grab bucket of a port gantry crane based on target detection according to claim 1 is characterized in that: The collecting of the gantry crane grab bucket image data collected by the camera includes obtaining the gantry crane grab bucket image collected by the gun camera; The gun camera is installed on the trunk of the portal machine.

3. The swing amplitude recognition and calculation method of the grab bucket of a port gantry crane based on target detection as claimed in claim 2 is characterized by: The data set includes gantry crane grab bucket images in various weather conditions and various working environments.

4. The swing amplitude recognition and calculation method of the grab bucket of a port gantry crane based on target detection as claimed in claim 3 is characterized by: The training data set includes dividing the data set into a training set, a validation set and a test set for training.

5. The swing amplitude recognition and calculation method of the grab bucket of a port gantry crane based on target detection as claimed in claim 4 is characterized by: The gantry crane grab bucket target detection model includes re-collecting image data according to actual on-site conditions and continuing training.

6. The swing amplitude recognition and calculation method of the grab bucket of a port gantry crane based on target detection as claimed in claim 5 is characterized by: The real-time calculation in combination with the image coordinates of the static initial point includes collecting the static image data of the grab bucket and calculating and correcting the image coordinates of the static initial point of the gantry crane grab bucket, which is expressed as: X0=X0+Htanα,Y0=Y0+Htanβ Among them, X'0 represents the X coordinate of the center point of the grab bucket after correction, Y'0 represents the Y coordinate of the center point of the grab bucket after correction, X0 represents the X coordinate of the center point of the grab bucket in the target recognition result at rest, α represents the horizontal deflection angle of the camera, Y0 represents the Y coordinate of the center point of the grab bucket in the target recognition result at rest, β represents the vertical deflection angle of the camera, and H represents the distance from the camera lens to the grab bucket; After calculation, the coordinates of the center point of the gantry crane grab bucket in the operating state are obtained.

7. The swing amplitude recognition and calculation method of the grab bucket of a port gantry crane based on target detection as claimed in claim 6 is characterized by: The real-time swing amplitude of the gantry crane grab bucket is obtained by converting the pixel size of the gantry crane grab bucket offset in the image into the actual swing amplitude size through a pixel density conversion formula; Among them, the pixel size conversion is expressed as: Wherein, h represents the pixel size of the deviation on the image, f represents the focal length of the lens, and the actual size L of the swing is calculated.

8. A system for swing amplitude recognition and calculation of grab bucket of port gantry crane based on target detection according to any one of claims 1 to 7, characterized in that: include: Data acquisition module, data processing module, model training module and real-time computing module; The data acquisition module collects the gantry crane grab bucket image data acquired by the camera; The data processing module processes the image data and produces a data set; The model training module trains the data set to obtain a gantry crane grab target detection model; The real-time calculation module performs real-time calculation in combination with the image coordinates of the static initial point to obtain the real-time swing amplitude of the gantry crane grab bucket.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the swing amplitude recognition and calculation method of the grab bucket of a port gantry crane based on target detection as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the swing amplitude recognition and calculation method of the grab bucket of a port gantry crane based on target detection as described in any one of claims 1 to 7 are implemented.

Images