Shipbuilding gantry crane cart anti-collision protection system and method based on video recognition

Abstract

The invention discloses a shipbuilding gantry crane cart anti-collision protection system and method based on video recognition, and belongs to the field of industrial automation and safety control. The camera transmits an encoded video stream to the algorithm server through an RTSP protocol, and the FFmpeg pulls the video stream and carries out hard decoding to obtain data as algorithm input; the CUDA kernel function completes preprocessing of video frames; a CUDA parallelization processing algorithm and a TensorRT loading INT8 quantitative model are used for executing rapid reasoning; an alarm area is selected on a video picture through a configuration file, and once foreign matters appear in the area, an algorithm sends alarm information to a PLC through the Ethernet within 200 ms; the PLC judges whether collision danger occurs or not according to the direction of the cart, and the corresponding area is prompted through the voice broadcaster. According to the system, precise recognition and millisecond-level active braking of small obstacles are achieved.

Description

TECHNICAL FIELD

[0001] The application relates to a shipbuilding gantry crane collision protection system and method based on video recognition. BACKGROUND

[0002] In the modern shipbuilding industry, the 800-ton large gantry crane as the core equipment of the dock, the safety protection of the trolley running track faces severe challenges. The dock environment has the typical characteristics of metal concentration, and dynamic obstacles such as personnel, transport vehicles and metal components often appear in the track area. The structural difference between the rigid leg and the flexible leg of the gantry crane leads to a significant visual blind area in the driver's room. According to industry accident statistics (such as the public case of China Shipbuilding Ninth Institute), more than 75% of the gantry crane collision accidents are caused by the failure of track foreign object identification, with an annual direct economic loss of hundreds of thousands of yuan. The traditional anti-collision scheme has fundamental limitations: millimeter / microwave radar generates a large number of false alarms due to multipath reflection in a metal-intensive scene, and has weak scattering signal capture ability for small metal targets (such as wrenches and bolts); the millimeter / microwave radar has a high multi-path reflection false alarm rate of 15%-30% under the interference of metal debris; the laser scanning device is suitable for lifting workpiece anti-collision, and the laser scanning device is significantly affected by rain and fog weather, with a rain and fog weather failure rate of more than 40%; it can only detect vertical plane obstacles and cannot detect planar obstacles, such as cannot identify track plane vehicles, personnel and other targets. The space modeling technology can realize multi-device cooperative collision avoidance, but it depends on high-precision communication network and cannot respond to non-device dynamic targets and has single-point failure risk.

[0003] The OpenCV framework only supports basic video file decoding, has weak multi-protocol processing capability, no hard decoding support, and low video preprocessing efficiency; the library UI function is limited and only suitable for debugging and simple display, making industrial-level UI design difficult; the ONNXRuntime / TFLite framework supports GPU, but mainly faces CPU, and has weak extension capability for specific hardware (such as embedded devices), wherein the GPU acceleration of the ONNX framework depends on DirectML or OpenVINO, and the performance fluctuates greatly; other frameworks mostly rely on CPU for pre-processing and post-processing, and the data transmission delay between GPU and CPU is high, and the plug-in support is limited. The industry urgently needs an anti-collision technology that can stably run in a metal-intensive environment, accurately identify small obstacles and achieve millisecond-level active braking. SUMMARY

[0004] To solve the problems in the prior art, the application provides a shipbuilding gantry crane anti-collision protection system and method based on video recognition, which improves the identification accuracy of occluded targets and small targets through an Anchor-free, Decoupled-head and Task-Aligned Assigner dynamic matching strategy, and realizes millisecond active braking through FFmpeg hard decoding of an RTSP video stream, TensorRT INT8 quantization inference and CUDA parallel post-processing.

[0005] The technical scheme adopted by the application is as follows: a shipbuilding gantry crane anti-collision protection system based on video recognition, comprising: A video acquisition module composed of four 2 million pixel true color cameras is installed on both sides of the gantry leg and the flexible leg, covers four protection areas of the track, and is used for acquiring track and surrounding environment video data and transmitting the data through an RTSP protocol; An algorithm processing module includes a video decoding unit, a preprocessing unit, an inference unit and a post-processing unit, the video decoding unit realizes RTSP video stream hard decoding through FFmpeg, the preprocessing unit completes LetterBox size conversion, decompression and color conversion through a CUDA kernel function, the inference unit performs target detection based on a TensorRT framework loaded with an INT8 quantization model, and the post-processing unit processes non-maximum suppression NMS through CUDA parallelization; A control module includes a PLC controller and a voice broadcaster, the PLC controller is used for receiving alarm information of the algorithm processing module, judging collision risks according to the running direction of the crane and dynamically activating corresponding warning areas, and the voice broadcaster is used for issuing risk prompt voice; An interaction module is built based on a Qt framework, supports alarm area configuration, distance threshold on-site adjustment and industrial grade UI display, and is suitable for multi-resolution and cross-platform deployment.

[0006] Further, the detection model of the algorithm processing module adopts a multi-level feature fusion visual algorithm, and the algorithm framework includes: A Backbone network adopts a structure with cross-layer connection, directly connects a lower-level feature map to a higher-level feature map, and retains spatial information; A detection head adopts an Anchor-free and Decoupled-head design, does not need to preset an anchor frame, and directly regresses a target center point and a width and height; A loss function adopts a combined loss function of classification BCE, regression CIOU+VFL; A box matching strategy adopts a Task-Aligned Assigner dynamic matching mode; Training optimization: turn off Mosaic operation, set the total number of epochs to 500.

[0007] Further, the end-to-end processing delay of the algorithm processing module is ≤200 ms, supporting seamless deployment of desktop platforms and embedded devices, and the video decoding unit is compatible with H.264 / H.265 encoding formats.

[0008] A working method of a shipbuilding gantry crane collision protection system based on video recognition, comprising the following steps: S1: video acquisition, through 4 cameras installed on the rigid and flexible legs of the gantry crane, real-time acquisition of track area video data and transmission to the algorithm server through the RTSP protocol; S2: video decoding and preprocessing, using FFmpeg hard decoding video stream to obtain Mat format data, completing LetterBox size conversion, decompression and color conversion through CUDA kernel function, and simultaneously converting the image storage format from [n, h, w, c] to [n, c, w, h]; S3: target detection inference, loading the INT8 quantization model through TensorRT, and executing fast inference on the preprocessed video frames to identify obstacles in the track area; S4: post-processing and risk judgment, using CUDA parallelization to execute non-maximum suppression NMS and filter redundant detection boxes, and if it is determined that the obstacle is located in the preset alarm area, generating an alarm information; S5: dynamic warning and control, the algorithm server sends the alarm information to the PLC within 200 ms, the PLC judges the collision risk according to the traveling direction of the crane, activates the corresponding area alarm logic, and issues a prompt voice through the voice broadcaster.

[0009] Further, the target detection inference in step S3 includes identifying metal small targets and occluded targets.

[0010] Further, the preset alarm area in step S4 is delimited through the configuration file of the system interaction module, and the area range can be adjusted according to the work scene; Further, the PLC in step S5 supports on-site adjustment of distance threshold values, including deceleration distance threshold value and stop distance threshold value.

[0011] Further, the whole process of steps S2 to S4 adopts GPU acceleration processing, including FFmpeg hard decoding, CUDA parallelized preprocessing and post-processing, and TensorRT low-precision inference, to ensure that the real-time performance meets the high-speed running requirements of the gantry crane.

[0012] The beneficial effects of the present application are: in view of the high false alarm rate and high missed alarm rate of the traditional radar scheme in the metal environment, the present application proposes to use the Backbone structure of Shortcut to enhance the small target feature reservation capability, combine the Anchor-free decoupled detection head and the Task-Aligned Assigner dynamic matching strategy, and improve the multi-level feature fusion visual algorithm of the occluded target and small target recognition accuracy; through FFmpeg hard decoding RTSP video stream and TensorRT INT8 quantization inference and CUDA parallel post-processing (including LetterBox size transformation and non-maximum suppression NMS), the end-to-end delay of the whole process GPU acceleration architecture is realized ≤200ms; the PLC activates the warning area alarm according to the walking direction of the large car, and supports the distance threshold adjustable warning system. Finally, the recognition and warning of 800t shipbuilding portal crane and various obstacles are realized, and the autonomous safety is improved.

[0013] In the present application, anchor-free does not need to preset anchor, only needs to regress the center point and width and height of different size feature maps, reduces the time consumption and computing power. At the same time, anchor-free can also avoid some missed detection or repeated detection problems caused by unreasonable anchor setting. Shortcut can directly connect the feature maps of lower level with the feature maps of higher level, so as to retain more spatial information and strengthen the small target detection capability. The combination of CIOU Loss+DFL (Distribution Focal Loss) can effectively process the boundary box regression problem and improve the detection accuracy. The anchor-free Decoupled-head and Task-Aligned Assigner dynamic matching strategy improve the recognition accuracy of the occluded target and small target.

[0014] The system can quickly respond, realize millisecond-level real-time, and fully meet the needs of various high-performance scenarios. Relying on the professional UI and OpenGL provided by Qt, the system has strong expressiveness in industrial interface design, and provides comprehensive and efficient solutions for industrial applications. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0016] Figure 1A flowchart of a shipbuilding gantry crane anti-collision protection system based on video recognition.

[0017] Figure 2 An algorithm flowchart of a shipbuilding gantry crane anti-collision protection system based on video recognition.

[0018] Figure 3 An algorithm deployment flowchart of a shipbuilding gantry crane anti-collision protection system based on video recognition. DETAILED DESCRIPTION

[0019] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0020] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The description of the at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0021] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form, unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0022] Unless specifically stated otherwise, the relative arrangement of components and steps, numerical expressions, and numerical values set forth in the various embodiments described herein are not limiting. It should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportions. The techniques, methods and devices known to those skilled in the relevant art can not be discussed in detail, but should be considered as part of the authorized specification. In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary, and not as a limitation. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0023] In the description of the present application, it should be understood that the orientation words such as "front, back, up, down, left, right", "transverse, vertical, perpendicular, horizontal" and "top, bottom" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and in the absence of contrary description, these orientation words do not indicate and imply that the devices or elements referred to must have a particular orientation or be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the scope of protection of the present application: the orientation words "inner, outer" refer to the inner and outer relative to the contour of each component itself.

[0024] For the convenience of description, spatial relative terms such as "over", "above", "upper surface", "upper" and the like can be used herein to describe the spatial positional relationship of one device or feature with other devices or features as shown in the drawings. It should be understood that the spatial relative terms are intended to include different orientations in use or operation in addition to the orientation of the devices described in the drawings. For example, if the devices in the drawings are inverted, the device described as "above" or "over" other devices or structures will be positioned "below" or "under" the other devices or structures. Thus, the exemplary term "above" can include both "above" and "below" orientations. The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein are interpreted accordingly.

[0025] In addition, it should be noted that the use of the words "first", "second" and the like to define parts only facilitates the differentiation of the corresponding parts, and the above words have no special meaning unless otherwise stated, and therefore cannot be understood as limiting the scope of protection of the present application.

[0026] Figure 1 A shipbuilding gantry crane trolley anti-collision protection system based on video recognition is shown, which is composed of four 200 million pixel true color cameras (installed on the rigid and flexible legs of the gantry crane on both sides, covering four protection areas of the track), an algorithm server and a PLC controller.

[0027] The camera transmits encoded video stream to the algorithm server through the RTSP protocol; FFmpeg pulls the video stream and hard decodes, and the decoded data is input to the algorithm; CUDA kernel function completes the preprocessing of video frames, such as scaling (LetterBox keeps aspect ratio), decompression and color conversion; CUDA parallel processing algorithm (non-maximum suppression NMS) and TensorRT load INT8 quantization model to perform fast inference; through the configuration file, the alarm area is selected on the video screen, and once foreign matter appears in the area, the algorithm sends the alarm information to the PLC through Ethernet within 200ms, the PLC judges whether a collision danger will occur through the walking of the car, and activates the corresponding area alarm logic, and finally prompts through the voice broadcaster.

[0028] FFmpeg efficiently supports multi-threading, built-in H.264 / H.265 hard decoding, and can process multiple protocol video streams (RTSP, etc.). CUDA accelerates video frame preprocessing, such as scaling, decompression and color conversion, which is suitable for high frame rate scenarios. Qt provides rich UI controls, supports hardware accelerated rendering (such as OpenGL), and is suitable for complex industrial interface design. It also supports cross-platform and multi-resolution adaptation. TensorRT is optimized for NVIDIA GPU, supporting FP16 / INT8 low-precision inference, greatly improving inference speed.

[0029] Through full-process GPU processing, CUDA and TensorRT are used to achieve extremely low inference delay and high throughput, meeting high-performance requirements. Combined with hard decoding, multi-threading and GPU acceleration, the system can quickly respond and achieve millisecond-level real-time performance. Flexible protocol, plug-in and model expansion capabilities ensure that it adapts to complex and variable industrial environment requirements, while supporting seamless deployment on desktop and embedded platforms (such as Jetson series), ensuring deployment flexibility. Relying on the professional UI development capabilities provided by Qt and OpenGL hardware accelerated rendering, the system has strong expressive power in industrial interface design, providing comprehensive and efficient solutions for industrial applications.

[0030] Figure 2 The algorithm module of the system is shown, which includes Backbone (main trunk), head (detection head), frame matching strategy module and data enhancement module; Backbone (main trunk): This module combines rich gradient flow information, connects lower-level feature maps with higher-level feature maps, and retains more spatial information, improving feature extraction capability.

[0031] head (detection head): The design of anchor-free + Decoupled-head enables the model to better adapt to objects of different sizes and shapes, improving detection accuracy.

[0032] Loss function: The combined loss function of classification BCE and regression CIOU + VFL is adopted. The introduction of VFL (Vision Transformer Loss) improves the model's ability to capture context information.

[0033] Box matching strategy: The box matching strategy is changed from static matching to Task-Aligned Assigner matching, which enables the model to more accurately predict object positions.

[0034] Data augmentation: During training, the Mosaic operation is turned off, and the total number of epochs is increased from 300 to 500, further improving the model's generalization ability.

[0035] Figure 3 The algorithm deployment scheme of the system, the running logic of the algorithm deployment process is: first, through the C++ / Qt front-end framework program to interface the camera device, call ffmpeg to pull and decode the video stream of the camera, and at the same time monitor the stream state in real time during the pulling process, if the stream is interrupted, it will automatically pull again; the decoded video stream is parallelized frame preprocessing through cuda / c++, then the tensorrt framework is used to complete model inference on GPU, and the detection result is post-processed through cuda / c++ to determine whether there is an obstacle; when it is determined that there is an obstacle, the system will simultaneously mark on the screen and trigger the voice broadcast device to issue warning information.

[0036] The algorithm needs to consider the throughput and delay generated during inference in the deployment process.

[0037] The anti-collision system needs to realize real-time inference of multiple cameras, so the first problem to be solved is the video stream delay. Therefore, this project uses Ffmpeg to pull the Rtsp stream of the camera, decodes the Mat data supported by opencv, and then inputs it into the algorithm.

[0038] The algorithm inference uses Nvidia's tensorrt framework, which uses cuda cores to accelerate the algorithm for inference. In order to simplify the model and speed up the inference, the onnx model middleware is optimized, and in the onnx2engine process, it is quantized from FP32 to INT8, reducing the size of the model.

[0039] The algorithm uses cuda / c++ compiled cuda kernel function to accelerate in pre-processing and post-processing. In order to avoid the image scale adjustment in this process, letter_box is used to modify the image size, and the storage of the image is converted from [n, h, w, c] to [n, c, w, h]. In order to reduce the time of nms, the multi-dimensional data is decoded by using the cuda kernel function in post-processing, and finally the detection data is obtained.

[0040] Single-stage target detection algorithm completes the target detection task in a single forward propagation without generating candidate regions. They directly predict the class and position of the target through dense grids or anchor boxes. (No need to generate candidate regions, and then regress prediction). Single-stage algorithm usually has faster inference speed, and is suitable for real-time application or scene with high speed requirement. The advantage of anchor free is that it does not need to preset anchor, but only needs to regress the target center point and width and height of different scale feature maps, reducing time-consuming and computing power. At the same time, the anchor free method can also avoid some missed detection or repeated detection problems caused by unreasonable anchor setting. The connection mode of shortcut can directly connect the lower level feature map to the higher level feature map, so as to effectively retain more spatial information. This algorithm pays more attention to the retention of spatial information, so it has better performance when dealing with smaller targets.

[0041] The combination of loss function CIOU Loss+DFL (Distribution Focal Loss) can more effectively handle the boundary box regression problem and improve the detection accuracy. The ask Aligned Assigner technology improves the detection ability of small targets and occluded targets by optimizing the task alignment assignment strategy.

[0042] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A shipbuilding gantry anti-collision protection system based on video recognition, characterized in that, Comprise: Video acquisition module: composed of 4 200 million pixel true color cameras, installed on both sides of the gantry crane rigid leg and flexible leg, covering 4 protection areas of the track, used for collecting track and surrounding environment video data and transmitting through RTSP protocol; Algorithm processing module: including video decoding unit, preprocessing unit, inference unit and post-processing unit, the video decoding unit uses FFmpeg to realize RTSP video stream hard decoding, the preprocessing unit completes LetterBox size transformation, decompression and color conversion through CUDA kernel function, the inference unit loads INT8 quantization model based on TensorRT framework to perform target detection, the post-processing unit executes non-maximum suppression NMS through CUDA parallel processing; Control module: including PLC controller and voice announcer, the PLC controller is used for receiving alarm information of the algorithm processing module, judging collision risk according to the direction of the cart and dynamically activating the corresponding warning area, the voice announcer is used to issue risk prompt voice; Interaction module: based on Qt framework, supporting alarm area configuration, distance threshold on-site adjustment and industrial grade UI display, adapting to multi-resolution and cross-platform deployment.

2. The shipbuilding gantry anti-collision protection system based on video recognition of claim 1, wherein, The detection model of the algorithm processing module adopts a multi-level feature fusion visual algorithm, including: Backbone network: adopts a structure with cross-layer connection, directly connects the lower-level feature map to the higher-level feature map, and retains spatial information; Detection head: adopts Anchor-free and Decoupled-head design, without preset anchor box, directly regresses target center point and width-height; Loss function: adopts combined loss function of classification BCE, regression CIOU+VFL; Frame matching strategy: adopts Task-Aligned Assigner dynamic matching method; Training optimization: close the Mosaic operation, and set the total training epoch number to 500.

3. The shipbuilding gantry anti-collision protection system based on video recognition of claim 1, wherein, The end-to-end processing delay of the algorithm processing module is ≤200ms, supporting seamless deployment of desktop platform and embedded device, and the video decoding unit is compatible with H.264 / H.265 encoding format.

4. The working method of a shipbuilding gantry crane anti-collision protection system based on video recognition according to claim 1, characterized in that, Comprise the following steps: S1: video acquisition, through 4 cameras installed on both sides of the rigid and flexible legs of the gantry crane, real-time acquisition of track area video data and transmission to the algorithm server through RTSP protocol; S2: video decoding and preprocessing, using FFmpeg to hard decode video stream to get Mat format data, completing LetterBox size transformation, decompression and color conversion through CUDA kernel function, and converting the image storage format from [n, h, w, c] to [n, c, w, h]; S3: target detection inference, loading INT8 quantization model through TensorRT, performing fast inference on the preprocessed video frame, and identifying obstacles in the track area; S4: post-processing and risk judgment, using CUDA to execute non-maximum suppression NMS in parallel, filtering redundant detection frames, and generating alarm information if the obstacle is located in the preset alarm area; S5: Dynamic early warning and control. The algorithm server sends alarm information to the PLC within 200ms. The PLC judges the collision risk based on the direction of travel of the vehicle, activates the alarm logic of the corresponding area, and issues a prompt voice through the voice broadcaster.

5. The method of claim 4, wherein, The target detection reasoning in step S3 includes the identification of small metallic targets and occluded targets.

6. The method of claim 4, wherein, In step S4, the preset alarm area is defined through the configuration file of the system interaction module, and the area range can be adjusted according to the operation scenario.

7. The method of claim 4, wherein, The PLC mentioned in step S5 supports on-site adjustment of distance thresholds, including deceleration distance threshold and stopping distance threshold.

8. The method of claim 4, wherein, The entire process from step S2 to step S4 is accelerated by GPU, including FFmpeg hardware decoding, CUDA parallel preprocessing and postprocessing, and TensorRT low-precision inference, to ensure that the real-time performance meets the high-speed operation requirements of the gantry crane.

Images