Field bridge personnel intrusion prevention method based on 360-degree panoramic perspective splicing
Through 360° panoramic perspective splicing and YOLO V5 target detection algorithm, the problems of blind spots and false alarms of field bridge monitoring are solved, full coverage of intrusion monitoring is achieved, a complete anti-intrusion system is established, and the production safety environment is optimized.
Patent Information
- Application Number
- CN202411152119.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2025-01-03
AI Technical Summary
The existing bridge intrusion prevention monitoring technology has problems such as blind spots in monitoring and frequent false alarms, making it difficult to achieve full coverage and unified intrusion monitoring functions.
The 360° panoramic perspective splicing method is adopted to collect images through the camera arrangement on the top and legs of the field bridge, and splicing and distortion correction are carried out at the top view angle. Combined with the YOLO V5 object detection algorithm, full coverage monitoring is achieved and false alarms are reduced.
Full coverage monitoring within the scope of the field bridge operation has been achieved, false alarms have been reduced, a unified and complete anti-intrusion mechanism has been established, and the production safety environment has been optimized.
Smart Images
Figure CN119251962A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of field bridge anti-intrusion monitoring, and in particular to a field bridge anti-personnel intrusion method based on 360° panoramic perspective splicing. Background Art
[0002] When the yard crane machinery at the port terminal is moving or operating, if there are people moving close to the bottom of the yard crane, or crossing the gap between the legs of the truck, there will be great safety hazards and personal injury accidents are likely to occur, so it is necessary to prevent such intrusions. To address this issue, the yard crane should be equipped with complete personnel intrusion warning and monitoring functions to prevent such risks. However, due to the complex mechanical structure at the bottom of the yard crane, there are prone to monitoring blind spots, so there are many camera points that need to be deployed and they are scattered, resulting in many monitoring screens and difficulty in unified monitoring. In addition, during the operation process, due to interference from factors such as the paint on the bottom container, the camera often misidentifies people and mispronounces warnings, resulting in frequent shutdowns and affecting operating efficiency. In summary, it is difficult to form an overall unified and complete intrusion monitoring and alarm function system for the yard crane. Summary of the invention
[0003] The purpose of the present invention is to overcome the shortcomings and defects of the prior art and to provide a method for preventing personnel intrusion on a field bridge based on 360° panoramic perspective splicing. The present invention can achieve full coverage monitoring within the operating range of the field bridge.
[0004] A method for preventing personnel intrusion in a field bridge based on 360° panoramic perspective stitching, comprising:
[0005] A first camera arranged on the top of the field bridge is used to collect multiple top-view images at different angles;
[0006] splicing the plurality of top-view images to form a spliced image from a top-view perspective;
[0007] The second camera arranged on the leg of the field bridge is used to collect the target image of the bottom of the field bridge in the blind spot monitored by the first camera;
[0008] Convert the target image of the bottom of the field bridge into a bottom image of the field bridge in a bird's-eye view, and splice it to the corresponding position of the spliced image to form an overall monitoring image that can cover the predetermined area of the field bridge;
[0009] Monitoring is performed based on the overall monitoring image, and an alarm is triggered when human intrusion is detected.
[0010] Among them, when the image is stitched, when the stitched pictures overlap, the overlapping parts are optimized, including optimization by using pixel weighting, brightness adjustment, and color balance methods.
[0011] Among them, the first camera is a top-down camera, and the second camera is a fisheye camera.
[0012] There are four first cameras, which are arranged at the four top corners of the field bridge, and there are sixteen second cameras, one of which is installed at the four corners of each leg of the field bridge.
[0013] The step of converting the bottom image of the field bridge into the bottom image of the field bridge in a top-down perspective includes:
[0014] The internal and external parameter data of each second camera are obtained, and then the distortion matrix A and the overhead perspective matrix B at the current angle of the second camera are used to perform view transformation on the captured image; wherein, firstly, the distortion matrix A is used to convert the spherical image captured by the second camera into a plane projection image to eliminate the line distortion in the image, and then the overhead perspective matrix B is used to convert the dedistorted image into a bird's-eye view, and finally an image of the bottom of the field bridge from a bird's-eye view is formed.
[0015] Wherein, based on the YOLO V5 target detection algorithm, the overall surveillance image is monitored to detect whether there is human intrusion.
[0016] Among them, an electronic monitoring fence is set on the overall monitoring image. Based on the YOLO V5 target detection algorithm, when a person is detected entering the electronic monitoring fence, an alarm is triggered and the field bridge is triggered to stop.
[0017] The electronic monitoring fence is arranged on the outside of the field bridge tires, and surrounds the field bridge tires on three sides of the tires outside the field bridge.
[0018] Wherein, when monitoring based on the overall monitoring image, monitoring is performed based on the YOLO V5 target detection algorithm, and dual discrimination technology is used for identification and alarm, including:
[0019] When the second camera detects a person and the confidence level is greater than the upper limit of the confidence threshold, an alarm is directly triggered;
[0020] When the second camera detects a person, if the confidence level is greater than the lower limit of the confidence threshold but less than the upper limit of the confidence threshold and the overall monitoring image also detects a person, an alarm is triggered; otherwise, no alarm is triggered.
[0021] The overall monitoring image is combined with the yard crane positioning data and the global map of the terminal to form a real-time alarm map for yard crane personnel intrusion. When a personnel intrusion is detected and an alarm is issued, the yard crane number, location and personnel location of the intrusion are displayed at the same time.
[0022] The present invention splices the monitoring images of various cameras at predetermined positions on the yard crane to achieve unified personnel intrusion monitoring, and reduces false alarms through multi-level intrusion monitoring identification and verification. It monitors personnel intrusion behaviors from three levels: the overall level of the terminal, the overall level of the yard crane, and the local level of the yard crane to achieve a complete anti-intrusion mechanism. While taking into account the production efficiency of the yard crane, a unified and complete anti-intrusion system is established to eliminate risks and hidden dangers during yard crane operations and optimize the production safety environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a flow chart of the field bridge anti-personnel intrusion method based on 360° panoramic perspective splicing of the present invention.
[0024] Figure 2 It is a schematic diagram of camera arrangement on the field bridge of the present invention.
[0025] Figure 3 It is a schematic diagram of the overall monitoring image of the field bridge formed by the method of the present invention. DETAILED DESCRIPTION
[0026] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0027] See also Figure 1 As shown, a method for preventing personnel intrusion of a field bridge based on 360° panoramic perspective stitching of the present invention includes the following implementation steps:
[0028] S1. Collect multiple top-view images at different angles by arranging a first camera on the top of the field bridge;
[0029] S2. stitching the plurality of top-view images to form a stitched image from a top-view perspective;
[0030] S3. The second camera arranged at the leg of the field bridge is used to collect the target image of the bottom of the field bridge in the blind spot monitored by the first camera;
[0031] S4. Convert the target image of the bottom of the field bridge into a bottom image of the field bridge from a top-down perspective, and splice it to the corresponding position of the spliced image to form an overall monitoring image that can cover the predetermined area of the field bridge location;
[0032] S5. Perform monitoring based on the overall monitoring image and generate an alarm when human intrusion is detected.
[0033] The first camera is used for primary monitoring, monitoring the intrusion of personnel at the bottom of the field bridge, and the second camera is used for auxiliary monitoring, supplementing the monitoring blind area of the first camera, and finally achieving full coverage of monitoring without blind spots. Moreover, by splicing the images of the primary monitoring to form the main spliced image, and then splicing the images of the auxiliary monitoring to the main spliced image, a perspective area is formed, and finally a picture of the overall personnel intrusion monitoring of the field bridge is formed from a bird's-eye view, which can realize the overall comprehensive monitoring of the personnel of the field bridge, including the monitoring around the tires of the field bridge trucks.
[0034] The present invention directly splices the images captured by the first camera 1 on the top of the field bridge 10 large vehicles and puts them in the corresponding position of the spliced images to initially form the field bridge overall personnel intrusion monitoring image 3 ( Figure 3 The spliced image is a bird's-eye view, and then the second camera 2 arranged on the leg 11 of the bridge is used to collect the bottom image of the bridge in the blind spot monitored by the first camera; the bottom image of the bridge is converted into a bottom image of the bridge under a bird's-eye view, and is spliced to the corresponding position of the spliced image to obtain a supplementary monitoring image of the blind spot monitored by the first camera 1, that is, the bottom monitoring image 4 of the bridge ( Figure 3 Finally, an overall monitoring image covering the predetermined area of the field bridge location can be formed. The spliced image, the field bridge bottom image and the overall monitoring image are used for monitoring. When human intrusion is detected, an alarm is triggered. This can achieve all-round intrusion monitoring of people, animals or targets outside the predetermined range of the field bridge, eliminate risks during field bridge operation, and optimize the production safety environment.
[0035] When stitching images, if there is overlap in the stitched images, the overlapping parts are optimized, including pixel weighting, brightness adjustment, and color balance.
[0036] In some embodiments, the first camera is a bird's-eye view camera, and the second camera is a fisheye camera. More preferably, there are four first cameras, which are arranged at the four top corners of the field bridge, so as to provide a large-scale, overall bird's-eye view. There are sixteen second cameras, one of which is installed at each of the four corners of each leg of the field bridge. Figure 2 As shown, ensure that all blind spots at the bottom of the field bridge where people may pass by are covered.
[0037] In some embodiments, the bottom image of the field bridge is converted into the bottom image of the field bridge from a bird's-eye view by using a dynamic affine transformation method to perform distortion correction and bird's-eye projection on the image captured by the fisheye camera. First, the distortion matrix A is used to convert the spherical image captured by the second camera into a plane projection image to eliminate line distortion in the image. Then, the bird's-eye perspective matrix B is used to convert the dedistorted image into a bird's-eye view, thereby finally forming the bottom image of the field bridge from a bird's-eye view.
[0038] Specifically, during processing, the checkerboard calibration algorithm is first used to solve the distortion matrix A of the fisheye spherical image obtained by the second camera to the plane image, and then according to the current angle of the second camera, the overhead perspective matrix B is calibrated again to convert the image to a top-down perspective; the distortion correction matrix A is multiplied by the top-down perspective matrix B to obtain a conversion matrix from the fisheye spherical image obtained by the second camera to the top-down image, and the fisheye spherical image can be converted to a top-down image through the conversion matrix.
[0039] The following takes one of the fisheye cameras as an example to explain the solution method of its distortion matrix A and bird's-eye perspective matrix B. The distortion matrix is obtained by using the chessboard calibration method, and the chessboard calibration tool is used.
[0040] The distortion matrix A is obtained:
[0041] The fisheye camera is used to shoot a checkerboard calibration object, a sample image is captured for the checkerboard calibration object, and the checkerboard coordinates in the sample image are calibrated one by one. For the calibrated sample image, the fisheye camera internal parameters and distortion coefficients are calculated using the calibrate function in the fisheye module of opencv, and then the distortion mapping matrix A is obtained using the initUndistortRectifyMap function.
[0042] The top-down perspective matrix B is obtained:
[0043] Place the checkerboard calibration object horizontally on the ground, and let the fisheye camera take an image M of the calibration object, and calibrate the checkerboard coordinate matrix in it as P M After the fisheye camera is distorted, the resulting image is T, and the checkerboard coordinate matrix is P T =AP M To stitch the top-down view, the image T needs to be converted into a top-down (bird’s-eye) view image F. The top view F of the checkerboard calibration object is captured by a planar camera, and its checkerboard coordinate matrix is P F . By P T Mapping to P F The relationship P F =BP T , the bird’s-eye view perspective matrix B can be solved.
[0044] According to the formula F=ABM, each frame captured by the fisheye camera can be converted into a bird's-eye view to form a surveillance video with a bird's-eye view.
[0045] Among them, when the converted bottom image of the field bridge is stitched, the converted overhead shots of each second camera are stitched. When stitching, each projection screen cuts off a portion of the outermost circle. When there are overlapping parts in the stitched screen, the overlapping parts are optimized, including pixel weighting, brightness adjustment, and color balance. Since the edge of the fisheye camera still has uneliminated distortion after being converted to a bird's-eye view, when conducting intrusion monitoring over a large range, the top corner of the field bridge is preferred. In the blind spot of the top corner camera, the top-view projection of the bottom fisheye camera is used to supplement it.
[0046] In some embodiments, the overall surveillance image is monitored based on the YOLO V5 target detection algorithm to detect whether there is a human intrusion. Other target detection algorithms can also be used to detect on the image, and the use of this algorithm is not limited.
[0047] In some embodiments, an electronic monitoring fence 5 is provided on the overall monitoring image, such as Figure 3 As shown, there are two electronic monitoring fences, located on the outside of the tires 13 on both sides below the main beam 12 of the field bridge. Based on the YOLO V5 target detection algorithm, when a person enters the electronic monitoring fence, an alarm is triggered and the field bridge is stopped. Once a person is identified to enter the fence, it is regarded as a human intrusion. An alarm is immediately triggered and the field bridge is stopped synchronously to prevent human intrusion and prevent injury from the walking mechanism of the field bridge vehicle. The position of the electronic monitoring fence and the distance from the field bridge can be set as needed to ensure safety needs. More preferably, the electronic monitoring fence is arranged on the outside of the field bridge tires, such as about 5m, and the field bridge tires are enclosed on the three sides of the tires outside the field bridge.
[0048] In some embodiments, in order to reduce false alarms or false triggering of alarms, when monitoring is performed based on the overall surveillance image, monitoring is performed based on the YOLO V5 target detection algorithm, and a double discrimination technology is used for identification and alarm, including: when the second camera detects a person and the confidence is greater than the upper limit of the confidence threshold, an alarm is directly triggered; when the second camera detects a person, if the confidence is greater than the lower limit of the confidence threshold but less than the upper limit of the confidence threshold and the overall surveillance image simultaneously monitors the person, an alarm is triggered; otherwise, no alarm is triggered. Through the above double discrimination, it is possible to avoid the situation where the second camera may mistakenly detect a person and trigger an alarm, such as the second camera mistakenly detecting a container as a person.
[0049] The above confidence threshold can be set to a range, such as a confidence threshold lower limit and upper limit, such as an upper limit of 0.7 and a lower limit of 0.5. Through double detection and judgment, the detected result can be guaranteed to be more accurate, and false detection can be prevented, and false alarms can be prevented from causing shutdowns.
[0050] In some embodiments, the overall monitoring image is further combined with the field bridge positioning data and the global map of the terminal to form a real-time alarm map for field bridge personnel intrusion. When a personnel intrusion is detected and an alarm is triggered, the field bridge number, location and personnel location of the intruder are displayed at the same time, thereby constructing a three-layer early warning and anti-intrusion system for the terminal as a whole, the field bridge as a whole, and the field bridge locally. In this way, after the corresponding field bridge triggers an alarm, the location of the field bridge can be immediately determined through the map, and intuitive display and positioning can be achieved on the display platform.
[0051] From the above description, it can be seen that the method for preventing personnel intrusion on the yard crane of the embodiment of the present invention splices the monitoring images of each camera at the predetermined position on the yard crane to realize unified personnel intrusion monitoring, and reduces false alarms through multi-level intrusion monitoring identification and verification, monitors personnel intrusion behaviors from three levels: the overall level of the terminal, the overall level of the yard crane, and the local level of the yard crane, to realize a complete anti-intrusion mechanism, establish a unified and complete anti-personnel intrusion system while taking into account the production efficiency of the yard crane, eliminate the risks and hidden dangers during the operation of the yard crane, and optimize the production safety environment.
[0052] The basic principles and main features of the present invention and the advantages of the present invention are shown and described above. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.
[0053] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the claims are included in the present invention.
[0054] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A method for preventing personnel intrusion in a field bridge based on 360° panoramic perspective stitching, characterized in that: include: A first camera arranged on the top of the field bridge is used to collect multiple top-view images at different angles; splicing the plurality of top-view images to form a spliced image from a top-view perspective; The second camera arranged on the leg of the field bridge is used to collect the target image of the bottom of the field bridge in the blind spot monitored by the first camera; Convert the target image of the bottom of the field bridge into a bottom image of the field bridge in a bird's-eye view, and splice it to the corresponding position of the spliced image to form an overall monitoring image that can cover the predetermined area of the field bridge; Monitoring is performed based on the overall monitoring image, and an alarm is triggered when human intrusion is detected.
2. The method for preventing personnel from intrusion into a field bridge according to claim 1, characterized in that: When image stitching is performed, when there is overlap in the stitching images, the overlapping parts are optimized, including optimization by using pixel weighting, brightness adjustment, and color balance methods.
3. The method for preventing personnel from intrusion into a field bridge according to claim 1, characterized in that: The first camera is a top-down camera, and the second camera is a fisheye camera.
4. The method for preventing personnel from intrusion into a field bridge according to claim 3, characterized in that: There are four first cameras, which are arranged at the four top corners of the field bridge, and there are sixteen second cameras, one of which is installed at the four corners of each leg of the field bridge.
5. The method for preventing personnel from intrusion into a field bridge according to claim 3 is characterized in that: The converting the bottom image of the field bridge into the bottom image of the field bridge in a top-down perspective comprises: The internal and external parameter data of each second camera are obtained, and then the distortion matrix A and the overhead perspective matrix B at the current angle of the second camera are used to perform view transformation on the captured image; wherein, firstly, the distortion matrix A is used to convert the spherical image captured by the second camera into a plane projection image to eliminate the line distortion in the image, and then the overhead perspective matrix B is used to convert the dedistorted image into a bird's-eye view, and finally an image of the bottom of the field bridge from a bird's-eye view is formed.
6. The method for preventing personnel from intrusion into a field bridge according to claim 1, characterized in that: Based on the YOLO V5 target detection algorithm, the overall surveillance image is monitored to detect whether there is human intrusion.
7. The method for preventing personnel from intrusion into a field bridge according to claim 1, characterized in that: An electronic monitoring fence is set on the overall monitoring image. Based on the YOLO V5 target detection algorithm, when a person is detected entering the electronic monitoring fence, an alarm is triggered and the field bridge is triggered to stop.
8. The method for preventing personnel from intrusion into a field bridge according to claim 1, characterized in that: The electronic monitoring fence is arranged on the outside of the field bridge tire, and surrounds the field bridge tire inside on three sides of the tire outside the field bridge.
9. The method for preventing personnel from intrusion into a field bridge according to claim 1, characterized in that: When monitoring based on the overall monitoring image, monitoring is performed based on the YOLO V5 target detection algorithm, and dual discrimination technology is used for identification and alarm, including: When the second camera detects a person and the confidence level is greater than the upper limit of the confidence threshold, an alarm is directly triggered; When the second camera detects a person, if the confidence level is greater than the lower limit of the confidence threshold but less than the upper limit of the confidence threshold and the overall monitoring image also detects a person, an alarm is triggered; otherwise, no alarm is triggered.
10. The method for preventing personnel intrusion in a field bridge according to claim 1, characterized in that: The overall monitoring image is combined with the yard crane positioning data and the global map of the terminal to form a real-time alarm map for yard crane personnel intrusion. When a personnel intrusion is detected and an alarm is issued, the yard crane number, location and personnel location of the intrusion are displayed at the same time.