An intelligent monitoring method for cross operation in a refinery plant area

Abstract

The present application provides a kind of intelligent monitoring method for cross operation in refinery device area, comprising the following steps: respectively in the horizontal and vertical direction of cross operation area, fixedly erect binocular camera, shoot cross operation area picture, real-time acquisition two binocular camera RTSP video stream is carried out intelligent analysis;Respectively to the picture of two binocular cameras is shot camera calibration, binocular correction and stereo matching, then the picture of two binocular cameras is shot intelligent analysis determination;If two binocular camera shooting picture is judged as suspected lower cross operation area station person, then lower cross operation area station person event alarm is carried out, two binocular cameras respectively output two shooting images, control alarm device carries out sound and light and voice reminder.The present application carries out alarm to abnormal situation, informs management personnel and operating personnel to dispose in time to illegal behavior, prevent cross operation process and appear object strike, mechanical injury etc. personnel casualty accident appears.

Description

Technical Field

[0001] This invention relates to the field of cross-operation monitoring technology, specifically to an intelligent monitoring method for cross-operations in a refining and chemical plant area. Background Technology

[0002] Cross-operations refer to different tasks being performed on the same work surface, or different or the same tasks being performed on different work surfaces in the same three-dimensional space. Construction sites often have vertically overlapping operations, as well as simultaneous high-altitude operations in a spatially connected environment. These all fall under the category of cross-operations and are highly prone to safety accidents such as falling objects causing injuries, falls from heights, mechanical impacts, chemical burns, and fires. They pose numerous safety hazards and are difficult to manage.

[0003] Existing safety protection measures for cross-operations mainly rely on establishing communication mechanisms to rationally allocate work activities to minimize overlap, strengthening personnel training and risk notification, and developing emergency plans. However, technical solutions are lacking. CN107247456A discloses a safety anti-collision system for multi-equipment cross-operation areas. This system uses multiple sensors installed on different equipment to continuously read data. By calculating the distance relationships between the equipment, it can determine the distance between multiple devices and trigger an audible and visual alarm to remind operators to operate with caution, improving detection efficiency and safety. CN101570304 discloses a tower crane cross-operation display device. A warning bell and camera are installed on the crane hook trolley. When the crane is operating, the bell and camera move with the trolley, using images and sounds to avoid collisions and remind workers to avoid injury from the suspended load. CN113095525A provides an intelligent control system for cross-operations in power grid maintenance, including a data storage unit, a detection unit, and an advanced application unit. This system helps maintenance personnel to understand maintenance work information in a timely and accurate manner, preventing cross-operations between maintenance and high-voltage testing, and effectively reducing operational risks.

[0004] Existing safety management technologies for cross-operations mainly focus on using auxiliary sensor detection equipment or predictive trajectory for early warning. However, these technologies are costly to implement and complex to set up on-site, making them unsuitable for petrochemical tank farms and other similar environments. Therefore, it is necessary to provide a three-dimensional cross-operation identification and detection method to alert managers to strengthen early warning monitoring and prevent personal injury during operations. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes an intelligent monitoring method for cross-operations in refining and chemical processing units. The method is rationally designed, overcomes the shortcomings of the prior art, and has good results.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for intelligent monitoring of cross-operations in a refining and chemical plant area includes the following steps:

[0008] S1. Set up binocular cameras in the horizontal and vertical directions of the cross-operation area to capture images of the cross-operation area. The cross-operation area includes two construction platforms, and the height of the binocular cameras is the same as the height of the lower construction platform. The distance between the cameras should be such that they can cover the cross-operation environment. The RTSP video streams from the binocular cameras are acquired in real time for intelligent analysis.

[0009] S2. Perform camera calibration, binocular correction, and stereo matching on the images captured by the two binocular cameras respectively, and then make a judgment on the images captured by the two binocular cameras.

[0010] S3. If the images captured by the binocular cameras in both directions are determined to be suspected of being people standing in the lower cross-operation area, an alarm for the incident of people standing in the lower cross-operation area will be triggered. The two binocular cameras will output two images respectively, and the alarm device will be controlled to provide sound, light and voice reminders.

[0011] Furthermore, in step S2, OpenCV is used to calibrate the two binocular cameras respectively. The specific implementation steps are as follows:

[0012] a. Place the binocular cameras horizontally and fix them on the same reference plane;

[0013] b. Place a checkerboard calibration board in front of the binocular camera, ensuring that the calibration board is fully displayed in the shooting frame and occupies 1 / 4 to 1 / 2 of the entire shooting frame. Keep the calibration board flat during the calibration process.

[0014] c. When the brightness of the calibration board is sufficient and uniform, place multiple calibration boards at equal intervals within the actual working distance range and collect 20 to 30 calibration images.

[0015] d. Use Matlab to automatically import the calibration image for automatic calibration, and then read the calibration results into OpenCV for subsequent image calibration and matching.

[0016] Furthermore, after calibration, the internal parameters of the two binocular cameras are obtained, including fx, fy, cx, cy, distortion coefficients [k1,k2,p1,p2,k3], and the relative positions between the left and right cameras.

[0017] Furthermore, in step S2, the specific implementation process of binocular calibration and stereo matching is as follows: after obtaining the various parameters of the camera through binocular calibration, the calibration rotation matrix R, projection matrix P, and reprojection matrix Q are obtained using stereoRectify in OpenCV. Then, the calibration mapping parameters are obtained using the initUndistortRectifyMap function. Then, remap is used to calibrate the input left and right images. The matching points of the two calibrated left and right images are on the same row. The disparity map is calculated using the SGBM algorithm in OpenCV. After binocular matching, the disparity map is obtained.

[0018] Furthermore, in step S2, the images captured by the two binocular cameras are simultaneously evaluated, including the following sub-steps:

[0019] S21. Perform personnel target detection on the captured image, and calculate the coordinates of the center point below each rectangle based on the personnel rectangle coordinates, that is, the coordinates corresponding to the personnel's feet, which represent the position of each personnel.

[0020] S22. Let the image height be H. Compare the ordinate value of each person's position with H / 2. If the ordinate value is greater than H / 2, then it is recorded as an upper-level worker. If it is less than H / 2, then it is recorded as a lower-level worker.

[0021] S23. Calculate the difference in the horizontal coordinate between each upper-level worker and each lower-level worker in turn, and determine whether it is less than the threshold d1. If it is less than the threshold, further determine the depth information distance relationship.

[0022] S24. Based on the disparity map, obtain the gray values ​​corresponding to the position coordinates of the upper and lower level workers, calculate the gray value difference and determine whether it is less than the threshold d2. If it is less than the threshold d2, the binocular camera in that direction is determined to be a suspected person standing in the lower level cross-operation area.

[0023] S25. If both binocular cameras in both directions determine that a person is suspected to be standing in the lower-level cross-operation area, then the incident is determined to be a person standing in the lower-level cross-operation area.

[0024] Furthermore, in step S21, the YOLOv3 person target detection framework is used to detect people in the captured image. First, feature information is extracted through the backbone network Darknet-53, and then classification and localization are performed through the detection network.

[0025] The Darknet-53 network architecture incorporates residual modules into the network, each consisting of two convolutional layers and a shortcut link.

[0026] YOLOv3 employs upsampling and fusion methods to predict bounding boxes on three feature maps: 13*13, 26*26, and 52*52. The three prediction branches use a fully convolutional structure.

[0027] Furthermore, for the YOLOv3 detection framework, in addition to the original classification and localization results, the output of the target key point heatmap is added, thereby integrating the tasks of target detection and human key point detection. The obtained human key point location information includes the head, chest, waist and feet. Based on the key point location information, the human height information is further obtained, and the setting of the adaptive correction discrimination thresholds d, d1 and d2 is set.

[0028] The beneficial technical effects of this invention are as follows:

[0029] This invention employs a deep learning-based video recognition method. Without altering the inherent structure of the equipment or facilities, it overcomes the challenges that cannot be solved by safety management alone by installing binocular cameras in the external area and collecting, transmitting, and analyzing video streams. It enables the identification of people standing illegally in the vertical direction and issues alarms for abnormal situations, notifying managers and workers to handle violations promptly. It has the advantages of high efficiency and high precision, preventing personnel injuries and deaths such as being struck by objects or mechanical injuries during cross-operations. It provides a new approach and solution for controlling dangerous behaviors in cross-operations. Attached Figure Description

[0030] Figure 1 This is a schematic diagram showing the mounting positions of the two binocular cameras proposed in this invention;

[0031] Figure 2 This is a schematic diagram of the chessboard calibration plate proposed in this invention;

[0032] Figure 3 This is a flowchart of the binocular calibration process proposed in this invention;

[0033] Figure 4 This is a schematic diagram of personnel positions proposed in this invention; Detailed Implementation

[0034] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and specific examples:

[0035] Example 1

[0036] A method for intelligent monitoring of cross-operations in a refining and chemical plant area includes the following steps:

[0037] S1, such as Figure 1As shown, binocular cameras are fixedly installed in the horizontal and vertical directions of the cross-operation area to capture images of the cross-operation area, which includes two construction platforms, one above the other. The binocular cameras are installed at the same height as the lower construction platform, and the installation distance is determined to cover the cross-operation environment. The RTSP video streams from the binocular cameras are acquired in real time to identify dangerous events such as people being on the lower platform when the two construction platforms are operating cross-platform.

[0038] S2. Perform camera calibration, binocular correction, and stereo matching on the images captured by the two binocular cameras respectively, and then perform intelligent analysis and judgment on the images captured by the two binocular cameras.

[0039] Specifically, OpenCV is a cross-platform computer vision library released under the BSD license (open source), which can run on Linux, Windows, Android, and Mac OS operating systems. The specific steps for calibrating two binocular cameras using OpenCV are as follows:

[0040] a. Place the binocular cameras horizontally and fix them on the same reference plane;

[0041] b. Place a reference object—a checkerboard calibration board—in front of the binocular camera, such as... Figure 2 As shown, the calibration board is displayed completely in the shooting frame and occupies 1 / 4 of the entire shooting frame. The calibration board remains flat during the calibration process.

[0042] c. When the brightness of the calibration board is sufficient and uniform, place multiple calibration boards at equal intervals within the actual working distance range. In this embodiment, the working distance is 5m, and the boards are placed at 5m intervals of 1m to collect 20 calibration images.

[0043] d. Use the stereo calibration toolbox in Matlab to perform the calibration. Enter "stereoCameraCalibrator" in the command line to enter the calibration module. The image will be automatically imported for automatic calibration. The calibration results will be read into OpenCV for subsequent image calibration and matching.

[0044] After calibration, the intrinsic parameters of the two binocular cameras are obtained. The intrinsic parameters include fx, fy, cx, cy, distortion coefficients [k1,k2,p1,p2,k3], and the relative positions between the left and right cameras. The relative positions are the translation vector t and rotation matrix R of the right camera relative to the left camera.

[0045] Specifically, such as Figure 3As shown, the purpose of binocular calibration is to ensure strict row correspondence between the two distorted images, so that the epipolar lines of the two images are exactly on the same horizontal line. This means that any point in one image will have the same row number as its corresponding point in the other image, allowing for a one-dimensional search within that row to find the matching point. After obtaining the camera parameters through binocular calibration, the `stereoRectify` function in OpenCV is used to obtain the calibration rotation matrix R, projection matrix P, and reprojection matrix Q. The `initUndistortRectifyMap` function is then used to obtain the calibration mapping parameters, and `remap` is used to calibrate the input left and right images. After obtaining the calibrated left and right images, the matching points are on the same row. The disparity map is calculated using the SGBM algorithm in OpenCV, and the disparity map is obtained after binocular matching. Disparity is the difference in the x-coordinate of the same spatial point in the images from two cameras. It can be encoded into a grayscale image to reflect the distance; the closer to the lens, the brighter the grayscale. (If the two cameras are perpendicular, the difference in the y-coordinate is used).

[0046] The process of evaluating the images captured by the two binocular cameras includes the following sub-steps:

[0047] S21. Perform personnel target detection on the captured image, and calculate the coordinates of the center point below each rectangle based on the personnel rectangle coordinates, that is, the coordinates corresponding to the personnel's feet, which represent the position of each personnel.

[0048] Among them, the YOLOv3 person target detection framework is used for personnel detection in the captured images. First, feature information is extracted through the backbone network Darknet-53, and then the YOLOv3 detection framework is used for classification and localization.

[0049] The Darknet-53 network architecture incorporates residual modules, which helps solve the gradient problem in deep networks. Each residual module consists of two convolutional layers and a shortcut link.

[0050] To enhance the accuracy of the algorithm in detecting small targets such as people, YOLOv3 adopts an upsampling and fusion approach similar to FPN, predicting target boxes on three feature maps: 13*13 (32x downsampling), 26*26 (16x downsampling), and 52*52 (8x downsampling). The three prediction branches adopt a fully convolutional structure.

[0051] For the YOLOv3 detection framework, in addition to the original classification and localization results, a target keypoint heatmap is added as an output, thereby integrating multiple tasks such as target detection and human keypoint detection to improve target detection accuracy. The obtained human keypoint location information includes the head, chest, waist, and feet. Based on the keypoint location information, human height information is further obtained, and the setting of the discrimination thresholds d, d1, and d2 is adaptively corrected.

[0052] S22, such as Figure 4 As shown, let the image height be H, and denote the ordinate value y of each person's position. downi Compared with H / 2, if the ordinate value y downi If the value is greater than H / 2, the worker is recorded as an upper-level worker; if the value is less than H / 2, the worker is recorded as a lower-level worker.

[0053] S23. Calculate the difference in the horizontal coordinate between each upper-level worker and each lower-level worker in turn. Determine whether it is less than the threshold d1 according to formula (1). If it is less than the threshold, the binocular camera in the horizontal direction is determined to be a suspected person standing in the lower-level cross-operation area. Then, further determine the depth information distance relationship.

[0054] ||x upi -x downj ||<d1 (1)

[0055] Where, x upi Let x be the x-coordinate of the workers on the upper level. downj The x-axis represents the horizontal axis of the workers at the lower level.

[0056] S24. Based on the parallax map, obtain the gray values ​​corresponding to the position coordinates of the upper and lower level workers, calculate the gray value difference and determine whether it is less than the threshold d2. If it is less than the threshold d2, the binocular camera in the vertical direction is identified as a suspected person standing in the lower level cross-operation area.

[0057] S25. If both binocular cameras in both directions determine that a person is suspected to be standing in the lower-level cross-operation area, then the incident is determined to be a person standing in the lower-level cross-operation area.

[0058] S3. If the images captured by the binocular cameras in both directions are determined to be suspected of being a person standing in the lower cross-operation area, an alarm for the incident of a person standing in the lower cross-operation area will be triggered. The two binocular cameras will output two images respectively, and the alarm device will be controlled to provide sound, light and voice reminders.

[0059] Example 2

[0060] A method for intelligent monitoring of cross-operations in a refining and chemical plant area includes the following steps:

[0061] S1, such as Figure 1As shown, binocular cameras are fixedly installed in the horizontal and vertical directions of the cross-operation area to capture images of the cross-operation area, which includes two construction platforms, one above the other. The binocular cameras are installed at the same height as the lower construction platform, and the installation distance is determined to cover the cross-operation environment. The RTSP video streams from the binocular cameras are acquired in real time to identify dangerous events such as people being on the lower platform when the two construction platforms are operating cross-platform.

[0062] S2. Perform camera calibration, binocular correction, and stereo matching on the images captured by the two binocular cameras respectively, and then perform intelligent analysis and judgment on the images captured by the two binocular cameras.

[0063] Specifically, OpenCV is a cross-platform computer vision library released under the BSD license (open source), which can run on Linux, Windows, Android, and Mac OS operating systems. The specific steps for calibrating two binocular cameras using OpenCV are as follows:

[0064] a. Place the binocular cameras horizontally and fix them on the same reference plane;

[0065] b. Place a reference object—a checkerboard calibration board—in front of the binocular camera, such as... Figure 2 As shown, the calibration board should be fully displayed in the shooting frame and occupy 3 / 8 of the entire shooting frame, and the calibration board should remain flat during the calibration process;

[0066] c. When the brightness of the calibration board is sufficient and uniform, place multiple calibration boards at equal intervals within the actual working distance range. In this embodiment, the working distance is 6m, and the boards are placed at 6m intervals of 1m to collect 25 calibration images.

[0067] d. Use the stereo calibration toolbox in Matlab to perform the calibration. Enter "stereoCameraCalibrator" in the command line to enter the calibration module. The image will be automatically imported for automatic calibration. The calibration results will be read into OpenCV for subsequent image calibration and matching.

[0068] After calibration, the intrinsic parameters of the two binocular cameras are obtained. The intrinsic parameters include fx, fy, cx, cy, distortion coefficients [k1,k2,p1,p2,k3], and the relative positions between the left and right cameras. The relative positions are the translation vector t and rotation matrix R of the right camera relative to the left camera.

[0069] Specifically, such as Figure 3As shown, the purpose of binocular calibration is to ensure strict row correspondence between the two distorted images, so that the epipolar lines of the two images are exactly on the same horizontal line. This means that any point in one image will have the same row number as its corresponding point in the other image, allowing for a one-dimensional search within that row to find the matching point. After obtaining the camera parameters through binocular calibration, the `stereoRectify` function in OpenCV is used to obtain the calibration rotation matrix R, projection matrix P, and reprojection matrix Q. The `initUndistortRectifyMap` function is then used to obtain the calibration mapping parameters, and `remap` is used to calibrate the input left and right images. After obtaining the calibrated left and right images, the matching points are on the same row. The disparity map is calculated using the SGBM algorithm in OpenCV, and the disparity map is obtained after binocular matching. Disparity is the difference in the x-coordinate of the same spatial point in the images from two cameras. It can be encoded into a grayscale image to reflect the distance; the closer to the lens, the brighter the grayscale. (If the two cameras are perpendicular, the difference in the y-coordinate is used).

[0070] The process of evaluating the images captured by the two binocular cameras includes the following sub-steps:

[0071] S21. Perform personnel target detection on the captured image, and calculate the coordinates of the center point below each rectangle based on the personnel rectangle coordinates, that is, the coordinates corresponding to the personnel's feet, which represent the position of each personnel.

[0072] Among them, the YOLOv3 person target detection framework is used for personnel detection in the captured images. First, feature information is extracted through the backbone network Darknet-53, and then the YOLOv3 detection framework is used for classification and localization.

[0073] The Darknet-53 network architecture incorporates residual modules, which helps solve the gradient problem in deep networks. Each residual module consists of two convolutional layers and a shortcut link.

[0074] To enhance the accuracy of the algorithm in detecting small targets such as people, YOLOv3 adopts an upsampling and fusion approach similar to FPN, predicting target boxes on three feature maps: 13*13 (32x downsampling), 26*26 (16x downsampling), and 52*52 (8x downsampling). The three prediction branches adopt a fully convolutional structure.

[0075] For the YOLOv3 detection framework, in addition to the original classification and localization results, a target keypoint heatmap is added as an output, thereby integrating multiple tasks such as target detection and human keypoint detection to improve target detection accuracy. The obtained human keypoint location information includes the head, chest, waist, and feet. Based on the keypoint location information, human height information is further obtained, and the setting of the discrimination thresholds d, d1, and d2 is adaptively corrected.

[0076] S22, such as Figure 4 As shown, let the image height be H, and denote the ordinate value y of each person's position. downi Compared with H / 2, if the ordinate value y downi If the value is greater than H / 2, the worker is recorded as an upper-level worker; if the value is less than H / 2, the worker is recorded as a lower-level worker.

[0077] S23. Calculate the difference in the horizontal coordinate between each upper-level worker and each lower-level worker in turn. Determine whether it is less than the threshold d1 according to formula (1). If it is less than the threshold, the binocular camera in the horizontal direction is determined to be a suspected person standing in the lower-level cross-operation area. Then, further determine the depth information distance relationship.

[0078] ||x upi -x downj ||<d1 (1)

[0079] Where, x upi Let x be the x-coordinate of the workers on the upper level. downj The x-axis represents the horizontal axis of the workers at the lower level.

[0080] S24. Based on the parallax map, obtain the gray values ​​corresponding to the position coordinates of the upper and lower level workers, calculate the gray value difference and determine whether it is less than the threshold d2. If it is less than the threshold d2, the binocular camera in the vertical direction is identified as a suspected person standing in the lower level cross-operation area.

[0081] S25. If both binocular cameras in both directions determine that a person is suspected to be standing in the lower-level cross-operation area, then the incident is determined to be a person standing in the lower-level cross-operation area.

[0082] S3. If the images captured by the binocular cameras in both directions are determined to be suspected of being a person standing in the lower cross-operation area, an alarm for the incident of a person standing in the lower cross-operation area will be triggered. The two binocular cameras will output two images respectively, and the alarm device will be controlled to provide sound, light and voice reminders.

[0083] Example 3

[0084] A method for intelligent monitoring of cross-operations in a refining and chemical plant area includes the following steps:

[0085] S1, such as Figure 1As shown, binocular cameras are fixedly installed in the horizontal and vertical directions of the cross-operation area to capture images of the cross-operation area, which includes two construction platforms, one above the other. The binocular cameras are installed at the same height as the lower construction platform, and the installation distance is determined to cover the cross-operation environment. The RTSP video streams from the binocular cameras are acquired in real time to identify dangerous events such as people being on the lower platform when the two construction platforms are operating cross-platform.

[0086] S2. Perform camera calibration, binocular correction, and stereo matching on the images captured by the two binocular cameras respectively, and then perform intelligent analysis and judgment on the images captured by the two binocular cameras.

[0087] Specifically, OpenCV is a cross-platform computer vision library released under the BSD license (open source), which can run on Linux, Windows, Android, and Mac OS operating systems. The specific steps for calibrating two binocular cameras using OpenCV are as follows:

[0088] a. Place the binocular cameras horizontally and fix them on the same reference plane;

[0089] b. Place a reference object—a checkerboard calibration board—in front of the binocular camera, such as... Figure 2 As shown, the calibration board is displayed completely in the shooting frame and occupies 1 / 2 of the entire shooting frame. The calibration board remains flat during the calibration process.

[0090] c. When the brightness of the calibration board is sufficient and uniform, place multiple calibration boards at equal intervals within the actual working distance range. In this embodiment, the working distance is 7m, and the boards are placed at 7m intervals of 1m to collect 30 calibration images.

[0091] d. Use the stereo calibration toolbox in Matlab to perform the calibration. Enter "stereoCameraCalibrator" in the command line to enter the calibration module. The image will be automatically imported for automatic calibration. The calibration results will be read into OpenCV for subsequent image calibration and matching.

[0092] After calibration, the intrinsic parameters of the two binocular cameras are obtained. The intrinsic parameters include fx, fy, cx, cy, distortion coefficients [k1,k2,p1,p2,k3], and the relative positions between the left and right cameras. The relative positions are the translation vector t and rotation matrix R of the right camera relative to the left camera.

[0093] Specifically, such as Figure 3As shown, the purpose of binocular calibration is to ensure strict row correspondence between the two distorted images, so that the epipolar lines of the two images are exactly on the same horizontal line. This means that any point in one image will have the same row number as its corresponding point in the other image, allowing for a one-dimensional search within that row to find the matching point. After obtaining the camera parameters through binocular calibration, the `stereoRectify` function in OpenCV is used to obtain the calibration rotation matrix R, projection matrix P, and reprojection matrix Q. The `initUndistortRectifyMap` function is then used to obtain the calibration mapping parameters, and `remap` is used to calibrate the input left and right images. After obtaining the calibrated left and right images, the matching points are on the same row. The disparity map is calculated using the SGBM algorithm in OpenCV, and the disparity map is obtained after binocular matching. Disparity is the difference in the x-coordinate of the same spatial point in the images from two cameras. It can be encoded into a grayscale image to reflect the distance; the closer to the lens, the brighter the grayscale. (If the two cameras are perpendicular, the difference in the y-coordinate is used).

[0094] The process of evaluating the images captured by the two binocular cameras includes the following sub-steps:

[0095] S21. Perform personnel target detection on the captured image, and calculate the coordinates of the center point below each rectangle based on the personnel rectangle coordinates, that is, the coordinates corresponding to the personnel's feet, which represent the position of each personnel.

[0096] Among them, the YOLOv3 person target detection framework is used for personnel detection in the captured images. First, feature information is extracted through the backbone network Darknet-53, and then the YOLOv3 detection framework is used for classification and localization.

[0097] The Darknet-53 network architecture incorporates residual modules, which helps solve the gradient problem in deep networks. Each residual module consists of two convolutional layers and a shortcut link.

[0098] To enhance the accuracy of the algorithm in detecting small targets such as people, YOLOv3 adopts an upsampling and fusion approach similar to FPN, predicting target boxes on three feature maps: 13*13 (32x downsampling), 26*26 (16x downsampling), and 52*52 (8x downsampling). The three prediction branches adopt a fully convolutional structure.

[0099] For the YOLOv3 detection framework, in addition to the original classification and localization results, a target keypoint heatmap is added as an output, thereby integrating multiple tasks such as target detection and human keypoint detection to improve target detection accuracy. The obtained human keypoint location information includes the head, chest, waist, and feet. Based on the keypoint location information, human height information is further obtained, and the setting of the discrimination thresholds d, d1, and d2 is adaptively corrected.

[0100] S22, such as Figure 4 As shown, let the image height be H, and denote the ordinate value y of each person's position. downi Compared with H / 2, if the ordinate value y downi If the value is greater than H / 2, the worker is recorded as an upper-level worker; if the value is less than H / 2, the worker is recorded as a lower-level worker.

[0101] S23. Calculate the difference in the horizontal coordinate between each upper-level worker and each lower-level worker in turn. Determine whether it is less than the threshold d1 according to formula (1). If it is less than the threshold, the binocular camera in the horizontal direction is determined to be a suspected person standing in the lower-level cross-operation area. Then, further determine the depth information distance relationship.

[0102] ||x upi -x downj ||<d1 (1)

[0103] Where, x upi Let x be the x-coordinate of the workers on the upper level. downj The x-axis represents the horizontal axis of the workers at the lower level.

[0104] S24. Based on the parallax map, obtain the gray values ​​corresponding to the position coordinates of the upper and lower level workers, calculate the gray value difference and determine whether it is less than the threshold d2. If it is less than the threshold d2, the binocular camera in the vertical direction is identified as a suspected person standing in the lower level cross-operation area.

[0105] S25. If both binocular cameras in both directions determine that a person is suspected to be standing in the lower-level cross-operation area, then the incident is determined to be a person standing in the lower-level cross-operation area.

[0106] S3. If the images captured by the binocular cameras in both directions are determined to be suspected of being a person standing in the lower cross-operation area, an alarm for the incident of a person standing in the lower cross-operation area will be triggered. The two binocular cameras will output two images respectively, and the alarm device will be controlled to provide sound, light and voice reminders.

[0107] Example 4

[0108] A method for intelligent monitoring of cross-operations in a refining and chemical plant area includes the following steps:

[0109] S1, such as Figure 1As shown, binocular cameras are fixedly installed in the horizontal and vertical directions of the cross-operation area to capture images of the cross-operation area, which includes two construction platforms, one above the other. The binocular cameras are installed at the same height as the lower construction platform, and the installation distance is determined to cover the cross-operation environment. The RTSP video streams from the binocular cameras are acquired in real time to identify dangerous events such as people being on the lower platform when the two construction platforms are operating cross-platform.

[0110] S2. Perform camera calibration, binocular correction, and stereo matching on the images captured by the two binocular cameras respectively, and then perform intelligent analysis and judgment on the images captured by the two binocular cameras.

[0111] Specifically, OpenCV is a cross-platform computer vision library released under the BSD license (open source), which can run on Linux, Windows, Android, and Mac OS operating systems. The specific steps for calibrating two binocular cameras using OpenCV are as follows:

[0112] a. Place the binocular cameras horizontally and fix them on the same reference plane;

[0113] b. Place a reference object—a checkerboard calibration board—in front of the binocular camera, such as... Figure 2 As shown, the calibration board is displayed completely in the shooting frame and occupies 1 / 2 of the entire shooting frame. The calibration board remains flat during the calibration process.

[0114] c. When the brightness of the calibration board is sufficient and uniform, place multiple calibration boards at equal intervals within the actual working distance range. In this embodiment, the working distance is 7m, and the boards are placed at 7m intervals of 1m to collect 25 calibration images.

[0115] d. Use the stereo calibration toolbox in Matlab to perform the calibration. Enter "stereoCameraCalibrator" in the command line to enter the calibration module. The image will be automatically imported for automatic calibration. The calibration results will be read into OpenCV for subsequent image calibration and matching.

[0116] After calibration, the intrinsic parameters of the two binocular cameras are obtained. The intrinsic parameters include fx, fy, cx, cy, distortion coefficients [k1,k2,p1,p2,k3], and the relative positions between the left and right cameras. The relative positions are the translation vector t and rotation matrix R of the right camera relative to the left camera.

[0117] Specifically, such as Figure 3As shown, the purpose of binocular calibration is to ensure strict row correspondence between the two distorted images, so that the epipolar lines of the two images are exactly on the same horizontal line. This means that any point in one image will have the same row number as its corresponding point in the other image, allowing for a one-dimensional search within that row to find the matching point. After obtaining the camera parameters through binocular calibration, the `stereoRectify` function in OpenCV is used to obtain the calibration rotation matrix R, projection matrix P, and reprojection matrix Q. The `initUndistortRectifyMap` function is then used to obtain the calibration mapping parameters, and `remap` is used to calibrate the input left and right images. After obtaining the calibrated left and right images, the matching points are on the same row. The disparity map is calculated using the SGBM algorithm in OpenCV, and the disparity map is obtained after binocular matching. Disparity is the difference in the x-coordinate of the same spatial point in the images from two cameras. It can be encoded into a grayscale image to reflect the distance; the closer to the lens, the brighter the grayscale. (If the two cameras are perpendicular, the difference in the y-coordinate is used).

[0118] The process of evaluating the images captured by the two binocular cameras includes the following sub-steps:

[0119] S21. Perform personnel target detection on the captured image, and calculate the coordinates of the center point below each rectangle based on the personnel rectangle coordinates, that is, the coordinates corresponding to the personnel's feet, which represent the position of each personnel.

[0120] Among them, the YOLOv3 person target detection framework is used for personnel detection in the captured images. First, feature information is extracted through the backbone network Darknet-53, and then the YOLOv3 detection framework is used for classification and localization.

[0121] The Darknet-53 network architecture incorporates residual modules, which helps solve the gradient problem in deep networks. Each residual module consists of two convolutional layers and a shortcut link.

[0122] To enhance the accuracy of the algorithm in detecting small targets such as people, YOLOv3 adopts an upsampling and fusion approach similar to FPN, predicting target boxes on three feature maps: 13*13 (32x downsampling), 26*26 (16x downsampling), and 52*52 (8x downsampling). The three prediction branches adopt a fully convolutional structure.

[0123] For the YOLOv3 detection framework, in addition to the original classification and localization results, a target keypoint heatmap is added as an output, thereby integrating multiple tasks such as target detection and human keypoint detection to improve target detection accuracy. The obtained human keypoint location information includes the head, chest, waist, and feet. Based on the keypoint location information, human height information is further obtained, and the setting of the discrimination thresholds d, d1, and d2 is adaptively corrected.

[0124] S22, such as Figure 4 As shown, let the image height be H, and denote the ordinate value y of each person's position. downi Compared with H / 2, if the ordinate value y downi If the value is greater than H / 2, the worker is recorded as an upper-level worker; if the value is less than H / 2, the worker is recorded as a lower-level worker.

[0125] S23. Calculate the difference in the horizontal coordinate between each upper-level worker and each lower-level worker in turn. Determine whether it is less than the threshold d1 according to formula (1). If it is less than the threshold, the binocular camera in the horizontal direction is determined to be a suspected person standing in the lower-level cross-operation area. Then, further determine the depth information distance relationship.

[0126] ||x upi -x downj ||<d1 (1)

[0127] Where, x upi Let x be the x-coordinate of the workers on the upper level. downj The x-axis represents the horizontal axis of the workers at the lower level.

[0128] S24. Based on the parallax map, obtain the gray values ​​corresponding to the position coordinates of the upper and lower level workers, calculate the gray value difference and determine whether it is less than the threshold d2. If it is less than the threshold d2, the binocular camera in the vertical direction is identified as a suspected person standing in the lower level cross-operation area.

[0129] S25. If both binocular cameras in both directions determine that a person is suspected to be standing in the lower-level cross-operation area, then the incident is determined to be a person standing in the lower-level cross-operation area.

[0130] S3. If the images captured by the binocular cameras in both directions are determined to be suspected of being a person standing in the lower cross-operation area, an alarm for the incident of a person standing in the lower cross-operation area will be triggered. The two binocular cameras will output two images respectively, and the alarm device will be controlled to provide sound, light and voice reminders.

[0131] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for intelligent monitoring of cross-operations in a refining and chemical plant area, characterized in that, Includes the following steps: S1. Set up binocular cameras in the horizontal and vertical directions of the cross-operation area to capture images of the cross-operation area. The cross-operation area includes two construction platforms, and the height of the binocular cameras is the same as the height of the lower construction platform. The distance between the cameras should be such that they can cover the cross-operation environment. The RTSP video streams from the binocular cameras are acquired in real time for intelligent analysis. S2. Perform camera calibration, binocular correction, and stereo matching on the images captured by the two binocular cameras respectively, and then make a judgment on the images captured by the two binocular cameras. S3. If the images captured by the binocular cameras in both directions are determined to be suspected of being people standing in the cross-operation area on the lower level, an alarm for the incident of people standing in the cross-operation area on the lower level will be triggered. The two binocular cameras will output two images respectively, and the alarm device will be controlled to provide sound, light and voice reminders. In step S2, the images captured by the two binocular cameras are judged simultaneously, including the following sub-steps: S21. Perform personnel target detection on the captured image, and calculate the coordinates of the center point below each rectangle based on the personnel rectangle coordinates, that is, the coordinates corresponding to the personnel's feet, which represent the position of each personnel. S22. Let the image height be H. Compare the vertical coordinate value of each person's position with H / 2. If the vertical coordinate value is greater than H / 2, then it is recorded as an upper-level worker. If it is less than H / 2, then it is recorded as a lower-level worker. S23. Calculate the difference in the horizontal coordinate between each upper-level worker and each lower-level worker in turn, and determine whether it is less than the threshold d1. If it is less than the threshold, further determine the depth information distance relationship. S24. Based on the disparity map, obtain the gray values ​​corresponding to the position coordinates of the upper and lower level workers, calculate the gray value difference and determine whether it is less than the threshold d2. If it is less than the threshold d2, the binocular camera in that direction is determined to be a suspected person standing in the lower level cross-operation area. S25. If both binocular cameras in both directions determine that a person is suspected to be standing in the lower-level cross-operation area, then the incident is determined to be a person standing in the lower-level cross-operation area.

2. The intelligent monitoring method for cross-operations in a refining and chemical plant area according to claim 1, characterized in that, In step S2, OpenCV is used to calibrate the two binocular cameras respectively. The specific implementation steps are as follows: a. Place the binocular cameras horizontally and fix them on the same reference plane; b. Place a checkerboard calibration board in front of the binocular camera, ensuring that the calibration board is fully displayed in the shooting image, and keep the calibration board flat during the calibration process; c. When the brightness of the calibration board is sufficient and uniform, place multiple calibration boards at equal intervals within the actual working distance range and collect 20 to 30 calibration images. d. Use Matlab to automatically import the calibration image for automatic calibration, and then read the calibration results into OpenCV for subsequent image calibration and matching.

3. The intelligent monitoring method for cross-operations in a refining and chemical plant area according to claim 2, characterized in that, In step b, the calibration plate occupies 1 / 4 to 1 / 2 of the entire shooting frame.

4. The intelligent monitoring method for cross-operations in a refining and chemical plant area according to claim 2, characterized in that, After calibration, the internal parameters of the two binocular cameras are obtained. The internal parameters include fx, fy, cx, cy, distortion coefficients [k1,k2,p1,p2,k3], and the relative positions between the left and right cameras.

5. The intelligent monitoring method for cross-operations in a refining and chemical plant area according to claim 4, characterized in that, In step S2, the specific implementation process of binocular calibration and stereo matching is as follows: After obtaining the various parameters of the camera through binocular calibration, the calibration rotation matrix R, projection matrix P, and reprojection matrix Q are obtained using stereoRectify in OpenCV. Then, the calibration mapping parameters are obtained using the initUndistortRectifyMap function. Finally, remap is used to calibrate the input left and right images. The matching points of the two calibrated left and right images are on the same row. The disparity map is calculated using the SGBM algorithm in OpenCV. After binocular matching, the disparity map is obtained.

6. The intelligent monitoring method for cross-operations in a refining and chemical plant area according to claim 1, characterized in that, In step S21, the YOLOv3 personnel target detection framework is used to detect people in the captured image. First, feature information is extracted through the backbone network Darknet-53, and then classification and localization are performed through the detection network.

7. The intelligent monitoring method for cross-operations in a refining and chemical plant area according to claim 6, characterized in that, The Darknet-53 network architecture incorporates residual modules into the network, each consisting of two convolutional layers and a shortcut link.

8. The intelligent monitoring method for cross-operations in a refining and chemical plant area according to claim 6, characterized in that, YOLOv3 employs upsampling and fusion methods to predict bounding boxes on three feature maps: 13*13, 26*26, and 52*52. The three prediction branches use a fully convolutional structure.

9. The intelligent monitoring method for cross-operations in a refining and chemical plant area according to claim 6, characterized in that, For the YOLOv3 detection framework, in addition to the original classification and localization results, the output of target key point heatmap is added, thereby integrating the tasks of target detection and human key point detection. The obtained human key point location information includes head, chest, waist and feet. Based on the key point location information, human height information is further obtained, and the setting of the discrimination thresholds d, d1 and d2 is adaptively corrected.

Images