Device for controlling operation of tube push bench through visual tracking

Abstract

The invention relates to the field of tube push bench operation, and discloses a device for controlling tube push bench operation through visual tracking, which comprises a visual perception module comprising a plurality of groups of industrial cameras arranged in a triangular array at intervals of 2-3m and equipped with 850-1200nm infrared light supplement and a self-adaptive exposure algorithm; the dynamic calibration and coordinate conversion module is used for carrying out SIFT feature matching based on a workshop fixed bracket and establishing a conversion matrix from an image coordinate system to a world coordinate system; and the deep learning identification module adopts a YOLOv5 and ResNet-50 double-model collaborative architecture and has static and dynamic shielding processing algorithms. According to the invention, multiple groups of industrial cameras are arranged in a triangular array, and an image splicing technology is combined, so that a single-visual-angle blind area is eliminated, and a detection area is fully covered; continuous and stable tracking of a moving target is ensured, tracking interruption and position deviation caused by view angle limitation or time asynchronization are avoided, infrared light supplement and a self-adaptive exposure algorithm are matched, ambient light interference is inhibited, strong light is coped, a clear image can still be obtained under complex illumination, and the positioning precision is improved.

Description

Technical Field

[0001] The invention relates to the technical field of pipe jacking machine operation, in particular to a visual tracking control device for controlling the operation of a pipe jacking machine. Background Art

[0002] In the field of pipeline construction, the automated operation and control of pipe jacking machines is crucial to construction accuracy and safety. Existing pipe jacking machine operation and control technologies mainly rely on traditional sensors or single vision systems, which have the following significant defects:

[0003] Single-view industrial cameras are prone to forming detection blind spots due to limited viewing angles, and the time synchronization errors of multiple cameras are large, resulting in interruptions in tracking of moving targets or position deviations, making it difficult to meet the needs of continuous and stable real-time tracking. Traditional vision systems lack efficient fill light and dynamic exposure adjustment mechanisms. The image clarity is low in strong or low light environments and is easily affected by ambient light interference, resulting in reduced target recognition accuracy.

[0004] Existing calibration methods often use fixed reference objects, lacking dynamic monitoring and a graded error response mechanism. Accumulated reference object displacement errors can easily lead to coordinate transformation errors and make fault tracing difficult. There is a lack of effective strategies for distinguishing and handling static and dynamic occlusions, and target data loss is common when the occlusion area exceeds 20%. Traditional anomaly detection models rely on single sensor data, resulting in delayed responses and high misjudgment rates. Control logic lacks flexibility and fault tolerance. Existing control systems often use fixed threshold response strategies, lack graded dynamic adjustment capabilities, and lack reliable backup modes in the event of data anomalies. Manual intervention has a low priority, making it difficult to respond to unexpected operating conditions. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a visual tracking control device for a pipe jacking machine, which solves the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a visual tracking control device for pipe jacking machine operation, including a visual perception module, including multiple groups of industrial cameras arranged in a triangular array with a spacing of 2-3m, equipped with 850-1200nm infrared fill light and an adaptive exposure algorithm;

[0007] The dynamic calibration and coordinate conversion module performs SIFT feature matching based on the workshop fixed bracket and establishes the conversion matrix from the image coordinate system to the world coordinate system;

[0008] The deep learning recognition module uses a YOLOv5 and ResNet-50 dual-model collaborative architecture and is equipped with static and dynamic occlusion processing algorithms;

[0009] Control logic and safety interlock modules implement hierarchical response control based on visual data.

[0010] Preferably, the industrial camera meets the following requirements: time synchronization error ≤ 1ms, image transmission delay ≤ 50ms, global shutter, frame rate ≥ 60fps, and the ability to maintain three-dimensional coordinate reconstruction through parallax calculation when a single camera is occluded.

[0011] Preferably, a conversion matrix from the image pixel coordinate system to the world coordinate system is established in the dynamic calibration and coordinate conversion module. If the deviation between the actual coordinates of the reference object and the calibration coordinates is greater than 5%, an alarm is triggered and recalibration is performed to achieve accurate conversion from image coordinates to world coordinates.

[0012] Preferably, the conversion model is as follows:

[0013]

[0014] Among them, K is the camera intrinsic parameter matrix, R is the rotation matrix, and T is the translation vector.

[0015] Preferably, calibration errors are divided into three levels, corresponding to different response strategies, which can promptly detect and handle calibration deviation problems, store the calibration records of the last 30-60 days, support fault tracing, and facilitate traceability and analysis of the calibration process.

[0016] Preferably, in the deep learning recognition module, YOLOv5 is responsible for rapid detection of the wild pipe head, and ResNet-50 performs refined coordinate regression on the detection area, wherein the hood occlusion defines the invalid area through hood edge detection; personnel occlusion shields the interference area based on human body contour recognition, and when the occlusion area is ≤30%, Kalman filtering is used to predict the movement trajectory of the wild pipe; when the occlusion area is greater than 30%, it is marked as unreliable data, triggering an alarm and an abnormal state determination.

[0017] Preferably, the dynamic occlusion area ratio is calculated by predicting the key points of the human body K={k1, k2, ..., k 18}, generate human body contour mask M human (x,y), calculate the occlusion area ratio η.

[0018]

[0019] Among them, M target (x,y) is the mask of the target area. When the occlusion area ratio η≤30%, the Kalman filter is used to predict the movement trajectory of the fan tube.

[0020] Preferably, the Kalman filter includes two steps: prediction and update. The predicted state vector x t|t-1 and the covariance matrix P t|t-1 .

[0021] x t|t-1 =Fx t-1|t-1 +Bu t

[0022] P t|t-1 =FP t-1|t-1 F T +Q

[0023] Among them, F is the state transfer matrix, B is the control input matrix, u t is the control input vector, and Q is the process noise covariance matrix.

[0024] Preferably, the control logic and safety interlocking module also sets a fault-tolerant mechanism, including data credibility verification, switching to the load observer backup mode when the visual data is abnormal (such as missing the target for 5 consecutive frames), and the manual intervention signal priority > automatic control.

[0025] Preferably, the static processing includes a Canny edge detection algorithm including Gaussian smoothing, gradient calculation, non-maximum suppression and double threshold processing, and a Gaussian filter G(x,y) is used to smooth the image I(x,y) to obtain a smoothed image S(x,y).

[0026] The present invention provides a visual tracking control device for pipe jacking machine operation. It has the following beneficial effects:

[0027] 1. This invention uses multiple groups of industrial cameras arranged in a triangular array and combined with image stitching technology to eliminate single-view blind spots and fully cover the detection area; ensure continuous and stable tracking of moving targets, avoid tracking interruptions and position deviations caused by view limitations or time asynchrony, and use infrared fill light and adaptive exposure algorithms to suppress ambient light interference and cope with strong light. It can still obtain clear images under complex lighting conditions and improve positioning accuracy.

[0028] 2. The present invention uses a fixed bracket of a workshop dust removal device as a reference object, regularly detects displacement through a laser rangefinder, and realizes corner point positioning of the reference object in combination with SIFT feature template matching. It establishes a conversion matrix from the image pixel coordinate system to the world coordinate system, sets a mechanism for triggering an alarm and recalibrating when the deviation is greater than 5%, matches different response strategies, and stores the calibration records of the last 30 days to support fault tracing, thereby ensuring the accuracy of coordinate conversion and the long-term stability of the system.

[0029] 3. The present invention realizes rapid detection of wild pipe heads and refined coordinate regression through the collaboration of YOLOv5 and ResNet-50 dual models, uses the Canny algorithm to detect the edge of the hood and generate an invalid area mask, calculates the occlusion area ratio based on OpenPose recognition of human contours, combines penetration judgment and pull-off judgment to achieve rapid response to abnormal conditions, and triggers pre-alarms or emergency shutdowns in a hierarchical manner through the PLC program, combined with data credibility verification and load observer backup mode, to improve system fault tolerance and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the present invention. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0032] Example:

[0033] Please see the attached Figure 1 An embodiment of the present invention provides a visual tracking control device for controlling the operation of a pipe jacking machine, including a visual perception module. The visual perception module includes multiple groups of industrial cameras, which can adopt Hikvision MV-CS050-60GC, 5 million pixels, global shutter, frame rate 60fps, installation parameters of horizontal viewing angle 60°, vertical viewing angle 40°, covering 18m×3m detection area, and industrial cameras are transmitted through optical fiber to ensure that the image transmission delay is ≤50ms, thereby ensuring the real-time nature of image information. The time synchronization error of multiple groups of industrial cameras is ≤1ms, which enables moving targets to be tracked continuously and stably, avoiding target tracking interruption or position deviation caused by time asynchrony. Multiple groups of industrial cameras are distributed in a triangular array layout with a spacing of 2m between each other. Single-view blind spots are eliminated through image stitching. The time synchronization error of multiple groups of industrial rubber is ≤1ms, ensuring continuous tracking of moving targets. 850nm infrared fill light is used to suppress ambient light interference, ensuring that clear images can still be obtained under complex lighting conditions, and an adaptive exposure algorithm (dynamic range ≥120dB) is used to cope with the strong light of rolling sparks. At the same time, when any single camera is blocked, the remaining two can still cover the rolling area. Three-dimensional coordinate reconstruction is achieved through parallax calculation (triangulation measurement method) to improve positioning accuracy.

[0034] The visual perception module is connected to the dynamic calibration and coordinate conversion module, and the dynamic calibration and coordinate conversion module

[0035] A calibration reference object is required. The reference object is generally a fixed bracket for a workshop dust removal device (1m×0.5m in size, with reflective markings on the surface). The reference object's displacement is regularly monitored using a laser rangefinder, with an error controlled to ≤±2mm / year. First, the reference object is identified: template matching (SIFT features) is used to locate the reference object's corner points. A conversion matrix is ​​then established from the image pixel coordinate system (u, v) to the world coordinate system (X, Y, Z). If the deviation between the reference object's actual coordinates and the calibration coordinates is greater than 5%, an alarm is triggered and recalibration is performed to achieve accurate conversion from image coordinates to world coordinates. The model is as follows:

[0036]

[0037] Among them, K is the camera intrinsic parameter matrix, R is the rotation matrix, and T is the translation vector

[0038] Threshold setting: If the actual coordinates of the reference object deviate by more than 5% from the calibrated coordinates, an alarm is triggered and recalibration is performed to ensure calibration accuracy. Calibration errors are divided into three levels (warning, slight deviation, and severe deviation), each with corresponding response strategies to promptly detect and address calibration deviations. Calibration records from the last 30 days are stored, supporting fault tracing and facilitating traceability and analysis of the calibration process.

[0039] The dynamic calibration and coordinate conversion module is connected to the deep learning recognition module, which includes a collaborative design of YOLOv5 and ResNet-50 dual models. YOLOv5 is responsible for rapid detection of the wild pipe head (inference speed ≤10ms / frame), and ResNet-50 performs refined coordinate regression on the detection area (output accuracy ±3mm). The hood occlusion (static) defines the invalid area through hood edge detection (Canny algorithm); personnel occlusion (dynamic) shields the interference area based on human body contour recognition (OpenPose). When the occlusion area is ≤30%, Kalman filtering is used to predict the wild pipe movement trajectory; when the occlusion area is greater than 30%, it is marked as "untrusted data", triggering an alarm and abnormal state judgment.

[0040] The specific calculation process is:

[0041] The target score s of the YOLOv5 target detection model is determined by the target confidence c and the category probability p(cls|box), and the confidence is normalized using the Sigmoid function σ.

[0042]

[0043] in, YOLOv5 predicts the center coordinates (x, y), width w and height h of the bounding box on feature maps of different scales. For each predicted box, its coordinate calculation depends on the anchor box and the predicted offset. Assume that the center coordinates of the anchor box are (x a ,y a ), width is w a , height is h a , the predicted offset is (t x ,t y ,t w ,t h ), the center coordinates (x, y) and width and height (w, h) of the predicted box are calculated as follows:

[0044] x=(σ(t x )+c x )·s

[0045] y=(σ(t y )+c y )·s

[0046]

[0047] Here, (c x ,c y ) is the coordinate of the upper left corner of the feature map grid, s is the stride of the feature map. The input image is adjusted to the size required by the model (such as 640×640), and the CSPDarknet backbone network is used to extract features of the input image to obtain feature maps of different scales. On the feature maps of different scales, the convolutional layer is used to predict the offset, confidence, and category probability of each anchor box, and the predicted boxes are screened to remove boxes with high overlap and low scores, and the boxes with the highest scores are retained.

[0048] The feature vector F extracted by ResNet-50 is mapped to three-dimensional coordinates through the fully connected layer

[0049]

[0050] Among them, f θ is a multi-layer perceptron (MLP) with parameters θ.

[0051] Loss function: The mean squared error (MSE) is used as the loss function to train the model.

[0052]

[0053] Among them, X i are real three-dimensional coordinates, is the predicted three-dimensional coordinate, and N is the number of samples.

[0054] The ROI area detected by YOLOv5 is input into ResNet-50, and deep features are extracted through 50 layers of residual blocks. The extracted feature map is globally average pooled and compressed into a feature vector of fixed length. The feature vector is input into a multi-layer perceptron, and the predicted three-dimensional coordinates are output. The model is trained using the mean square error loss function, and the model parameters are updated through back propagation.

[0055] Occlusion processing algorithm static occlusion (hood occlusion):

[0056] The Canny edge detection algorithm includes Gaussian smoothing, gradient calculation, non-maximum suppression and double threshold processing. It uses a Gaussian filter G(x,y) to smooth the image I(x,y) to obtain a smoothed image S(x,y).

[0057] S(x,y) = I(x,y) * G(x,y)

[0058] Calculate the gradient G of the image in the x and y directions x and G y , and then calculate the gradient magnitude G and direction θ.

[0059]

[0060] Perform non-maximum suppression on the gradient magnitude to retain local maxima.

[0061] Set two thresholds T1 and T2 (T1 < T2), mark pixels with gradient magnitude greater than T2 as strong edges, and pixels between T1 and T2 as weak edges. Connect weak edges to strong edges

[0062] Generate a mask M mask (x,y) according to the edge detection result.

[0063]

[0064] Calculation process: Convert the color image of the hood area to a grayscale image, smooth the grayscale image using a Gaussian filter to reduce noise effects, calculate the gradient magnitude and direction of the image, perform non-maximum suppression on the gradient magnitude to retain local maxima, set appropriate thresholds to determine the final edges, generate a mask based on the edge detection result, and mark invalid areas;

[0065] Dynamic occlusion area ratio calculation: Predict human key points K = {k1, k2,..., k 18} through OpenPose, generate a human contour mask M human (x,y), and calculate the occlusion area ratio η.

[0066]

[0067] Among them, M target (x,y) is the mask of the target area. When the occlusion area ratio η ≤ 30%, use Kalman filter to predict the motion trajectory of the fan tube. The Kalman filter includes two steps: prediction and update. Predict the state vector x<00><000035>and covariance matrix P t|t-1 .

[0068] x t|t-1 = Fx t-1|t-1 + Bu t

[0069] P t|t-1 = FP t-1|t-1 F T + Q

[0070] Among them, F is the state transfer matrix, B is the control input matrix, u t is the control input vector, Q is the process noise covariance matrix. According to the observation value z t Update the state vector x t|t and the covariance matrix P t|t .

[0071] y t =z t -Hx t|t-1

[0072] S t =HP t|t-1 H T +R

[0073]

[0074] x t|t =x t|t-1 +K t y t

[0075] P t|t =(IK t H)P t|t-1

[0076] Where H is the observation matrix, R is the observation noise covariance matrix, and y t is the residual vector, S t is the innovation covariance matrix, K t is the Kalman gain matrix; the OpenPose model is used to detect key points of the human body, generate a human silhouette mask, calculate the overlap area between the human silhouette mask and the target area mask, and obtain the occlusion area ratio. Decisions are made based on the occlusion area ratio. If η ≤ 30%, a Kalman filter is used to predict the trajectory of the wild controller. If η > 30%, the data is marked as "untrusted" and an alarm is triggered.

[0077] Puncture determination formula

[0078]

[0079] Where L is the actual length of the rough pipe, measured by the vision system; L 阈值 Dynamically adjust according to the steel type, for example, 600mm for carbon steel and 800mm for stainless steel; use the visual system to measure the actual length L of the rough pipe, compare the measured length L with the threshold L threshold of the corresponding steel type, and make a penetration judgment based on the comparison result.

[0080] Break judgment

[0081] formula

[0082] Breaking judgment = (N≥2)∧(T static>1s)∧(F<0.5F rated)

[0083] Where N is the number of target detections, Tstatic is the static time of the subsequent waste pipe, F is the load observer data of the transmission system, and Frated is the rated load of the transmission system.

[0084] Calculation process: Count the number of unused pipe targets N detected by YOLOv5, monitor the subsequent stationary time T of unused pipes, obtain the load observer data F of the transmission system, and perform logical judgment based on the above three conditions. If all three conditions are met, it is determined to be broken.

[0085] The deep learning recognition module runs as follows:

[0086] S1.YOLOv5 fast detection: Input the input image into the YOLOv5 model, quickly detect the wild pipe head, and output the ROI area.

[0087] S2. ResNet-50 Refinement Regression: The ROI region output by YOLOv5 is input into the ResNet-50 model, and the location of the waste pipe is refined and regressed to output 3D coordinates.

[0088] S3. Occlusion processing: Perform occlusion detection on the detection area, distinguish between static occlusion (hood occlusion) and dynamic occlusion (personnel occlusion), and perform corresponding processing based on the proportion of the occlusion area, such as using Kalman filtering to predict or trigger an alarm.

[0089] S4. Abnormal judgment: According to the punch-through and break judgment formula, the status of the waste pipe is judged and the control signal is output.

[0090] The coordinate conversion module is connected to the control logic and safety interlocking module. The control logic in the control logic and safety interlocking module is implemented using a PLC program. It receives the coordinate data of the visual system (10ms / time), dynamically compares the deviation, and triggers the graded response as shown in the following table:

[0091]

[0092] A fault-tolerant mechanism is also set up, including data credibility verification. When the visual data is abnormal (such as the target is lost for 5 consecutive frames), it switches to the load observer backup mode, and the manual intervention signal priority is greater than automatic control.

[0093] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A visual tracking control device for pipe jacking machine operation, characterized in that: Including: A visual perception module, which includes multiple industrial cameras arranged in a triangular array with a spacing of 2 - 3m, equipped with 850 - 1200nm infrared supplementary lighting and an adaptive exposure algorithm; A dynamic calibration and coordinate transformation module, which performs SIFT feature matching based on fixed brackets in the workshop to establish a transformation matrix from the image coordinate system to the world coordinate system; A deep learning recognition module, which adopts a collaborative architecture of YOLOv5 and ResNet-50 and has static and dynamic occlusion processing algorithms; A control logic and safety interlock module, which realizes hierarchical response control based on visual data.

2. A visual tracking control pipe jacking machine operation device according to claim 1, characterized in that: The industrial cameras meet the requirements that the time synchronization error ≤ 1ms, the image transmission delay ≤ 50ms, use a global shutter, the frame rate ≥ 60fps, and maintain the three-dimensional coordinate reconstruction ability through parallax calculation when a single camera is occluded.

3. A visual tracking control pipe jacking machine operation device according to claim 1, characterized in that: In the dynamic calibration and coordinate transformation module, a transformation matrix from the image pixel coordinate system to the world coordinate system is established. If the deviation between the actual coordinates and the calibrated coordinates of the reference object > 5%, an alarm is triggered and recalibration is performed to achieve accurate transformation from image coordinates to world coordinates.

4. A visual tracking control pipe jacking machine operation device according to claim 3, characterized in that: The transformation model is as follows: Where, K is the camera internal parameter matrix, R is the rotation matrix, and T is the translation vector.

5. A visual tracking control pipe jacking machine operation device according to claim 3, characterized in that: The calibration error is divided into three levels, corresponding to different response strategies, which can timely detect and handle calibration deviation problems, store the calibration records of the last 30 - 60 days, support fault tracing, and facilitate the tracing and analysis of the calibration process.

6. A visual tracking control pipe jacking machine operation device according to claim 1, characterized in that: In the deep learning recognition module, YOLOv5 is responsible for quickly detecting the head of the blank pipe, and ResNet-50 performs refined coordinate regression on the detected area. The invalid area is defined by detecting the edge of the hood for hood occlusion; for personnel occlusion, the interference area is shielded based on human contour recognition. When the occlusion area ≤ 30%, the Kalman filter is used to predict the movement trajectory of the blank pipe; when the occlusion area > 30%, it is marked as untrusted data, an alarm is triggered, and an abnormal state is determined.

7. The visual tracking control pipe jacking machine operation device according to claim 1 is characterized in that: Dynamic occlusion area ratio calculation is performed by predicting the key points of the human body through OpenPose K = {k1, k2, ..., k 18 }, generate human body contour mask M human (x, y), calculate the occlusion area ratio η; Among them, M target (x, y) is the mask of the target area; when the occlusion area ratio η≤30%, the Kalman filter is used to predict the motion trajectory of the fan.

8. A visual tracking control pipe jacking machine operation device according to claim 7, characterized in that: Kalman filtering includes two steps: prediction and update; the predicted state vector x t|t-1 and the covariance matrix P t|t-1 ; x t|t-1 =Fx t-1|t-1 +Bu t P t|t-1 =FP t-1|t-1 F T +Q Among them, F is the state transfer matrix, B is the control input matrix, u t is the control input vector, and Q is the process noise covariance matrix.

9. The visual tracking control pipe jacking machine operation device according to claim 1, characterized in that: The control logic and safety interlock module also sets a fault tolerance mechanism, including data credibility verification. When visual data is abnormal (such as losing the target for 5 consecutive frames), it switches to the standby mode of the load observer, and the priority of the manual intervention signal > automatic control.

10. The visual tracking control pipe jacking machine operation device according to claim 1, characterized in that: Static processing includes the Canny edge detection algorithm, which includes Gaussian smoothing, gradient calculation, non-maximum suppression, and double-threshold processing. The Gaussian filter G(x, y) is used to smooth the image I(x, y) to obtain the smoothed image S(x, y); S(x, y) = I(x, y) * G(x, y) Calculate the gradient G of the image in the x and y directions x and G y , then calculate the gradient magnitude G and direction θ; Perform non-maximum suppression on the gradient magnitude to retain local maxima; Set two thresholds T1 and T2 (T1 < T2), mark the pixels with gradient magnitude greater than T2 as strong edges, and the pixels between T1 and T2 as weak edges, and connect the weak edges to the strong edges.

Images