A shield grease consumption monitoring device and method based on machine vision

Abstract

The present application relates to the technical field of shield construction monitoring, and discloses a shield grease consumption monitoring device and method based on machine vision. The present application aims to solve the technical problem that the traditional method for monitoring grease consumption in the prior art mainly relies on a pull rope type displacement sensor, has a high mechanical failure rate, is difficult to maintain, has data drift, and is susceptible to interference. The present application comprises an image acquisition module, which is communicatively connected to an image processing and algorithm module. The image processing and algorithm module is embedded in an industrial computer or an edge computing device. The image processing and algorithm module comprises a preprocessing unit, an instance segmentation model unit, a length calculation unit, and a calibration and conversion unit. The image processing and algorithm module is communicatively connected to a consumption calculation and output module. Through an instance segmentation algorithm, the present application can effectively distinguish between oil stains, dust, and complex backgrounds, and only identify the pull rod body. The instance segmentation algorithm based on deep learning can accurately identify the pull rod body.

Description

Technical Field

[0001] This invention relates to the field of shield tunneling construction monitoring technology, and in particular to a machine vision-based device and method for monitoring shield grease consumption. Background Technology

[0002] During shield tunnel construction, controlling the amount of grout injected at the tail of the shield is crucial. Insufficient grout will lead to failure of the tail seal, causing groundwater inrush or grout backflow; excessive grout will result in resource waste and environmental pollution.

[0003] Currently, traditional methods for monitoring grease consumption mainly rely on pull-string displacement sensors. These sensors are mounted on top of the grease container, with the end of the pull string connected to the piston rod of the grease pump or a follower plate on the grease surface. As grease is consumed, the pull string is extended, and the sensor calculates the remaining grease or the amount consumed by measuring the length of the extended string. However, existing technologies have the following drawbacks: High mechanical failure rate: The internal environment of the tunnel boring machine is harsh (high humidity, high vibration, and high dust). The springs inside the rope sensor are prone to fatigue and breakage, or the rope may get stuck due to oil and fail to spring back.

[0004] Maintenance difficulties: The sensor is installed inside the grease tank or close to mechanical parts. Once damaged, replacement and recalibration require downtime, which affects the construction progress.

[0005] Data drift: After prolonged use, mechanical wear of contact sensors can lead to a decrease in measurement accuracy and zero drift.

[0006] Measurements are susceptible to interference: the viscosity of grease may cause the pull rope or follower plate to stick together, resulting in measurement errors.

[0007] Therefore, developing a non-contact, interference-resistant, and easy-to-maintain oil consumption monitoring system is of significant engineering importance. Summary of the Invention

[0008] In view of the above technical problems, this disclosure provides a machine vision-based device and method for monitoring the consumption of grease in tunnel boring machines (TBMs), solving the technical problems in the prior art. Utilizing computer vision and deep learning technologies, the system performs non-contact real-time monitoring of the consumption of grease in the tail seal and main bearings of the TBM. Through non-contact image acquisition and intelligent image analysis, the system calculates grease consumption in real time and accurately, exhibiting high reliability, low maintenance costs, and strong environmental adaptability.

[0009] The core concept of this invention is to use a fixed camera to capture real-time images of the movement area of ​​the pneumatic grease pump's lever, accurately identify and segment the lever target using an instance segmentation algorithm, calculate its pixel length, convert it into the actual physical length using a pre-calibrated scaling factor, and finally calculate the actual volume of grease consumed based on the change in lever length and the cross-sectional area of ​​the grease container.

[0010] According to one aspect of this disclosure, a machine vision-based shield grease consumption monitoring device is provided, including an image acquisition module, the image acquisition module including an industrial camera installed on the side or above a pneumatic grease pump, the industrial camera being used to capture real-time motion images of the grease pump lever; The image acquisition module is communicatively connected to the image processing and algorithm module, which is embedded in an industrial computer or edge computing device. The image processing and algorithm module includes a preprocessing unit, an instance segmentation model unit, a length calculation unit, and a calibration and conversion unit. The preprocessing unit performs denoising and contrast enhancement on the input image. The instance segmentation model unit loads and runs a pre-trained deep learning instance segmentation model to segment the pull rod target from the motion image and output a pixel-level mask. The length calculation unit calculates the pixel length of the pull rod in the main axis direction based on the pull rod mask. The calibration and conversion unit converts the pixel length into the actual physical length. The image processing and algorithm module is communicatively connected to the consumption calculation and output module, used to calculate the consumption based on the real-time value L of the physical length of the lever. current With initial full load length L full The difference is used to calculate the change in length AL=L. full -L current， Calculate the volume of grease consumed (V) based on the cross-sectional area S inside the grease container. consumed =AL*S, and output the consumed data through the communication interface.

[0011] In some embodiments of this disclosure, the image acquisition module further includes a fill light deployed next to the industrial camera to provide auxiliary lighting for the lever.

[0012] In some embodiments of this disclosure, the deep learning instance segmentation model pre-trained by the instance segmentation model unit is a YOLOv8-Seg model or a YOLACT model.

[0013] In some embodiments of this disclosure, the consumption calculation and output module establishes a communication connection with the main control room or cloud platform of the tunnel boring machine through an industrial Ethernet or RS485 communication interface to output data including real-time consumption volume, remaining volume and consumption rate.

[0014] In some embodiments of this disclosure, an auxiliary module is also included. The auxiliary module includes a model training dataset and a human-computer interaction interface. The model training dataset includes labeled images of the lever under different lighting, angles, and levels of contamination. The human-computer interaction interface is used for parameter setting, status display, and alarm.

[0015] A machine vision-based method for monitoring shield tunneling grease consumption includes the following steps: S1. Image Acquisition: Acquire a video stream or timed frame image containing the pneumatic grease pump lever; S2. Instance segmentation: Using a pre-trained deep learning instance segmentation model, the lever region in the image is segmented from the background at the pixel level, and the lever mask is extracted. S3. Feature Extraction and Calculation: Morphological processing is performed on the segmented lever mask to extract the physical features of the lever exposed outside the grease bucket, and the pixel length of the lever in the image is converted into the actual physical length based on the camera calibration parameters. S4. Consumption Calculation: Based on the change in the length of the tie rod. Calculate the volume of grease consumed based on the inner cross-sectional area S of the grease container. It accumulates and uploads the consumption data in real time.

[0016] The deep learning instance segmentation model described in S2 is pre-trained using a training dataset consisting of images of grease pump levers under different lighting conditions and different levels of oil contamination, along with their labeled mask regions.

[0017] In some embodiments of this disclosure, the camera calibration parameters in step S3 are obtained in the following manner: With the grease drum fully loaded, measure the actual physical length L of a section of the tie rod. real ref ; In the images acquired during the initial system run, obtain the pixel length L corresponding to this section of the lever. pixel ref ; Calculate the camera calibration parameters K = L real ref / L pixel ref .

[0018] In some embodiments of this disclosure, step S4 further includes: comparing the change in the length of the lever at the current moment with a preset vibration threshold; if the change in length is less than the vibration threshold, it is determined to be invalid data and filtered out; if the change in length is greater than or equal to the vibration threshold, it is determined to be valid data, and the corresponding single oil consumption is calculated.

[0019] In some embodiments of this disclosure, in step S1, the industrial camera is fixedly mounted on a bracket at a horizontal distance of 0.5 to 1.5 meters from the grease pump lever, and the optical axis of the lens is perpendicular to the direction of movement of the lever. The industrial camera acquires motion images of the lever at a fixed frequency.

[0020] In some embodiments of this disclosure, S4 further includes data output: the calculated grease consumption volume, consumption rate and remaining amount data are transmitted in real time to the shield tunnel main control room or cloud platform through an industrial network communication interface and displayed on the local human-machine interface.

[0021] In some embodiments of this disclosure, an anomaly alarm is also included: when the unique valid pull rod target is not successfully segmented during the S2 instance segmentation process, a visual detection anomaly alarm is issued; when the grease consumption rate is abnormal or the remaining grease is lower than a preset threshold, a warning signal is issued.

[0022] The beneficial effects of this invention are as follows: Non-contact measurement, high reliability: The camera does not directly contact grease or mechanical moving parts, completely eliminating mechanical failures such as rope breakage and jamming. It adopts non-contact vision measurement, without any physical contact with moving parts, fundamentally eliminating sensor failure problems caused by mechanical wear, jamming, and corrosion, and is particularly suitable for the harsh construction environment of tunnel boring machines.

[0023] High measurement accuracy and strong anti-interference capability: Through instance segmentation algorithm, it can effectively distinguish oil stains, dust and complex backgrounds, and only identify the main body of the pull rod. The deep learning-based instance segmentation algorithm can accurately identify the main body of the pull rod and effectively overcome interference such as complex backgrounds, oil stains and reflections on the pull rod surface, and changes in ambient light. It is more robust than traditional methods such as simple color recognition or edge detection.

[0024] Easy to install and maintain: The camera is mounted on the external bracket of the grease pump, so the normal operation of the grease pump is not affected when maintaining or wiping the lens, and there is no need to stop the machine.

[0025] Visual verification: The system can save image snapshots of the operation process. When the data is abnormal, the photos can be retrieved manually for intuitive verification, which is something that traditional sensors cannot do.

[0026] Achieve true real-time and remote monitoring: Data is digitized, making it easy to transmit, store, and analyze, providing real-time data support and decision-making basis for construction management. Attached Figure Description

[0027] Figure 1 The flowchart shows the logic for monitoring shield grease consumption based on machine vision. Figure 2 A flowchart of the shield tunneling grease consumption monitoring process based on machine vision; Detailed Implementation

[0028] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Example 1

[0029] This example discloses a machine vision-based device for monitoring the grease consumption of tunnel boring machines. See [link to relevant documentation]. Figures 1 to 2 , It includes an image acquisition module, which includes an industrial camera mounted on the side or above the pneumatic grease pump. The industrial camera is used to capture real-time images of the movement of the grease pump lever. The image acquisition module is communicatively connected to the image processing and algorithm module, which is embedded in an industrial computer or edge computing device. The image processing and algorithm module includes a preprocessing unit, an instance segmentation model unit, a length calculation unit, and a calibration and conversion unit. The preprocessing unit performs denoising and contrast enhancement on the input image. The instance segmentation model unit loads and runs a pre-trained deep learning instance segmentation model to segment the pull rod target from the moving image and output a pixel-level mask. The length calculation unit calculates the pixel length of the pull rod along its main axis based on the pull rod mask. The calibration and conversion unit converts the pixel length into the actual physical length. The image processing and algorithm module communicates with the consumption calculation and output module to calculate the consumption based on the real-time value L of the physical length of the lever. current With initial full load length L full The difference is used to calculate the change in length AL=L. full -L current， Calculate the volume of grease consumed (V) based on the cross-sectional area S inside the grease container. consumed =AL*S, and output the consumed data through the communication interface.

[0030] The image acquisition module also includes a supplementary light deployed next to the industrial camera to provide auxiliary lighting for the lever.

[0031] The deep learning instance segmentation model pre-trained by the instance segmentation model unit is either the YOLOv8-Seg model or the YOLACT model.

[0032] The consumption calculation and output module establishes a communication connection with the main control room or cloud platform of the tunnel boring machine through an industrial Ethernet or RS485 communication interface to output data including real-time consumption volume, remaining volume and consumption rate.

[0033] It also includes an auxiliary module, which includes a model training dataset and a human-computer interaction interface. The model training dataset includes labeled images of the lever under different lighting, angles, and pollution levels. The human-computer interaction interface is used for parameter settings, status display, and alarms.

[0034] A machine vision-based method for monitoring shield tunneling grease consumption includes the following steps: S1. Image Acquisition: Acquire a video stream or timed frame image containing the pneumatic grease pump lever; S2. Instance segmentation: Using a pre-trained deep learning instance segmentation model, the lever region in the image is segmented from the background at the pixel level, and the lever mask is extracted. S3. Feature Extraction and Calculation: Morphological processing is performed on the segmented lever mask to extract the physical features of the lever exposed outside the grease bucket, and the pixel length of the lever in the image is converted into the actual physical length based on the camera calibration parameters. S4. Consumption Calculation: Based on the change in the length of the tie rod. Calculate the volume of grease consumed based on the inner cross-sectional area S of the grease container. It accumulates and uploads the consumption data in real time.

[0035] The deep learning instance segmentation model in S2 is pre-trained using a training dataset consisting of images of grease pump levers under different lighting conditions and different levels of oil contamination, along with their labeled mask regions.

[0036] The camera calibration parameters in S3 are obtained in the following way: With the grease drum fully loaded, measure the actual physical length L of a section of the tie rod. real ref ; In the images acquired during the initial system run, obtain the pixel length L corresponding to this section of the lever. pixel ref ; Calculate the camera calibration parameters K = L real ref / L pixel ref .

[0037] S4 also includes: comparing the change in the length of the lever at the current moment with a preset vibration threshold; if the change in length is less than the vibration threshold, it is determined to be invalid data and filtered out; if the change in length is greater than or equal to the vibration threshold, it is determined to be valid data and the corresponding single oil consumption is calculated.

[0038] In S1, the industrial camera is fixedly mounted on a bracket at a horizontal distance of 0.5 to 1.5 meters from the grease pump pull rod, and the optical axis of the lens is perpendicular to the direction of movement of the pull rod. The industrial camera acquires motion images of the pull rod at a fixed frequency.

[0039] S4 also includes data output: the calculated grease consumption volume, consumption rate and remaining amount data are transmitted in real time to the shield tunnel main control room or cloud platform through the industrial network communication interface and displayed on the local human-machine interface.

[0040] It also includes anomaly alarms: when the unique valid lever target is not successfully segmented during the S2 instance segmentation process, a visual detection anomaly alarm is issued; when the grease consumption rate is abnormal or the remaining grease is lower than a preset threshold, a warning signal is issued.

[0041] Hardware deployment Camera selection: Select an industrial camera with a dustproof and waterproof housing (IP65 rating), a resolution of at least 1920x1080, and a fixed-focus lens.

[0042] Installation location: The camera should be fixed on a bracket at a horizontal distance of 0.5 to 1.5 meters from the grease pump lever, and the lens optical axis should be kept as perpendicular as possible to the direction of lever movement.

[0043] Auxiliary lighting: LED explosion-proof supplementary lights are deployed next to the camera to ensure that the outline of the tie rod is clear in the dim environment at the bottom of the tunnel boring machine trolley.

[0044] Calibration reference: A calibration block of known size (such as a black and white checkerboard) is attached to a fixed stationary part next to the lever to calculate the "pixel to millimeter" conversion coefficient in real time and eliminate the slight distance variation error caused by vibration.

[0045] The industrial camera and supplementary lighting were mounted on a sturdy, vibration-free bracket near the pneumatic grease pump. The angle and focus were adjusted to ensure that the field of view fully covered the entire stroke of the lever from "extended" (grease full) to "retracted" (grease depleted). The camera was connected via cable to an industrial control computer located in a relatively clean area of ​​the tunnel boring machine.

[0046] The system operation process is as follows: 1. System startup and initialization: Load the pre-trained instance segmentation model and read system parameters (such as calibration coefficient K, full load length Lfull, and barrel cross-sectional area S).

[0047] 2. Image Acquisition: The camera acquires video streams at a fixed frequency (e.g., 1Hz).

[0048] 3. Keyframe Extraction and Preprocessing: Extract valid frames (e.g., 1 frame per second) based on the lever's movement speed. Preprocess the image to improve quality. 4. Instance Segmentation: Input the preprocessed image into the instance segmentation model. The model outputs a pixel-level mask of the "lever" target in the image. 5. Target Determination: Determine whether a unique and valid lever target has been successfully segmented. If it fails, issue a "visual detection anomaly" alarm, indicating possible lens damage or abnormal lighting.

[0049] 6. Length Calculation and Conversion: For a successful mask, calculate its pixel length L-pixel along the main axis. Using a pre-calibrated scaling factor K, calculate the actual physical length Lreal of the current lever.

[0050] 7. Consumption Calculation: Compare Lreal with the initial full-load length LfulI recorded by the system to obtain the change AL. Multiply AL by the known internal cross-sectional area S of the grease container to obtain the cumulative grease consumption volume V-consumed since monitoring began. The consumption rate per unit time can then be calculated.

[0051] 8. Data Output: The calculation results (real-time length, consumed volume, consumed rate) are sent to the tunnel boring machine's main control room via the industrial network and displayed on the local human-machine interface. An early warning is issued when the consumed rate is abnormal or the remaining quantity falls below the threshold.

[0052] 9. Loop Monitoring: The system returns to step 2 and continues to acquire and process the next frame of image to achieve real-time monitoring. 1. Model training phase: Images of the grease pump lever under different lighting conditions and varying levels of oil contamination were acquired. The "lever" region in the images was manually labeled. An instance segmentation network (e.g., using a lightweight YOLOv8-Seg model for edge computing devices) was trained to accurately output the pixel mask of the lever.

[0053] Training the instance segmentation model needs to be done offline beforehand. A large number of images of grease pump pull rods taken at different construction sites, under different lighting conditions, and with varying levels of grease contamination are collected. The precise outlines of the pull rods are manually annotated to form a training dataset. This dataset is then used to train and optimize the selected instance segmentation network until it achieves satisfactory accuracy and recall on the independent test set. Finally, the model is deployed to the algorithm module on-site.

[0054] 2. Calibration Process: With the grease tank fully loaded, manually measure the actual length of a section of the exposed pull rod (e.g., the distance from a fixed reference point to the end of the piston rod, denoted as L real ref). In the initial system running image, select the same section of the pull rod and record its pixel length L pixel ref. The calibration coefficient K = L real ref / L pixel ref. This coefficient remains unchanged after the camera position and focal length are fixed.

[0055] 2. Real-time monitoring phase: Step S1: Image Acquisition. The camera captures images at a fixed frequency (e.g., 1 frame per second).

[0056] Step S2: Distortion Correction and Preprocessing. The image is denoised and lens distortion corrected.

[0057] Step S3: Instance segmentation inference. Input the model, output the mask region of the lever.

[0058] Step S4: Key Point Location. In the Mask area, identify the "top endpoint" and "root entry point" of the lever.

[0059] Step S5: Length Calculation. Calculate the pixel distance P between the top and bottom. Using the calibration coefficient K (mm / pixel), determine the current exposed length of the lever. .

[0060] Step S6: Consumption calculation.

[0061] Let the length of the previous time step be Llast.

[0062] Length change .

[0063] like If the vibration level is less than the set vibration threshold, it is ignored (filtered); if valid, the single-cycle consumption is calculated. (r is the radius of the grease container).

[0064] Step S7: Data Output. Send the results to the tunnel boring machine PLC system via Modbus or Ethernet.

[0065] Although some preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0066] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A machine vision-based device for monitoring the consumption of grease in tunnel boring machines, characterized in that: The system includes an image acquisition module, which includes an industrial camera mounted on the side or above the pneumatic grease pump. The industrial camera is used to capture real-time images of the movement of the grease pump lever. The image acquisition module is communicatively connected to the image processing and algorithm module, which is embedded in an industrial computer or edge computing device. The image processing and algorithm module includes a preprocessing unit, an instance segmentation model unit, a length calculation unit, and a calibration and conversion unit. The preprocessing unit is used to perform noise reduction and contrast enhancement processing on the input image. The instance segmentation model unit loads and runs a pre-trained deep learning instance segmentation model to segment the pull rod target from the motion image and output a pixel-level mask; the length calculation unit calculates the pixel length of the pull rod in the main axis direction based on the pull rod mask; the calibration and conversion unit is used to convert the pixel length into the actual physical length. The image processing and algorithm module is communicatively connected to the consumption calculation and output module, used to calculate the consumption based on the real-time value L of the physical length of the lever. current With initial full load length L full The difference is used to calculate the change in length AL=L. full -L current， Calculate the volume of grease consumed (V) based on the cross-sectional area S inside the grease container. consumed =AL*S, and output the consumed data through the communication interface.

2. The shield tunneling grease consumption monitoring device based on machine vision as described in claim 1, characterized in that: The image acquisition module also includes a fill light deployed next to the industrial camera to provide auxiliary lighting for the lever.

3. The shield tunneling grease consumption monitoring device based on machine vision as described in claim 1, characterized in that: The deep learning instance segmentation model pre-trained by the instance segmentation model unit is either the YOLOv8-Seg model or the YOLACT model.

4. The shield tunneling grease consumption monitoring device based on machine vision as described in claim 1, characterized in that: The consumption calculation and output module establishes a communication connection with the main control room or cloud platform of the tunnel boring machine through an industrial Ethernet or RS485 communication interface to output data including real-time consumption volume, remaining volume and consumption rate.

5. The shield grease consumption monitoring device based on machine vision as described in claim 1, characterized in that: It also includes an auxiliary module, which includes a model training dataset and a human-computer interaction interface. The model training dataset includes labeled images of the lever under different lighting, angles, and levels of contamination. The human-computer interaction interface is used for parameter setting, status display, and alarms.

6. A machine vision-based method for monitoring shield tunneling grease consumption, applicable to the machine vision-based shield tunneling grease consumption monitoring device as described in claim 1, characterized in that, Includes the following steps: S1. Image Acquisition: Acquire a video stream or timed frame image containing the pneumatic grease pump lever; S2. Instance segmentation: Using a pre-trained deep learning instance segmentation model, the lever region in the image is segmented from the background at the pixel level, and the lever mask is extracted. S3. Feature Extraction and Calculation: Morphological processing is performed on the segmented lever mask to extract the physical features of the lever exposed outside the grease bucket, and the pixel length of the lever in the image is converted into the actual physical length based on the camera calibration parameters. S4. Consumption Calculation: Based on the change in the length of the tie rod. Calculate the volume of grease consumed based on the inner cross-sectional area S of the grease container. It accumulates and uploads the consumption data in real time.

7. The method for monitoring shield grease consumption based on machine vision as described in claim 6, characterized in that: The deep learning instance segmentation model described in S2 is pre-trained using a training dataset consisting of images of grease pump levers under different lighting conditions and different levels of oil contamination, along with their labeled mask regions. It also includes anomaly alarms: when the unique valid lever target is not successfully segmented during the instance segmentation process in S2, a visual detection anomaly alarm is issued; when the grease consumption rate is abnormal or the remaining grease is below a preset threshold, a warning signal is issued.

8. The method for monitoring shield grease consumption based on machine vision as described in claim 6, characterized in that: The camera calibration parameters in S3 are obtained by measuring the actual physical length L of a section of the pull rod while the grease drum is fully loaded. real ref In the images acquired during the initial system operation, obtain the pixel length L corresponding to that section of the lever. pixel ref ; Calculate the camera calibration parameters K = L real ref / L pixel ref .

9. The method for monitoring shield tunneling grease consumption based on machine vision as described in claim 6, characterized in that: S4 further includes: comparing the change in the length of the lever at the current moment with a preset vibration threshold; if the change in length is less than the vibration threshold, it is determined to be invalid data and filtered out; if the change in length is greater than or equal to the vibration threshold, it is determined to be valid data, and the corresponding single oil consumption is calculated. S4 also includes data output: the calculated grease consumption volume, consumption rate and remaining amount data are transmitted in real time to the shield tunnel main control room or cloud platform through the industrial network communication interface and displayed on the local human-machine interface.

10. The method for monitoring shield grease consumption based on machine vision as described in claim 6, characterized in that: In S1, the industrial camera is fixedly mounted on a bracket at a horizontal distance of 0.5 to 1.5 meters from the grease pump pull rod, and the optical axis of the lens is perpendicular to the direction of movement of the pull rod. The industrial camera acquires motion images of the pull rod at a fixed frequency.

Images