Shield attitude control method and system based on machine vision

Abstract

The invention discloses a shield attitude control method and system based on machine vision, and the method comprises the steps: constructing a motion state model of a shield tunneling machine according to the motion track of a feature point, and carrying out the prediction and correction of a motion state through a Kalman filtering algorithm, and obtaining the real-time attitude data of the shield tunneling machine; vibration data of the shield tunneling machine are obtained, the vibration data are obtained based on a vibration compensation model of an inertial measurement unit, the vibration data and visual sensor data are combined, a data fusion algorithm based on particle filtering is adopted to suppress vibration interference, and stable attitude information is obtained; and constructing a multi-modal data fusion model based on a graph convolutional neural network, and fusing the visual data, the inertial data and the motion state data to obtain a high-precision attitude estimation result.

Description

Technical Field

[0001] The present invention belongs to the technical field of shield machine posture control, and in particular relates to a shield machine posture control method and system based on machine vision. Background Art

[0002] During the shield construction process, the attitude control of the shield machine is crucial and directly affects the quality and efficiency of excavation. Traditional manual monitoring and control methods have problems such as slow response and insufficient precision. In order to achieve real-time and accurate attitude monitoring and control, introducing a method based on machine vision is a feasible technical route. However, in practical applications, machine vision technology faces many challenges and difficulties. First, the shield construction environment is complex, the lighting conditions are poor, the shooting angle is limited, and it is difficult to obtain high-quality images. Secondly, the structure of the shield machine is complex, the key feature points are easily blocked, and the feature extraction and matching accuracy are difficult to guarantee. Furthermore, the motion state of the shield machine is changeable, including translation, steering, rolling, etc., and it is difficult to fully characterize its attitude changes from a single perspective. In addition, the shield machine vibrates frequently, the stability of the visual sensor is difficult to maintain, and the data quality is easily affected. Finally, the real-time requirements are high, and the processing and analysis of massive visual data pose challenges to the algorithm efficiency and hardware performance. How to stably collect high-quality visual data in harsh environments, how to extract and match key features to accurately estimate the posture of the shield machine, how to integrate multi-perspective information to comprehensively characterize the motion state of the shield machine, how to suppress vibration interference to improve data quality, and how to improve algorithm efficiency to achieve real-time control, these are all technical challenges that need to be overcome urgently. Summary of the invention

[0003] The present invention proposes a shield machine posture control method and system based on machine vision to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above object, the present invention provides a shield posture control method based on machine vision, comprising the following steps:

[0005] Acquire the image of the surrounding environment of the shield machine by using multi-spectral imaging technology, correct the image of the surrounding environment of the shield machine to obtain a corrected image; acquire the image of the shield machine at different angles by using a multi-view visual sensor array;

[0006] A feature extraction network based on deep learning is used to identify the key feature points of the shield machine in the correction image and the shield machine image. The feature points in adjacent frame images are matched by a feature matching method based on the RANSAC algorithm to obtain the motion trajectory of the feature points.

[0007] According to the motion trajectory of the characteristic points, a motion state model of the shield machine is constructed, and the motion state is predicted and corrected by using a Kalman filter algorithm to obtain predicted posture data of the shield machine;

[0008] The vibration data of the shield machine is obtained through the vibration compensation model of the inertial measurement unit, the vibration data is combined with the visual sensor data, and the data fusion algorithm based on particle filtering is used to suppress vibration interference to obtain stable posture information;

[0009] Construct a multimodal data fusion model based on graph convolutional neural network to fuse visual data, inertial data, stable posture information and predicted posture data to obtain posture estimation results;

[0010] According to the posture estimation result and the preset excavation trajectory, an adaptive control algorithm is used to generate control instructions to perform precise posture control.

[0011] Preferably, correcting the image of the surrounding environment of the shield machine includes:

[0012] Performing denoising and image enhancement processing on the image of the surrounding environment of the shield machine to obtain a preprocessed image;

[0013] According to the preset illumination compensation model, the preprocessed image is subjected to histogram equalization processing, the contrast and brightness distribution of the image are adjusted, and an illumination compensation image is obtained;

[0014] Adopting adaptive gamma correction algorithm, the gamma value is adaptively adjusted according to the local features of the preprocessed image to enhance the dark area of ​​the image;

[0015] The multi-band images processed by histogram equalization and adaptive gamma correction are fused to obtain the corrected image.

[0016] Preferably, using the Kalman filter algorithm to predict the motion state includes:

[0017] The actual measurement value of the shield machine at the current moment is obtained, and the posterior estimation value of the motion state of the shield machine at the current moment is determined by combining the prior estimation value and the measurement value through the updating step of the Kalman filter algorithm; it is judged whether the residual between the posterior estimation value and the prior estimation value exceeds the preset threshold value. If it exceeds the threshold value, it is considered that the motion state of the shield machine has changed suddenly and the early warning mechanism is triggered; according to the posterior estimation value, the state transfer matrix is ​​used to predict the prior estimation value of the motion state of the shield machine at the next moment, and the prior estimation value is used as the input of the next iteration; the posterior estimation values ​​of multiple consecutive moments constitute the time series of the motion state of the shield machine, and the changing trend of the motion state of the shield machine is judged through the time series analysis algorithm.

[0018] Preferably, obtaining the posture estimation result includes:

[0019] A multimodal data fusion model is constructed based on graph convolutional neural network. The node features are updated and fused through graph convolution operation to capture the correlation between different modal data. The attention mechanism is introduced into the graph convolutional neural network to adaptively assign weights to different nodes, highlight the role of key nodes, and improve the fusion effect. By stacking multiple layers of graph convolutional neural network, the high-level semantic features of multimodal data are gradually extracted and fused to obtain the global feature representation. The global feature representation is input into the fully connected layer, and the posture estimation result is obtained after nonlinear transformation and regression prediction.

[0020] The present invention also provides a shield attitude control system based on machine vision, comprising:

[0021] The data acquisition module is used to obtain the image of the surrounding environment of the shield machine through multi-spectral imaging technology, correct the image of the surrounding environment of the shield machine to obtain a corrected image; and obtain the image of the shield machine at different angles through a multi-view visual sensor array;

[0022] The feature point motion trajectory acquisition module is used to identify the key feature points of the shield machine in the correction image and the shield machine image through a feature extraction network based on deep learning, and to match the feature points in adjacent frame images through a feature matching method based on the RANSAC algorithm to obtain the feature point motion trajectory;

[0023] A motion state modeling module is used to construct a motion state model of the shield machine according to the motion trajectory of the feature points, and use a Kalman filter algorithm to predict and correct the motion state to obtain predicted posture data of the shield machine;

[0024] A vibration data fusion module is used to obtain the vibration data of the shield machine through the vibration compensation model of the inertial measurement unit, combine the vibration data with the visual sensor data, and use a data fusion algorithm based on particle filtering to suppress vibration interference and obtain stable posture information;

[0025] The multimodal data fusion module is used to build a multimodal data fusion model based on graph convolutional neural network, which fuses visual data, inertial data, stable posture information and predicted posture data to obtain posture estimation results;

[0026] The posture control module is used to generate control instructions using an adaptive control algorithm based on the posture estimation result and the preset excavation trajectory to perform precise posture control.

[0027] Preferably, the data acquisition module includes:

[0028] The acquisition unit is used to obtain images of the surrounding environment of the shield machine through multi-spectral imaging technology, and to obtain images of the shield machine at different angles through a multi-view visual sensor array;

[0029] The correction unit is used to perform correction processing on the image of the surrounding environment of the shield machine.

[0030] Preferably, the correction unit comprises:

[0031] A preprocessing unit, used for performing denoising and image enhancement processing on the image of the surrounding environment of the shield machine to obtain a preprocessed image;

[0032] An illumination compensation unit, used to perform histogram equalization processing on the preprocessed image according to a preset illumination compensation model, adjust the contrast and brightness distribution of the image, and obtain an illumination compensated image;

[0033] A gamma correction unit, which is used to use an adaptive gamma correction algorithm to adaptively adjust the gamma value according to the local features of the preprocessed image, so as to enhance the dark area of ​​the image;

[0034] The fusion unit is used to fuse the multi-band images processed by histogram equalization and adaptive gamma correction to obtain a corrected image.

[0035] Preferably, the feature point motion trajectory acquisition module includes:

[0036] A feature extraction unit, used for identifying key feature points of the shield machine in the correction image and the shield machine image through a feature extraction network based on deep learning;

[0037] The feature matching unit is used to match feature points in adjacent frame images by a feature matching method based on the RANSAC algorithm.

[0038] The present invention also provides a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0039] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method are implemented.

[0040] Compared with the prior art, the present invention has the following advantages and technical effects:

[0041] The present invention discloses a shield attitude control method and system based on machine vision. The method obtains the image of the surrounding environment of the shield machine through multi-spectral imaging technology, and corrects it in combination with the illumination compensation model. The deep learning feature extraction network and the RANSAC algorithm are used to perform feature matching on the multi-view images to construct a motion state model. At the same time, the vibration data of the inertial measurement unit is fused, and the particle filter algorithm is used to suppress interference. Finally, based on the multimodal data fusion model of the graph convolutional neural network, the visual, inertial and motion state data are fused to obtain a high-precision attitude estimation result. According to the real-time attitude estimation and the preset excavation trajectory, the adaptive control algorithm is used to generate control instructions, adjust the shield machine propulsion and guidance system, and realize precise attitude control. The present invention effectively improves the attitude control accuracy and construction efficiency of the shield machine in a complex geological environment through multi-source data fusion and intelligent algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Embodiment 1

[0047] like Figure 1 As shown, this embodiment provides a shield posture control method based on machine vision, including the following steps:

[0048] Acquire the image of the surrounding environment of the shield machine by using multi-spectral imaging technology, correct the image of the surrounding environment of the shield machine to obtain a corrected image; acquire the image of the shield machine at different angles by using a multi-view visual sensor array;

[0049] A feature extraction network based on deep learning is used to identify the key feature points of the shield machine in the correction image and the shield machine image. The feature points in adjacent frame images are matched by a feature matching method based on the RANSAC algorithm to obtain the motion trajectory of the feature points.

[0050] According to the motion trajectory of the characteristic points, a motion state model of the shield machine is constructed, and the motion state is predicted and corrected by using a Kalman filter algorithm to obtain predicted posture data of the shield machine;

[0051] The vibration data of the shield machine is obtained through the vibration compensation model of the inertial measurement unit, the vibration data is combined with the visual sensor data, and the data fusion algorithm based on particle filtering is used to suppress vibration interference to obtain stable posture information;

[0052] Construct a multimodal data fusion model based on graph convolutional neural network to fuse visual data, inertial data, stable posture information and predicted posture data to obtain posture estimation results;

[0053] According to the posture estimation result and the preset excavation trajectory, an adaptive control algorithm is used to generate control instructions to perform precise posture control.

[0054] The specific implementation steps are as follows:

[0055] Step S101, obtaining a first image of the surrounding environment of the shield machine, wherein the first image is obtained by using multispectral imaging technology, and illumination correction is performed on the first image in combination with a preset illumination compensation model to obtain a second image, wherein the illumination compensation model is based on histogram equalization and adaptive gamma correction, and is used to correct illumination non-uniformity for images of different bands.

[0056] Specifically, a multispectral image of the surrounding environment of the shield machine is obtained, and preprocessing is performed on images of different bands, including denoising, enhancement and other operations, to obtain preprocessed multiband images; according to a preset illumination compensation model, the preprocessed multiband images are subjected to histogram equalization processing, and the contrast and brightness distribution of the images are adjusted to make the brightness distribution of the images in various areas more uniform; on the basis of histogram equalization, an adaptive gamma correction algorithm is further adopted to adaptively adjust the gamma value according to the local features of the image, and the dark areas of the image are enhanced to make the overall brightness distribution of the image more balanced; the multiband images subjected to histogram equalization and adaptive gamma correction processing are fused to obtain high-quality images after illumination correction.

[0057] For example, when the shield machine is working underground, the surrounding environment is complex and changeable, and obtaining high-quality environmental images is crucial for safe construction. Multispectral imaging technology can capture image information in different bands, such as visible light, infrared, etc., which helps to fully understand the surrounding environment. Preprocessing these multi-band images is the first step to improve image quality, including removing noise and enhancing details. For example, median filtering can be used to remove salt and pepper noise, Gaussian filtering can be used to smooth the image, and then the edge details can be enhanced by sharpening algorithms. Light compensation is the key to solving the problem of insufficient light in underground environments. Histogram equalization can effectively adjust the contrast and brightness distribution of the image. In specific implementation, the grayscale histogram of the image can be calculated first, and then the original grayscale value can be mapped to the new grayscale range through the cumulative distribution function, so that the brightness distribution of the image is more uniform. The processed image can highlight the details that were originally difficult to identify in the dark. Adaptive gamma correction further optimizes the brightness distribution of the image. The algorithm dynamically adjusts the gamma value according to the brightness characteristics of the local area of ​​the image. For example, for darker areas, a smaller gamma value (such as 0.6) can be used to enhance the brightness; for brighter areas, a larger gamma value (such as 1.2) can be used to suppress overexposure. This method can effectively improve the visibility of dark details while maintaining the overall contrast of the image. Multi-band image fusion is the process of integrating information from different bands into a high-quality image. The weighted average method can be used to assign different weights to each band according to the amount of information and importance. For example, in an environment with weak visible light, the weight of the infrared band can be increased to highlight the outline of heat sources and obstacles. Image feature extraction is the basis for identifying potential risks in the environment. The Sobel operator can be used to detect edges, and the texture features can be analyzed through the grayscale co-occurrence matrix. Combined with the pre-established shield machine surrounding environment model, it can be determined whether there are abnormal conditions. For example, if a sharp edge is detected suddenly, it may mean that the geological structure has changed or an obstacle has appeared. Based on the detected abnormality, the system needs to evaluate its impact on the construction of the shield machine. A risk assessment model can be established, taking factors such as the location, size, and shape of the abnormality as input and outputting a risk level. For example, a large obstacle within 5 meters in front of the shield machine may be judged as high risk, requiring immediate shutdown and manual inspection. The generation and transmission of early warning information is a key link in ensuring construction safety. The system can automatically generate corresponding early warning information according to the risk level, such as "a suspected pipeline is found 5 meters ahead, it is recommended to slow down and conduct detection." This information is transmitted to the shield machine control system through a real-time communication network, so that the operator can take timely countermeasures. In order to adapt to different construction environments, the system needs to have self-learning capabilities. The parameters of the illumination compensation model and the environmental perception model can be continuously optimized by recording the accuracy of each early warning and the feedback of the control system.For example, if the system repeatedly misidentifies a certain type of texture as an obstacle, it can adjust the threshold of the feature extraction algorithm to improve the accuracy of recognition. This dynamic adjustment mechanism enables the shield machine to maintain efficient and safe construction under various complex geological conditions.

[0058] Step S102: for the second image and the third image, a feature extraction network based on deep learning is used to identify the key feature points of the shield machine, and the feature points in adjacent frame images are matched by a feature matching method based on the RANSAC algorithm to obtain the motion trajectory of the feature points. The third image is obtained from the deployed multi-view visual sensor array to obtain images of the shield machine at different angles.

[0059] Specifically, the second image and the third image are acquired, wherein the third image is acquired by a multi-view visual sensor array deployed on the shield machine to obtain images of the shield machine at different angles. The second image and the third image are processed by a feature extraction network based on deep learning to identify the key feature points of the shield machine. According to the identified feature points, the feature points in adjacent frame images are matched using a feature matching method based on the RANSAC algorithm. The motion trajectory of the feature points between adjacent frame images is obtained through the feature point matching results. According to the motion trajectory of the feature points, the motion state and posture change of the shield machine are judged. If the motion trajectory is abnormal, it is judged that the shield machine may have a risk of failure. Images of the shield machine at multiple different angles are acquired, and the three-dimensional motion state of the shield machine is further determined by comparing and analyzing the position changes of the feature points in images at different angles.

[0060] For example, during the construction of a shield machine, it is crucial to monitor its motion state and posture changes in real time. By acquiring the second image and the third image, the external environment and the state of the shield machine can be fully captured. The second image is corrected for illumination and provides clear environmental information, while the third image is acquired by a multi-view visual sensor array, which can present detailed views of the shield machine from different angles. When processing these images, the deep learning feature extraction network can identify the key feature points of the shield machine. For example, the edge contour of the shield machine cutter head, the spiral blades of the screw conveyor, the connection points of the thrust cylinder, etc. can be identified. The accurate positioning of these feature points lays the foundation for subsequent analysis. Feature matching is a key step in judging the motion state of the shield machine. The RANSAC algorithm plays an important role in this process. It can effectively remove erroneous matching points and improve matching accuracy. Assuming that 100 feature points on the edge of the shield machine cutter head are identified in two adjacent frames of images, the RANSAC algorithm may remove 20 erroneous matching points and retain 80 reliable matching pairs. By analyzing the motion trajectory of these matching feature points, the motion state of the shield machine can be inferred. For example, if the motion trajectory of the feature point on the edge of the cutter disc shows regular circular motion, it means that the cutter disc rotates normally; if the motion trajectory of the connection point of the thrust cylinder shows linear displacement, it means that the shield machine is moving forward. Abnormal motion trajectories may indicate that the shield machine is at risk of failure. For example, if the motion trajectory of the feature point of the screw conveyor blade suddenly becomes irregular or stagnant, it may mean that the conveyor is blocked or damaged. At this time, the system will issue an early warning to remind the operator to check the status of the equipment. Multi-angle image analysis can more accurately determine the three-dimensional motion state of the shield machine. By comparing the position changes of the same feature point in images from different angles, the pitch angle, yaw angle and other attitude parameters of the shield machine can be calculated. For example, if the relative position of the feature points of the shield machine fuselage changes significantly in the images from the top and side perspectives, it may indicate that the shield machine is tilted or twisted.

[0061] Step S103, constructing a shield machine motion state model according to the motion trajectory of the feature points, using a Kalman filter algorithm to predict and correct the motion state, and obtaining real-time posture data of the shield machine.

[0062] Specifically, the motion trajectory data of the characteristic points of the shield machine are obtained, and the data are preprocessed to remove outliers and noise interference, so as to obtain a stable and reliable motion trajectory data set. According to the preprocessed motion trajectory data, the motion state model of the shield machine is established, and a suitable mathematical model is used to describe the motion characteristics of the shield machine, such as state variables such as position, velocity, and acceleration. The relevant parameters of the Kalman filter algorithm are initialized, including the state transfer matrix, observation matrix, process noise covariance matrix, and observation noise covariance matrix, etc., to ensure the convergence and stability of the algorithm. The real-time motion trajectory data of the characteristic points of the shield machine are obtained at each moment as the observation value of the Kalman filter algorithm, which is used to update the estimated value of the motion state of the shield machine. Through the prediction step of the Kalman filter algorithm, the state estimation value and state transfer matrix of the previous moment are used to predict the prior estimation value of the motion state of the shield machine at the current moment. Using the observation value and observation matrix at the current moment, the prior estimation value is corrected through the correction step of the Kalman filter algorithm to obtain the posterior estimation value as the optimal estimation of the motion state of the shield machine at the current moment. The estimated value of the shield machine's motion state obtained by the Kalman filter algorithm is converted into attitude data, including position, direction and other information, and the attitude data of the shield machine is output in real time for subsequent control and decision-making.

[0063] Exemplarily, the preprocessing of the motion trajectory data of the characteristic points of the shield machine is a key step to ensure the accuracy of subsequent analysis. By removing outliers and noise interference, a more stable and reliable data set can be obtained. For example, the median filter method can be used to remove sudden noise, and the sliding average method can be used to smooth the data curve. In practical applications, suppose that the position data of a shield machine suddenly jumps from the coordinates (10, 20, 30) to (100, 200, 300) instantly. This is obviously an outlier and needs to be eliminated by the preprocessing algorithm. Establishing a shield machine motion state model is the basis for describing its motion characteristics. A linear motion model or a nonlinear motion model can be used, and a suitable model can be selected according to the actual working characteristics of the shield machine. For example, for a shield machine with uniform linear motion, a simple linear model can be used; for a shield machine under complex geological conditions, nonlinear factors such as the influence of stratum changes on motion may need to be considered. The parameter initialization of the Kalman filter algorithm is crucial to the performance of the algorithm. The state transfer matrix describes the transformation relationship of the shield machine from one state to the next, and the observation matrix reflects the relationship between the observation value and the state variable. For example, for the position-velocity model, the state transfer matrix may contain the update relationship between position and velocity, while the observation matrix may only contain position information. The process noise covariance matrix and the observation noise covariance matrix represent the uncertainty in the system model and the observation process, respectively. When acquiring the motion trajectory data of the characteristic points of the shield machine in real time, the multi-sensor fusion technology can be used to improve the reliability of the data. For example, combining the data of multiple sensors such as visual sensors, inertial measurement units (IMUs), and odometers can more comprehensively describe the motion state of the shield machine. When the visual sensor fails due to changes in lighting conditions, the IMU data can be used as a supplement to ensure the continuity of the motion trajectory. The prediction step of the Kalman filter algorithm uses the state estimation and state transfer model of the previous moment to predict the current state. This step takes into account the dynamic characteristics of the system and can provide reasonable predictions when the observation data is missing or unreliable. For example, when the shield machine enters the communication blind spot, the prediction step can estimate the approximate position and posture of the shield machine based on the previous motion trend. The correction step is the core of the Kalman filter algorithm, which fuses the predicted value with the observed value to obtain a more accurate state estimate. The advantage of this step is that it can dynamically adjust the degree of trust in the predicted value and the observed value. For example, when the observation noise is large, the algorithm will rely more on the predicted value; when the observation data quality is high, the observed value will be given a greater weight. Finally, the motion state estimate obtained by the Kalman filter algorithm is converted into attitude data to provide a basis for the control and decision-making of the shield machine. These attitude data can include information such as the three-dimensional coordinates, pitch angle, and yaw angle of the shield machine. By outputting these data in real time, it can help operators to adjust the propulsion direction and speed of the shield machine in time to ensure the accuracy and efficiency of the construction.For example, when it is detected that the shield machine deviates from the planned route, the control system can automatically make fine adjustments based on the posture data to keep the shield machine on the correct track.

[0064] Furthermore, through the prediction step of the Kalman filter algorithm, the state estimation value and the state transfer matrix at the previous moment are used to predict the prior estimation value of the motion state of the shield machine at the current moment.

[0065] Specifically, according to the state estimation value of the shield machine at the previous moment and the pre-established state transfer matrix, the prediction step of the Kalman filter algorithm is adopted to obtain the prior estimation value of the motion state of the shield machine at the current moment; the actual measurement value of the shield machine at the current moment is obtained, and the posterior estimation value of the motion state of the shield machine at the current moment is determined by combining the prior estimation value and the measurement value through the update step of the Kalman filter algorithm; it is judged whether the residual between the posterior estimation value and the prior estimation value exceeds the preset threshold value. If it exceeds the threshold value, it is considered that the motion state of the shield machine has a sudden change, and the early warning mechanism is triggered; according to the posterior estimation value, the state transfer matrix is ​​used to predict the next The prior estimate of the motion state of the shield machine at a moment is obtained, and the prior estimate is used as the input of the next iteration; the posterior estimates of multiple consecutive moments constitute the time series of the motion state of the shield machine, and the changing trend of the motion state of the shield machine is judged through the time series analysis algorithm; the design parameters and geological condition parameters of the shield machine are obtained, and combined with the changing trend of the motion state, the motion state of the shield machine in the future is predicted through the machine learning algorithm; according to the predicted future motion state, the operating parameters of the shield machine, such as propulsion speed, cutter head speed, slag discharge, etc., are automatically adjusted to achieve intelligent control and ensure excavation efficiency and safety.

[0066] For example, the application of Kalman filter algorithm in the estimation of shield machine motion state demonstrates its powerful prediction and correction capabilities. The prediction step uses the state estimate and state transfer matrix of the previous moment to calculate the prior estimate of the current moment. For example, assuming that the position of the shield machine at a certain moment is (10, 5, 2) and the speed is (0.5, 0.3, 0.1), the approximate position of the next moment can be predicted by the state transfer matrix. The update step fuses the prior estimate and the actual measurement value to obtain a more accurate posterior estimate. This process takes into account the uncertainty of the system model and measurement, and dynamically adjusts the weights of the predicted value and the observed value. For example, when the geological conditions are complex and the measurement error increases, the algorithm will rely more on the predicted value; and when the measurement accuracy in open areas is improved, the weight of the observed value will increase accordingly. Residual analysis is the key to detecting sudden changes in the motion state of the shield machine. By comparing the difference between the posterior estimate and the prior estimate, abnormal conditions can be discovered in time. It is crucial to set a reasonable threshold. Too low may lead to frequent false alarms, and too high may miss important alarms. For example, if the residual suddenly exceeds the preset threshold of 5 meters, the system will trigger an early warning to alert the operator to possible geological changes or equipment failures. Time series analysis can reveal the long-term trend of the motion state of the shield machine. Through the posterior estimation values ​​of multiple consecutive moments, linear trends, periodic changes or emergencies can be identified. This is of great significance for predicting future excavation paths and optimizing construction plans. For example, the analysis found that the shield machine gradually deviated from the predetermined route over a period of time, which may imply a gradual change in geological conditions and the need to adjust the excavation parameters in time. The introduction of machine learning algorithms makes the prediction of the motion state of the shield machine more intelligent. By combining historical data, design parameters and geological conditions, the model can be trained to predict the motion state in the future. This prediction not only takes into account historical trends, but also adapts to complex nonlinear relationships. For example, the model may find that under certain specific geological conditions, the advancement speed of the shield machine will naturally decrease, thereby adjusting the construction plan in advance. The intelligent control system automatically adjusts the operating parameters of the shield machine based on the prediction results to achieve closed-loop control. For example, when it is predicted that it is about to enter a hard rock layer, the system may automatically increase the cutter head speed and reduce the advancement speed to ensure cutting efficiency and equipment safety. This adaptive control not only improves construction efficiency, but also greatly reduces the risk of human error. Through the comprehensive application of this series of technologies, the motion state monitoring and control of the shield machine has reached a new level, providing strong technical support for the precise construction and safety management of underground projects.

[0067] Step S104, obtaining vibration data of the shield machine, wherein the vibration data is obtained based on a vibration compensation model of an inertial measurement unit, the vibration data is combined with visual sensor data, and a data fusion algorithm based on particle filtering is used to suppress vibration interference to obtain stable posture information.

[0068] Specifically, the vibration data of the shield machine is obtained. The data is based on the vibration compensation model of the inertial measurement unit. By collecting and analyzing the vibration signal of the shield machine, a vibration compensation model is established to obtain accurate vibration data. The obtained vibration data is synchronized with the image data collected by the visual sensor. Through data preprocessing and feature extraction, the two heterogeneous data are converted into a unified feature representation form to prepare for subsequent data fusion. According to the characteristics of vibration data and visual sensor data, a particle filter model is constructed. Through the prediction, update and resampling process of particles, the estimation and tracking of the shield machine posture are realized, and the influence of vibration interference on posture estimation is effectively suppressed. In the particle filtering process, according to the confidence of vibration data and visual sensor data, the weight distribution of particles is dynamically adjusted to improve the robustness and adaptability of posture estimation, and ensure stable posture information in complex vibration environments. The posture estimation results obtained by particle filtering are smoothed, and high-frequency vibration and noise in posture estimation are eliminated through methods such as Kalman filtering or average filtering to obtain a stable posture trajectory. According to the smoothed posture trajectory, the dynamic parameters of the shield machine such as the angular velocity and acceleration are calculated. Through threshold judgment and pattern recognition, the abnormal vibration and posture mutation of the shield machine are detected in real time, providing a basis for fault diagnosis and safety warning. The obtained stable posture information is integrated with the control system of the shield machine. Through posture feedback and vibration compensation, the precise positioning and trajectory tracking of the shield machine are achieved, improving the automation level and engineering quality of shield construction.

[0069] For example, the acquisition of shield machine vibration data is the key to construction precision control. The inertial measurement unit (IMU) collects the vibration signal of the shield machine through accelerometers and gyroscopes to establish a vibration compensation model. For example, in a shield construction project, the raw data collected by the IMU shows the periodic vibration caused by the rotation of the cutter head, with a frequency of about 1.5Hz and an amplitude of 0.5g. The vibration characteristics are analyzed by Fourier transform, and a compensation model containing the main vibration mode is established, which effectively eliminates 90% of the periodic vibration and reduces the residual vibration to less than 0.05g. The time synchronization of vibration data and visual data is crucial. The timestamp alignment method can be used to match the IMU data (usually with a sampling rate of 100Hz) with the visual sensor image (frame rate of 30fps). Through the interpolation algorithm, the two types of data can be unified to the same time scale. In the data preprocessing stage, the IMU data is low-pass filtered to remove high-frequency noise, and the image data is distorted and feature points are extracted. For example, the SURF algorithm is used to extract 200 feature points from each frame of the image, and the feature vector is formed with the vibration data at the corresponding moment. The construction of the particle filter model needs to consider the motion characteristics of the shield machine. The state vector may include position, attitude angle, linear velocity, and angular velocity. The observation model combines the reprojection error of the visual feature points and the IMU measurement value. In practice, 1000 particles can be used for state estimation. Each particle represents a possible shield machine state, which is continuously optimized through the prediction-update-resampling cycle. For example, when the shield machine encounters a hard rock layer, the vibration amplitude increases suddenly. At this time, increasing the weight of the IMU data can effectively suppress the interference of vibration on visual positioning. Dynamic adjustment of particle weights is the key to improving the robustness of estimation. The weight can be dynamically calculated based on the variance of IMU and visual data. When the IMU detects strong vibration, its weight is reduced; when the visual tracking is blurred or blocked, the weight of the IMU data is increased. For example, in a certain construction, when the shield machine passed through the soft stratum, the weight of the visual data accounted for 80%; when passing through the gravel layer, due to the intensified vibration, the weight of the IMU data was increased to 60%, ensuring the continuity of the attitude estimation. The smoothing of the attitude estimation results can be done by sliding window Kalman filtering. The window size is set to 1 second (corresponding to 100 IMU data points), and the attitude output by the particle filter is smoothed. This not only eliminates high-frequency jitter, but also retains the true movement trend of the shield machine. For example, the standard deviation of the pitch angle of the shield machine in the original estimate is 0.5°, which is reduced to 0.1° after smoothing, greatly improving the stability of the attitude data. Dynamic parameter calculation and anomaly detection are important links to ensure construction safety. The attitude angular velocity and acceleration are calculated by the differential method, and the threshold is set for abnormal judgment. For example, when it is detected that the pitch angular velocity of the shield machine suddenly increases from 0.1° / s to 1° / s, the system immediately issues an early warning, prompting the operator to pay attention to geological changes or equipment failures. This real-time monitoring mechanism successfully warned of potential equipment failures three times in a certain project, effectively avoiding construction interruptions.Integrating stable posture information into the control system can achieve accurate navigation of the shield machine. For example, in the construction of a long-distance tunnel, through real-time posture feedback, the control system can control the trajectory deviation of the shield machine within the range of ±5cm, which is 50% higher than the traditional method. This not only ensures the geometric accuracy of the tunnel, but also optimizes the construction efficiency and reduces the later finishing work.

[0070] Step S105, constructing a multimodal data fusion model based on a graph convolutional neural network, fusing the visual data, inertial data and motion state data to obtain a high-precision posture estimation result.

[0071] Specifically, multimodal sensor data containing visual, inertial and motion state information is obtained, and the data is preprocessed, including denoising, normalization and other operations, to improve data quality. The preprocessed multimodal data is converted into graph structure data, and a graph model containing node features and edge connection relationships is constructed. The node features fuse information of different modalities. A multimodal data fusion model based on graph convolutional neural network is designed, and node features are updated and fused through graph convolution operations to capture the correlation between different modal data. The attention mechanism is introduced into the graph convolutional neural network to adaptively assign weights to different nodes, highlight the role of key nodes, and improve the fusion effect. By stacking multiple layers of graph convolutional neural networks, high-level semantic features of multimodal data are gradually extracted and fused to obtain global feature representation. The output features of the graph convolutional neural network are sent to the fully connected layer, and after nonlinear transformation and regression prediction, the final high-precision posture estimation result is obtained.

[0072] For example, the acquisition of multimodal sensor data is the basis for achieving high-precision posture estimation. Taking the shield machine as an example, data from sensors such as visual cameras, inertial measurement units, and odometers can be collected simultaneously. Visual data contains rich environmental information, inertial data reflects posture changes, and motion state data records displacement and speed. It is crucial to preprocess these raw data. Median filtering can be used to remove outliers, wavelet transform can be used to eliminate high-frequency noise, and then normalization can be performed to make different modal data comparable. Converting preprocessed multimodal data into a graph structure is a novel representation method. The multimodal data of each time step can be used as a node, and the node features contain a fusion representation of visual, inertial, and motion state information. Edge connections are established between nodes in adjacent time steps to form a time series graph structure. This representation method can effectively capture the spatiotemporal relationship between data. The multimodal data fusion model based on graph convolutional neural network can make full use of the advantages of graph structure. By designing a specific graph convolution operator, local aggregation and update of node features can be achieved. For example, spatial graph convolution operations can be used to fuse information of different modalities in the same time step, and then temporal graph convolution operations can be used to capture the dependency between adjacent time steps. This hierarchical fusion method can effectively extract the joint representation of multimodal data. Introducing the attention mechanism in the graph convolutional neural network can further improve the fusion effect. The attention weight based on node similarity can be designed so that there is stronger information interaction between nodes with higher correlation. For example, when the shield machine turns, the attention mechanism can be used to highlight the role of visual and inertial data, while when it is driving in a straight line, it pays more attention to the motion state data. By stacking multiple layers of graph convolutional neural networks, high-level semantic features of multimodal data can be gradually extracted. The shallow network mainly focuses on local features, such as edge and texture information in visual data, instantaneous acceleration changes in inertial data, etc. The deep network can capture more abstract global features, such as the motion mode and environment type of the shield machine. This hierarchical feature extraction process helps to obtain a more robust and generalized representation. Finally, the output features of the graph convolutional neural network are sent to the fully connected layer for posture estimation.

[0073] Step S106, based on the real-time posture estimation result and the preset tunneling trajectory, an adaptive control algorithm is used to generate control instructions, adjust the shield machine propulsion system and guidance system, and achieve precise posture control. The preset tunneling trajectory is determined based on geological exploration data and engineering design parameters.

[0074] Specifically, according to the geological exploration data and engineering design parameters, the tunneling trajectory of the shield machine is preset, and the trajectory data is stored in the control system database. The shield machine posture data, including pitch angle, yaw angle, roll angle, etc., are collected in real time through the sensors on the shield machine, and the collected data is transmitted to the control system. After the control system obtains the real-time posture data, it compares it with the preset tunneling trajectory and calculates the posture deviation value. If the deviation value exceeds the preset threshold, the adaptive control algorithm is triggered. According to the posture deviation value, the adaptive control algorithm combines the historical posture data of the shield machine and the tunneling parameters, and estimates the posture of the shield machine at the next moment through the Kalman filter algorithm. According to the posture estimation result, the adaptive control algorithm generates control instructions for the propulsion system and the guide system, including propulsion speed, propulsion force, and extension amount of the guide cylinder. The generated control instructions are sent to the propulsion system and the guide system of the shield machine to adjust the propulsion speed, direction and posture of the shield machine to keep it consistent with the preset tunneling trajectory. The control system continuously obtains the real-time posture data of the shield machine, repeats steps 3-6, and continuously adjusts the control instructions through the adaptive control algorithm to achieve precise posture control of the shield machine, ensuring that the tunneling process always follows the preset trajectory.

[0075] Exemplarily, the tunneling trajectory of the shield machine is preset based on geological exploration data and engineering design parameters. For example, in a subway tunnel project, the stratum distribution, lithology characteristics and groundwater conditions along the line were obtained through geological drilling and geophysical exploration. Combined with the design slope of the tunnel, curve radius and other parameters, engineers can develop an optimal tunneling route. This route will be converted into a series of three-dimensional coordinate points and stored in the database of the control system as a reference for the guidance of the shield machine. The shield machine is equipped with a variety of sensors to monitor its posture in real time. Common sensors include gyroscopes, accelerometers and tilt sensors. These sensors can accurately measure the pitch angle, yaw angle and roll angle of the shield machine. For example, a certain model of shield machine uses a fiber optic gyroscope, which has a measurement accuracy of up to 0.01° and can capture tiny posture changes. The control system calculates the deviation value by comparing the real-time posture data with the preset trajectory. Assuming that the preset trajectory requires the shield machine to maintain a pitch angle of 2° to move forward, and the actual measured pitch angle is 2.5°, the deviation value is 0.5°. If the threshold is set to ±0.3°, the adaptive control algorithm will be triggered. The adaptive control algorithm uses Kalman filtering to estimate the future posture of the shield machine. Kalman filtering can comprehensively consider measurement noise and system noise to provide the best estimate. For example, the algorithm combines the current posture, historical data and tunneling parameters (such as propulsion speed, earth pressure, etc.) to predict the possible posture deviation of the shield machine at the next moment. Based on the posture estimation results, the control algorithm will generate corresponding control instructions. If it is predicted that the shield machine has a tendency to deviate upward, the algorithm may instruct the upper guide cylinder to retract and the lower guide cylinder to extend, while appropriately reducing the propulsion speed to correct the posture deviation. These instructions are accurate to specific values, such as the cylinder extension amount can be accurate to millimeters. The control instructions are transmitted to the execution system of the shield machine through the data bus. The propulsion system adjusts the propulsion speed and thrust according to the instructions, while the guidance system controls the extension and retraction of each guide cylinder. This synergy enables the shield machine to maintain a stable tunneling posture in complex formations. The entire control process is a closed-loop system that repeats the above steps. This adaptive control method can cope with changes in geological conditions and external interference. For example, when the shield machine encounters hard rock, the system automatically increases the thrust and torque while reducing the thrust speed to ensure that the excavation accuracy is not affected. Through this continuous posture adjustment, the shield machine can maintain centimeter-level trajectory deviation control accuracy in tunnels several kilometers long, providing a strong guarantee for the safety and quality of tunnel engineering.

[0076] This embodiment also provides a shield attitude control system based on machine vision, including:

[0077] The data acquisition module is used to obtain the image of the surrounding environment of the shield machine through multi-spectral imaging technology, correct the image of the surrounding environment of the shield machine to obtain a corrected image; and obtain the image of the shield machine at different angles through a multi-view visual sensor array;

[0078] The feature point motion trajectory acquisition module is used to identify the key feature points of the shield machine in the correction image and the shield machine image through a feature extraction network based on deep learning, and to match the feature points in adjacent frame images through a feature matching method based on the RANSAC algorithm to obtain the feature point motion trajectory;

[0079] A motion state modeling module is used to construct a motion state model of the shield machine according to the motion trajectory of the feature points, and use a Kalman filter algorithm to predict and correct the motion state to obtain predicted posture data of the shield machine;

[0080] A vibration data fusion module is used to obtain the vibration data of the shield machine through the vibration compensation model of the inertial measurement unit, combine the vibration data with the visual sensor data, and use a data fusion algorithm based on particle filtering to suppress vibration interference and obtain stable posture information;

[0081] The multimodal data fusion module is used to build a multimodal data fusion model based on graph convolutional neural network, which fuses visual data, inertial data, stable posture information and predicted posture data to obtain posture estimation results;

[0082] The posture control module is used to generate control instructions using an adaptive control algorithm based on the posture estimation result and the preset excavation trajectory to perform precise posture control.

[0083] Furthermore, the data acquisition module includes:

[0084] The acquisition unit is used to obtain images of the surrounding environment of the shield machine through multi-spectral imaging technology, and to obtain images of the shield machine at different angles through a multi-view visual sensor array;

[0085] The correction unit is used to perform correction processing on the image of the surrounding environment of the shield machine.

[0086] Furthermore, the correction unit comprises:

[0087] A preprocessing unit, used for performing denoising and image enhancement processing on the image of the surrounding environment of the shield machine to obtain a preprocessed image;

[0088] An illumination compensation unit, used to perform histogram equalization processing on the preprocessed image according to a preset illumination compensation model, adjust the contrast and brightness distribution of the image, and obtain an illumination compensated image;

[0089] A gamma correction unit, which is used to use an adaptive gamma correction algorithm to adaptively adjust the gamma value according to the local features of the preprocessed image, so as to enhance the dark area of ​​the image;

[0090] The fusion unit is used to fuse the multi-band images processed by histogram equalization and adaptive gamma correction to obtain a corrected image.

[0091] Furthermore, the feature point motion trajectory acquisition module includes:

[0092] A feature extraction unit, used for identifying key feature points of the shield machine in the correction image and the shield machine image through a feature extraction network based on deep learning;

[0093] The feature matching unit is used to match feature points in adjacent frame images by a feature matching method based on the RANSAC algorithm.

[0094] This embodiment further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0095] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method are implemented.

[0096] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A shield machine posture control method based on machine vision, characterized in that: The following steps are involved: Acquire the image of the surrounding environment of the shield machine by using multi-spectral imaging technology, correct the image of the surrounding environment of the shield machine to obtain a corrected image; acquire the image of the shield machine at different angles by using a multi-view visual sensor array; A feature extraction network based on deep learning is used to identify the key feature points of the shield machine in the correction image and the shield machine image. The feature points in adjacent frame images are matched by a feature matching method based on the RANSAC algorithm to obtain the motion trajectory of the feature points. According to the motion trajectory of the characteristic points, a motion state model of the shield machine is constructed, and the motion state is predicted and corrected by using a Kalman filter algorithm to obtain predicted posture data of the shield machine; The vibration data of the shield machine is obtained through the vibration compensation model of the inertial measurement unit, the vibration data is combined with the visual sensor data, and the data fusion algorithm based on particle filtering is used to suppress vibration interference to obtain stable posture information; Construct a multimodal data fusion model based on graph convolutional neural network to fuse visual data, inertial data, stable posture information and predicted posture data to obtain posture estimation results; According to the posture estimation result and the preset excavation trajectory, an adaptive control algorithm is used to generate control instructions to perform precise posture control.

2. The method according to claim 1, characterized in that Correcting the image of the surrounding environment of the shield machine includes: Performing denoising and image enhancement processing on the image of the surrounding environment of the shield machine to obtain a preprocessed image; According to the preset illumination compensation model, the preprocessed image is subjected to histogram equalization processing, the contrast and brightness distribution of the image are adjusted, and an illumination compensation image is obtained; Adopting adaptive gamma correction algorithm, the gamma value is adaptively adjusted according to the local features of the preprocessed image to enhance the dark area of ​​the image; The multi-band images processed by histogram equalization and adaptive gamma correction are fused to obtain the corrected image.

3. The method according to claim 1, characterized in that The Kalman filter algorithm is used to predict the motion state, including: The actual measurement value of the shield machine at the current moment is obtained, and the posterior estimation value of the motion state of the shield machine at the current moment is determined by combining the prior estimation value and the measurement value through the updating step of the Kalman filter algorithm; it is judged whether the residual between the posterior estimation value and the prior estimation value exceeds the preset threshold value. If it exceeds the threshold value, it is considered that the motion state of the shield machine has changed suddenly and the early warning mechanism is triggered; according to the posterior estimation value, the state transfer matrix is ​​used to predict the prior estimation value of the motion state of the shield machine at the next moment, and the prior estimation value is used as the input of the next iteration; the posterior estimation values ​​of multiple consecutive moments constitute the time series of the motion state of the shield machine, and the changing trend of the motion state of the shield machine is judged through the time series analysis algorithm.

4. The method according to claim 1, characterized in that: The posture estimation results include: A multimodal data fusion model is constructed based on graph convolutional neural network. The node features are updated and fused through graph convolution operation to capture the correlation between different modal data. The attention mechanism is introduced into the graph convolutional neural network to adaptively assign weights to different nodes, highlight the role of key nodes, and improve the fusion effect. By stacking multiple layers of graph convolutional neural network, the high-level semantic features of multimodal data are gradually extracted and fused to obtain the global feature representation. The global feature representation is input into the fully connected layer, and the posture estimation result is obtained after nonlinear transformation and regression prediction.

5. A shield machine attitude control system based on machine vision, characterized in that: include: The data acquisition module is used to obtain the image of the surrounding environment of the shield machine through multi-spectral imaging technology, correct the image of the surrounding environment of the shield machine to obtain a corrected image; and obtain the image of the shield machine at different angles through a multi-view visual sensor array; The feature point motion trajectory acquisition module is used to identify the key feature points of the shield machine in the correction image and the shield machine image through a feature extraction network based on deep learning, and to match the feature points in adjacent frame images through a feature matching method based on the RANSAC algorithm to obtain the feature point motion trajectory; A motion state modeling module is used to construct a motion state model of the shield machine according to the motion trajectory of the feature points, and use a Kalman filter algorithm to predict and correct the motion state to obtain predicted posture data of the shield machine; A vibration data fusion module is used to obtain the vibration data of the shield machine through the vibration compensation model of the inertial measurement unit, combine the vibration data with the visual sensor data, and use a data fusion algorithm based on particle filtering to suppress vibration interference and obtain stable posture information; The multimodal data fusion module is used to build a multimodal data fusion model based on graph convolutional neural network, which fuses visual data, inertial data, stable posture information and predicted posture data to obtain posture estimation results; The posture control module is used to generate control instructions using an adaptive control algorithm based on the posture estimation result and the preset excavation trajectory to perform precise posture control.

6. The system according to claim 5, characterized in that The data acquisition module includes: The acquisition unit is used to obtain images of the surrounding environment of the shield machine through multi-spectral imaging technology, and to obtain images of the shield machine at different angles through a multi-view visual sensor array; The correction unit is used to perform correction processing on the image of the surrounding environment of the shield machine.

7. The system according to claim 6, characterized in that The correction unit comprises: A preprocessing unit, used for performing denoising and image enhancement processing on the image of the surrounding environment of the shield machine to obtain a preprocessed image; An illumination compensation unit, used to perform histogram equalization processing on the preprocessed image according to a preset illumination compensation model, adjust the contrast and brightness distribution of the image, and obtain an illumination compensated image; A gamma correction unit, which is used to use an adaptive gamma correction algorithm to adaptively adjust the gamma value according to the local features of the preprocessed image, so as to enhance the dark area of ​​the image; The fusion unit is used to fuse the multi-band images processed by histogram equalization and adaptive gamma correction to obtain a corrected image.

8. The system according to claim 5, characterized in that The feature point motion trajectory acquisition module includes: A feature extraction unit, used for identifying key feature points of the shield machine in the correction image and the shield machine image through a feature extraction network based on deep learning; The feature matching unit is used to match feature points in adjacent frame images by a feature matching method based on the RANSAC algorithm.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images