A method for evaluating self-cleaning and mud removal efficiency of a deep-sea mining vehicle and predicting residual life of brush wires

Abstract

This invention provides a method for evaluating the self-cleaning and desliming efficiency of deep-sea mining vehicles and predicting the remaining lifespan of their brushes, belonging to the field of intelligent operation and maintenance technology for deep-sea mining equipment based on machine learning. First, it proposes a multi-source heterogeneous data fusion architecture based on physical information soft constraints and Granger causality tests. This mechanism strictly aligns dynamic temporal features and environmental perception image features in terms of spatiotemporal dimensions before inputting them into three temporal convolutional layers, ultimately outputting a fused state vector. Second, it proposes an end-to-end dynamic evaluation and prediction strategy based on a lightweight semantic segmentation network and a hybrid expert system. In the desliming efficiency evaluation stage, it achieves high-precision analysis and outputs the residual clay area ratio and average residual clay thickness, reflecting the actual desliming efficiency. In the lifespan prediction stage, it abandons static weight allocation and intelligently and adaptively switches specific computational logic according to complex seabed environmental conditions, outputting a high-precision predicted value for the remaining lifespan of the brushes.

Description

Technical Field

[0001] This invention belongs to the field of intelligent operation and maintenance technology of deep-sea mining equipment based on machine learning, and particularly relates to a method for evaluating the self-cleaning and mud removal efficiency of deep-sea mining vehicles and predicting the remaining life of brushes. Background Technology

[0002] In the field of intelligent operation and maintenance technology for deep-sea mining equipment, tracked walking devices for deep-sea mining vehicles are widely used in several key areas, such as deep-sea polymetallic nodule collection, seabed cobalt-rich crust mining, and deep-sea extreme environment exploration, due to their low ground pressure, high traction, and good adaptability to complex terrain. These devices typically include, but are not limited to, tracks, drive wheels, plumes, quick-release mud covers, and self-cleaning brush systems. Their walking performance is mainly reflected in the stripping efficiency of high-viscosity sediments in the deep sea, the control level of drive energy consumption, the wear life of key vulnerable parts such as brushes, and the traction stability under complex hydrodynamic conditions. The aforementioned macroscopic walking and self-cleaning performance are not determined by a single factor, but are fundamentally constrained by the synergistic effect of complex microscopic interaction mechanisms between the tracks and deep-sea sediments. These interaction mechanisms specifically involve multi-scale, multi-modal dynamic and environmental coupling characteristics, such as the proportion of residual clay area, the counteracting effect of water flow angle and velocity, the dynamic fluctuation of drive wheel torque, and the step evolution of brush wear stages.

[0003] From the perspective of existing technologies, the operation and maintenance of deep-sea equipment mainly employs mechanistic dynamics simulation and traditional threshold determination methods. While mechanistic dynamics simulation can reveal the coupling mechanism between water flow and track surface at the physical flow field scale, its computational complexity is extremely high, and its time delay is significant, making it difficult to run effectively in real-time on-board edge computing controllers, and even more difficult to directly generate feedback signals that can guide second-level vehicle speed control. Traditional time-series prediction methods typically focus on establishing a simple mapping between single dynamic data and wear state, lacking the ability to explicitly model the correlation of multi-source heterogeneous visual features, making it difficult to output reliable prediction results that integrate hydrodynamic physical information and causal relationships, thus making it difficult to directly translate decision results into specific adaptive optimization objectives. In summary, the existing technological system cannot yet meet the actual needs of deep-sea mining vehicle self-cleaning tracks in multimodal fusion. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a method for evaluating the self-cleaning and desliming efficiency of deep-sea mining vehicles and predicting the remaining lifespan of brush bristles. This method is applied to the self-cleaning tracked walking device of a deep-sea mining vehicle. A plume is installed on the track, and a mud-covering brush with brush bristles installed inside is installed near the drive wheel. The method includes the following steps: S1, collects and preprocesses images of the track plate surface, time-series data of water flow velocity inside the plume, time-series data of water flow direction inside the plume, time-series data of drive wheel torque and track rotation speed; S2, the preprocessed track plate surface image, water flow velocity time series data, and water flow direction time series data are input into a physical information-guided convolutional neural network to obtain environmental perception image features of hydrodynamic influence; the preprocessed drive wheel torque time series data and track rotation speed time series data are synthesized into two-dimensional dynamic time series features, which are input into a time series convolutional network based on Granger causality test, and combined with environmental perception image features, output a fusion state vector reflecting the dynamic relationship between walking mud removal and the environment; S3. The preprocessed track plate surface image is input into the encoder of the mud removal efficiency evaluation network to output high-level semantic features. In the decoder, the high-level semantic features are concatenated with the fused state vector through the feature fusion module to output a segmentation probability map, and then the residual clay area ratio and the average thickness of residual clay are calculated. Next, the fused state vector is input into the brush bristle remaining life prediction network to output the predicted value of brush bristle remaining life.

[0005] Preferably, the preprocessing includes: Based on the differences in the physical reflection characteristics of the track plate surface in the track plate surface image, the track plate surface image is annotated at the pixel level. Based on grayscale threshold segmentation and local binary mode analysis, the track plate region in the track plate surface image is segmented and annotated into the track tooth region, the track plate base region, and the residual clay region.

[0006] Preferably, the S2 process specifically includes: First, image feature deep mining based on hydrodynamic conditions is performed. The track plate surface image is input into the encoder of the semantic segmentation network to extract and output a deep semantic feature map. At the same time, the time series data of water flow velocity and water flow direction inside the plume are input into the physical information layer to calculate and output a spatial weight mask. The spatial weight mask is multiplied element-wise with the deep semantic feature map and then input into the decoder. Finally, the environmental perception image features that have been fused with hydrodynamic influence are output. Secondly, a dynamic visual feature correlation analysis based on causal inference is performed. The time series data of drive wheel torque and track rotation speed are combined into dynamic time series features as input. The Granger causality test module is used to test the dynamic time series features and environmental perception image features to output the optimal lag order. Then, the two are aligned according to the optimal lag order and input into the feature stitching layer to output multimodal stitched features. Finally, the multimodal stitched features are input into three temporal convolutional layers to extract and output a fusion state vector that comprehensively reflects the dynamic correlation between walking mud removal and the environment.

[0007] Preferably, the physical information-guided convolutional neural network includes a semantic segmentation network based on an encoder-decoder structure and a physical information layer; The semantic segmentation network uses four convolutional layers as the encoder. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to progressively extract the spatial features of the input track plate surface image. The fourth convolutional layer of the encoder outputs a deep semantic feature map. The physical information layer takes flow velocity time series data and flow direction time series data as input, and outputs a spatial weight mask with the same size as the deep semantic feature map by calculating the angle between the water flow direction and the normal direction of the track plate at each spatial location, i.e., the water flow angle of attack and the flow velocity magnitude. The spatial weight mask is multiplied element-wise with the deep semantic feature map to achieve soft weighting of the feature map, guiding the network to pay more attention to image regions with significant water flow impact effects. The weighted feature map is input into the decoder of the semantic segmentation network. The decoder consists of four deconvolutional layers. The output of the fourth deconvolutional layer of the decoder is an environmental perception image feature with hydrodynamic influence fused at the same resolution as the input track plate surface image.

[0008] Preferably, the temporal convolutional network based on Granger causality test includes a Granger causality test module, a feature concatenation layer, and three temporal convolutional layers; First, the Granger causality test module performs Granger causality tests with lag orders from 1 to 10 on the dynamic time-series features and the output environmental perception image features: for the lag order... The Granger causality test module constructs a constrained autoregressive model based on the lag order. The model uses the value of the environmental perception image features at the current time t as the dependent variable, and the environmental perception image features at... , Until Moment Using numerical values ​​as independent variables, the linear regression coefficients are estimated using the least squares method to obtain the constrained model fit values. The sum of squares of the differences between the dependent variable and the constrained model fit values ​​is then calculated and denoted as the constrained model residual sum of squares. Simultaneously, the Granger causality test module constructs an unconstrained autoregressive model, which, based on the independent variables of the constrained autoregressive model, incorporates dynamic time-series characteristics. , Until Moment The numerical values ​​are used as additional independent variables. The linear regression coefficients are estimated by the least squares method to obtain the unconstrained model fit values. The sum of squares of the differences between the dependent variable and the unconstrained model fit values ​​is calculated and denoted as the unconstrained model residual sum of squares. Then calculate the F-statistic: First, subtract the sum of squared residuals of the unconstrained model from the sum of squared residuals of the constrained model, and divide by the lag order. Obtain the mean of the residual sum of squares, then divide the mean of the residual sum of squares by the residual sum of squares of the unconstrained model, and then divide by the time series length minus ( The difference between the two values ​​is used to obtain the F-statistic; the F-statistic is used to look up the F-distribution table to obtain the corresponding significance level, and the lag order that makes the F-statistic reach its maximum value is recorded as the optimal lag order. Subsequently, the feature stitching layer aligns the dynamic temporal features with a significance level below 0.05 with the generated environmental perception image features in the time dimension according to the optimal lag order, and then stitches them in the channel dimension to form multimodal stitched features. Finally, the multimodal concatenated features are input into three temporal convolutional layers, and the output is a fused state vector.

[0009] Preferably, the S3 process specifically includes: First, visual semantic parsing of desliming efficiency is performed. The track plate surface image is input into the encoder of the desliming efficiency evaluation network, which outputs high-level semantic features. In the decoder, the high-level semantic features are concatenated with the fused state vector through the feature fusion module to output a segmentation probability map. Then, the residual clay area ratio and the average thickness of residual clay are calculated and output. Second, the remaining life of the brush bristles is predicted. The fused state vector is input into two layers of long short-term memory network, which outputs the first and second temporal hidden states in sequence. Then, the second temporal hidden state is concatenated with the residual clay area ratio, the average thickness of residual clay, the driving wheel torque time series data, and the cumulative number of track rotations, and then input into three expert subnetworks, which output the desliming efficiency expert feature, the driving torque expert feature, and the cumulative rotation expert feature in sequence. At the same time, the second temporal hidden state is input into a gating network to output an expert weight vector. Finally, the expert weight vector is weighted and summed with the above three expert features to obtain the fused expert feature, and this feature is input into the fully connected layer to output the predicted value of the remaining life of the brush bristles.

[0010] Preferably, the desludge removal efficiency evaluation network adopts an encoder-decoder structure. The track plate surface image is input into the encoder, which outputs high-level semantic features. The encoder part contains three depthwise separable convolutional blocks, each consisting of a depthwise separable convolutional layer, a batch normalization layer, and a ReLU activation function, used to progressively compress the spatial resolution of the track plate surface image and extract high-level semantic information. The decoder part uses bilinear interpolation upsampling layers to progressively restore the high-level semantic features to the original image resolution, and introduces a feature fusion module after each upsampling layer. This module maps the generated fusion state vector through a fully connected layer and then concatenates it with the feature map obtained from the upsampling layer in terms of channel dimension. The decoder finally outputs a segmentation probability map with the same resolution as the input image, where each pixel in the map has the probability of belonging to the tooth region, the track plate base region, and the residual clay region. Based on the segmentation probability map, the residual clay area ratio is obtained by calculating the percentage of the total number of pixels in the residual clay region to the total number of pixels in the track plate region, and the average thickness of the residual clay is obtained by calculating the average of the actual physical thickness corresponding to each pixel in the residual clay region.

[0011] Preferably, the remaining lifespan prediction network for the brush bristles includes two long short-term memory network layers, three expert sub-networks, a gated network, and a fully connected layer; First, the fused state vector is input into two layers of long short-term memory (LSTM) network. The first LSM network layer models the temporal dependency of the input fused state vector and outputs the first temporal hidden state. The second LSM network layer takes the first temporal hidden state as input and further captures the degradation trend of features over time, outputting the second temporal hidden state. Secondly, the second-layer time-series hidden state is concatenated with the residual clay area ratio and the average residual clay thickness and then input into the first expert sub-network to output the mud removal efficiency characteristics; the second-layer time-series hidden state is concatenated with the drive wheel torque time-series data and then input into the second expert sub-network to output the power load characteristics; the second-layer time-series hidden state is concatenated with the cumulative number of track rotations and then input into the third expert sub-network to output the physical wear characteristics; each expert sub-network is a two-layer fully connected network; The gating network is a two-layer fully connected network. The first fully connected layer contains 24 neurons connected to the ReLU activation function, and the second fully connected layer contains 3 neurons connected to the Softmax activation function. The sludge removal efficiency features, dynamic load features, and physical wear features are input into the gating network. The weights of each expert subnetwork are calculated and selected through the Softmax activation function, and the expert weight vector is output. The expert weight vector is then weighted and summed with the outputs of the three expert subnetworks to obtain the fused expert features. The expert features are fused and input into the fully connected layer, and then the Softmax activation function is used to output the predicted value of the remaining brush life.

[0012] Compared with the prior art, the present invention has the following innovative features: First, an architecture for physical information-guided and causal inference-based state feature extraction based on track sediment interaction mechanisms is proposed: a multi-source heterogeneous data fusion architecture based on soft constraints of physical information and Granger causality tests is put forward. This architecture first constructs a physical information-guided convolutional neural network. Using time-series data of water flow velocity and direction within the plume as input, it calculates the angle of attack and velocity magnitude, generating a spatially weighted mask of consistent size. This mask is then used to perform element-wise soft weighting on the deep semantic feature map extracted by the semantic segmentation network encoder, thereby deeply mapping the macroscopic physical effects of water flow impact during the generation of environmental perception image features. Subsequently, a temporal convolutional network based on Granger causality tests is constructed. Abandoning the traditional blind input splicing method, it uses constrained and unconstrained autoregressive models to calculate the residual sum of squares and F-statistic, dynamically determining the optimal lag order that complies with the significance level. This mechanism strictly aligns the temporal and spatial dimensions of dynamic temporal features with environmental perception image features and then inputs them into three temporal convolutional layers, finally outputting a fused state vector, thereby achieving deep deterministic fusion of multi-source heterogeneous multimodal data at the underlying physical causal logic. Second, an embedded visual semantic parsing and condition-oriented hybrid expert collaborative prediction strategy is designed: an end-to-end dynamic evaluation and prediction strategy based on a lightweight semantic segmentation network and a hybrid expert system is proposed. In the sludge removal efficiency evaluation stage, the feature fusion module in the sludge removal efficiency evaluation network decoder is used to concatenate the real-time flowing fused state vector with the high-level semantic feature map after bilinear interpolation along the channel dimension. This allows for real-time, high-precision parsing and output of the residual clay area ratio and average residual clay thickness, reflecting the actual sludge removal efficiency. In the lifespan prediction stage, a brush remaining lifespan prediction network containing three expert sub-networks and one gating network is constructed. The second-layer temporal hidden state is decoupled and concatenated with the residual clay area ratio and average residual clay thickness, the time-series data of the drive wheel torque, and the cumulative number of track rotations. This independently maps and outputs sludge removal efficiency features, power load features, and physical wear features. Simultaneously, the gating network outputs expert weight vectors based on the above three types of features and performs dynamic weighted combination to obtain the fused expert features. This predictive architecture can abandon static weight allocation and intelligently and adaptively switch specific computational logic according to complex seabed environment conditions, thereby eliminating the error accumulation caused by cross-condition generalization, and then outputting high-precision brush filament remaining life prediction values ​​through a fully connected layer. Attached Figure Description

[0013] Figure 1 This is a structural diagram of the self-cleaning tracked walking device for a deep-sea mining vehicle.

[0014] Figure 2 The design drawing is for a mud cover brush that can be quickly removed, where A represents the bristle bundle.

[0015] Figure 3 for Figure 2 Enlarged view of section A.

[0016] Figure 4 This is a flowchart illustrating the overall technical route of the present invention.

[0017] Figure 5 This is a structural diagram of the multi-source heterogeneous data fusion and state feature extraction based on physical information guidance and causal inference in this invention.

[0018] Figure 6 This is an experimental diagram of multimodal dataset feature analysis in an embodiment of the present invention.

[0019] Figure 7 This is a diagram showing the experimental results of real-time semantic segmentation performance comparison of sludge removal efficiency in an embodiment of the present invention.

[0020] Figure 8 This is a graph showing the experimental results comparing the accuracy of remaining life prediction during the brush bristle wear stage in an embodiment of the present invention. Detailed Implementation

[0021] The self-cleaning tracked walking device for deep-sea mining vehicles used in this invention has a specific structure as follows: Figure 1 As shown, the main structure of the device includes a track 1, track teeth 2, drive wheel 3, plume hood 4, quick-removable mud cover brush 5, brush bristles 6, disassembly handle 7, positioning slot 8, and limit pointer 9. The plume hood 4 is installed on the track to reduce the spread of plumes caused by track disturbance to sediment. Near the drive wheel, a quick-removable mud cover brush 5 is installed, containing two rows of brush bristles 6 arranged in a specific pattern. As the track rotates, the soft soil adhering to the track teeth is squeezed by the brush bristles and discharged under the plume hood under the action of gravity. The plumes generated during the desludge removal process can be effectively suppressed and diffused by the plume hood, reducing emissions. At the same time, considering the wear of the brush bristles, the desludge cover brush is designed with a quick-removable structure, which can be replaced and maintained periodically using an underwater robot.

[0022] Specifically, the plume shroud 4 connects the track frame to the vehicle body via a crossbeam, covering the entire track. The plume shroud section near the drive wheel 3 is designed as a detachable structure (equivalent to being divided into two pieces that can be assembled), forming the quick-removable mud shroud brush 5. Its center coincides with the center of the drive wheel 3 and is fixed in place by the positioning slot 8 and the limit pointer 9. The quick-removable mud shroud brush 5 has two rows of inverted "V"-shaped brush bristles 6 inside, with the bristle length slightly longer than the vertical distance between the inner wall of the quick-removable mud shroud brush and the track plate. When the mining vehicle moves, the clay adhering to the track rotates to the desliming position and is squeezed off by the contact of the two rows of brush bristles, then discharged below the plume shroud 4, achieving self-cleaning of the track. Except for the ground contact section, the entire track is blocked by the plume shroud 4, which can suppress the diffusion of the plume generated by the track. When the track rotates, it drives the nearby water flow. The water flow is blocked by the plume shroud, and the water flow reverses to create a water flow counter-current effect, which can enhance the desliming effect. When the teeth pass over the bristles, they disturb the bristles, causing them to undergo "deformation-rebound," during which the mud accumulated between the bristles is shaken off.

[0023] Figure 2 This is a design drawing for a quick-removable mudguard brush. The brush bristles are arranged in two inverted "V" shapes, one above the other, perpendicular to the inner wall of the brush and pointing towards the center of the drive wheel. (See diagram below.) Figure 3 As shown, the diameter of the brush filament bundle is 8 mm, and the center-to-center distance of the holes in the brush filament bundle is 25 mm.

[0024] This invention discloses a method for evaluating the self-cleaning and desliming efficiency of deep-sea mining vehicles and predicting the remaining lifespan of their bristles. The overall process is as follows: Figure 4 As shown: S1, Multi-source heterogeneous data acquisition and preprocessing; This step aims to construct a multimodal dataset containing visual dynamics and environmental parameters for the operating environment of the self-cleaning tracked walking device of the deep-sea mining vehicle, providing basic data input with strictly time-stamped alignment for subsequent state feature extraction; The specific process is as follows: First, multi-source heterogeneous data acquisition is performed, deploying miniature pressure-resistant cameras, hydrodynamic sensor arrays, torque sensors, and speed encoders at appropriate locations to acquire and output track surface images, time-series data of water flow velocity inside the plume, time-series data of water flow direction inside the plume, time-series data of drive wheel torque, and time-series data of track rotation speed; Second, data preprocessing and annotation are performed, using the acquired track surface images, time-series data of water flow velocity inside the plume, time-series data of water flow direction inside the plume, time-series data of drive wheel torque, and time-series data of track rotation speed as input, and aligning them with timestamps; S2, Multi-source heterogeneous data fusion and state feature extraction based on physical information guidance and causal inference; this step aims to explore the spatiotemporal causal relationship between hydrodynamic conditions and multimodal data, mapping discrete low-level time series and image data into high-dimensional features that comprehensively reflect the system's operating state; firstly, deep mining of image features based on fused hydrodynamic conditions is performed, inputting the track plate surface image into the encoder of the semantic segmentation network to extract and output a deep semantic feature map, and simultaneously inputting the time series data of water flow velocity and direction inside the plume into the physical information layer to calculate and output a spatial weight mask, which is then element-wise compared with the deep semantic feature map. The multiplier is then input into the decoder, and the final output is an environmental perception image feature that incorporates the effects of hydrodynamics. Next, a dynamic visual feature correlation analysis based on causal inference is performed. The time series data of the drive wheel torque and the time series data of the track rotation speed are combined into a dynamic time series feature as input. The Granger causality test module is used to test the dynamic time series feature and the environmental perception image feature to output the optimal lag order. Then, the two are aligned according to the optimal lag order and input into the feature stitching layer to output a multimodal stitched feature. Finally, the multimodal stitched feature is input into a three-layer temporal convolutional layer to extract and output a fused state vector that comprehensively reflects the dynamic correlation between walking mud removal and the environment. S3, Desliming Efficiency Assessment and Brush Filament Remaining Life Prediction: This step aims to establish a visual analysis and wear prediction model based on the extracted fusion state features, achieving a quantitative assessment of current desliming performance and accurate prediction of future service life. First, visual semantic analysis of desliming efficiency is performed. The track plate surface image is input into the encoder of the desliming efficiency assessment network, outputting high-level semantic features. In the decoder, the high-level semantic features are concatenated with the fusion state vector through a feature fusion module, outputting a segmentation probability map. Then, the residual clay area ratio and average residual clay thickness are calculated and output. Second, brush filament remaining life prediction is performed. The fusion state vector is input into... Two layers of long short-term memory network output the first and second layer of temporal hidden states in sequence. Then, the second layer of temporal hidden states are concatenated with the residual clay area ratio, the average thickness of residual clay, the driving wheel torque time series data, and the cumulative number of track rotations, and then input into three expert subnetworks, which output the mud removal efficiency expert feature, the driving torque expert feature, and the cumulative rotation expert feature in sequence. At the same time, the second layer of temporal hidden states are input into the gating network to output the expert weight vector. Finally, the expert weight vector is weighted and summed with the above three expert features to obtain the fused expert feature, and this feature is input into the fully connected layer to output the predicted value of the remaining brush life.

[0025] S1, Multi-source heterogeneous data acquisition S1-1, used for data acquisition for desliming efficiency evaluation and brush bristle remaining life prediction, employs a miniature pressure-resistant camera deployed on the top of the quick-release mud hood 5 of the deep-sea mining vehicle's self-cleaning tracked walking device. This camera acquires images of the track plate surface within the working area of ​​the brush bristles 6 at a fixed acquisition frequency, providing a direct view of the clay adhesion and removal process on the track. Hydrodynamic sensor arrays are deployed on both sides of the inner wall of the plume hood to collect real-time data on the water flow velocity and direction within the plume hood 4, quantifying the auxiliary effect of the water flow counter-current on desliming. Through data acquisition... The mining truck controller's local area network bus connects to the drive wheel torque sensor and the track speed encoder, which are used to collect real-time drive wheel torque time-series data and track rotation speed time-series data, respectively, to characterize the energy consumption and traction status of the walking device. Inside the on-board edge computing controller, continuous time-dimensional integration is performed on the real-time track rotation speed time-series data to obtain the cumulative physical distance actually traveled by the track. The calculated cumulative physical distance is divided by the preset fixed mechanical circumference of a single track rotation to obtain the cumulative number of track rotations. S1-2 Data Preprocessing and Labeling: The collected track plate surface images, water flow velocity time series data, water flow direction time series data, drive wheel torque time series data, and track rotation speed time series data are time-stamped to ensure that different modal data at the same time form a one-to-one correspondence. Based on the differences in the physical reflection characteristics of the track plate surface in the image, the track plate surface image is annotated at the pixel level. The track tooth area, due to its metal material, exhibits a high gray value range and regular linear texture features. The track plate base area, due to its original anti-rust coating, exhibits a medium gray value range and uniform texture. The residual clay area, due to its water-containing sediments, exhibits a low gray value range and fine but irregular texture. Therefore, based on gray value threshold segmentation and local binary mode analysis, the track plate area in the track plate surface image is segmented and annotated into the track tooth area, the track plate base area, and the residual clay area.

[0026] S2. Multi-source heterogeneous data fusion and state feature extraction based on physical information guidance and causal inference. S2-1 Deep mining of image features fused with hydrodynamic conditions: The track plate surface images, time-series data of water velocity inside the plume, and time-series data of water direction inside the plume acquired in S1-1 are input into a convolutional neural network guided by physical information. This network includes a semantic segmentation network based on an encoder-decoder structure and a physical information layer; The semantic segmentation network uses four convolutional layers as the encoder. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to progressively extract the spatial features of the input track plate surface image. The fourth convolutional layer of the encoder outputs a deep semantic feature map. The physical information layer takes flow velocity time series data and flow direction time series data as input, and outputs a spatial weight mask with the same size as the deep semantic feature map by calculating the angle between the water flow direction and the normal direction of the track plate at each spatial location, i.e., the water flow angle of attack and the flow velocity magnitude. The spatial weight mask is multiplied element-wise with the deep semantic feature map to achieve soft weighting of the feature map, guiding the network to pay more attention to image areas with significant water flow impact. The weighted feature map is input into the decoder of the semantic segmentation network. The decoder consists of four deconvolution layers. The output of the fourth deconvolution layer of the decoder is an environmental perception image feature with hydrodynamic influence fused at the same resolution as the input track plate surface image. The semantic segmentation network is pre-trained using the track tooth region, track plate base region and residual clay region marked in S1-2 as supervision targets, so that the preliminary segmentation feature map has the ability to accurately identify key components and clay distribution in the image.

[0027] S2-2 Dynamic-Visual Feature Correlation Analysis Based on Causal Inference: The time-series data of drive wheel torque and track rotation speed collected in S1-1 are combined into a two-dimensional dynamic time-series feature, which is then input into a time-series convolutional network based on Granger causality test.

[0028] The temporal convolutional network based on Granger causality test includes a Granger causality test module, a feature concatenation layer, and three temporal convolutional layers. First, the Granger causality test module performs Granger causality tests with lag orders from 1 to 10 on the dynamic time series features and the environmental perception image features output by S2-1. The specific processing procedure is as follows: For the lag order... The Granger causality test module constructs a constrained autoregressive model based on the lag order. This model uses the value of the environmental perception image features at the current time t as the dependent variable, and the environmental perception image features at... , Until Moment Using numerical values ​​as independent variables, the linear regression coefficients are estimated using the least squares method to obtain the constrained model fit values. The sum of squares of the differences between the dependent variable and the constrained model fit values ​​is then calculated and denoted as the constrained model residual sum of squares. Simultaneously, the Granger causality test module constructs an unconstrained autoregressive model, which, based on the independent variables of the constrained autoregressive model, incorporates dynamic time-series characteristics. , Until Moment The numerical values ​​are used as additional independent variables. The linear regression coefficients are estimated by the least squares method to obtain the unconstrained model fit values. The sum of squares of the differences between the dependent variable and the unconstrained model fit values ​​is calculated and denoted as the unconstrained model residual sum of squares. The F-statistic is calculated based on the sum of squared residuals of the constrained and unconstrained models. Specifically, the difference between the sum of squared residuals of the constrained and unconstrained models is divided by the lag order. Obtain the mean of the residual sum of squares, then divide the mean of the residual sum of squares by the sum of squares of the unconstrained model residuals, and then divide by the time series length minus The difference between the F-statistic and the F-statistic is used to obtain the F-statistic; the corresponding significance level is obtained by looking up the F-distribution table based on the F-statistic, and the lag order that makes the F-statistic reach its maximum value is recorded as the optimal lag order.

[0029] Then, the feature stitching layer aligns the dynamic temporal features with a significance level below 0.05 with the environmental perception image features generated by S2-1 in the time dimension according to the optimal lag order, and then stitches them in the channel dimension to form multimodal stitched features.

[0030] Finally, the multimodal splicing features are input into three temporal convolutional layers. The first temporal convolutional layer uses a causal convolution with a kernel size of 3 to extract local temporal patterns from the multimodal splicing features and output primary temporal features. The second temporal convolutional layer uses a causal convolution with a kernel size of 5 to extract temporal patterns with an expanded receptive field using the primary temporal features as input and output intermediate temporal features. The third temporal convolutional layer uses a causal convolution with a kernel size of 7 to extract global temporal dependencies using the intermediate temporal features as input and outputs a fusion state vector that comprehensively reflects the dynamic relationship between walking, mud removal, and the environment.

[0031] The overall process of this step is as follows: Figure 5 As shown.

[0032] S3. Desliming efficiency assessment and bristle remaining life prediction S3-1 Visual semantic analysis of desliming efficiency: A desliming efficiency evaluation network is constructed, which adopts an encoder-decoder structure. The track plate surface image is input into the encoder, which outputs high-level semantic features. The encoder part contains three depthwise separable convolutional blocks. Each convolutional block consists of a depthwise separable convolutional layer, a batch normalization layer, and a ReLU activation function, which are used to progressively compress the spatial resolution of the track plate surface image and extract high-level semantic information. The decoder part uses bilinear interpolation upsampling layers to progressively restore the high-level semantic features to the original image resolution. After each upsampling layer, a feature fusion module is introduced. This module maps the fusion state vector generated in S2-2 through a fully connected layer and concatenates it with the feature map obtained from the upsampling layer in terms of channel dimension to achieve real-time and accurate segmentation of the residual clay area. The decoder finally outputs a segmentation probability map with the same resolution as the input image, where each pixel in the map has the probability of belonging to the track tooth area, the track plate base area, and the residual clay area. Based on the segmentation probability map, the residual clay area ratio is obtained by calculating the percentage of the total number of pixels in the residual clay area to the total number of pixels in the track plate area, and the average thickness of the residual clay is obtained by calculating the average of the actual physical thickness of each pixel in the residual clay area. The residual clay area ratio and the average thickness of the residual clay together reflect the mud removal efficiency. S3-2 Brush Filament Remaining Life Prediction Network: The fused state vector obtained from S2-2, the residual clay area ratio and average residual clay thickness output from S3-1, the drive wheel torque time series data collected in S1-1, and the cumulative rotation number feature composed of the cumulative rotation number of the track are simultaneously input into the brush filament remaining life network; the brush filament remaining life network includes two long short-term memory network layers, three expert sub-networks, a gated network, and a fully connected layer; First, the fused state vector is input into two layers of long short-term memory (LSTM) network. The first LSM network layer contains 128 hidden units, which model the temporal dependency of the input fused state vector and output the first layer of temporal hidden state. The second LSM network layer contains 64 hidden units, which take the first layer of temporal hidden state as input and further capture the degradation trend of features over time, and output the second layer of temporal hidden state.

[0033] Secondly, the second-layer temporal hidden state is concatenated with the residual clay area ratio and the average residual clay thickness and then input into the first expert sub-network to output the mud removal efficiency feature; the second-layer temporal hidden state is concatenated with the driving wheel torque temporal data and then input into the second expert sub-network to output the power load feature; the second-layer temporal hidden state is concatenated with the cumulative number of track rotations and then input into the third expert sub-network to output the physical wear feature; each expert sub-network is a two-layer fully connected network, the first fully connected layer contains 32 neurons connected to the ReLU activation function, and the second fully connected layer contains 16 neurons connected to the ReLU activation function; The gating network is a two-layer fully connected network. The first fully connected layer contains 24 neurons connected to the ReLU activation function, and the second fully connected layer contains 3 neurons connected to the Softmax activation function. Sludge removal efficiency features, dynamic load features, and physical wear features are input into the gating network. The Softmax activation function is used to calculate and select the weights of each expert subnetwork, outputting an expert weight vector. The expert weight vector is then weighted and summed with the outputs of the three expert subnetworks to obtain the fused expert features. The fusion expert features are input into a fully connected layer containing one neuron and then connected to a Softmax activation function to output the predicted value of the remaining lifespan of the brush bristles.

[0034] S4. Experimental Analysis and Verification To verify the effectiveness of the proposed multimodal data fusion and desliming efficiency evaluation method based on the track-sediment interaction mechanism, three sets of experiments were designed based on real operational data collected by the self-cleaning tracked walking device of a deep-sea mining vehicle. The experiments evaluated the operational condition representation capability of the multimodal dataset, the real-time parsing accuracy of the lightweight semantic segmentation network, and the brush wear life prediction performance based on a hybrid expert system. All comparative experiments used the same data partitioning and evaluation metrics.

[0035] 1. Feature analysis of multimodal datasets: The experiment first analyzed the variation of the residual clay area ratio and the drive wheel torque fluctuation with track speed in the constructed multimodal dataset. For example... Figure 6 As shown, the solid blue line represents the percentage of residual clay area, the dashed red line represents the torque fluctuation of the drive wheel, and the horizontal axis represents the track speed. The background color in the figure is divided into three regions according to the desliming efficiency levels defined in S1-2: green represents the high-efficiency desliming zone (torque fluctuation less than 5% of rated torque and clay percentage less than 10%), orange represents the low-efficiency desliming zone (torque fluctuation 5%-10%, clay percentage 10%-30%), and red represents the failed desliming zone (torque fluctuation exceeding 10% and clay percentage higher than 30%). The curve trend shows that when the track speed is below 0.7 m / s, there is less clay adhesion and the torque is stable, indicating that the system is in a high-efficiency desliming state. As the speed increases, clay residue and torque fluctuation rise simultaneously, entering the low-efficiency zone. After the speed exceeds 1.3 m / s, a large amount of clay accumulates and the torque fluctuates drastically, indicating a significant decrease in the brush cleaning ability. This visualization result intuitively verifies the dataset's ability to accurately characterize key physical quantities. The data distribution under different working conditions conforms to theoretical expectations, providing a reliable foundation for subsequent fusion modeling.

[0036] 2. Comparison of real-time semantic segmentation performance in mud removal efficiency To verify the advantages of the lightweight deep semantic segmentation network proposed in S3-1 in real-time mud removal efficiency analysis, this invention compares it with two mainstream semantic segmentation models, DeepLabv3 and PSPNet. All models were trained on the same high-resolution image dataset, and the mean Intersection over Union (mIoU) was used as the accuracy metric. Figure 7 The mIoU convergence curves of the three models over 50 training epochs are presented. Our lightweight network employs a depthwise separable convolution and feature fusion design, significantly reducing computational cost while maintaining high segmentation accuracy. The curves show that our network converges fastest, with mIoU stabilizing at around 0.92 after approximately 20 epochs, significantly outperforming DeepLabv3's 0.87 and PSPNet's 0.83. This is attributed to the introduction of fused state vectors in the encoder-decoder structure, enabling the network to leverage hydrodynamic and dynamic information to enhance its ability to identify clay regions. Real-time inference tests demonstrate that our network achieves a single-frame processing time of only 28 ms on an embedded controller, fully meeting real-time evaluation requirements.

[0037] 3. Comparison of the accuracy of remaining life prediction during the bristle wear stage Predicting the remaining life of brush bristles is crucial for maintenance decisions. This invention, based on the Long Short-Term Memory Network-Hybrid Expert System LSTM-MoE proposed in S3-2, compares it with standard LSTM, GRU, and Support Vector Regression (SVR). The experiment uses the same input features (fused state vector, desliming efficiency index, drive wheel torque timing, and cumulative rotations), and evaluates the remaining life prediction using the root mean square error (RMSE) as the metric, across three wear stages. Figure 8 A bar chart comparing the RMSE of each model at the initial, intermediate, and severe wear stages is provided. The LSTM-MoE model presented in this paper achieves the lowest error across all stages: 2.1 days for initial wear, 3.5 days for intermediate wear, and 4.8 days for severe wear; while the standard LSTM models achieve 2.8, 4.2, and 6.1 days respectively, with GRU and SVR showing even larger errors. The hybrid expert system dynamically combines three sub-networks of expert data for different operating conditions through a gating network, enabling adaptive adjustment of the prediction mode for different wear stages and flow conditions, thereby significantly improving accuracy.

[0038] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0039] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for evaluating the self-cleaning and desliming efficiency of deep-sea mining vehicles and predicting the remaining lifespan of their bristles, characterized in that: A self-cleaning tracked walking device for deep-sea mining vehicles is provided, comprising a plume cover installed on the track and a mud cover brush with bristles installed inside near the drive wheel; and includes the following steps: S1, collects and preprocesses images of the track plate surface, time-series data of water flow velocity inside the plume, time-series data of water flow direction inside the plume, time-series data of drive wheel torque and track rotation speed; S2, the preprocessed track plate surface image, water flow velocity time series data, and water flow direction time series data are input into a physical information-guided convolutional neural network to obtain environmental perception image features of hydrodynamic influence; the preprocessed drive wheel torque time series data and track rotation speed time series data are synthesized into two-dimensional dynamic time series features, which are input into a time series convolutional network based on Granger causality test, and combined with environmental perception image features, output a fusion state vector reflecting the dynamic relationship between walking mud removal and the environment; S3. The preprocessed track plate surface image is input into the encoder of the mud removal efficiency evaluation network to output high-level semantic features. In the decoder, the high-level semantic features are concatenated with the fused state vector through the feature fusion module to output a segmentation probability map, and then the residual clay area ratio and the average thickness of residual clay are calculated. Next, the fused state vector is input into the brush bristle remaining life prediction network to output the predicted value of brush bristle remaining life.

2. The method for evaluating the self-cleaning and desliming efficiency of deep-sea mining vehicles and predicting the remaining lifespan of brush bristles as described in claim 1, characterized in that: The preprocessing includes: Based on the differences in the physical reflection characteristics of the track plate surface in the track plate surface image, the track plate surface image is annotated at the pixel level. Based on grayscale threshold segmentation and local binary mode analysis, the track plate region in the track plate surface image is segmented and annotated into the track tooth region, the track plate base region, and the residual clay region.

3. The method for evaluating the self-cleaning and desliming efficiency of deep-sea mining vehicles and predicting the remaining lifespan of brush bristles as described in claim 1, characterized in that: The specific process of S2 includes: First, image feature deep mining based on hydrodynamic conditions is performed. The track plate surface image is input into the encoder of the semantic segmentation network to extract and output a deep semantic feature map. At the same time, the time series data of water flow velocity and water flow direction inside the plume are input into the physical information layer to calculate and output a spatial weight mask. The spatial weight mask is multiplied element-wise with the deep semantic feature map and then input into the decoder. Finally, the environmental perception image features that have been fused with hydrodynamic influence are output. Secondly, a dynamic visual feature correlation analysis based on causal inference is performed. The time series data of drive wheel torque and track rotation speed are combined into dynamic time series features as input. The Granger causality test module is used to test the dynamic time series features and environmental perception image features to output the optimal lag order. Then, the two are aligned according to the optimal lag order and input into the feature stitching layer to output multimodal stitched features. Finally, the multimodal stitched features are input into three temporal convolutional layers to extract and output a fusion state vector that comprehensively reflects the dynamic correlation between walking mud removal and the environment.

4. The method for evaluating the self-cleaning and desliming efficiency of deep-sea mining vehicles and predicting the remaining lifespan of brush bristles as described in claim 3, characterized in that: The physical information-guided convolutional neural network includes a semantic segmentation network based on an encoder-decoder structure and a physical information layer. The semantic segmentation network uses four convolutional layers as the encoder. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to progressively extract the spatial features of the input track plate surface image. The fourth convolutional layer of the encoder outputs a deep semantic feature map. The physical information layer takes flow velocity time series data and flow direction time series data as input, and outputs a spatial weight mask with the same size as the deep semantic feature map by calculating the angle between the water flow direction and the normal direction of the track plate at each spatial location, i.e., the water flow angle of attack and the flow velocity magnitude. The spatial weight mask is multiplied element-wise with the deep semantic feature map to achieve soft weighting of the feature map, guiding the network to pay more attention to image regions with significant water flow impact effects. The weighted feature map is input into the decoder of the semantic segmentation network. The decoder consists of four deconvolutional layers. The output of the fourth deconvolutional layer of the decoder is an environmental perception image feature with hydrodynamic influence fused at the same resolution as the input track plate surface image.

5. The method for evaluating the self-cleaning and desliming efficiency of deep-sea mining vehicles and predicting the remaining lifespan of brush bristles as described in claim 3, characterized in that: The temporal convolutional network based on Granger causality test includes a Granger causality test module, a feature splicing layer, and three temporal convolutional layers. First, the Granger causality test module performs Granger causality tests with lag orders from 1 to 10 on the dynamic time-series features and the output environmental perception image features: for the lag order... The Granger causality test module constructs a constrained autoregressive model based on the lag order. The model uses the value of the environmental perception image features at the current time t as the dependent variable, and the environmental perception image features at... , Until Moment Using numerical values ​​as independent variables, the linear regression coefficients are estimated using the least squares method to obtain the constrained model fit values. The sum of squares of the differences between the dependent variable and the constrained model fit values ​​is then calculated and denoted as the constrained model residual sum of squares. Simultaneously, the Granger causality test module constructs an unconstrained autoregressive model, which, based on the independent variables of the constrained autoregressive model, incorporates dynamic time-series characteristics. , Until Moment The numerical values ​​are used as additional independent variables. The linear regression coefficients are estimated by the least squares method to obtain the unconstrained model fit values. The sum of squares of the differences between the dependent variable and the unconstrained model fit values ​​is calculated and denoted as the unconstrained model residual sum of squares. Then calculate the F-statistic: First, subtract the sum of squared residuals of the unconstrained model from the sum of squared residuals of the constrained model, and divide by the lag order. Obtain the mean of the residual sum of squares, then divide the mean of the residual sum of squares by the residual sum of squares of the unconstrained model, and then divide by the time series length minus ( The difference between the two values ​​is used to obtain the F-statistic; the F-statistic is used to look up the F-distribution table to obtain the corresponding significance level, and the lag order that makes the F-statistic reach its maximum value is recorded as the optimal lag order. Subsequently, the feature stitching layer aligns the dynamic temporal features with a significance level below 0.05 with the generated environmental perception image features in the time dimension according to the optimal lag order, and then stitches them in the channel dimension to form multimodal stitched features. Finally, the multimodal concatenated features are input into three temporal convolutional layers, and the output is a fused state vector.

6. The method for evaluating the self-cleaning and desliming efficiency of a deep-sea mining vehicle and predicting the remaining lifespan of its bristles as described in claim 1, characterized in that: The specific process of S3 includes: First, visual semantic parsing of desliming efficiency is performed. The track plate surface image is input into the encoder of the desliming efficiency evaluation network, which outputs high-level semantic features. In the decoder, the high-level semantic features are concatenated with the fused state vector through the feature fusion module to output a segmentation probability map. Then, the residual clay area ratio and the average thickness of residual clay are calculated and output. Second, the remaining life of the brush bristles is predicted. The fused state vector is input into two layers of long short-term memory network, which outputs the first and second temporal hidden states in sequence. Then, the second temporal hidden state is concatenated with the residual clay area ratio, the average thickness of residual clay, the driving wheel torque time series data, and the cumulative number of track rotations, and then input into three expert subnetworks, which output the desliming efficiency expert feature, the driving torque expert feature, and the cumulative rotation expert feature in sequence. At the same time, the second temporal hidden state is input into a gating network to output an expert weight vector. Finally, the expert weight vector is weighted and summed with the above three expert features to obtain the fused expert feature, and this feature is input into the fully connected layer to output the predicted value of the remaining life of the brush bristles.

7. The method for evaluating the self-cleaning and desliming efficiency of a deep-sea mining vehicle and predicting the remaining lifespan of its bristles as described in claim 6, characterized in that: The desliming efficiency evaluation network adopts an encoder-decoder structure. The track plate surface image is input into the encoder and the high-level semantic features are output. The encoder part contains three depth-separable convolutional blocks. Each convolutional block consists of a depth-separable convolutional layer, a batch normalization layer and a ReLU activation function, which are used to progressively compress the spatial resolution of the track plate surface image and extract high-level semantic information. The decoder part gradually restores the high-level semantic features to the original image resolution through bilinear interpolation upsampling layers, and introduces a feature fusion module after each upsampling layer. This module maps the generated fusion state vector through a fully connected layer and then concatenates it with the feature map obtained from the upsampling layer along the channel dimension. The decoder outputs a segmentation probability map with the same resolution as the input image, where each pixel has the probability of belonging to the tooth region, track plate base region, and residual clay region. Based on the segmentation probability map, the residual clay area ratio is obtained by calculating the percentage of the total number of pixels in the residual clay region to the total number of pixels in the track plate region, and the average thickness of the residual clay is obtained by calculating the average of the actual physical thickness of each pixel in the residual clay region.

8. The method for evaluating the self-cleaning and desliming efficiency of a deep-sea mining vehicle and predicting the remaining lifespan of its bristles as described in claim 6, characterized in that: The remaining life prediction network for the bristles includes two long short-term memory network layers, three expert sub-networks, a gated network, and a fully connected layer. First, the fused state vector is input into two layers of long short-term memory network. The first layer of long short-term memory network performs time-dependent modeling on the input fused state vector and outputs the first layer of temporal hidden state. The second layer of the Long Short-Term Memory network takes the first layer's temporal hidden state as input, further captures the degradation trend of features over time, and outputs the second layer's temporal hidden state. Next, the second-layer temporal hidden state is spliced ​​with the residual clay area ratio and the average thickness of residual clay and then input into the first expert sub-network to output the mud removal efficiency characteristics. The second-layer hidden temporal state is concatenated with the driving wheel torque timing data and then input into the second expert sub-network to output the power load characteristics; the second-layer hidden temporal state is concatenated with the cumulative number of track rotations and then input into the third expert sub-network to output the physical wear characteristics; each expert sub-network is a two-layer fully connected network; The gating network is a two-layer fully connected network. The first fully connected layer contains 24 neurons connected to the ReLU activation function, and the second fully connected layer contains 3 neurons connected to the Softmax activation function. The sludge removal efficiency features, dynamic load features, and physical wear features are input into the gating network. The weights of each expert subnetwork are calculated and selected through the Softmax activation function, and the expert weight vector is output. The expert weight vector is then weighted and summed with the outputs of the three expert subnetworks to obtain the fused expert features. The expert features are fused and input into the fully connected layer, and then the Softmax activation function is used to output the predicted value of the remaining brush life.

Images