Prefabricated bridge rubber support deformation detection method and system based on MVMFormer

Abstract

The invention relates to an MVMFormer-based prefabricated bridge rubber support deformation detection method and system, which are used for carrying out compression and shear performance tests on prefabricated bridge rubber supports made of different materials. Obtaining different deformation diagrams under a vertical load in the testing process, a vertical load limit diagram when deformation and damage occur, a deformation diagram in the simulation process, a limit diagram under single-side stress when deformation and damage occur, a deformation diagram in the simulation process and a transverse force limit diagram when deformation and damage occur; forming a classification data set by using the deformation graph, the corresponding experiment working condition and the non-deformation original graph, and training a classifier by using the classification data set; and identifying the experimental working condition of the image to be detected by using the classifier, and then performing segmentation. Corresponding limit diagrams and deformation diagrams under different experiment working conditions are obtained according to compression and shear performance tests, semantic segmentation is carried out on the basis, and detection and timely early warning of the phase rubber support deformation are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of deformation measurement of precast bridge rubber bearings, and particularly to a deformation detection method and system for precast bridge rubber bearings based on MVMFormer. Background Art

[0002] Precast bridge rubber bearings are key structural components connecting the upper and lower structures of bridges. They provide a support surface between bridge piers and bridge decks for transmitting loads such as seismic loads, traffic loads, wind loads, and the self-weight of the upper structure. Sliding bearings allow the upper structure to move due to reasons such as traffic and thermal expansion. Anti-pulling bearings are used to prevent the upward pull of the main beam of the bridge and maintain structural stability. Bearings will inevitably be deformed due to factors such as the external environment, material aging, and overloading, thus affecting the safety, serviceability, and durability of the bridge. Therefore, it is necessary to monitor the bridge deformation for a long time to avoid the safety risk of bridge collapse.

[0003] In current practice, precast bridge rubber bearings are mainly inspected through regular manual measurement. The above method requires the use of large-scale measurement equipment, and the whole process is labor-intensive, time-consuming, and subjective. Currently, with the rapid development of intelligent technologies, their applications in target deformation measurement are becoming more and more extensive. Intelligent theories and technologies mainly include artificial intelligence, machine learning, image processing, etc. By applying these technologies, automatic recognition, classification, and evaluation of target deformation can be achieved, improving the detection efficiency and accuracy. The development of image analysis methods based on deep learning provides a new way to solve the deformation of precast bridge rubber bearings, and the present invention proposes a deformation detection method for precast bridge rubber bearings based on MVMFormer. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above-mentioned technical drawbacks and provide a stable, high-precision, and intelligent deformation detection method and system for precast bridge rubber bearings based on MVMFormer. Realize the measurement and analysis of precast bridge rubber bearings, and can accurately and efficiently detect the deformation of precast bridge rubber bearings and quantify it, providing an effective means for the maintenance of precast bridge rubber bearings.

[0005] To achieve the above purpose, the technical solution of the present invention is as follows:

[0006] A deformation detection method for precast bridge rubber bearings based on MVMFormer includes the following steps:

[0007] Perform compression and shear performance tests on precast bridge rubber bearings of different materials, apply vertical pressure to the rubber bearings to test their compression performance, and obtain different deformation diagrams under vertical loads during the test and the vertical load limit diagram when deformation failure occurs;

[0008] And simulate the deformation caused by the excessive unilateral force on the precast bridge rubber bearing in actual application, and obtain the deformation diagram during the simulation process and the limit diagram under the unilateral force when deformation failure occurs;

[0009] Simulate the deformation caused by the influence of the lateral force in extreme cases, and obtain the deformation diagram during the simulation process and the lateral force limit diagram when deformation failure occurs;

[0010] Mark the above three deformation diagrams with corresponding experimental conditions respectively, and form a classification data set with the deformation diagrams, the corresponding experimental conditions, and the original diagram without deformation. Use the classification data set to train a classifier to identify the experimental conditions to which the picture belongs and whether there is deformation;

[0011] Perform pixel-level annotation on the deformation diagram and the original diagram without deformation, assign corresponding semantic labels to each pixel, and generate a binary diagram after annotation. The black pixel value is 0, representing the background, and the white pixel value is 255, representing the target, to form a data set with target annotation information;

[0012] Construct an MVMFormer semantic segmentation model, and train the MVMFormer semantic segmentation model with the data set with target annotation information;

[0013] Use the classifier to identify the experimental conditions to which the image to be measured belongs, and then after being processed by the trained MVMFormer semantic segmentation model, output the segmentation diagram of the target object in the image to be measured;

[0014] Obtain the pixel area A of the target object in the segmentation diagram and the pixel area B of the target object in the limit diagram corresponding to the experimental conditions;

[0015] Set a first warning threshold and a second warning threshold for each experimental condition. If A / B is less than the first warning threshold, it is a first-level warning deformation alarm; if A / B is not less than the first warning threshold and less than the second warning threshold, it is a second-level warning deformation alarm, reminding for timely processing; the first warning threshold is less than the second warning threshold.

[0016] Furthermore, in the compression and shear performance test of the precast bridge rubber bearing, first increase the force linearly and then decrease the force linearly, observe whether it can return to the initial state after the applied force, collect the deformation diagram during the process, and use the deformation diagram when it cannot return to the initial state as the limit diagram.

[0017] Furthermore, during the construction of the data set, the installation parameters of the equipment used for taking pictures of the deformation diagram are the same as those in actual application.

[0018] Furthermore, the MVMFormer semantic segmentation model includes: a framework that integrates mask image uniformity and unsupervised domain adaptation, simply referred to as the MIU framework; and a main network composed of a MambaVision encoder and a context-aware feature fusion decoder.

[0019] Furthermore, the framework for integrating mask image uniformity and unsupervised domain adaptation includes a teacher network, a source image and a corresponding one-hot label for training the main network f θ with the supervised loss L S ; After that, the source image generates a source prediction image through the main network, and the supervised loss between the source prediction image and the real image is calculated using the following formula;

[0020]

[0021] where the superscript s represents the corresponding data of the source image, Ns is the number of samples for each label, and the subscript k represents the k-th element;

[0022] At the same time, the target image is input into the main network, the mask operation, and the teacher network respectively. Among them, the weights of the teacher network are updated by exponential moving average EMA between the teacher network and the main network to transfer knowledge from the source domain to the target domain; the unlabeled image in the target domain is input into the main network, and the adaptation loss is calculated using the following formula

[0023]

[0024] where H and W represent the height and width of the target image, H×W is the image pixel area, C represents the number of classes, represents the quality estimate corresponding to the pseudo-label of the target image, represents the true label of the target image, represents the output of the main network; the superscript T represents the corresponding data of the target image; θ is the model parameter of the main network;

[0025] When the target image is input into the mask operation, by masking random patches of the target image, the MIU module trains the network to predict the semantic segmentation of the entire image, and the MIU loss is L M : First, a mask M is randomly sampled from a uniform distribution to obtain a masked image. The masked target image is obtained by element-wise multiplication of the mask and the target image, and the masked target image is input into the main network to obtain a masked prediction result;

[0026] When the target image is input into the teacher network, the pseudo-label of the target domain data is obtained according to the formula ;

[0027] where Denote the pseudo-label of the target domain data;

[0028]

[0029] Denote the output of the teacher network, i.e., the probability distribution of the feature vectors of the target domain data updated based on the source domain; c is the class label, taking integer values from 1 to C; Denote the class label that obtains the maximum probability value among all class labels c';

[0030] Use the teacher network to generate pseudo-labels for the target domain data, and utilize local and context information to improve the quality of the pseudo-labels in the target domain, thereby generating more reliable labels;

[0031] The pseudo-labels of the target images and their quality estimates are used for further training on the target domain; the mask prediction and the pseudo-labels are input into the MIU loss calculation, and the MIU loss calculation formula is:

[0032] L M = q T H(y M , p T )

[0033] where p T denotes the pseudo-label, q T is the quality estimate of the target image; y M is the predicted probability of the main network for the masked target image; H(·) represents the maximum entropy function;

[0034] The final output of the neural network is obtained through the constraint training of three loss functions, and the image after semantic segmentation is obtained.

[0035] Furthermore, in the main network, the encoder includes four stages, where the first two stages are composed of residual convolutional blocks and downsamplers, while the 3rd and 4th stages include MambaVision hybrid layers and MLP layers and self-attention layers.

[0036] Furthermore, the MambaVision hybrid layer includes two branches, one branch without an SSM module, one branch includes a linear layer, a one-dimensional convolution, and activation by a sigmoid function connected in sequence, and the other branch includes a linear layer, a one-dimensional convolution, activation by a sigmoid function, and an SSM connected in sequence; the outputs of the two branches are processed by a connection layer and a linear layer to obtain the output of the MambaVision hybrid layer.

[0037] The self-attention layer uses the lightning self-attention mechanism.

[0038] ​The present invention also protects a deformation detection system for precast bridge rubber bearings, characterized in that the method is executed, and the system includes:

[0039] An image acquisition module for obtaining a cross-sectional deformation map of a precast bridge rubber bearing;

[0040] A classifier module for identifying the experimental conditions to which the picture belongs and whether there is deformation;

[0041] An image processing module for segmenting and labeling the deformation map of the image acquisition module at the pixel level according to the category, and generating a binary map after the labeling is completed. The black pixel value is 0, representing the background, and the white pixel value is 255, representing the target, to obtain a dataset with target annotation information, including deformation maps and non-deformed normal maps;

[0042] A semantic segmentation module for training a semantic segmentation model using the dataset with target annotation information; after the classifier identifies the deformation and the experimental conditions to which it belongs, and then processes it through the trained semantic segmentation model, an output segmentation map of the target object is obtained;

[0043] An area comparison module for comparing the pixel area of the bearing in the segmentation map output by the semantic segmentation module with the pixel area of the target object in the limit map under the corresponding experimental conditions, and calculating the area ratio;

[0044] An early warning and feedback module for determining whether to alarm and process according to the relationship between the area ratio and the corresponding early warning threshold.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] The method of the present invention obtains the corresponding limit maps and deformation maps under different experimental conditions according to the compression and shear performance tests, and performs semantic segmentation based on this, realizing the detection and timely early warning of the deformation of the rubber bearing.

[0047] The present invention applies the MVMFormer semantic segmentation model to the deformation detection of precast bridge rubber bearings, which has the function of finely identifying image information, can identify and quantify the deformation of precast bridge rubber bearings, can overcome the deficiencies of the difficulties in manually monitoring the deformation of precast bridge rubber bearings, avoid potential hazards to operators and the need for equipment maintenance, realize monitoring and control, improve construction efficiency, reduce dependence on professional and technical personnel, reduce labor costs, and relieve the pressure of staff working in harsh environments, and provide a more reliable and safe solution for the deformation detection of precast bridge rubber bearings. It shows obvious advantages in terms of mRecall, mF1 and mIoU indicators compared with the existing algorithms.

[0048] The present invention provides the possibility for realizing the deformation detection of precast bridge rubber bearings on a larger scale. A large amount of collected data can be used for further analysis, providing data support for improving the deformation detection of precast bridge rubber bearings. After training, it can adapt to different production environments and various types of bridge rubber bearing structures. It further promotes the development of this field, promotes the transformation of traditional manufacturing to intelligent manufacturing, has great significance and broad application prospects, promotes the innovation and application of related technologies, and at the same time provides strong support and guarantee for the development of the deformation detection of precast bridge rubber bearings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of the overall structure of the MVMFormer semantic segmentation model in an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of the structure of the MambaVision encoder of the MVMFormer semantic segmentation model in the present invention;

[0051] Figure 3(a) is a schematic diagram of the internal structure of the Stem block in the MambaVision encoder of the present invention;

[0052] Figure 3(b) is a schematic diagram of the internal structure of the residual convolution block in the MambaVision encoder of the present invention;

[0053] Figure 3(c) is a schematic diagram of the internal structure of the downsampler in the MambaVision encoder of the present invention;

[0054] Figure 3(d) is a schematic diagram of the internal structure of the MambaVision hybrid layer in the MambaVision encoder of the present invention;

[0055] Figure 4 is a schematic diagram of the structure of the context-aware feature fusion decoder in an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of the longitudinal force on the bearing when testing the precast bridge rubber bearing in the present invention;

[0057] Figure 6 is a schematic diagram of the longitudinal misalignment force on the bearing when testing the precast bridge rubber bearing in the present invention;

[0058] Figure 7 is a schematic diagram of the lateral misalignment force on the bearing when testing the precast bridge rubber bearing in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] To more clearly describe the technical problems, technical solutions, and advantages of the present invention, the following will be described in detail with reference to the drawings and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as a limitation of the present invention.

[0060] Embodiment 1

[0061] The deformation detection method of the precast bridge rubber bearing based on MVMFormer in this embodiment includes the following steps:

[0062] Step 1: Obtain a deformation image dataset of the precast bridge rubber bearing. The dataset includes deformation images and normal images, and perform pixel-level annotation on the images, assigning corresponding semantic labels to each pixel; obtain source domain data (annotated images) and target domain data (unannotated images);

[0063] Step 2: Use the MVMFormer semantic segmentation model for semantic segmentation defect measurement

[0064] The MVMFormer semantic segmentation model includes: a framework integrating Masked Image Uniformity (MIU) and Unsupervised Domain Adaptation (UDA), simply referred to as the MIU framework; a main network composed of a MambaVision encoder and a context-aware feature fusion decoder.

[0065] The framework integrating Masked Image Uniformity and Unsupervised Domain Adaptation includes a teacher network, source images and a corresponding one-hot label for training the main network f θ and the supervised loss L S ; After that, the source image generates a source prediction image through the main network, and the supervised loss between the source prediction image and the real image is calculated using the following formula;

[0066]

[0067] where the superscript s represents the corresponding data of the source image, Ns is the number of samples for each label, and the subscript k represents the kth element;

[0068] At the same time, the target image is respectively input into the main network, the masking operation, and the teacher network. Among them, the weights of the teacher network are updated by exponential moving average (EMA) between the teacher network and the main network to transfer knowledge from the source domain to the target domain; the unlabeled image of the target domain is input into the main network, and the adaptation loss is calculated using the following formula

[0069]

[0070] Among them, H and W represent the height and width of the target image, H×W is the image pixel area, and C represents the number of categories. represents the quality estimate corresponding to the pseudo-label of the target image. represents the true label of the target image. represents the output of the main network; the superscript T represents the corresponding data of the target image.

[0071] The adaptation loss is used to improve the ability to effectively generalize to unseen data. The model parameters θ of the main network are optimized with the adaptation loss, so that the predictions of the model on the target domain are closer to the true labels.

[0072] When the target image is input to the masking operation, by masking random patches of the target image, the MIU module trains the network to predict the semantic segmentation of the entire image. The MIU loss is L M : First, a mask M is randomly sampled from a uniform distribution to obtain a mask. The masked target image is obtained by element-wise multiplication of the mask and the target image. The masked target image is input into the main network to obtain the masked prediction result.

[0073] When the target image is input to the teacher network, according to the formula the pseudo-label of the target domain data is obtained.

[0074] Among them represents the pseudo-label of the target domain data. represents the output of the teacher network, that is, the probability distribution of the feature vector of the target domain data updated based on the source domain; c is the class label, taking integer values from 1 to C. represents the class label that obtains the maximum probability value among all class labels c'.

[0075] The teacher network is used to generate pseudo-labels for the target domain data, and local and context information is used to improve the quality of the pseudo-labels in the target domain, thus generating more reliable labels.

[0076] The pseudo-label of the target image and its quality estimate are used for further training on the target domain; the masked prediction and the pseudo-label are input into the calculation of the MIU loss. The MIU loss calculation formula is:

[0077] L M =q T H(y M ,p T )

[0078] Among them, p T represents the pseudo-label, q T is the quality estimate of the target image; y M is the prediction probability of the main network for the masked target image; H(·) represents the maximum entropy function.

[0079] The final output of the neural network is obtained through the constraint training of three loss functions, and the image after semantic segmentation is obtained, and then the pixel area of the target object is calculated.

[0080] In the main network, the encoder consists of four stages. The first two stages are composed of residual convolutional blocks and downsamplers, which are used for fast feature extraction at higher input resolutions. The third and fourth stages include MambaVision modules, MLP layers, and self-attention layers. Specifically, given an image of size H*W*3, the input is first converted into overlapping image patches of size H / 4*W / 4*C, where H represents the height pixels of the image and W represents the width pixels of the image, and is projected into a C-dimensional embedding space through a Stem layer (consisting of two consecutive 3*3 Conv layers with a stride of 2). The downsampler between stages consists of a 3*3 Conv layer with batch normalization and a stride of 2, which reduces the image resolution by half. The residual convolutional block consists of a 3*3 Conv layer followed by a BN layer, then a GELU activation function, then another 3*3 Conv layer and a BN layer, and finally added to the initial input.

[0081] The architecture of the MambaVision layer in the third and fourth stages. In addition to replacing the causal convolutional layers with their conventional counterparts, a symmetric path without SSM is created as a token mixer to enhance the modeling of global context.

[0082] The architecture of hierarchical MambaVision. The first two stages use residual convolutional blocks for fast feature extraction. The third and fourth stages use MambaVision, MLP layers, and self-attention layers simultaneously. The Transformer block in the final layer allows for the recovery of lost global context and the capture of long-range spatial dependencies. The Transformer block contains self-attention layers and MLP layers.

[0083] The MambaVision mixing layer is shown in Figure 3(d). The original Mamba mixing layer is redesigned to be more suitable for visual tasks. First, the causal convolution is replaced with a conventional convolution because it restricts unidirectional effects, which are unnecessary and restrictive for visual tasks. In addition, a symmetric branch without SSM is added, which contains an additional convolution and sigmoid function activation to compensate for any content loss due to the sequence limitations of SSMs. Then the outputs of the two branches are concatenated and projected through a final linear layer. This combination ensures that the final feature representation fuses sequence and spatial information, leveraging the advantages of both branches. And the output of each branch is projected into an embedding space of size C / 2 (i.e., half of the original embedding dimension) to maintain a similar number of parameters to the original block design. Given the input X in, the output X of the MambaVision mixing layer out is calculated as follows:

[0084]

[0085] where Linear(Cin, Cout)() represents a linear layer with Cin and Cout as input and output embedding dimensions, and Scan is a selective scanning operation.

[0086] where the self-attention layer uses the Lightning self-attention mechanism, and the formula is as follows:

[0087] O = Norm(Q(K T V))

[0088] where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively.

[0089] Example 2

[0090] The precast bridge rubber bearing detection method based on MVMformer in this example includes:

[0091] Conduct compression and shear performance tests on precast bridge rubber bearings of different materials, apply vertical pressure to the rubber bearings to test their compression performance, and obtain different deformation diagrams under vertical load during the test and the vertical load limit diagram when deformation failure occurs;

[0092] And simulate the deformation caused by the influence when the unilateral force is too large in the actual application of the precast bridge rubber bearing, and obtain the deformation diagram during the simulation process and the limit diagram under unilateral force when deformation failure occurs;

[0093] Simulate the deformation caused by the influence of lateral force in extreme cases, and obtain the deformation diagram during the simulation process and the lateral force limit diagram when deformation failure occurs;

[0094] Mark the above three deformation diagrams with corresponding experimental conditions respectively, and form a classification data set with the deformation diagrams, the corresponding experimental conditions, and the original diagrams without deformation. Use the classification data set to train a classifier to identify the experimental conditions to which the pictures belong and whether there is deformation;

[0095] Perform pixel-level annotation on the deformation diagrams and the original diagrams without deformation, assign corresponding semantic labels to each pixel, and generate a binary diagram after annotation. The black pixel value is 0, representing the background, and the white pixel value is 255, representing the target, to form a data set with target annotation information;

[0096] There are three experimental results for different precast materials, namely extreme cases under vertical load, extreme cases under unilateral force, and extreme cases under lateral force.

[0097] Compression tests were conducted on the bearings. As Figure 5 shown, this test simulated the deformation of precast bridge rubber bearings caused by excessive vertical pressure overload and other effects during actual application. For the longitudinal misalignment experiment, as Figure 6 shown, a difference of 80 mm was created at both ends of the specimen to simulate the deformation of precast bridge rubber bearings caused by the influence of excessive unilateral force during actual application. For the transverse misalignment experiment, as Figure 7 shown, the displacement control box on one side of the bearing was loosened, and a horizontal force was applied to the loosened control box with a horizontal jack until the crossbeam inclination angle reached 2.5°, then the displacement control box was fixed and the experiment was carried out. This was to simulate the deformation of the bridge caused by the influence of lateral forces under extreme conditions such as earthquakes or typhoons during actual application. Images of the deformation degree of the bearing under different extreme conditions were collected during each experiment, and the deformation diagrams during the simulation process and the limit diagrams when deformation failure occurred were obtained and saved in real time to the computer for detection and adjustment.

[0098] Throughout the process, the camera was fixed at the same position, and the distance (focal length) between the lens and the bridge rubber bearing was maintained at 40 mm. The image size captured by the camera was 1280×1024 pixels, and the highest sampling frequency was 9 frames per 5 minutes. When conducting actual detection, the parameters of the camera for taking pictures were the same as these parameters.

[0099] In this embodiment, the above three experiments can all be implemented according to the existing technical standards. During the compression and shear performance tests of precast bridge rubber bearings, the force was first increased linearly and then decreased linearly to see if it could return to the original state. The deformation diagram when the force increased to the last force value was collected. If the deformation was not restored, the image corresponding to the force at which deformation occurred was collected, which was the limit diagram, and at the same time, the original diagram without deformation was output.

[0100] Construct an MVMFormer semantic segmentation model and train the MVMFormer semantic segmentation model with a dataset with target annotation information;

[0101] Use a classifier to identify the experimental condition to which the image to be tested belongs, and then after being processed by the trained MVMFormer semantic segmentation model, output the segmentation map of the target object in the image to be tested;

[0102] Obtain the pixel area A of the target object in the segmentation map and the pixel area B of the target object in the limit diagram corresponding to the experimental condition;

[0103] For each experimental condition, a first warning threshold and a second warning threshold are set. If A / B is less than the first warning threshold, it is a first-level warning deformation alarm; if A / B is not less than the first warning threshold and less than the second warning threshold, it is a second-level warning deformation alarm to remind for timely handling; the first warning threshold is less than the second warning threshold.

[0104] The area to be recognized in the acquired image of the prefabricated bridge rubber bearing is labeled at the pixel level using the Labelme tool, and corresponding semantic labels are assigned to each pixel to form a dataset with target annotation information. After the annotation is completed, a binary image is generated, where the black pixel value is 0, representing the background, and the white pixel value is 255, representing the target area. The annotation information indicates the category to which each pixel in the image belongs at the pixel level. The dataset with target annotation information is divided into a training set plus a validation set and a test set in a ratio of 8:2.

[0105] Read the deformed images in the training set into the MVMFormer semantic segmentation model, and train to obtain the trained MVMFormer semantic segmentation model;

[0106] In this embodiment, the data in the training set is divided into 10 parts. Each time, 1 part out of the 10 parts is used as the validation set, and the rest are all used for training. After 10 times of training, 10 different models are obtained. Evaluate the effects of the 10 models, select the hyperparameters with the best mIOU, use the optimal hyperparameters, and then retrain the model with all 10 parts of the data as the training set to obtain the trained MVMFormer semantic segmentation model. After 160,000 iterations, when the training of the model reaches convergence, that is, when the training gradient of the model is close to 0 (less than 0.01 can be considered close to 0), stop the training, extract the optimal network parameters for prediction. If the training gradient is not close to 0, adjust the weight parameters. During prediction, the network first loads the trained parameters, loads the input image from the test set, calculates the category to which each pixel belongs through the trained MVMFormer semantic segmentation model, and thus realizes semantic segmentation.

[0107] The calculation method of the pixel area in the picture is as follows: Obtain the binary mask of the bearing through the image segmentation algorithm, that is, mark the bearing part as the foreground and the non-defective part as the background. Use the pixel counting method to calculate the number of pixels in the defective part, that is, the number of foreground pixels, denoted by N. Multiply the number of pixels N by the square of the pixel size to obtain the area A of the bearing part, with the unit of pixel square.

[0108] The binary mask refers to the process of converting a certain area of an image into a binary image. In the semantic segmentation task, the binary mask is used to represent the category information of each pixel, that is, each pixel point is marked as the foreground (target) or the background (non-target).

[0109] It should be noted that for the calculation of the pixel scale, the length of the scale in the image needs to be accurately measured, and the scale should be on the same plane as the bearing.

[0110] Example 3:

[0111] The deformation detection method of precast bridge rubber bearings based on MVMFormer in this embodiment uses the MVMFormer semantic segmentation model for semantic segmentation defect measurement. The MVMFormer semantic segmentation model includes a framework that integrates mask image uniformity and unsupervised domain adaptation, and a main network composed of a MambaVision encoder and a context-aware feature fusion decoder;

[0112] As Figure 2 shown, the main network image is projected into a C-dimensional embedding space through the Stem layer in the MambaVision encoder. In the first stage, the resolution of the image is reduced by half through the residual convolution block and the downsampler, and then through the residual convolution block and the downsampler in the second stage, and then enters the third stage. The first two stages use the residual convolution block for fast feature extraction. The third and fourth stages use the MambaVision mixing layer, the self-attention layer, and the MLP layer at the same time, and the two stages are connected by a downsampler. The self-attention layer and the MLP layer block in the final layer allow the restoration of the lost global context and the capture of long-range spatial dependencies.

[0113] Finally, after a two-dimensional average pooling layer and a linear layer, the image is input into the context-aware feature fusion decoder as Figure 7 shown. The context-aware feature fusion decoder uses multiple parallel 3×3 depthwise separable convolutions with different dilation rates and a 1×1 convolution. In this embodiment, a parallel 3×3 depthwise separable convolution with a dilation rate of 1, a parallel 3×3 depthwise separable convolution with a dilation rate of 4, and a 1×1 convolution are used to process the output of the stack, and then the results on all parallel branches are concatenated, and then passed through a 1×1 convolution to obtain the output prediction image.

[0114] The main network of the present invention can be used in bottleneck characteristics, can fuse all stacked multi-level characteristics, and can reduce the overfitting phenomenon of the source domain.

[0115] The structure of the Stem layer is shown in Figure 3, and the Stem layer is composed of two consecutive 3*3 Conv layers with a stride of 2 in series. The structure of the residual convolution block is as Figure 4 shown, and the residual convolution block is composed of a 3*3 Conv layer passing through a BN layer, a GELU activation function, then passing through a 3*3 Conv layer and a BN layer, and finally adding to the initial input.

[0116] The structure of the downsampler is as Figure 5 shown, and it is composed of a BN layer and a 3*3 Conv layer with a stride of 2.

[0117] The structure of the MambaVision mixing layer is as Figure 6As shown, it consists of two branches. The first branch obtains the output of the first branch after passing through a linear layer, one-dimensional convolution, a Sigmoid activation function, and then through a spatio-temporal state model (SSM). The second branch consists of a linear layer, one-dimensional convolution, and a Sigmoid activation function. The outputs of the two branches are merged through a connection layer and finally output through a linear layer to obtain the output of the MambaVision hybrid layer.

[0118] The self-attention layer uses the Lightning self-attention mechanism, and the formula is as follows:

[0119] O = Norm(Q(K T V))

[0120] where Q, K, and V represent the query matrix, key matrix, and value matrix respectively. This mechanism achieves a significant acceleration of attention calculation by removing the traditional softmax operation and using the associative law of matrix multiplication. By simplifying the calculation process, the high-complexity operations in the traditional attention mechanism are reduced to linear time complexity, i.e., O(N), where N is the sequence length.

[0121] In the overall network structure of MVMformer, a framework integrating masked image uniformity and unsupervised domain adaptation, the source image and a corresponding one-hot label are used to train the main network f θ with the supervised loss L S . Then, the source image generates a source prediction image through the main network, and then makes a comparison with the real image through the supervised loss

[0122]

[0123] At the same time, the target image is input into the main network and the MIU framework respectively. In the MIU framework, the teacher network is updated by exponential moving average (EMA) and the parameters of the main network are corrected.

[0124] First, the unlabeled images in the target domain are input into the main network, and then the adaptation loss is used to improve the ability to effectively generalize to unseen data:

[0125]

[0126] where H and W represent the height and width of the target image, H×W is the pixel area of the image, C represents the number of classes, represents the quality estimate corresponding to the pseudo-label of the target image, represents the true label of the target image, represents the output of the main network; the superscript T represents the corresponding data of the target image;

[0127] The purpose of this loss function is to optimize the parameters θ of the main network model so that the prediction of the main network on the target domain is closer to the true label.

[0128] When the target image is input into the MIU framework, by masking random patches of the target image, the MIU framework trains the network to predict the semantic segmentation of the entire image, and the MIU loss is L M : First, a mask M is randomly sampled from a uniform distribution to obtain the masked image. The masked target image is obtained by element-wise multiplication of the mask and the target image. The masked target image is input into the main network to obtain the masked prediction result for predicting the semantic segmentation of the entire image. Then, the MIU loss L M =q T H(y M ,p T ) is optimized. Among them, p T represents the pseudo-label of the target image, and q T represents the quality estimate of the corresponding target image, and y M is the prediction probability of the main network for the masked target image.

[0129] The teacher network updates the moving average EMA processed by the main network using the masked target image. When the target image is input into the teacher network, the output of the teacher network is obtained Then, the pseudo-label of the target domain data is obtained according to the following formula

[0130]

[0131] Among them, represents the probability distribution of the feature vector of the target domain data updated based on the source domain, c is the class label, represents the class label that obtains the maximum probability value among all class labels c'.

[0132] The teacher network is used to generate pseudo-labels for the target domain data, and local and context information is utilized during the pseudo-label generation process to improve the quality of the pseudo-labels in the target domain, thereby generating more reliable labels. This pseudo-label represents the transfer of knowledge from the source domain to the target domain. In addition, the pseudo-label and its quality estimate are used for further training on the target domain and for calculating the MIU loss. The masked prediction and the pseudo-label are input into the MIU loss calculation. The MIU loss calculation formula is:

[0133] L M =q T H(y M ,p T )

[0134] Among them, p T represents the pseudo-label of the target image, and q TQuality estimation of the target image; y M is the predicted probability of the main network for the masked target image; H(.) represents the maximum entropy function.

[0135] Finally, the supervision loss, adaptation loss, and MIU loss jointly constrain the training to obtain the final output of the MVMFormer semantic segmentation model.

[0136] In this embodiment, the experimental data is collected by an industrial camera. The image to be measured is the image of the precast bridge rubber bearing collected by the camera in the field. The training pictures are obtained from the compression and shear performance tests of the precast bridge rubber bearing in the laboratory. A vertical pressure is applied to the rubber bearing to test its compression performance, including the permanent deformation rate, and the horizontal displacement of the bridge under earthquakes and other conditions is simulated to test the shear performance of the rubber bearing. During the test, images of the precast bridge rubber bearing at different deformation degrees are acquired and saved to the computer in real time for detection and adjustment. Throughout the process, the camera is fixed at the same position, and the distance (focal length) between the lens and the bridge rubber bearing is maintained at 40 mm. The image size captured by the camera is 1280×1024 pixels, and the highest sampling frequency is 9 frames per 5 minutes.

[0137] The experimental dataset consists of 10,000 images of different precast bridge rubber bearings. Each image in the dataset is an RGB image of 1280×1024 pixels.

[0138] The regions to be identified in the non-deformed and deformed images of the obtained precast bridge rubber bearing are pixel-level labeled using the Labelme tool, and corresponding semantic labels are assigned to each pixel to form a dataset with annotation information. After the annotation is completed, a binary image is generated, where the black pixel value is 0, representing the background, and the white pixel value is 255, representing the target area. The annotation information indicates the category to which each pixel in the image belongs at the pixel level.

[0139] The annotated images are used as source domain data, and the corresponding unannotated images are used as target domain data. The source domain data and target domain data are a corresponding sample, and all corresponding samples constitute a dataset; the pictures in the dataset are allocated to the training set and test set in a ratio of 8:2.

[0140] In this embodiment, hyper-supervised semantic segmentation experiments are conducted using different training parameters and learning rate schedules: These parameters include a learning rate schedule with 40,000 iterations in DeeplabV3+, a learning rate schedule with 16,000 iterations in MVMFormer, a learning rate schedule with 80,000 iterations in DAFormer, a learning rate schedule with 80,000 iterations in MIU-Transformer, and a learning rate schedule with 80,000 iterations in PSPNET. When the loss value decreases and stabilizes, and at the same time evaluation metrics such as mIoU (mean intersection over union: calculates the ratio of the intersection and union between the predicted output and the ground truth label, measuring the accuracy of the model prediction) and mF1 (mean F1 score, which is calculated using the harmonic mean by combining precision and recall, and is especially suitable for dealing with imbalanced datasets) increase and stabilize, it is considered that the model has converged. And the evaluation is carried out on a dataset with target annotation information. And the performances of models such as MVMformer, MIU-Transformer, DAFormer, PSPNet, and DeepLabV3+ are compared, and the comparison results are as follows:

[0141]

[0142] The detection method of the present invention aims to solve the problems that in the current monitoring process of precast bridge rubber bearings, manual monitoring is relied on and cannot be inspected all-weather, and the detection accuracy of existing intelligent algorithms is not high, and omissions or misjudgments are likely to occur. It can monitor places beyond the human visual range itself, and realize the whole-process deformation monitoring and early warning of the monitoring area of precast bridge rubber bearings. For deformation, it can be quantitatively analyzed and automatically alarm when the deformation size exceeds the limit. Compared with the prior art, the technical solution of the present invention has the following advantages and application prospects: reducing labor costs and improving detection efficiency. This is of great significance for ensuring the safety and stability of precast bridge rubber bearings and maintenance, and has broad application prospects.

[0143] Example 4

[0144] The deformation detection system of precast bridge rubber bearings in this embodiment, the system includes:

[0145] An image acquisition module, used to obtain the cross-sectional deformation map of the precast bridge rubber bearing;

[0146] A classifier module, used to identify the experimental conditions to which the picture belongs and whether there is deformation;

[0147] An image processing module, used to perform pixel-level segmentation and annotation of the deformation map of the image acquisition module. After the annotation is completed, a binary map is generated, where the black pixel value is 0, representing the background, and the white pixel value is 255, representing the target, to obtain a dataset with target annotation information, including deformation maps and non-deformed normal maps;

[0148] A semantic segmentation module that trains a semantic segmentation model using a dataset with target annotation information; after the classifier identifies the deformation and the corresponding experimental working condition, it is processed by the trained semantic segmentation model to output a segmentation map of the target object.

[0149] An area comparison module that is used to compare the pixel area of the bearing in the segmentation map output by the semantic segmentation module with the pixel area of the target object in the limit map of the corresponding experimental working condition to calculate the area ratio.

[0150] An early warning and feedback module that is used to determine whether to give an alarm and take corresponding actions according to the relationship between the area ratio and the corresponding early warning threshold.

[0151] The early warning and feedback process will follow the following sub-processes:

[0152] 1) Transmit the deformation information to the background.

[0153] 2) The background analyzes the deformation information. For each experimental working condition, a first early warning threshold and a second early warning threshold are set. If A / B is less than the first early warning threshold, it is a first-level early warning deformation alarm; if A / B is not less than the first early warning threshold and less than the second early warning threshold, it is a second-level early warning deformation alarm, reminding to take timely actions and execute step 3); the first early warning threshold is less than the second early warning threshold.

[0154] 3) Relevant personnel further check, take appropriate measures to solve the defects, and perform repairs or replacements.

[0155] 4) Re-detect the processed bearing and continue with the deformation detection of the precast bridge rubber bearing.

[0156] The MVMFormer semantic segmentation model adopted in the present invention greatly improves the accuracy of segmentation and prediction, enabling the system to achieve real-time high-precision measurement, successfully improving the adaptability and performance of the model on target data without labels, and enhancing the model's ability to capture instance-level features. The MIU framework improves the segmentation accuracy of the model in complex scenarios by explicitly modeling the dependencies between instances (mainly referring to spatial context, predicting the content of the occluded area based on the area not occluded by the mask). The MIU framework provides additional information by learning the spatial context relationship of the target domain, which enables the teacher network to generate more robust pseudo-labels using local and context clues. These improvements not only enhance the robustness of the system when dealing with complex scenarios but also reduce the dependence on computing resources, making the model more suitable for practical application scenarios. It enables the system to achieve real-time high-precision monitoring of precast bridge rubber bearings.

[0157] It should be understood that each part of the present invention can be implemented by hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or hardware stored in a memory and executed by a suitable instruction execution system.

[0158] Those of ordinary skill in the art can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0159] Matters not described in the present invention are applicable to the prior art.

Claims

1. A method for detecting the deformation of precast bridge rubber bearings based on MVMFormer, characterized in that, The detection method includes the following steps: Conduct compression and shear performance tests on precast bridge rubber bearings of different materials. Apply a vertical pressure to the rubber bearing to test its compression performance, and obtain different deformation diagrams under the vertical load during the test and the vertical load limit diagram when deformation failure occurs. Moreover, simulate the deformation caused by the excessive unilateral force on the precast bridge rubber bearing in actual application, and obtain the deformation diagram during the simulation process and the limit diagram under the unilateral force when deformation failure occurs. Simulate the deformation caused by the influence of the lateral force under extreme conditions, and obtain the deformation diagram during the simulation process and the lateral force limit diagram when deformation failure occurs. Mark the above three deformation diagrams with corresponding experimental conditions respectively. Use the deformation diagrams, the corresponding experimental conditions, and the original diagram without deformation to form a classification data set, and train a classifier with the classification data set to identify the experimental conditions to which the picture belongs and whether there is deformation. Perform pixel-level annotation on the deformation diagrams and the original diagrams without deformation. Assign corresponding semantic labels to each pixel. After the annotation is completed, generate a binary diagram. The black pixel value is 0, representing the background, and the white pixel value is 255, representing the target, to form a data set with target annotation information. Construct an MVMFormer semantic segmentation model, and train the MVMFormer semantic segmentation model with the data set with target annotation information. Use the classifier to identify the experimental conditions to which the image to be tested belongs, and then after being processed by the trained MVMFormer semantic segmentation model, output the segmentation diagram of the target object in the image to be tested. Obtain the pixel area A of the target object in the segmentation diagram and the pixel area B of the target object in the limit diagram corresponding to the experimental conditions. Set a first warning threshold and a second warning threshold for each experimental condition. If A / B is less than the first warning threshold, it is a first-level warning deformation alarm; if A / B is not less than the first warning threshold and less than the second warning threshold, it is a second-level warning deformation alarm, reminding for timely processing; the first warning threshold is less than the second warning threshold.

2. The method according to claim 1, characterized in that, In the compression and shear performance test of the precast bridge rubber bearing, first increase the force linearly and then decrease the force linearly, observe whether it can return to the initial state after the applied force, collect the deformation diagrams during the process, and use the deformation diagram when it cannot return to the initial state as the limit diagram.

3. The method according to claim 1, wherein During the construction of the data set, the installation parameters of the equipment used for taking the deformation diagrams are the same as those in actual application.

4. The method according to claim 1, wherein The MVMFormer semantic segmentation model includes: a framework that integrates mask image uniformity and unsupervised domain adaptation, simply referred to as the MIU framework; a main network composed of a MambaVision encoder and a context-aware feature fusion decoder.

5. The method according to claim 4, characterized in that The framework integrating mask image uniformity and unsupervised domain adaptation includes a teacher network, a source image and a corresponding one-hot label for training the main network f θ with the supervised loss L S ; then, the source image generates a source prediction image through the main network, and the supervised loss between the source prediction image and the real image is calculated using the following formula; Among them, the superscript s represents the corresponding data of the source image, Ns is the number of samples for each label, and the subscript k represents the kth element. Meanwhile, the target image is input into the main network, the mask operation, and the teacher network respectively. Among them, the weights of the teacher network are updated by exponential moving average (EMA) between the teacher network and the main network to transfer knowledge from the source domain to the target domain. The unlabeled images in the target domain are input into the main network, and the adaptation loss is calculated by the following formula Among them, H and W represent the height and width of the target image, H×W is the image pixel area, and C represents the number of categories. represents the quality estimate corresponding to the pseudo-label of the target image. represents the true label of the target image. represents the output of the main network; the superscript T represents the corresponding data of the target image; θ represents the model parameters of the main network. When the target image is input into the masking operation, the MIU module trains the network to predict the semantic segmentation of the entire image by masking random patches of the target image, and the MIU loss is L M : First, a mask M is randomly sampled from a uniform distribution to obtain a mask. The masked target image is obtained by element-wise multiplication of the mask and the target image. The masked target image is input into the main network to obtain a masked prediction result; When the target image is input into the teacher network, according to the formula the pseudo-label of the target domain data is obtained; wherein represents the pseudo-label of the target domain data; ] Represents the output of the teacher network, that is, the probability distribution of the feature vectors of the target domain data updated based on the source domain; c is the class label, taking integer values from 1 to C; Represents the class label that obtains the maximum probability value among all class labels c'; Use the teacher network to generate pseudo-labels for the target domain data, and use local and context information to improve the quality of the pseudo-labels in the target domain, so as to generate more reliable labels. The pseudo-labels of the target image and their quality estimates are used for further training on the target domain; the mask prediction is input into the MIU loss calculation together with the pseudo-labels, and the MIU loss calculation formula is as follows: L M = q T H(y M , p T ) Among them, where p T represents the pseudo-label, q T is the quality estimate of the target image; y M is the prediction probability of the main network for the masked target image; H(.) represents the maximum entropy function; Through the constrained training of three loss functions, the final output of the neural network is obtained, and the image after semantic segmentation is obtained.

6. The method according to claim 4, characterized in that, In the main network, the encoder includes four stages. The first two stages are composed of residual convolutional blocks and downsamplers, while the third and fourth stages include MambaVision hybrid layers, MLP layers, and self-attention layers.

7. The method according to claim 6, wherein The MambaVision hybrid layer includes two branches. One branch has no SSM module, and the other branch includes a linear layer, a one-dimensional convolution, and a sigmoid function activation connected in sequence. Another branch includes a linear layer, a one-dimensional convolution, a sigmoid function activation, and an SSM connected in sequence. The outputs of the two branches are processed through a connection layer and a linear layer to obtain the output of the MambaVision hybrid layer. The self-attention layer uses the lightning self-attention mechanism.

8. A deformation detection system for a precast bridge rubber bearing, characterized in that, Implement the method according to claim 1, the system includes: An image acquisition module for obtaining a cross-sectional deformation map of a prefabricated bridge rubber bearing; A classifier module for identifying the experimental conditions to which the picture belongs and whether there is deformation; An image processing module for segmenting and labeling the deformation map of the image acquisition module at the pixel level according to the category, generating a binary map after labeling, with a black pixel value of 0 representing the background and a white pixel value of 255 representing the target, and obtaining a dataset with target annotation information, including deformation maps and non-deformed normal maps; A semantic segmentation module for training a semantic segmentation model using the dataset with target annotation information; after the classifier identifies the deformation and the experimental conditions to which it belongs, it is then processed by the trained semantic segmentation model to output a segmentation map of the target object; An area comparison module for comparing the pixel area of the bearing in the segmentation map output by the semantic segmentation module with the pixel area of the target object in the limit map of the corresponding experimental conditions to calculate the area ratio; An early warning and feedback module for determining whether to give an alarm and take measures according to the relationship between the area ratio and the corresponding early warning threshold.

Images