Concrete dry and wet state identification method and device based on multi-modal data fusion

Abstract

The invention discloses a concrete dry and wet state recognition method and device based on multi-modal data fusion, relates to the technical field of image recognition, can be applied to the field of intelligent buildings, and mainly aims to solve the problem of low accuracy of concrete dry and wet state recognition based on multi-modal data fusion. The method mainly comprises the steps of performing multi-scale image feature extraction on visible light image data and infrared image data by using a feature extraction model, and performing structured feature extraction on resistivity data by using a multi-layer sensing network model to obtain resistivity features; according to the illumination intensity and resistivity data of the environment where the to-be-identified concrete surface is located, weights corresponding to the different modal data are determined; and performing classification prediction on fusion features of the visible light features, the infrared features and the resistivity features by using the trained classification identification model to obtain a dry and wet state identification result of the to-be-identified concrete surface. The method is mainly used for identifying concrete dry-wet states.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology and can be applied to the field of intelligent buildings, and in particular to a method and device for identifying the dry and wet states of concrete based on multimodal data fusion. Background Art

[0002] As one of the most widely used building materials in modern construction, concrete's quality is directly related to the safety, durability, and load-bearing capacity of the entire building structure. Concrete curing is a crucial step in building quality control. The key to curing is controlling the rate of water evaporation and maintaining appropriate humidity and temperature to prevent rapid water loss, leading to surface cracking and insufficient structural strength. Appropriate curing measures can effectively prevent concrete cracking and surface damage, ensuring its strength and durability. Therefore, real-time monitoring of the wet / dry state of the concrete surface is crucial for optimizing curing measures and improving construction quality.

[0003] With the development of intelligent construction technology, more and more sensors and IoT devices are being used on construction sites to enable real-time monitoring and intelligent management of the construction process. However, existing methods for identifying wet-dry conditions primarily rely on infrared images of the concrete surface's temperature and evaporation response. However, this approach has a limited coverage area and cannot identify large concrete wall surfaces, resulting in low recognition accuracy. Summary of the Invention

[0004] In view of this, the present invention provides a method and device for identifying the dry and wet state of concrete based on multimodal data fusion, the main purpose of which is to solve the problem of low recognition accuracy of existing methods for identifying the dry and wet state of concrete.

[0005] According to one aspect of the present invention, a method for identifying the dry and wet state of concrete based on multimodal data fusion is provided, comprising:

[0006] Acquire multimodal data of the concrete surface to be identified, including visible light image data, infrared image data, and resistivity data;

[0007] Using a feature extraction model to perform multi-scale image feature extraction on the visible light image data and the infrared image data to obtain visible light features and infrared features, and using a multi-layer perception network model to perform structured feature extraction on the resistivity data to obtain resistivity features;

[0008] Determining weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data, and performing feature fusion based on the visible light feature, the infrared feature, the resistivity feature, and the weights to obtain a fused feature;

[0009] The trained classification and recognition model is used to perform classification prediction on the fusion features to obtain a dry and wet state recognition result of the concrete surface to be identified.

[0010] Furthermore, determining the weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data includes:

[0011] By introducing the light intensity and history fitting parameters into the activation function, the confidence of the visible light image is calculated;

[0012] By introducing the illumination intensity correction term and the infrared image temperature difference into the activation function, the infrared image confidence is calculated;

[0013] Calculating a resistivity mean and a standard deviation based on the resistivity data, and calculating a resistivity confidence level based on the mean and the standard deviation;

[0014] The visible light image confidence, the infrared image confidence, and the resistivity confidence are normalized to obtain a visible light image weight corresponding to the visible light image confidence, an infrared image weight corresponding to the infrared image confidence, and a resistivity weight corresponding to the resistivity confidence.

[0015] Furthermore, before performing classification prediction on the fusion features using the trained classification recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified, the method further includes:

[0016] Acquire multiple sets of historical multimodal data of concrete surfaces collected at different historical times under different lighting conditions, as well as the ambient temperature and concrete age when each set of historical multimodal data was collected, wherein the historical multimodal data includes historical visible light image data, historical infrared image data, and historical resistivity data;

[0017] Determine the dryness and wetness state labels of each set of historical multimodal data, and mark the fusion features corresponding to the historical multimodal data based on the dryness and wetness state labels to obtain a marked fusion feature sample;

[0018] An initial classification and recognition model is trained based on the labeled fusion feature samples to obtain a trained classification and recognition model.

[0019] Furthermore, determining the dry and wet state labels of each set of historical multimodal data includes:

[0020] For each set of historical multimodal data, a dynamic resistivity threshold is calculated using a multivariate linear regression algorithm according to the corresponding ambient temperature and concrete age;

[0021] If the historical resistivity value is greater than or equal to the resistivity dynamic threshold, determining that the dry-wet state label is in a wet state;

[0022] If the historical resistivity value is less than the resistivity dynamic threshold, it is determined that the dry / wet state tag is in a dry state.

[0023] Furthermore, the feature extraction model is an Inception-v3 model that completes transfer learning in the last fully connected layer. The feature extraction model is used to perform multi-scale image feature extraction on the visible light image data and the infrared image data to obtain visible light features and infrared features, including:

[0024] resizing the visible light image data and the infrared image data so that the visible light image and the infrared image have the same size, thereby obtaining a preprocessed visible light image and a preprocessed infrared image;

[0025] Performing multi-scale image feature extraction on the preprocessed visible light image and the preprocessed infrared image using the Inception-v3 model that completes transfer learning using the last fully connected layer, thereby obtaining initial visible light image features at multiple scales and initial infrared image features at multiple scales;

[0026] The initial visible light image features of multiple scales are globally averaged pooled to obtain visible light features, and the initial infrared image features of multiple scales are globally averaged pooled to obtain infrared features.

[0027] Furthermore, the classification and recognition model that has been trained is used to perform classification and prediction on the fusion features to obtain the dry and wet state recognition result of the concrete surface to be identified, including:

[0028] Re-weighting and fusing the fused features through the channel attention network in the classification and recognition model to obtain a fused feature after secondary weighted fusion;

[0029] The fusion features after the quadratic weighted fusion are classified and predicted by the regression model in the classification and recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified.

[0030] Furthermore, the resistivity data is subjected to structured feature extraction using a multi-layer perception network model to obtain resistivity features, including:

[0031] Extracting low-dimensional numerical vectors from the resistivity data, and the real-time ambient temperature and real-time concrete age of the concrete location where each resistivity value in the resistivity data is collected, using the multi-layer perception network model, to obtain a resistivity value vector, an ambient temperature vector, and a concrete age vector;

[0032] The resistivity feature is obtained by learning the combined relationship between the resistivity value vector, the ambient temperature vector and the concrete age vector through a two-layer fully connected network and an activation function.

[0033] According to another aspect of the present invention, a device for identifying the dry and wet state of concrete based on multimodal data fusion is provided, comprising:

[0034] An acquisition module, used to acquire multimodal data of the concrete surface to be identified, including visible light image data, infrared image data and resistivity data;

[0035] a feature extraction module, configured to perform multi-scale image feature extraction on the visible light image data and the infrared image data using a feature extraction model to obtain visible light features and infrared features, and to perform structured feature extraction on the resistivity data using a multi-layer perception network model to obtain resistivity features;

[0036] a determination module, configured to determine weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data, and to perform feature fusion based on the visible light feature, the infrared feature, the resistivity feature, and the weights to obtain a fused feature;

[0037] The prediction module is used to perform classification prediction on the fusion features using the trained classification recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified.

[0038] Furthermore, the determining module includes:

[0039] A first calculation unit is configured to calculate a confidence level of a visible light image by introducing the light intensity and the history fitting parameter into an activation function;

[0040] A second calculation unit is used to calculate the infrared image confidence by introducing a light intensity correction term and an infrared image temperature difference into the activation function;

[0041] a third calculation unit, configured to calculate a mean value and a standard deviation of the resistivity according to the resistivity data, and calculate a resistivity confidence level according to the mean value and the standard deviation;

[0042] A normalization unit is used to normalize the visible light image confidence, the infrared image confidence, and the resistivity confidence to obtain a visible light image weight corresponding to the visible light image confidence, an infrared image weight corresponding to the infrared image confidence, and a resistivity weight corresponding to the resistivity confidence.

[0043] Furthermore, the device further comprises:

[0044] The acquisition module is further configured to acquire multiple sets of historical multimodal data of concrete surfaces collected at different historical times under different lighting conditions, as well as the ambient temperature and concrete age at the time each set of historical multimodal data was collected, wherein the historical multimodal data includes historical visible light image data, historical infrared image data, and historical resistivity data;

[0045] a labeling module, configured to determine the dryness and wetness state labels of each set of historical multimodal data, and label the fusion features corresponding to the historical multimodal data based on the dryness and wetness state labels to obtain labeled fusion feature samples;

[0046] The training module is used to train the initial classification and recognition model based on the marked fusion feature samples to obtain a trained classification and recognition model.

[0047] Furthermore, the marking module includes:

[0048] a fourth calculation unit, configured to calculate, for each set of historical multimodal data, a dynamic resistivity threshold value based on the corresponding ambient temperature and the concrete age using a multivariate linear regression algorithm;

[0049] a first determining unit, configured to determine that the dry-wet state label is a wet state if the historical resistivity value is greater than or equal to the resistivity dynamic threshold;

[0050] The second determining unit is configured to determine that the dry / wet state tag is in a dry state if the historical resistivity value is less than the resistivity dynamic threshold value.

[0051] Furthermore, the feature extraction module includes:

[0052] an adjusting unit, configured to resize the visible light image data and the infrared image data so that the visible light image and the infrared image have the same size, thereby obtaining a preprocessed visible light image and a preprocessed infrared image;

[0053] an extraction unit, configured to perform multi-scale image feature extraction on the preprocessed visible light image and the preprocessed infrared image using an Inception-v3 model that uses a last fully connected layer to complete transfer learning, thereby obtaining initial visible light image features at multiple scales and initial infrared image features at multiple scales;

[0054] The pooling unit is used to perform global average pooling on the initial visible light image features of multiple scales to obtain visible light features, and to perform global average pooling on the initial infrared image features of multiple scales to obtain infrared features.

[0055] Furthermore, the prediction module includes:

[0056] A weighted fusion unit, configured to re-weight and fuse the fused features through the channel attention network in the classification and recognition model to obtain a fused feature after secondary weighted fusion;

[0057] The classification prediction unit is used to perform classification prediction on the fusion features after the quadratic weighted fusion through the regression model in the classification recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified.

[0058] Furthermore, the resistivity data is subjected to structured feature extraction using a multi-layer perception network model to obtain resistivity features, including:

[0059] an extraction unit, configured to extract low-dimensional numerical vectors from the resistivity data and the real-time ambient temperature and real-time concrete age of the concrete location where each resistivity value in the resistivity data is collected, using the multi-layer perception network model, to obtain a resistivity value vector, an ambient temperature vector, and a concrete age vector;

[0060] A fully connected processing unit is used to learn the combination relationship between the resistivity value vector, the ambient temperature vector and the concrete age vector through a two-layer fully connected network and an activation function to obtain a resistivity feature.

[0061] According to another aspect of the present invention, a storage medium is provided, in which at least one executable instruction is stored. The executable instruction enables a processor to perform operations corresponding to the above-mentioned method for identifying the dry and wet state of concrete based on multimodal data fusion.

[0062] According to another aspect of the present invention, there is provided a terminal, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0063] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned method for identifying the dry and wet state of concrete based on multimodal data fusion.

[0064] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0065] The present invention provides a method and device for identifying the wetness and dryness of concrete based on multimodal data fusion. The present invention acquires multimodal data of the concrete surface to be identified, including visible light image data, infrared image data, and resistivity data. A feature extraction model is used to perform multiscale image feature extraction on the visible light and infrared image data to obtain visible light and infrared features, respectively. A multilayer perceptron network model is used to perform structured feature extraction on the resistivity data to obtain resistivity features. Weights corresponding to the different modal data are determined based on the ambient light intensity and resistivity data of the concrete surface to be identified. Feature fusion is performed based on the visible light, infrared, and resistivity features, along with the weights, to obtain fused features. A trained classification and recognition model is used to classify and predict the fused features to obtain the wetness and dryness identification result of the concrete surface to be identified. This method significantly improves the richness of the data modalities, enabling the feature fusion process to adaptively adjust the fusion weights of the different modal features based on varying light intensities, thereby significantly improving the accuracy of wetness and dryness identification.

[0066] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0068] Figure 1 A flow chart of a method for identifying the dry and wet state of concrete based on multimodal data fusion provided by an embodiment of the present invention is shown;

[0069] Figure 2 A flowchart of another method for identifying the dry and wet state of concrete based on multimodal data fusion provided by an embodiment of the present invention is shown;

[0070] Figure 3 The overall structure of the convolutional layer and pooling layer of Inception-v3 provided by an embodiment of the present invention is shown;

[0071] Figure 4 A schematic diagram of an algorithm structure combining a channel and spatial attention mechanism provided by an embodiment of the present invention is shown;

[0072] Figure 5A schematic diagram of a network architecture of a multi-layer perceptron provided by an embodiment of the present invention is shown;

[0073] Figure 6 A schematic diagram of the entire process of identifying the dryness and wetness of a concrete wall surface provided by an embodiment of the present invention is shown;

[0074] Figure 7 A block diagram of a device for identifying the dry and wet state of concrete based on multimodal data fusion provided by an embodiment of the present invention is shown;

[0075] Figure 8 A schematic structural diagram of a terminal provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0076] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0077] Aiming at the problem of low recognition accuracy of existing concrete dry and wet state recognition methods, the present invention provides a concrete dry and wet state recognition method based on multimodal data fusion. Figure 1 As shown, the method includes:

[0078] 101. Acquire multimodal data of the concrete surface to be identified, including visible light image data, infrared image data, and resistivity data.

[0079] In an embodiment of the present invention, the concrete surface to be identified is a building wall that needs to be monitored for dry and wet status. It can be a wall that needs concrete curing during construction, or a wall that needs to be identified for water seepage, etc., which is not specifically limited in the embodiment of the present invention. In order to accurately identify the current dry and wet status of the concrete surface, a high-resolution RGB camera, an infrared thermal imager, and a resistivity sensor are installed on the intelligent inspection robot to collaboratively collect concrete surface status information to obtain multimodal data of the concrete surface. Specifically, visible light image data can be collected by a high-resolution RGB camera, infrared image data can be collected by an infrared thermal imager, and resistivity data can be collected by a resistivity sensor. Among them, the intelligent inspection robot can autonomously navigate through SLAM technology, cover a large area of ​​concrete surface, and realize automated and efficient monitoring.

[0080] It's important to note that using a high-resolution RGB camera to capture visible light images of the concrete surface can capture color and texture variations (wet areas typically appear dark gray or dark, while dry areas appear light gray or white). Infrared thermal imaging can capture temperature differences caused by water evaporation (wet areas absorb heat due to evaporation and are therefore cooler; dry areas are hotter due to the absence of water evaporation), thereby revealing defects and unevenness in the concrete. Resistivity sensors, typically consisting of two or more electrodes, measure the resistivity of the concrete surface or near-surface area when attached or embedded via contact or embedded probes, reflecting moisture content. Because changes in moisture content within concrete pores significantly affect its electrical conductivity (when wet, ion migration is strong, resulting in low resistivity; when dry, the conductive path is interrupted, resulting in a significant increase in resistivity), resistivity can be used as an indirect indicator of moisture content.

[0081] 102. Use a feature extraction model to perform multi-scale image feature extraction on the visible light image data and the infrared image data to obtain visible light features and infrared features, and use a multi-layer perception network model to perform structured feature extraction on the resistivity data to obtain resistivity features.

[0082] In an embodiment of the present invention, the feature extraction model can be a deep convolutional neural network. Preferably, the Inception-v3 network structure adopts an asymmetric convolution design. By decomposing a large convolution kernel into asymmetric small convolution kernels, the operation time of a large number of parameters is significantly reduced, overfitting is alleviated, and a nonlinear layer is added to expand the expression ability of the model. This asymmetric convolution structure splitting can extract more diverse features and improve the richness of feature representation compared with the symmetric convolution structure splitting. Therefore, it performs well in multiple computer vision tasks such as image classification, object detection, and image segmentation. It can extract richer features from visible light image data and infrared image data, providing a diverse data basis for subsequent feature fusion and prediction. Since resistivity is strongly correlated with temperature and material age, structural parameters such as ambient temperature and material age are also needed in the process of resistivity feature extraction. In response to the problem of limited feature expression ability of structural parameters such as resistivity data, a multi-layer perceptron network is used to perform nonlinear mapping and semantic extraction on structural parameters such as resistivity value, ambient temperature, and material age. Since the multilayer perceptron network includes multiple hidden layers, different hidden layers can learn features at different levels and degrees of abstraction, extract more valuable information from the original data, and help the model better perform linear mapping and semantic extraction of structural parameters.

[0083] 103. Determine weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data, and perform feature fusion based on the visible light feature, the infrared feature, the resistivity feature, and the weights to obtain a fused feature.

[0084] In an embodiment of the present invention, in order to adapt to complex environmental conditions such as strong direct sunlight and shadow obstruction at the construction site, the collected visible light image data, infrared thermal imaging data, and resistivity data are fused into a multi-dimensional feature vector. In the fusion process, a higher weight is assigned to the data of the modality with better data accuracy and performance in the current environment. For example, in the case of strong light, a lower weight is assigned to the visible light feature; in the case of insufficient light, a higher weight is assigned to the infrared feature; under normal light, the weights of the three modal data are evenly distributed so that the fused feature can more accurately describe the information displayed by the current concrete. Set the fused feature to F and the visible light feature to F RGB , the infrared characteristic is F IRT , the resistivity characteristic is F R , the feature fusion formula is expressed as:

[0085] F=W RGB `F RGB +W IRT `F IRT +W R ·F R (1);

[0086] Among them, W RGB Represents the visible light weight, W IRT Indicates infrared weight, W R Represents the resistivity weight.

[0087] It should be noted that the actual construction site environment is very complex. For example, the data collection environment varies greatly between sunny and cloudy days, and between daytime and nighttime. The accuracy and reliability of different modal data vary under different environments. For example, in scenes with strong lighting, visible light images are prone to overexposure, and infrared images are interfered with by solar radiation, resulting in a significant decrease in reliability. In scenes with insufficient lighting, infrared temperature differences are even more significant. Therefore, a dynamic weight adjustment method based on adaptive illumination intensity is adopted to learn the correlation between illumination intensity and modal data confidence. Based on this correlation, the weight distribution of each modal feature is dynamically adjusted to achieve robust recognition in complex lighting environments.

[0088] 104. Use the trained classification and recognition model to perform classification prediction on the fusion features to obtain a dry and wet state recognition result of the concrete surface to be recognized.

[0089] In an embodiment of the present invention, the classification and recognition model is a binary classification model that can be constructed based on a Softmax regression model. By constructing a multimodal feature fusion network, the temperature gradient characteristics of infrared thermal imaging, the texture and color information of visible light images, the conductivity changes of resistivity, and the environmental parameters of light intensity are jointly analyzed to achieve the complementarity and enhancement of each modal data in the feature space. This avoids the problem that a single sensor (such as relying only on infrared thermal imaging or visible light images) is susceptible to environmental interference (such as changes in lighting, surface reflections, and material aging), which leads to misjudgment or missed detection of the dry and wet state of concrete.

[0090] In one embodiment of the present invention, for further explanation and limitation, as Figure 2 As shown, the steps of determining the weights corresponding to different modal data according to the light intensity of the environment where the concrete surface to be identified is located and the resistivity data include:

[0091] 201. Calculate and obtain the visible light image confidence by introducing the light intensity and history fitting parameters into the activation function.

[0092] 202. By introducing the illumination intensity correction term and the infrared image temperature difference into the activation function, the infrared image confidence is calculated.

[0093] 203. Calculate a resistivity mean and a standard deviation based on the resistivity data, and calculate a resistivity confidence level based on the mean and the standard deviation.

[0094] 204. Normalize the visible light image confidence, the infrared image confidence, and the resistivity confidence to obtain a visible light image weight corresponding to the visible light image confidence, an infrared image weight corresponding to the infrared image confidence, and a resistivity weight corresponding to the resistivity confidence.

[0095] In an embodiment of the present invention, a corresponding confidence function is designed for each modal data to reflect the credibility score (confidence) of the feature corresponding to the data in the current environment, so that weighted fusion is performed according to the weight determined by this confidence during the feature fusion process. The visible light image confidence uses a Sigmoid function for illumination adjustment, and considers the historical fitting parameters obtained by fitting the illumination intensity and historical sample data in the calculation process. Among them, the illumination intensity can be collected in real time by the illumination intensity sensor additionally equipped on the inspection robot. The calculation function of the visible light image confidence is expressed as:

[0096]

[0097] Where L represents the light intensity of the current environment; k1 and k2 represent historical fitting parameters, which are obtained by fitting samples representing different reliability levels of visible light under different light intensities within the historical time period.

[0098] Since infrared thermal imaging is easily interfered by solar radiation under strong light, a light intensity correction term is added to the calculation process of infrared thermal imaging confidence. The specific calculation function is:

[0099] C IRT =f(k1·ΔT)·(1-f(k2·(L-L0)))(3);

[0100] Where ΔT represents the temperature difference of the infrared image, ΔT = T max -T min ;T max Indicates the maximum temperature value in the infrared image, T min represents the minimum temperature value in the infrared image; f(x) is the Sigmoid function; L0 represents the solar interference threshold, L represents the light intensity of the current environment; k1 and k2 represent the history fitting parameters.

[0101] Since resistivity in complex scenes with strong or weak light has higher credibility in areas with small fluctuations and strong stability, the coefficient of variation is used to calculate the resistivity confidence level. The calculation formula is expressed as:

[0102]

[0103] Where: σ represents the standard deviation of multiple resistivity values ​​in the collected resistivity data; μ represents the average value of multiple resistivity values ​​in the collected resistivity data; when C R The closer it is to 1, the more stable the resistivity is.

[0104] After obtaining the confidence corresponding to different modal data, the confidence under each mode is normalized. The normalization formula is expressed as:

[0105]

[0106] Among them, i∈{RGB,IRT,R} represents each modality, and the sum of the weights of each modality feature is 1.

[0107] Because the confidence pairs corresponding to different modalities are determined based on light intensity, the weights of the different modalities can automatically adapt to changes in scene lighting intensity. For example, in bright light scenes, the weights of visible light and infrared thermal imaging are automatically reduced, while the weight of resistivity is increased. In dim light, the weight of visible light is automatically reduced, while the weights of infrared and resistivity are increased. In normal light, the weights of the three modalities are evenly distributed, achieving optimal recognition results.

[0108] In one embodiment of the present invention, for further explanation and limitation, before using the trained classification and recognition model to perform classification prediction on the fusion features to obtain the dry and wet state recognition result of the concrete surface to be identified, the method further includes:

[0109] Obtain multiple sets of historical multimodal data of concrete surfaces under different historical collection conditions and different lighting conditions, as well as the ambient temperature and concrete age when each set of historical multimodal data was collected;

[0110] Determine the dryness and wetness state labels of each set of historical multimodal data, and mark the fusion features corresponding to the historical multimodal data based on the dryness and wetness state labels to obtain a marked fusion feature sample;

[0111] An initial classification and recognition model is trained based on the labeled fusion feature samples to obtain a trained classification and recognition model.

[0112] In an embodiment of the present invention, the historical multimodal data includes historical visible light image data, historical infrared image data, and historical resistivity data. By performing feature extraction and feature fusion on the above data respectively, the fusion features corresponding to the historical multimodal data can be obtained, and the dry and wet states of the concrete wall corresponding to the historical multimodal data when the data was collected are used as the fusion features corresponding to the label mark to obtain the fused feature samples with the completed label, and then the initial classification recognition model is trained based on the fused feature samples with the completed label. Among them, the labels used to mark the fusion features can be based on manual markings, or they can be judged in real time based on the resistivity data collected within the historical time period. The historical time period can be a preset time length from the current moment forward, and can be customized according to actual needs. The resistivity data outputs labels in real time and marks the sample data, which can be applied to the adaptive update training process of the model, thereby improving the life of the model.

[0113] The initial classification recognition model is built based on the Softmax regression model. The Softmax function is expressed as:

[0114]

[0115] Where z represents the input prediction array, C represents the total number of classifications; it is assumed that the training set in this patent consists of n labeled samples, denoted as {(x1,y1),(x2,y2,),...,(x n ,y n )}, where the category label y i ∈{1,2,...,k} has k different values, size j; use the hypothesis function to estimate the probability value p(y=j|x) for each category j; therefore, use the hypothesis function to output a k-dimensional vector to represent the k estimated probability value, and use By normalizing the probability distribution by the term, we can obtain the following function:

[0116]

[0117] Among them, θ represents the Softmax regression model parameters, which is an n×k-order matrix. The cost function corresponding to θ is the cross entropy loss function, which is expressed as:

[0118]

[0119] Where m is the characteristic number of variable x, and the value of the indicator function {·} is: I{expression with true value} = 1, I{expression with false value} = 0. For the minimization problem of J(θ), the gradient descent method in the iterative optimization algorithm is used to solve it. The gradient formula obtained by derivation of equation (8) is:

[0120]

[0121] in, is the first vector, whose lth element is J(θ) versus θ j The partial derivative of the lth component of . Since Softmax has redundant parameter sets, we can add Modify the cost function to penalize excessively large parameter values. The cost function can be transformed into:

[0122]

[0123] Where n is the number of input data. After introducing the weight decay term in the loss function (regularization parameter λ>0), the iterative function becomes a strictly convex function. The gradient descent optimization method can ensure convergence to a unique solution, and this solution is the global optimal solution. First, λ is assigned in ascending order; then, the model parameters are iteratively learned on the training set, and the validation error is calculated using the cross-validation set. The model with the smallest error is selected, that is, the corresponding λ value is determined; finally, the selected model is evaluated on the test set to verify and obtain the optimal λ value. In order to use the optimization algorithm, the new function J(θ) is differentiated to obtain the following formula:

[0124]

[0125] The initial classification and recognition model is trained and optimized by minimizing J(θ), thereby obtaining a classification and recognition model for identifying the dry and wet states of the concrete surface to be identified.

[0126] In one embodiment of the present invention, for further explanation and limitation, determining the dry and wet state labels of each set of historical multimodal data includes:

[0127] For each set of historical multimodal data, a dynamic resistivity threshold is calculated using a multivariate linear regression algorithm according to the corresponding ambient temperature and concrete age;

[0128] If the historical resistivity value is greater than or equal to the resistivity dynamic threshold, determining that the dry-wet state label is in a wet state;

[0129] If the historical resistivity value is less than the resistivity dynamic threshold, it is determined that the dry / wet state tag is in a dry state.

[0130] In this embodiment of the present invention, the wet / dry state label is determined based on the ambient temperature, concrete age, and resistivity. In real-world scenarios, the resistivity of concrete is significantly affected by factors such as ambient temperature, age, and the ratio of construction materials. Using a fixed threshold to determine the wet / dry state label can easily affect the label's accuracy. Therefore, an adaptive threshold generation method is introduced to generate a dynamic resistivity threshold. The calculation formula is:

[0131] R=a·T emp +b·log(Age+1)+c (12);

[0132] Where R represents the resistivity threshold value (unit: Ω·m) for judging the “wet” state in the current scene; T emp represents the ambient temperature (unit: °C). In the embodiment of the present invention, the ambient temperature is measured in real time by an environmental sensor additionally carried by the inspection robot. Age represents the age of the concrete (unit: day), which is calculated from the construction records. a, b, and c represent empirical coefficients obtained by fitting historical measurement data.

[0133] In one application example, an inspection robot can read the ambient temperature in real time along a pre-planned route, infer the project age based on the construction site, and calculate a dynamic resistivity threshold using the formula above. This dynamic resistivity threshold is then compared with the collected resistivity value to determine the dryness / wetness state. This dryness / wetness state is then used to label the fused features corresponding to the visible light and infrared images collected during the same time period as the resistivity value, generating training samples.

[0134] In one embodiment of the present invention, for further explanation and limitation, the use of a feature extraction model to perform multi-scale image feature extraction on the visible light image data and the infrared image data to obtain visible light features and infrared features includes:

[0135] resizing the visible light image data and the infrared image data so that the visible light image and the infrared image have the same size, thereby obtaining a preprocessed visible light image and a preprocessed infrared image;

[0136] Performing multi-scale image feature extraction on the preprocessed visible light image and the preprocessed infrared image using the Inception-v3 model that completes transfer learning using the last fully connected layer, thereby obtaining initial visible light image features at multiple scales and initial infrared image features at multiple scales;

[0137] The initial visible light image features of multiple scales are globally averaged pooled to obtain visible light features, and the initial infrared image features of multiple scales are globally averaged pooled to obtain infrared features.

[0138] In an embodiment of the present invention, before feature extraction is performed on the infrared image data and the visible light image data, data preprocessing is required to uniformly adjust the RGB image and infrared data to 299×299 (to meet the input requirements of Inception-v3) to form a multimodal input tensor, namely the preprocessed visible light image and the preprocessed infrared image. The preprocessing process of the visible light image specifically includes: first, denoising the high-resolution RGB image by using Gaussian filtering or median filtering to eliminate noise interference; then converting the image from the RGB color space to HSV or Lab space to more accurately extract color features (such as the difference between dark gray in wet areas and light gray in dry areas); enhancing the image contrast through histogram equalization or adaptive histogram equalization, and combining the Canny edge detection algorithm to extract the concrete surface texture features; finally, cropping the irrelevant background area and uniformly adjusting the image to a fixed resolution, while annotating the wet / dry areas to provide labels for subsequent supervised learning. The infrared thermal imaging data preprocessing process specifically includes: first, temperature calibration is performed on the infrared image data collected by the infrared thermal imager to eliminate the influence of ambient temperature and instrument errors, and then thermal imaging noise is removed through median filtering or Gaussian filtering; a threshold segmentation algorithm is used to extract temperature anomaly areas (such as low temperature areas caused by water evaporation in humid areas), and the temperature gradient distribution is calculated to quantify the temperature non-uniformity of the concrete surface; finally, the thermal imaging data is mapped into a uniform-sized thermal map to ensure alignment with the RGB image in the spatial dimension.

[0139] The feature extraction model is the Inception-v3 model that completes transfer learning with the last fully connected layer. Inception-v3 is a deep convolutional neural network architecture. Its network structure is designed in sequence from the input end: the initial stage contains three convolution layers, followed by a pooling layer for preliminary feature extraction and spatial dimensionality reduction; then two convolution layers are configured, and then a pooling layer is connected to further compress the resolution; finally, the main part is composed of eleven mixed layers (Inception modules). These modules are composed of convolution kernels of different sizes and maximum pooling operations. They work in parallel and then connect the outputs together. This design allows the network to capture features of different scales in the same layer, improving the expressive power of the model. The original model also includes a Dropout layer, a fully connected layer, and a Softmax layer for regularization and classification output. The overall structure of the convolution layer and pooling layer of Inception-v3 is as follows Figure 3 The model's transfer training process involves using the Inception-v3 model as the source model. All parameters in the trained Inception-v3 model are retained, with only the last fully connected layer replaced. Using a transfer learning strategy, a dataset of dry and wet concrete collected in the target domain is imported into the model. This dataset includes visible light images of concrete surfaces captured by a high-resolution RGB camera and thermal imaging data collected by an infrared thermal imager. The last layer is then retrained based on the preprocessing results of the dry and wet concrete dataset, and backpropagation is used to train the last layer. Specifically, low-level features such as edges and textures are extracted from the first 10 layers of Inception-v3. These are then processed in parallel by the Inception modules to extract multi-scale features. Finally, global average pooling is performed to output a 1024-dimensional feature vector. The preprocessing of the dry and wet concrete dataset is identical to the data preprocessing performed before feature extraction for the infrared and visible light image data described above and will not be repeated here.

[0140] In one embodiment of the present invention, for further explanation and limitation, the classification prediction of the fusion features using the trained classification recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified includes:

[0141] Re-weighting and fusing the fused features through the channel attention network in the classification and recognition model to obtain a fused feature after secondary weighted fusion;

[0142] The fusion features after the quadratic weighted fusion are classified and predicted by the regression model in the classification and recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified.

[0143] In this embodiment of the present invention, in order to improve the expressive power of the fused feature vector F when entering the classification stage, a lightweight and efficient attention mechanism (CBAM, Convolutional Block Attention Module) is introduced before the Softmax classifier. This module re-weights the original fused feature vector, which can enhance the system's perception of important features such as key modal feature dimensions, thereby improving recognition accuracy and environmental adaptability. The feature vector F output by CBAM is ′ The data is imported into the Softmax regression model for classification. By calculating the error between the output of the Softmax layer and the label vector of a given sample category, the weight parameters are adjusted using the cross entropy function. Finally, a high-accuracy classification model is obtained to realize the recognition of the dry and wet state of concrete.

[0144] CBAM is an algorithm model that combines channel and spatial attention mechanisms. The overall structure of the algorithm is as follows: Figure 4 As shown in the figure, the input fusion feature vector F is first subjected to a channel-wise attention mechanism and then a spatial-wise attention mechanism, and finally a refined feature is output. This achieves the goal of strengthening the region of interest from both channel and spatial perspectives. The channel-wise attention module is used to model the dependencies between channels, while the spatial-wise attention module is used to model the dependencies between spatial positions. It can adaptively adjust the importance distribution of each channel of the feature vector under different environments (such as strong lighting, image blur, and resistivity fluctuations), highlighting the representational power of key modalities and suppressing redundant or interfered feature channels. This mechanism significantly enhances the model's ability to perceive key information in multimodal input and improves the recognition accuracy and robustness of the classification module in complex lighting scenarios.

[0145] In one embodiment of the present invention, for further explanation and limitation, the resistivity data is subjected to structured feature extraction using a multi-layer perception network model to obtain resistivity features, including:

[0146] Extracting low-dimensional numerical vectors from the resistivity data, and the real-time ambient temperature and real-time concrete age of the concrete location where each resistivity value in the resistivity data is collected, using the multi-layer perception network model, to obtain a resistivity value vector, an ambient temperature vector, and a concrete age vector;

[0147] The resistivity feature is obtained by learning the combined relationship between the resistivity value vector, the ambient temperature vector and the concrete age vector through a two-layer fully connected network and an activation function.

[0148] In the embodiment of the present invention, the multilayer perceptron is a feedforward artificial neural network composed of multiple neurons (neural nodes). These neurons are arranged in a hierarchical structure, including an input layer, a hidden layer, and an output layer. The neurons between layers are connected by weights. Information is transmitted forward from the input layer to the output layer in sequence. There is no feedback connection. The network architecture is as follows: Figure 5 As shown in the figure, the input layer passes the input data to the next layer, and the neurons in the hidden layer then perform nonlinear transformation and feature extraction on the input data. Each hidden layer neuron receives the output of the neurons in the previous layer and performs a weighted sum based on its own weight and bias. It then performs nonlinear processing through an activation function and outputs the processed result to the next layer. The input features include structural parameters such as the current resistivity value, ambient temperature, and material age, which constitute a low-dimensional numerical vector. Through a two-layer fully connected network and activation function, the combination relationship between features is learned, and a structural modal embedding vector of uniform length is output. This vector is fused with the image modal features in a subsequent stage to enhance the system's comprehensive judgment ability on the dry and wet state of concrete, especially providing structural compensation when the image modality is degraded.

[0149] In an application example, Figure 6 As shown in the figure, the overall process for identifying the wetness of concrete wall surfaces involves multi-sensor data acquisition, including visible light image data, thermal imaging data (infrared image data), and resistivity data. After preprocessing, this multi-sensor data is used for wetness identification under complex light intensities. This involves extracting resistivity features using a multi-layer perceptron (MLP) and extracting features from visible light and infrared images using a feature extraction model trained through transfer learning. During feature fusion, the weights corresponding to different modal data are determined based on light intensity. During this feature fusion process, an attention mechanism (CBAM) is introduced to improve the classification prediction accuracy of the softmax method, thereby achieving accurate wetness identification results.

[0150] The present invention provides a method for identifying the wetness and dryness of concrete based on multimodal data fusion. The present invention acquires multimodal data of the concrete surface to be identified, including visible light image data, infrared image data, and resistivity data. A feature extraction model is used to perform multiscale image feature extraction on the visible light and infrared image data to obtain visible light and infrared features, respectively. A multilayer perception network model is used to perform structured feature extraction on the resistivity data to obtain resistivity features. Weights corresponding to the different modal data are determined based on the ambient light intensity and resistivity data of the concrete surface to be identified. Feature fusion is performed based on the visible light, infrared, and resistivity features, along with the weights, to obtain fused features. A trained classification recognition model is used to classify and predict the fused features to obtain the wetness and dryness identification result of the concrete surface to be identified. This method significantly improves the richness of the data modalities and enables the feature fusion process to adaptively adjust the fusion weights of the different modal features based on varying light intensities, thereby significantly improving the accuracy of wetness and dryness identification.

[0151] Furthermore, as a response to the above Figure 1 The embodiment of the present invention provides a device for identifying the dry and wet state of concrete based on multimodal data fusion, such as Figure 7 As shown, the device includes:

[0152] An acquisition module 31 is used to acquire multimodal data of the concrete surface to be identified, including visible light image data, infrared image data and resistivity data;

[0153] a feature extraction module 32 for performing multi-scale image feature extraction on the visible light image data and the infrared image data using a feature extraction model to obtain visible light features and infrared features, and performing structured feature extraction on the resistivity data using a multi-layer perception network model to obtain resistivity features;

[0154] a determination module 33 for determining weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data, and performing feature fusion based on the visible light feature, the infrared feature, the resistivity feature, and the weights to obtain a fused feature;

[0155] The prediction module 34 is used to perform classification prediction on the fusion features using the trained classification recognition model to obtain a dry and wet state recognition result of the concrete surface to be identified.

[0156] Furthermore, the determining module 33 includes:

[0157] A first calculation unit is configured to calculate a confidence level of a visible light image by introducing the light intensity and the history fitting parameter into an activation function;

[0158] A second calculation unit is used to calculate the infrared image confidence by introducing a light intensity correction term and an infrared image temperature difference into the activation function;

[0159] a third calculation unit, configured to calculate a mean value and a standard deviation of the resistivity according to the resistivity data, and calculate a resistivity confidence level according to the mean value and the standard deviation;

[0160] A normalization unit is used to normalize the visible light image confidence, the infrared image confidence, and the resistivity confidence to obtain a visible light image weight corresponding to the visible light image confidence, an infrared image weight corresponding to the infrared image confidence, and a resistivity weight corresponding to the resistivity confidence.

[0161] Furthermore, the device further comprises:

[0162] The acquisition module is further configured to acquire multiple sets of historical multimodal data of concrete surfaces collected at different historical times under different lighting conditions, as well as the ambient temperature and concrete age at the time each set of historical multimodal data was collected, wherein the historical multimodal data includes historical visible light image data, historical infrared image data, and historical resistivity data;

[0163] a labeling module, configured to determine the dryness and wetness state labels of each set of historical multimodal data, and label the fusion features corresponding to the historical multimodal data based on the dryness and wetness state labels to obtain labeled fusion feature samples;

[0164] The training module is used to train the initial classification and recognition model based on the marked fusion feature samples to obtain a trained classification and recognition model.

[0165] Furthermore, the marking module includes:

[0166] a fourth calculation unit, configured to calculate, for each set of historical multimodal data, a dynamic resistivity threshold value based on the corresponding ambient temperature and the concrete age using a multivariate linear regression algorithm;

[0167] a first determining unit, configured to determine that the dry-wet state label is a wet state if the historical resistivity value is greater than or equal to the resistivity dynamic threshold;

[0168] The second determining unit is configured to determine that the dry / wet state tag is in a dry state if the historical resistivity value is less than the resistivity dynamic threshold value.

[0169] Furthermore, the feature extraction module 32 includes:

[0170] an adjusting unit, configured to resize the visible light image data and the infrared image data so that the visible light image and the infrared image have the same size, thereby obtaining a preprocessed visible light image and a preprocessed infrared image;

[0171] an extraction unit, configured to perform multi-scale image feature extraction on the preprocessed visible light image and the preprocessed infrared image using an Inception-v3 model that uses a last fully connected layer to complete transfer learning, thereby obtaining initial visible light image features at multiple scales and initial infrared image features at multiple scales;

[0172] The pooling unit is used to perform global average pooling on the initial visible light image features of multiple scales to obtain visible light features, and to perform global average pooling on the initial infrared image features of multiple scales to obtain infrared features.

[0173] Furthermore, the prediction module 34 includes:

[0174] A weighted fusion unit, configured to re-weight and fuse the fused features through the channel attention network in the classification and recognition model to obtain a fused feature after secondary weighted fusion;

[0175] The classification prediction unit is used to perform classification prediction on the fusion features after the quadratic weighted fusion through the regression model in the classification recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified.

[0176] Furthermore, the feature extraction module 32 includes:

[0177] an extraction unit, configured to extract low-dimensional numerical vectors from the resistivity data and the real-time ambient temperature and real-time concrete age of the concrete location where each resistivity value in the resistivity data is collected, using the multi-layer perception network model, to obtain a resistivity value vector, an ambient temperature vector, and a concrete age vector;

[0178] A fully connected processing unit is used to learn the combination relationship between the resistivity value vector, the ambient temperature vector and the concrete age vector through a two-layer fully connected network and an activation function to obtain a resistivity feature.

[0179] The present invention provides a device for identifying the wetness and dryness of concrete based on multimodal data fusion. The present invention acquires multimodal data of the concrete surface to be identified, including visible light image data, infrared image data, and resistivity data. A feature extraction model is used to perform multiscale image feature extraction on the visible light and infrared image data to obtain visible light and infrared features, respectively. A multilayer perception network model is used to perform structured feature extraction on the resistivity data to obtain resistivity features. Weights corresponding to the different modal data are determined based on the ambient light intensity and resistivity data of the concrete surface to be identified. Feature fusion is performed based on the visible light, infrared, and resistivity features, along with the weights, to obtain fused features. A trained classification and recognition model is used to classify and predict the fused features to obtain a wetness and dryness identification result for the concrete surface to be identified. This method significantly improves the richness of the data modalities and enables the feature fusion process to adaptively adjust the fusion weights of the different modal features based on varying light intensities, thereby significantly improving the accuracy of wetness and dryness identification.

[0180] According to one embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction. The computer-executable instruction can execute the method for identifying the dry and wet state of concrete based on multimodal data fusion in any of the above method embodiments.

[0181] Figure 8 A schematic structural diagram of a terminal provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the terminal.

[0182] like Figure 8 As shown, the terminal may include: a processor (processor) 402 , a communications interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0183] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0184] The communication interface 404 is used for network communication with other devices such as clients or other servers.

[0185] The processor 402 is configured to execute the program 410 , and specifically to execute the relevant steps in the above-mentioned embodiment of the method for identifying the dry and wet state of concrete based on multimodal data fusion.

[0186] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0187] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the terminal may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0188] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0189] The program 410 may be specifically configured to cause the processor 402 to perform the following operations:

[0190] Acquire multimodal data of the concrete surface to be identified, including visible light image data, infrared image data, and resistivity data;

[0191] Using a feature extraction model to perform multi-scale image feature extraction on the visible light image data and the infrared image data to obtain visible light features and infrared features, and using a multi-layer perception network model to perform structured feature extraction on the resistivity data to obtain resistivity features;

[0192] Determining weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data, and performing feature fusion based on the visible light feature, the infrared feature, the resistivity feature, and the weights to obtain a fused feature;

[0193] The trained classification and recognition model is used to perform classification prediction on the fusion features to obtain a dry and wet state recognition result of the concrete surface to be identified.

[0194] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0195] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for identifying the dry and wet state of concrete based on multimodal data fusion, characterized in that: include: Acquire multimodal data of the concrete surface to be identified, including visible light image data, infrared image data, and resistivity data; Using a feature extraction model to perform multi-scale image feature extraction on the visible light image data and the infrared image data to obtain visible light features and infrared features, and using a multi-layer perception network model to perform structured feature extraction on the resistivity data to obtain resistivity features; Determining weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data, and performing feature fusion based on the visible light feature, the infrared feature, the resistivity feature, and the weights to obtain a fused feature; The trained classification and recognition model is used to perform classification prediction on the fusion features to obtain a dry and wet state recognition result of the concrete surface to be identified.

2. The method according to claim 1, characterized in that Determining the weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data includes: By introducing the light intensity and history fitting parameters into the activation function, the confidence of the visible light image is calculated; By introducing the illumination intensity correction term and the infrared image temperature difference into the activation function, the infrared image confidence is calculated; Calculating a resistivity mean and a standard deviation based on the resistivity data, and calculating a resistivity confidence level based on the mean and the standard deviation; The visible light image confidence, the infrared image confidence, and the resistivity confidence are normalized to obtain a visible light image weight corresponding to the visible light image confidence, an infrared image weight corresponding to the infrared image confidence, and a resistivity weight corresponding to the resistivity confidence.

3. The method according to claim 1, characterized in that Before performing classification prediction on the fusion features using the trained classification recognition model to obtain the dry and wet state recognition result of the concrete surface to be recognized, the method further includes: Acquire multiple sets of historical multimodal data of concrete surfaces collected at different historical times under different lighting conditions, as well as the ambient temperature and concrete age when each set of historical multimodal data was collected, wherein the historical multimodal data includes historical visible light image data, historical infrared image data, and historical resistivity data; Determine the dryness and wetness state labels of each set of historical multimodal data, and mark the fusion features corresponding to the historical multimodal data based on the dryness and wetness state labels to obtain a marked fusion feature sample; An initial classification and recognition model is trained based on the labeled fusion feature samples to obtain a trained classification and recognition model.

4. The method according to claim 3, characterized in that Determining the dry and wet state labels of each set of historical multimodal data includes: For each set of historical multimodal data, a dynamic resistivity threshold is calculated using a multivariate linear regression algorithm according to the corresponding ambient temperature and concrete age; If the historical resistivity value is greater than or equal to the resistivity dynamic threshold, determining that the dry-wet state label is in a wet state; If the historical resistivity value is less than the resistivity dynamic threshold, it is determined that the dry / wet state tag is in a dry state.

5. The method according to claim 1, wherein The feature extraction model is an Inception-v3 model with the last fully connected layer completing transfer learning. The feature extraction model is used to perform multi-scale image feature extraction on the visible light image data and the infrared image data to obtain visible light features and infrared features, including: resizing the visible light image data and the infrared image data so that the visible light image and the infrared image have the same size, thereby obtaining a preprocessed visible light image and a preprocessed infrared image; Performing multi-scale image feature extraction on the preprocessed visible light image and the preprocessed infrared image using the Inception-v3 model that completes transfer learning using the last fully connected layer, thereby obtaining initial visible light image features at multiple scales and initial infrared image features at multiple scales; The initial visible light image features of multiple scales are globally averaged pooled to obtain visible light features, and the initial infrared image features of multiple scales are globally averaged pooled to obtain infrared features.

6. The method according to claim 1, wherein The method of using the trained classification recognition model to perform classification prediction on the fusion features to obtain the dry and wet state recognition result of the concrete surface to be identified includes: Re-weighting and fusing the fused features through the channel attention network in the classification and recognition model to obtain a fused feature after secondary weighted fusion; The fusion features after the quadratic weighted fusion are classified and predicted by the regression model in the classification and recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified.

7. The method according to claim 1, characterized in that The method of extracting structured features from the resistivity data using a multi-layer perception network model to obtain resistivity features includes: Extracting low-dimensional numerical vectors from the resistivity data, and the real-time ambient temperature and real-time concrete age of the concrete location where each resistivity value in the resistivity data is collected, using the multi-layer perception network model, to obtain a resistivity value vector, an ambient temperature vector, and a concrete age vector; The resistivity feature is obtained by learning the combined relationship between the resistivity value vector, the ambient temperature vector and the concrete age vector through a two-layer fully connected network and an activation function.

8. A device for identifying the dry and wet state of concrete based on multimodal data fusion, characterized in that: include: An acquisition module, used to acquire multimodal data of the concrete surface to be identified, including visible light image data, infrared image data and resistivity data; a feature extraction module, configured to perform multi-scale image feature extraction on the visible light image data and the infrared image data using a feature extraction model to obtain visible light features and infrared features, and to perform structured feature extraction on the resistivity data using a multi-layer perception network model to obtain resistivity features; a determination module, configured to determine weights corresponding to different modal data based on the light intensity of the environment in which the concrete surface to be identified is located and the resistivity data, and to perform feature fusion based on the visible light feature, the infrared feature, the resistivity feature, and the weights to obtain a fused feature; The prediction module is used to perform classification prediction on the fusion features using the trained classification recognition model to obtain the dry and wet state recognition result of the concrete surface to be identified.

9. A storage medium storing at least one executable instruction, wherein the executable instruction causes a processor to execute operations corresponding to the method for identifying the dry and wet state of concrete based on multimodal data fusion according to any one of claims 1 to 7.

10. A terminal comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method for identifying the dry and wet state of concrete based on multimodal data fusion according to any one of claims 1 to 7.

Images