Multi-modal wall water seepage detection method and device, computer device and medium

Abstract

The application discloses a kind of based on multimode wall water seepage detection method, device, computer equipment and medium, belongs to detection technical field.The application uses optical lens and infrared lens to monitor water-permeable wall, specific steps include: S10, obtain the image data collected by optical lens and infrared lens;S20, frame cutting operation is carried out to image data, obtain the RGB picture and infrared picture of each frame;S30, the RGB picture and infrared picture are input into water seepage detection model and are detected to obtain water seepage area feature;S40, water seepage area feature is predicted to obtain water seepage prediction result.The application can be well monitored the process of water seepage, it is convenient to find the source position of water seepage, it is convenient to carry out water seepage area maintenance, when the severity of water seepage reaches the set water seepage threshold value, can promptly carry out safety prompt, avoids long-term incubation of security risk.

Description

Technical Field

[0001] This invention relates to the field of detection technology, and more specifically, to a method, apparatus, computer equipment, and medium for detecting wall seepage based on multimodal methods. Background Technology

[0002] Cable tunnels are crucial infrastructure in the metallurgical industry. Because they are typically underground, leaks are common after a period of use, posing significant safety hazards if left untreated. Currently, two common methods are used for seepage detection:

[0003] The first method involves inserting a miniature infrared camera, connected to a flexible, foldable hose, into the interior or bottom of the pipe to be inspected through gaps in the pipe. The infrared detector on the camera then detects the infrared energy of the pipe, converting it into an electrical signal. This signal is transmitted through the hose to a display, amplified, and converted into an infrared thermal image. By observing the thermal image, the condition of the pipe can be determined, thus accurately locating the leak. However, this method relies solely on infrared light for leak detection and requires converting the infrared signal into an electrical signal, making the detection accuracy uncertain. Furthermore, it requires manual inspection to assess the pipe's condition, making real-time unattended monitoring and timely leak detection and alarm impossible.

[0004] The second method involves installing a water immersion mechanism and a detection mechanism on both sides of the wall. The detection mechanism includes a detection box, which is a square box. Each of the four edges of the detection box has a fixing block with a guide hole, and a fixing post is installed at each guide hole. One end of each fixing post has a suction cup, and the other end of each of the four fixing posts has a cross-shaped fixing plate. The center of the cross-shaped fixing plate has a threaded through hole, and an adjusting screw is installed at the threaded through hole. The end of the adjusting screw has a through hole. A water immersion sensor is installed inside the detection box. This water immersion and detection mechanism allows for a convenient and intuitive display of wall seepage. However, this device needs to be installed at every point on the wall, resulting in high costs. Furthermore, this method relies on a water immersion sensor for detection, and it cannot accurately detect the extent of small-area seepage or different types of wall seepage, thus failing to provide accurate alarms.

[0005] A search revealed numerous solutions for detecting water seepage in building walls, highway tunnels, and railway tunnels. Patent publication number CN114419470A discloses a method and system for detecting water seepage in the inner walls of cable tunnels based on an inspection robot. This application first obtains a training sample set, then trains a feature extraction network based on the training sample set to obtain a trained feature extraction network, further generating feature maps. Next, it trains a candidate box network based on the feature maps to obtain a trained candidate box network, further generating candidate boxes. Then, it trains a detection network based on the feature maps and candidate boxes to obtain a trained detection network. Finally, it uses the trained feature extraction network, candidate box network, and detection network to identify images acquired by the inspection robot to determine whether water seepage has occurred. This application allows for real-time unmanned monitoring, avoids the influence of human subjective factors, and requires no installation; however, this application is based on a patrol inspection method, which is not ideal for effectively monitoring water seepage. Summary of the Invention

[0006] 1. The technical problem that the invention aims to solve

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, device, computer equipment and medium for detecting wall seepage based on multimodal technology. This invention can effectively monitor the seepage process, facilitate the location of the seepage source, and facilitate the maintenance of the seepage area. When the severity of the seepage reaches the set seepage threshold, it can provide timely safety warnings and avoid the long-term lurking of safety hazards.

[0008] 2. Technical Solution

[0009] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0010] The multimodal wall seepage detection method of the present invention uses optical and infrared lenses to monitor easily seeping walls. Specific steps include:

[0011] S10. Acquire image data captured by the optical lens and infrared lens;

[0012] S20. Perform frame-slicing operation on the image data to obtain the RGB image and infrared image of each frame;

[0013] S30. Input the RGB image and infrared image into the seepage detection model for detection to obtain the features of the seepage area;

[0014] S40. Predict the characteristics of the seepage area to obtain the seepage prediction results.

[0015] Furthermore, the processing method for the seepage detection model described in step S30 includes:

[0016] S301. Input the RGB image into the native convolution operator, dilated convolution and deformable convolution respectively for processing to obtain the first feature map, the second feature map and the third feature map;

[0017] S302. Concatenate the first feature map, the second feature map, and the third feature map to obtain the first merged feature map;

[0018] S303. The first merged feature is processed by native convolution to obtain the first processed feature;

[0019] S304. Input the first processing feature into the residual network for processing to obtain the second processing feature, the third processing feature and the fourth processing feature;

[0020] S305. The first feature map is scaled to obtain a scaled feature, and the spatial size of the scaled feature is the same as the spatial size of the fourth feature.

[0021] S306. Perform a 1×1 convolution on the scaled features to obtain the fifth processed feature;

[0022] S307. Multiply the fifth processing feature by the fourth processing feature to obtain the sixth processing feature;

[0023] S308. Input the infrared image into the convolution operator for processing to obtain the fourth feature map;

[0024] S309. Input the fourth feature map into the SPPM module to obtain the seventh processing feature;

[0025] S3010. Input the seventh processing feature into a dynamic convolution for processing to obtain the eighth processing feature;

[0026] S3011. After scaling the eighth processing feature, concatenate it with the sixth processing feature to obtain the second merged feature;

[0027] S3012. Upsample the second merged feature to obtain the first upsampled feature;

[0028] S3013. Input the first upsampled feature and the fourth processed feature into the UAFM module to obtain the ninth processed feature;

[0029] S3014. Upsample the ninth processing feature to obtain the second upsampled feature;

[0030] S3015. Input the second upsampling feature and the seventh processing feature into the UAFM module to obtain the tenth processing feature;

[0031] S3016. The tenth processing feature is processed by resizing and 1×1 convolution, and then activated by the sigmoid activation function to make its channel number 1, so as to obtain the water seepage area feature.

[0032] Furthermore, step S309 specifically includes the following steps:

[0033] The S3091 and SPPM modules perform pooling of the fourth feature map at different sizes to obtain pooled features;

[0034] S3092. Perform convolution and scaling on the pooling features respectively to obtain convolution-processed features and scaling-processed features;

[0035] S3093. Add the convolution processing features and the scaling processing features to obtain the seventh processing feature.

[0036] Furthermore, step S3013 specifically includes the following steps:

[0037] S30131. The fourth processing feature is depooled to obtain a depooled feature, and the depooled feature has the same size as the first upsampled feature.

[0038] S30132. Input the unpooling features and the first upsampling features into the spatial attention mechanism and the channel attention mechanism respectively for processing to obtain the ninth processed features.

[0039] Furthermore, step S30132 specifically includes the following steps:

[0040] S301321, The spatial attention mechanism performs channel averaging and maximization operations on the input unpooling features and the first upsampled features respectively, and then merges them to obtain the first fused features;

[0041] S301322, The channel attention mechanism performs global average pooling and global max pooling on the input unpooled features and the first upsampled features, and then merges them to obtain the second fused features;

[0042] S301323. Perform convolution and sigmoid processing on the first fusion feature and the second fusion feature respectively to obtain the first processing result and the second processing result;

[0043] S301324. Multiply the first fusion feature by the first processing result to obtain the third processing result;

[0044] S301325. Perform the operation of subtracting the second processing result from 1 to obtain the calculation result;

[0045] S301326. Multiply the second fusion feature by the calculation result to obtain the fourth processing result;

[0046] S301327. Add the third processing result and the fourth processing result together to obtain the ninth processing feature.

[0047] Furthermore, step S40 specifically includes the following steps:

[0048] S401. Calculate the area of ​​the seepage zone based on its characteristics.

[0049] S402. If the area of ​​the seepage zone is greater than the set threshold, then record this seepage zone as the source of the initial seepage.

[0050] S403. Record the changes in the seepage area according to the set interval time.

[0051] S404. Based on the recorded changes in the seepage area, obtain the rate of change of the seepage area.

[0052] S405. The seepage rate is calculated based on the rate of change of the seepage area.

[0053] S406. Compare the seepage rate with the set seepage threshold to obtain the seepage prediction result.

[0054] Furthermore, in step S405, based on the obtained binarized seepage segmentation map, the edge gradient of the seepage area is calculated using the Canny operator. The edge gradient values ​​are summed to obtain Sum_gradient, and the sum of the corresponding infrared seepage area edge pixel values, Sum_pixel, is calculated. The depth of wall seepage is represented by the formula β = Sum_gradient / Sum_pixel. Finally, the formula is used to...

[0055] water_ratio=γα+δβ;

[0056] The water infiltration rate water_ratio is obtained, where γ=0.8, δ=0.2, α=(ΔS_area) / ∈, ΔS_area represents the magnitude of the change in infiltration area, and ∈ is the set interval time.

[0057] The present invention provides a multimodal wall seepage detection device, which uses an optical lens and an infrared lens to monitor walls prone to seepage. The device includes an acquisition unit, a processing unit, a detection unit, and a prediction unit.

[0058] The acquisition unit is used to acquire image data collected by the optical lens and the infrared lens;

[0059] The processing unit is used to perform frame-slicing operations on the image data to obtain an RGB image and an infrared image for each frame.

[0060] The detection unit is used to input RGB images and infrared images into the seepage detection model for detection in order to obtain the features of the seepage area;

[0061] The prediction unit is used to predict the characteristics of the seepage area to obtain the seepage prediction result.

[0062] A computer device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal wall seepage detection steps described above.

[0063] The present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the multimodal wall seepage detection steps.

[0064] 3. Beneficial effects

[0065] Compared with existing known technologies, the technical solution provided by this invention has the following significant advantages:

[0066] This invention combines infrared and RGB images for multimodal recognition, enabling better extraction of seepage feature information and improving sensitivity to seepage area characteristics. Furthermore, this invention can effectively monitor the seepage process, facilitating the location of the seepage source and enabling maintenance of the seepage area. When the severity of the seepage reaches a set threshold, it can provide timely safety alerts, preventing long-term latent safety hazards. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of a scenario based on multimodal wall seepage detection according to the present invention;

[0068] Figure 2 This is a flowchart of the multimodal wall seepage detection method of the present invention;

[0069] Figure 3 This is a schematic block diagram of the multimodal wall seepage detection device of the present invention;

[0070] Figure 4 This is a schematic block diagram of a computer device provided by the present invention. Detailed Implementation

[0071] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments.

[0072] Example 1

[0073] Combination Figure 1 and Figure 2The multimodal wall seepage detection method in this embodiment is applied to a server, and the method is executed by application software installed on the server.

[0074] like Figure 2 As shown, based on the multimodal wall seepage detection method, optical and infrared lenses are used to monitor easily seeping walls, including the following steps:

[0075] S10. Acquire image data captured by the optical lens and infrared lens.

[0076] The device monitors water-permeable walls by using optical and infrared lenses. Both optical and infrared lenses can acquire image data. The optical and infrared lenses can be standalone monitoring devices or monitoring devices with both types of lenses. This application does not limit this. In addition, the optical and infrared lenses can be those commonly available on the market. This application does not limit this either.

[0077] S20. Perform frame segmentation on the image data to obtain the RGB image and infrared image of each frame.

[0078] Specifically, by performing frame-slicing on the image data acquired by the optical lens, each frame of the RGB image can be obtained; similarly, by performing frame-slicing on the image data acquired by the infrared lens, each frame of the infrared image can be obtained.

[0079] S30. Input the RGB and infrared images into the seepage detection model for detection to obtain the features of the seepage area.

[0080] In this embodiment, step S30 specifically includes steps S301-S3016.

[0081] S301. Input the RGB three-channel image Input_rgb into Vanilla Conv (native convolution operator), Dilated Conv (dilated convolution), and Deformable Conv (deformable convolution) respectively to obtain features F_vc, F_dic, and F_dec.

[0082] S302. Concatenate the three feature maps F_vc, F_dic, and F_dec to obtain feature F_cd.

[0083] S303. The first merged feature F_cd is processed by native convolution to obtain the first processed feature F_vc1.

[0084] S304. The first processing feature F_vc1 is input into the residual network for processing to obtain the second processing feature F_r1, the third processing feature F_r2, and the fourth processing feature F_r3. The residual network consists of a residual function, BatchNorm fuzziness, and a ReLU activation function, respectively.

[0085] S305. The size of the first feature map F_vc is scaled to obtain the scaled processing feature F_vcs, and the spatial size of the scaled processing feature F_vcs is consistent with the spatial size of the fourth processing feature F_r3.

[0086] S306. Perform a 1×1 convolution on the scaled feature F_vcs to obtain the fifth feature F_vcs1.

[0087] S307. Multiply the fifth processing feature F_vcs1 with the fourth processing feature F_r3 to obtain the sixth processing feature F_r4.

[0088] S308. Input the infrared image Input_red into the convolution operator for processing to obtain the fourth feature map F_rc.

[0089] The fourth feature map F_rc is input into the SPPM (Simple Pyramid Pooling Module) module to obtain the seventh processing feature F_SPPM.

[0090] In this embodiment, step S309 specifically includes steps S3091-S3093.

[0091] The S3091 and SPPM modules perform pooling of the fourth feature map F_rc at different sizes to obtain pooled features.

[0092] S3092. Perform convolution and scaling on the pooling features respectively to obtain convolution-processed features and scaling-processed features.

[0093] S3093. Add the convolution processing features and the scaling processing features to obtain the seventh processing feature F_SPPM.

[0094] S3010. Input the seventh processing feature F_SPPM into Dynamic Conv for processing to obtain the eighth processing feature F_dc.

[0095] S3011. After scaling the eighth processing feature F_dc, it is concatenated with the sixth processing feature F_r4 to obtain the second merged feature F_fm.

[0096] S3012. Upsample the second merged feature F_fm to obtain the first upsampled feature F_u1.

[0097] S3013. The first upsampling feature F_u1 and the fourth processing feature F_r3 are input into the UAFM module as F_low and F_high features, respectively, to obtain the ninth processing feature F_UAFM1, which can better integrate semantic feature information from different spaces.

[0098] In this embodiment, step S3013 specifically includes steps S30131-S30132.

[0099] S30131. The fourth processing feature F_r3 is depooled to obtain the depooled feature, and the depooled feature has the same size as the first upsampled feature F_u1.

[0100] UAFM obtains feature F_up by unpooling feature F_high, making its feature size consistent with that of feature F_low, both being F_up∈R^(C×H×W), where C, H, and W represent the channel, height, and width of the feature, respectively.

[0101] S30132. Input the unpooling feature and the first upsampling feature into the spatial attention mechanism and the channel attention mechanism respectively for processing to obtain the ninth processing feature F_UAFM1.

[0102] In this embodiment, step S30132 specifically includes steps S301321-S301327.

[0103] S301321. The spatial attention mechanism performs channel averaging and maximization operations on the input unpooling features and the first upsampled features respectively, and then merges them to obtain the first fusion feature F_(spatial-concate), the size of which is F_(spatial-concate)∈R^(4×H×W).

[0104] S301322. The channel attention mechanism performs global average pooling and global max pooling on the input unpooling features and the first upsampled features, and then concatenates them to obtain the second fused feature F_(channel-concate), the size of which is F_(channel-concate)∈R^(C×1×1).

[0105] S301323. Perform convolution and sigmoid processing on the first fusion feature F_(spatial-concate) and the second fusion feature F_(channel-concate) respectively to obtain the first processing result and the second processing result F_ccb.

[0106] S301324. Multiply the first fusion feature F_(spatial-concate) with the first processing result F_cca to obtain the third processing result F_cca1.

[0107] S301325. Perform the operation of subtracting the second processing result F_ccb from 1 to obtain the result.

[0108] S301326. Multiply the second fusion feature F_(channel-concate) with the calculation result to obtain the fourth processing result F_ccb1.

[0109] S301327. Add the third processing result F_cca1 and the fourth processing result F_ccb1 to obtain the ninth processing feature F_UAFM1.

[0110] S3014. Upsample the ninth processing feature F_UAFM1 to obtain the second upsampled feature F_u2.

[0111] S3015. Input the second upsampling feature F_u2 and the seventh processing feature F_SPPM into the UAFM module to obtain the tenth processing feature F_UAFM2.

[0112] S3016. The tenth processing feature F_UAFM2 is processed by resizing and 1×1 convolution, and then activated by the sigmoid activation function to make its number of channels 1. The binary segmentation image Prediction_(water-area) is used to distinguish the background and the water-seepage area by pixel values ​​0 and 1, so as to obtain the water-seepage area feature.

[0113] The loss functions used in the seepage detection model are the cross-entropy loss function and the Dice loss function.

[0114] Specifically, the ninth processing feature F_UAFM1 and the tenth processing feature F_UAFM2 are subjected to 1×1 convolution and resize operations, and then the loss function is calculated with the corresponding seepage semantic annotation map. At the same time, the loss function of Prediction_(water-area) is calculated with the corresponding seepage semantic annotation map. Among them, the seepage semantic annotation map is the labeled data, which is obtained by labeling during the training of the seepage detection model.

[0115] This seepage detection model uses a combination of cross-entropy loss function and Dice loss function to represent the loss function for semantic segmentation.

[0116] Loss=Loss_dice+Loss_(cross entropy);

[0117] The final loss function is:

[0118] Loss_total=Loss_UAFM1+Loss_UAFM2+Loss_(water-area).

[0119] S40. Predict the characteristics of the seepage area to obtain the seepage prediction results.

[0120] In this embodiment, step S40 specifically includes steps S401-S406.

[0121] S401. Calculate the area of ​​the seepage zone based on its characteristics.

[0122] In this embodiment, the seepage area can be obtained by using the semantic segmentation information predicted by the seepage detection model, and the area of ​​the region can be calculated to obtain S_area.

[0123] S402. If the area of ​​the seepage zone is greater than the set threshold μ, then record this seepage zone as the source of the initial seepage.

[0124] S403. Based on the set interval time ∈, record the change in seepage area to obtain ΔS_area.

[0125] S404. Based on the recorded changes in the seepage area, the rate of change of the seepage area is obtained according to the formula α=(ΔS_area) / ∈. This rate of change represents the seepage velocity within the seepage space.

[0126] S405. The seepage rate is calculated based on the rate of change of the seepage area.

[0127] In this embodiment, based on the obtained binarized seepage segmentation map Prediction_(water-area), the edge gradient of the seepage area is calculated using the Canny operator. The edge gradient values ​​are summed to obtain Sum_gradient, and the sum of the corresponding infrared seepage area edge pixel values, Sum_pixel, is calculated. The depth of wall seepage is represented by the formula β = Sum_gradient / Sum_pixel. Finally, the formula is used to...

[0128] water_ratio=γα+δβ;

[0129] The water infiltration ratio, water_ratio, is obtained, where γ = 0.8 and δ = 0.2.

[0130] S406. Compare the water_ratio with the set water_ratio to obtain the water_ratio prediction result. If the water_ratio reaches a certain water_ratio threshold, an alarm will be triggered to notify relevant personnel to investigate the situation.

[0131] This embodiment combines infrared and RGB images for multimodal recognition, which can better extract seepage feature information and improve the sensitivity to seepage area characteristics. Furthermore, it can effectively monitor the seepage process, easily locate the source of the seepage, facilitate maintenance of the seepage area, and provide timely safety alerts when the severity of the seepage reaches a set threshold, preventing long-term latent safety hazards.

[0132] Example 2

[0133] Corresponding to the above-described multimodal wall seepage detection method, this embodiment provides a multimodal wall seepage detection device 100. Combined with... Figure 3 The water seepage detection device 100 includes a unit for performing the above-described multimodal wall seepage detection method, and the device can be configured in a server.

[0134] The multimodal wall seepage detection device 100 uses optical and infrared lenses to monitor easily seeping walls. The device includes an acquisition unit 110, a processing unit 120, a detection unit 130, and a prediction unit 140.

[0135] The acquisition unit 110 is used to acquire image data collected by the optical lens and the infrared lens.

[0136] The device monitors the easily permeable wall surface using both optical and infrared lenses. Both optical and infrared lenses can acquire image data. The optical and infrared lenses can be standalone monitoring devices or monitoring devices that combine both types of lenses. This application does not limit the choice of these two types of lenses. Furthermore, any commonly used optical and infrared lenses can be selected. This application does not limit the choice of these two types of lenses either.

[0137] The processing unit 120 is used to perform frame-slicing operations on the image data to obtain RGB and infrared images for each frame.

[0138] The detection unit 130 is used to input RGB images and infrared images into the seepage detection model for detection in order to obtain the features of the seepage area.

[0139] In this embodiment, the detection unit 130 includes a first processing module, a merging module, a second processing module, a third processing module, a fourth processing module, a fifth processing module, a sixth processing module, a seventh processing module, an eighth processing module, a ninth processing module, a tenth processing module, an eleventh processing module, a twelfth processing module, a thirteenth processing module, a fourteenth processing module, and a fifteenth processing module.

[0140] The first processing module is used to input the RGB three-channel image into Vanilla Conv (native convolution operator), Dilated Conv (dilated convolution), and Deformable Conv (deformable convolution) for processing to obtain the first feature map F_vc, the second feature map F_dic, and the third feature map F_dec.

[0141] The merging module is used to concatenate the first feature map F_vc, the second feature map F_dic, and the third feature map F_dec to obtain the first merged feature F_cd.

[0142] The second processing module is used to process the first merged feature F_cd through native convolution to obtain the first processed feature F_vc1.

[0143] The third processing module is used to input the first processing feature F_vc1 into the residual network for processing to obtain the second processing feature F_r1, the third processing feature F_r2 and the fourth processing feature F_r3.

[0144] The fourth processing module is used to scale the first feature map F_vc to obtain the scaled processing feature F_vcs, and the spatial size of the scaled processing feature is the same as the spatial size of the fourth processing feature F_r3.

[0145] The fifth processing module is used to perform 1x1 convolution on the scaled processing feature F_vcs to obtain the fifth processing feature F_vcs1.

[0146] The sixth processing module is used to multiply the fifth processing feature F_vcs1 with the fourth processing feature F_r3 to obtain the sixth processing feature F_r4.

[0147] The seventh processing module is used to input the infrared image Input_red into the convolution operator for processing to obtain the fourth feature map F_rc.

[0148] The eighth processing module is used to input the fourth feature map F_rc into the SPPM module to obtain the seventh processing feature F_SPPM.

[0149] In this embodiment, the eighth processing module includes a first processing submodule, a second processing submodule, and a third processing submodule.

[0150] The first processing submodule is used by the SPPM module to perform pooling of the fourth feature at different sizes to obtain pooled features.

[0151] The second processing submodule is used to perform convolution and scaling processing on the pooled features to obtain convolution-processed features and scaling-processed features.

[0152] The third processing submodule is used to add the convolutional processing features and the scaling processing features to obtain the seventh processing feature.

[0153] The ninth processing module is used to input the seventh processing feature F_SPPM into Dynamic Conv for processing to obtain the eighth processing feature F_dc.

[0154] The tenth processing module is used to scale the eighth processing feature F_dc and then concatenate it with the sixth processing feature F_r4 to obtain the second merged feature F_fm.

[0155] The eleventh processing module is used to upsample the second merged feature F_fm to obtain the first upsampled feature F_u1.

[0156] The twelfth processing module is used to input the first upsampled feature F_u1 and the fourth processing feature F_r3 into the UAFM module to obtain the ninth processing feature F_UAFM1.

[0157] In this embodiment, the twelfth processing module includes a fourth processing submodule and a fifth processing submodule.

[0158] The fourth processing submodule is used to depool the fourth processing feature to obtain the depooled feature, and the depooled feature has the same size as the first upsampled feature.

[0159] In this embodiment, UAFM obtains feature F_up by unpooling feature F_high, making its feature size consistent with that of feature F_low. Both feature sizes are F_up∈R^(C×H×W), where C, H, and W represent the channel, height, and width of the feature, respectively.

[0160] The fifth processing submodule is used to input the unpooling features and the first upsampling features into the spatial attention mechanism and the channel attention mechanism respectively for processing, so as to obtain the ninth processing features.

[0161] In this embodiment, the fifth processing submodule includes a sixth processing submodule, a seventh processing submodule, an eighth processing submodule, a ninth processing submodule, a tenth processing submodule, an eleventh processing submodule, and a twelfth processing submodule.

[0162] The sixth processing submodule is used for the spatial attention mechanism to perform channel averaging and maximization operations on the input unpooling features and the first upsampled features respectively, and then merge them to obtain the first fused feature F_(spatial-concate), the size of which is F_(spatial-concate)∈R^(4×H×W).

[0163] The seventh processing submodule is used to perform global average pooling and global max pooling on the input unpooled features and the first upsampled features by the channel attention mechanism, and then merge them to obtain the second fused feature F_(channel-concate), the size of which is F_(channel-concate)∈R^(C×1×1).

[0164] The eighth processing submodule is used to perform convolution and sigmoid processing on the first fusion feature and the second fusion feature respectively to obtain the first processing result F_cca and the second processing result F_ccb.

[0165] The ninth processing submodule is used to multiply the first fusion feature F_(spatial-concate) with the first processing result F_cca to obtain the third processing result F_cca1.

[0166] The tenth processing submodule is used to perform the operation of subtracting the second processing result F_ccb from 1 to obtain the result.

[0167] The eleventh processing submodule is used to multiply the second fusion feature F_(cannel-concate) with the calculation result to obtain the fourth processing result F_ccb1.

[0168] The twelfth processing submodule is used to add the third processing result F_cca1 and the fourth processing result F_ccb1 to obtain the ninth processing feature F_UAFM1.

[0169] The thirteenth processing module is used to upsample the ninth processing feature F_UAFM1 to obtain the second upsampled feature F_u2.

[0170] The fourteenth processing module inputs the second upsampled feature F_u2 and the seventh processing feature F_SPPM into the UAFM module to obtain the tenth processing feature F_UAFM2.

[0171] The fifteenth processing module resizes the tenth processing feature F_UAFM2 and processes it with a 1×1 convolution, then applies a sigmoid activation function to make its channel number 1. It then uses pixel values ​​0 and 1 to distinguish the background and the water-seepage area in a binarized segmentation image Prediction_(water-area) to obtain the water-seepage area features.

[0172] In addition, the loss functions used in the seepage detection model are the cross-entropy loss function and the Dice loss function.

[0173] Specifically, features F_UAFM1 and F_UAFM2 are subjected to 1×1 convolution and resize operations, and then the loss function is calculated with the corresponding seepage semantic annotation map. Simultaneously, Prediction_(water-area) is calculated with the corresponding seepage semantic annotation map, and the loss function is also calculated. The seepage semantic annotation map is the labeled data obtained during the training of the seepage detection model.

[0174] This seepage detection model uses a combination of cross-entropy loss function and Dice loss function to represent the loss function for semantic segmentation.

[0175] Loss=Loss_dice+Loss_(cross entropy);

[0176] The final loss function is:

[0177] Loss_total=Loss_UAFM1+Loss_UAFM2+Loss_(water-area).

[0178] Prediction unit 140 is used to predict the characteristics of the seepage area to obtain the seepage prediction result.

[0179] In this embodiment, the prediction unit 140 includes a first calculation module, a first recording module, a second recording module, a second calculation module, a third calculation module, and a comparison module.

[0180] The first calculation module can determine the seepage area by using the semantic segmentation information predicted by the seepage detection model, and calculate the area S_area of ​​the seepage area based on the characteristics of the seepage area.

[0181] The first recording module is used to record the seepage area as the source of the initial seepage if the area of ​​the seepage area is greater than a set threshold μ.

[0182] The second recording module is used to record the change in seepage area ΔS_area according to the set interval time ∈.

[0183] The second calculation module is used to obtain the rate of change of the seepage area based on the recorded change in seepage area and according to the formula α=(ΔS_area) / ∈, which represents the seepage rate of the seepage space.

[0184] The third calculation module is used to calculate the seepage rate based on the rate of change of the seepage area.

[0185] In this embodiment, based on the obtained binarized seepage segmentation map Prediction_(water-area), the edge gradient of the seepage area is calculated using the Canny operator. The edge gradient values ​​are summed to obtain Sum_gradient, and the sum of the corresponding infrared seepage area edge pixel values, Sum_pixel, is calculated. The depth of wall seepage is represented by the formula β = Sum_gradient / Sum_pixel. Finally, the formula is used to...

[0186] water_ratio=γα+δβ;

[0187] The water infiltration ratio, water_ratio, is obtained, where γ = 0.8 and δ = 0.2.

[0188] The comparison module compares the water_ratio with a set seepage threshold to obtain a seepage prediction result. If the water_ratio reaches a certain seepage threshold, an alarm is triggered to notify relevant personnel to investigate the situation.

[0189] Example 3

[0190] The aforementioned multimodal wall seepage detection device can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the computer device shown.

[0191] See Figure 4 The computer device 700 can be a server, which can be a standalone server or a server cluster composed of multiple servers.

[0192] The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the multimodal wall seepage detection method described above.

[0193] The computer device 700 can be a terminal or a server. The computer device 700 includes a processor 720, a memory, and a network interface 750 connected via a system bus 710, wherein the memory may include a non-volatile storage medium 730 and internal memory 740.

[0194] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. When the computer program 732 is executed, it enables the processor 720 to execute any multimodal wall seepage detection method.

[0195] The processor 720 provides computing and control capabilities to support the operation of the entire computer device 700.

[0196] The internal memory 740 provides an environment for the operation of the computer program 732 in the non-volatile storage medium 730. When the computer program 732 is executed by the processor 720, the processor 720 can execute any multimodal wall seepage detection method.

[0197] This network interface 750 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 700 to which the present application is applied. A specific computer device 700 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. The processor 720 is used to run program code stored in memory to implement the steps described in Embodiment 1.

[0198] It should be understood that in the embodiments of this application, the processor 720 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0199] Example 4

[0200] This embodiment provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multimodal wall seepage detection method disclosed in Embodiment 1 of this invention.

[0201] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0202] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0203] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0204] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0205] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

Claims

1. A multimodal wall seepage detection method, characterized in that: Monitoring of water-permeable walls using optical and infrared lenses involves the following steps: S10. Acquire image data captured by the optical lens and infrared lens; S20. Perform frame-slicing operation on the image data to obtain the RGB image and infrared image of each frame; S30. Input the RGB image and infrared image into the seepage detection model for detection to obtain the features of the seepage area; the processing method of the seepage detection model includes: S301. Input the RGB image into the native convolution operator, dilated convolution and deformable convolution respectively for processing to obtain the first feature map, the second feature map and the third feature map; S302. Concatenate the first feature map, the second feature map, and the third feature map to obtain the first merged feature map; S303. The first merged feature is processed by native convolution to obtain the first processed feature; S304. Input the first processing feature into the residual network for processing to obtain the second processing feature, the third processing feature and the fourth processing feature; S305. The first feature map is scaled to obtain a scaled feature, and the spatial size of the scaled feature is the same as the spatial size of the fourth feature. S306. Perform a 1×1 convolution on the scaled features to obtain the fifth processed feature; S307. Multiply the fifth processing feature by the fourth processing feature to obtain the sixth processing feature; S308. Input the infrared image into the convolution operator for processing to obtain the fourth feature map; S309. Input the fourth feature map into the SPPM module to obtain the seventh processing feature; S3010. Input the seventh processing feature into a dynamic convolution for processing to obtain the eighth processing feature; S3011. After scaling the eighth processing feature, concatenate it with the sixth processing feature to obtain the second merged feature; S3012. Upsample the second merged feature to obtain the first upsampled feature; S3013. Input the first upsampled feature and the fourth processed feature into the UAFM module to obtain the ninth processed feature; S3014. Upsample the ninth processing feature to obtain the second upsampled feature; S3015. Input the second upsampling feature and the seventh processing feature into the UAFM module to obtain the tenth processing feature; S3016. The tenth processing feature is processed by resizing and 1×1 convolution, and then activated by the sigmoid activation function to make its channel number 1, so as to obtain the seepage area feature. S40. Predict the characteristics of the seepage area to obtain the seepage prediction results.

2. The method for detecting wall seepage based on multimodal dynamics according to claim 1, characterized in that: Step S309 specifically includes the following steps: The S3091 and SPPM modules perform pooling of the fourth feature map at different sizes to obtain pooled features; S3092. Perform convolution and scaling on the pooling features respectively to obtain convolution-processed features and scaling-processed features; S3093. Add the convolution processing features and the scaling processing features to obtain the seventh processing feature.

3. The method for detecting wall seepage based on multimodal dynamics according to claim 2, characterized in that: Step S3013 specifically includes the following steps: S30131. The fourth processing feature is depooled to obtain a depooled feature, and the depooled feature has the same size as the first upsampled feature. S30132. Input the unpooling features and the first upsampling features into the spatial attention mechanism and the channel attention mechanism respectively for processing to obtain the ninth processed features.

4. The method for detecting wall seepage based on multimodal dynamics according to claim 3, characterized in that: Step S30132 specifically includes the following steps: S301321, the spatial attention mechanism performs channel averaging and maximization operations on the input unpooling features and the first upsampled features respectively, and then merges them to obtain the first fused feature; S301322, The channel attention mechanism performs global average pooling and global max pooling on the input unpooled features and the first upsampled features, and then merges them to obtain the second fused features; S301323. Perform convolution and sigmoid processing on the first fusion feature and the second fusion feature respectively to obtain the first processing result and the second processing result; S301324. Multiply the first fusion feature by the first processing result to obtain the third processing result; S301325. Perform the operation of subtracting the second processing result from 1 to obtain the calculation result; S301326. Multiply the second fusion feature by the calculation result to obtain the fourth processing result; S301327. Add the third processing result and the fourth processing result together to obtain the ninth processing feature.

5. The method for detecting wall seepage based on multimodal dynamics according to claim 4, characterized in that: Step S40 specifically includes the following steps: S401. Calculate the area of ​​the seepage zone based on its characteristics. S402. If the area of ​​the seepage zone is greater than the set threshold, then record this seepage zone as the source of the initial seepage. S403. Record the changes in the seepage area according to the set interval time. S404. Based on the recorded changes in the seepage area, obtain the rate of change of the seepage area. S405. The seepage rate is calculated based on the rate of change of the seepage area. S406. Compare the seepage rate with the set seepage threshold to obtain the seepage prediction result.

6. The method for detecting wall seepage based on multimodal dynamics according to claim 5, characterized in that: Step S405 uses the Canny operator to calculate the edge gradient of the seepage area based on the obtained binarized seepage segmentation map. The edge gradient values ​​are summed to obtain Sum_gradient, and the corresponding sum of infrared seepage area edge pixel values, Sum_pixel, is calculated. The depth of wall seepage is represented by the formula β = Sum_gradient / Sum_pixel. Finally, the formula is used to... water_ratio=γα+δβ; The water infiltration rate water_ratio is obtained, where γ=0.8, δ=0.2, α=(ΔS_area) / ∈, ΔS_area represents the magnitude of the change in infiltration area, and ∈ is the set interval time.

7. A multimodal wall seepage detection device, characterized in that: The method for detecting water seepage in walls based on any one of claims 1-6, wherein the detection device uses an optical lens and an infrared lens to monitor walls prone to water seepage, and the device includes an acquisition unit, a processing unit, a detection unit, and a prediction unit; The acquisition unit is used to acquire image data collected by the optical lens and the infrared lens; The processing unit is used to perform frame-slicing operations on the image data to obtain an RGB image and an infrared image for each frame. The detection unit is used to input RGB images and infrared images into the seepage detection model for detection in order to obtain the features of the seepage area; The prediction unit is used to predict the characteristics of the seepage area to obtain the seepage prediction result.

8. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal wall seepage detection steps as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor performs the multimodal wall seepage detection steps as described in any one of claims 1-6.

Images