Water conservancy inspection robot inspection method and system based on multi-modal data fusion

Abstract

The invention discloses a water conservancy inspection robot inspection method based on multi-modal data fusion, and relates to the field of intelligent inspection, and the method comprises the steps: obtaining the historical inspection information of a target dam; generating a water conservancy inspection strategy based on the historical inspection information and the dike data twinborn model; collecting multi-mode inspection information of the target dam; extracting multi-modal inspection features of the dam inspection area based on the multi-modal inspection information; and grade division of the multi-mode inspection features is completed, and dam abnormity grades are obtained. The accuracy of the inspection result can be effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of intelligent inspection, and in particular to a water conservancy inspection robot inspection method and system based on multimodal data fusion. Background Art

[0002] As core infrastructure for ensuring water security, the safe operation of water conservancy projects is directly linked to major livelihood issues such as flood prevention and disaster reduction, water resource allocation, and ecological protection. Levees, as key components of water conservancy projects, face constant risks such as erosion, seepage deformation, and geological disasters. Therefore, regular inspections and potential hazard detection are crucial to ensuring their structural stability.

[0003] Existing inspection methods include regular inspections using drone imagery, but this method is limited in scope and prone to significant errors in monitoring results, which can lead to serious consequences. Some regions also combine drone imagery with ground-based sensors for embankment inspections, but the temporal and spatial variations in the different sensor data make centralized analysis and processing difficult, potentially leading to biased monitoring results. Summary of the Invention

[0004] The embodiments of the present application provide a water conservancy inspection robot inspection method and system based on multimodal data fusion, which is used to solve the problems of single inspection information dimension and low inspection result accuracy in the existing inspection mode.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, a water conservancy inspection robot inspection method based on multimodal data fusion is provided, the method comprising: Obtain historical inspection information of the target dam through a pre-built dam data twin model; Generating a water conservancy inspection strategy based on the historical inspection information and the dam data twin model; Controlling the inspection robot to go to each dam inspection area according to the water conservancy inspection route in the water conservancy inspection strategy to collect multimodal inspection information of the target dam; For any of the dam inspection areas, extracting multimodal inspection features of the dam inspection area based on the multimodal inspection information; The multimodal inspection features are graded by forward propagation of a gradient decision tree model to obtain the dam anomaly grade of the dam inspection area.

[0006] Optionally, the historical inspection information includes historical inspection areas and historical abnormality levels, and generating a water conservancy inspection strategy based on the historical inspection information and the dam data twin model includes the following steps: Verify the location information of the preset dam inspection area and the historical inspection area, and allocate regional inspection time for all the dam inspection areas based on the inspection location verification result and the historical abnormality level; Locating the dam obstacle area of ​​the target dam based on the dam data twin model; Generate an initial inspection route based on the location information of all the inspection areas and using a shortest path algorithm; Using the dam obstacle area as a constraint condition, the initial inspection route is modified to obtain a water conservancy inspection route; The water conservancy inspection route and the regional inspection time are integrated to generate a water conservancy inspection strategy.

[0007] Optionally, the multimodal inspection information includes dam image information, dam infrared information and dam acoustic wave information.

[0008] Optionally, extracting multimodal inspection features of the dam inspection area based on the multimodal inspection information includes the following steps: Iteratively refining the dam image information using the ZHANG iterative refinement algorithm, and extracting crack size features of the dam image information based on the iterative refinement result; converting the dam infrared information into dam temperature information according to pre-acquired inspection equipment parameters, and extracting dam temperature characteristics of the dam inspection area according to the dam temperature information; Extracting the acoustic wave time-frequency characteristics of the dam acoustic wave information based on Fourier transform; The crack size characteristics, the dam temperature characteristics and the acoustic wave time-frequency characteristics are integrated into multimodal inspection characteristics of the dam inspection area.

[0009] Optionally, iteratively refining the dam image information using the ZHANG iterative refinement algorithm and extracting crack size features of the dam image information according to the iterative refinement result includes the following steps: pre-processing the dam image information; detecting an edge point set of the pre-processed dam image information using an edge detection algorithm, and separating the dam crack image from the dam image background in the dam image information based on the edge point set; completing morphological optimization of the dam crack image through morphological opening operation to obtain a complete crack image; Traversing all connected domains in the complete crack image, calculating the total pixel values ​​of all connected domains, and eliminating connected domains whose total pixel values ​​are less than a preset total pixel value threshold to obtain a target crack image; Iteratively thinning the target crack image using the ZHANG iterative thinning algorithm to obtain a full crack skeleton and a partial crack skeleton of the target crack image; Combining the full crack skeleton and the incomplete crack skeleton to extract a crack salient point set of the target crack image; Fitting the crack salient point set using the least square method, and calculating the crack pixel width of the dam crack image according to the fitting result; Calculating the sum of the Euclidean distances between all adjacent pixels in the full crack skeleton to obtain the crack pixel length of the dam crack image; The crack pixel width and the crack pixel length are integrated to obtain the crack size feature of the dam image information.

[0010] Optionally, converting the dam infrared information into dam temperature information according to pre-acquired inspection equipment parameters, and extracting the dam temperature characteristics of the dam inspection area according to the dam temperature information includes the following steps: Complete the radiometric brightness mapping of the inspection equipment parameters acquired in advance, and convert the infrared information of the dam into the radiometric brightness of the dam according to the radiometric brightness mapping result; Inversely calculate the dam temperature information of the dam inspection area according to the dam radiation brightness and based on Planck's law; Calculating a regional temperature mean of the dam inspection area based on the dam temperature information, and extracting a regional temperature extreme value of the dam inspection area; The regional temperature mean and the regional temperature extreme are integrated to obtain the dam temperature characteristics of the dam inspection area.

[0011] Optionally, extracting the acoustic wave time-frequency characteristics of the dam acoustic wave information based on Fourier transform includes the following steps: Using a wavelet transform algorithm to remove noise signals from the dam sound wave information to obtain a noise-free wave signal; Decomposing the noise-free wave signal into a plurality of local sound wave signals using a sliding window method; For any of the local sound wave signals, complete time domain analysis of all local sound wave signals to obtain local time domain features of the local sound wave signals, wherein the local time domain features include a local signal mean, a local signal variance, and a local signal peak value; Performing time-frequency conversion of the local sound wave signal by Fourier transform to obtain a local frequency domain signal, and extracting local frequency domain features of the local frequency domain signal, wherein the local frequency domain features include local spectrum amplitude, local spectrum phase and local spectrum frequency; The local time domain features and local frequency domain features of all the local acoustic wave signals are integrated to obtain the acoustic wave time-frequency features of the dam acoustic wave information.

[0012] Optionally, completing the level classification of the multimodal inspection features by forward propagation of a gradient decision tree model to obtain the dam anomaly level of the dam inspection area includes the following steps: Constructing a sample data set based on historical inspection information, and using the sample data set to train a gradient decision tree model; Normalizing the multimodal inspection features; Inputting the multimodal inspection features into the gradient decision tree model that has completed model training, performing node splitting judgment on the multimodal inspection features through the gradient decision tree model, and generating a predicted contribution value based on the node splitting judgment result; A weighted summation process of all the predicted contribution values ​​is completed through multiple decision trees in the gradient decision tree model to obtain an abnormality level probability distribution of the multimodal inspection feature; The abnormality level probability distribution is mapped to the dam abnormality level of the dam inspection area through a gradient decision tree model.

[0013] In a second aspect, the present application provides a machine-readable storage medium, characterized in that instructions are stored on the machine-readable storage medium, and the instructions are used to enable the machine to execute the method of water conservancy inspection robot based on multimodal data fusion according to any one of the first aspects.

[0014] In a third aspect, the present application provides a water conservancy inspection robot inspection system based on multimodal data fusion, characterized by including: a memory configured to store instructions; and The processor is configured to call the instructions from the memory and to implement the method for water conservancy inspection robot based on multimodal data fusion according to any one of the first aspects when executing the instructions.

[0015] Through the above technical solution, by planning the water conservancy inspection route for the inspection robot based on historical inspection information and generating a water conservancy inspection strategy, it is possible to effectively predict risks and avoid the inspection robot from being trapped during the inspection process, thereby avoiding the failure of the inspection task. At the same time, according to the historical anomaly level, the regional inspection time is allocated to each dam inspection area, and high-frequency monitoring is performed on the dam inspection area with a higher historical risk level, which can effectively ensure the accuracy of the monitoring results. By collecting multimodal data from each dam inspection area, complementary information of different modal data can be provided for the subsequent analysis of the dam anomaly level, avoiding the misjudgment of a single-dimensional sensor. By iteratively constructing a gradient decision tree, intelligent grading of multimodal inspection features can be achieved, thereby improving the accuracy of dam anomaly identification. In summary, the present application can effectively solve the problems of the single inspection information dimension and low inspection result accuracy in the existing inspection model, thereby promoting the digital and intelligent transformation of water conservancy project monitoring and providing core technical support for basin safety control.

[0016] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a water conservancy inspection robot inspection method based on multimodal data fusion is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0019] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0020] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] Figure 1 The following schematically shows a flow chart of a method for water conservancy inspection robot inspection based on multimodal data fusion according to an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides a method for water conservancy inspection robot inspection based on multimodal data fusion, which may include the following steps: S101. Obtain historical inspection information of the target dam through the pre-built dam data twin model.

[0022] In this embodiment, the dam data twin model integrates a multi-source heterogeneous data storage system, and calls the historical inspection information module through the data interface of the dam data twin model. This module stores historical inspection information divided by time series and spatial areas. The multi-source heterogeneous data storage system will automatically filter out the historical inspection information within the corresponding time period, and at the same time associate the abnormality level label recorded in the historical inspection information for each historical inspection area. During the data acquisition process, the dam data twin model will perform integrity verification and validity verification on the data, eliminate invalid records, such as information without abnormality level labels, and integrate the historical inspection areas and corresponding historical abnormality levels according to the preset data structure, and output them in the form of a table or structured file to ensure that the acquired information contains the spatial location coordinates, inspection timestamps and corresponding abnormality level classifications of the specific historical inspection areas, providing accurate historical data support for subsequent data analysis and model training.

[0023] S102. Generate water conservancy inspection strategies based on historical inspection information and dam data twin models.

[0024] In this embodiment, the spatial overlay function of the dam digital twin model is utilized to visualize the geographic boundaries of the dam inspection area and the historical inspection area using different color layers. Areas of complete overlap, partial overlap, or complete non-overlap are marked, and the area percentage of the overlapping areas is calculated. Coordinates of non-overlapping areas are compared, and the Euclidean distance between the two points is calculated to determine whether the positional deviation is within the allowable error range. For example, if the error range is within 5-10 meters, regional inspection time is allocated to each dam inspection area based on the historical anomaly level corresponding to the historical inspection area. The higher the corresponding historical anomaly level, the longer the allocated inspection time and the higher the inspection frequency.

[0025] Areas difficult for inspection robots to pass through, known as dam obstacle zones, are located within the dam data twin model. These areas include steep slopes with a gradient greater than 30°, water pits greater than 50 cm deep, or areas designated as prohibited areas with a radius of 50 meters in the dam data twin model due to landslide hazards. An initial inspection route is then generated based on the location information of all inspection areas using a shortest path algorithm. Common shortest path algorithms include the Dijkstra algorithm and the Floyd-Warshall algorithm. The initial inspection route is then verified to determine whether it passes through the dam obstacle zone. If not, the initial inspection route is used as the hydraulic inspection route. If it does, the initial inspection route is generated using the location information from the Dijkstra algorithm, the Floyd-Warshall algorithm, or other algorithms. Common shortest path algorithms replan the shortest path between nodes that pass through the dam obstacle zone to obtain the hydraulic inspection route. The hydraulic inspection route and regional inspection time are combined to form a hydraulic inspection strategy.

[0026] S103. Control the inspection robot to go to each dam inspection area according to the water conservancy inspection route in the water conservancy inspection strategy to collect multimodal inspection information of the target dam.

[0027] In this embodiment, the inspection robot is equipped with infrared cameras, millimeter-wave radars, and visual cameras. It travels to various dam inspection areas according to the water conservancy inspection route specified in the water conservancy inspection strategy. These devices collect image information, infrared information, and acoustic information from the target dams. Furthermore, because the inspection robot is equipped with millimeter-wave radars and visual cameras, it can identify obstacles along the inspection route in real time, such as temporarily stored construction materials, and reroute itself to avoid them, ensuring successful completion of the inspection mission.

[0028] S104. For any dam inspection area, extract the multimodal inspection features of the dam inspection area based on the multimodal inspection information.

[0029] The ZHANG iterative refinement algorithm is used to iteratively refine the dam image information, and the crack size characteristics of the dam image information are extracted based on the iterative refinement results. The dam infrared information is converted into dam temperature information according to the inspection equipment parameters acquired in advance, and the dam temperature characteristics of the dam inspection area are extracted according to the dam temperature information; The acoustic time-frequency characteristics of the dam acoustic information are extracted based on Fourier transform; The crack size characteristics, embankment temperature characteristics and acoustic wave time-frequency characteristics are integrated into the multimodal inspection characteristics of the embankment inspection area.

[0030] In this embodiment, the dam image information is first processed through grayscale conversion, filtering and noise reduction, edge detection, and morphological opening operations to obtain a target crack image. The target crack image is then iteratively refined using the ZHANG iterative refinement algorithm to obtain a full crack skeleton and a partial crack skeleton for the target crack image. The full crack skeleton refers to a skeleton structure in the target crack image that only retains the crack centerline and has a width of one pixel. The partial crack skeleton also includes some non-skeleton pixels. The least squares method is used to fit the non-skeleton pixels to a tangent line, and an orthogonal line to this tangent line is drawn. The intersection of this orthogonal line with the crack edge is the point of maximum width, and the pixel distance between the two intersections is the crack pixel width. The sum of the Euclidean distances between all adjacent pixels in the full crack skeleton is then calculated to obtain the crack pixel length of the dam crack image. By integrating the crack pixel width and crack pixel length, the crack size characteristics of the dam crack image information are obtained.

[0031] The infrared camera on the inspection robot was calibrated using a blackbody furnace. The DN values ​​at different temperatures were recorded, and a linear relationship between the infrared camera's grayscale value (DN value) and radiance was fitted. Based on this fitted linear relationship, the dam's infrared information was converted into dam radiance. Then, based on the detection wavelength parameters specified in the infrared camera's technical documentation and using Planck's law, the dam temperature information of the inspection area was inverted. The same method was used to obtain the dam temperature information of all inspection points within the dam inspection area. The temperature sum of all inspection points was calculated, and the regional temperature mean of the dam inspection area was calculated based on the temperature sum. The maximum and minimum temperatures of all inspection points were selected as the regional temperature extremes of the dam inspection area. The regional temperature mean and regional temperature extremes were integrated to obtain the dam temperature characteristics of the inspection area.

[0032] First, wavelet decomposition is used to remove noise signals from the levee acoustic wave signal to obtain a noise-free wave signal. A sliding window algorithm is then used to localize the noise-free wave signal to a local acoustic wave signal. The local signal mean, local signal variance, and local signal peak of the local acoustic wave signal are then calculated and integrated into the local time domain features of the local acoustic wave signal. The local frequency domain features of the local acoustic wave signal are then calculated using Fourier transform. The local frequency domain features include the local spectrum amplitude, local spectrum phase, and local spectrum frequency. The local time domain features and local frequency domain features are integrated to obtain the acoustic time-frequency features of the levee acoustic wave information. This step fuses the time domain and frequency domain information to form a multidimensional feature vector, providing comprehensive data support for subsequent levee anomaly identification. Finally, the crack size features, levee temperature features, and acoustic wave time-frequency features are integrated into the multimodal inspection features of the levee inspection area.

[0033] S105. Complete the classification of multimodal inspection features through forward propagation of the gradient decision tree model to obtain the dam anomaly level in the dam inspection area.

[0034] In this embodiment, previously labeled historical inspection information is used as a sample dataset, which is then divided into a training set, a validation set, and a test set. The basic parameters of the gradient decision tree model, including the number of trees, maximum depth, and learning rate, are then set. The gradient decision tree model is then trained on the training set, using the multimodal inspection features of the training set as input and the historical anomaly level labels as output. Multiple decision trees are iteratively constructed using a gradient boosting method. Each decision tree is constructed based on the residuals of the previous tree. The splitting criteria for each node are determined by minimizing a loss function (such as mean squared error or cross entropy). The features and splitting thresholds that result in the fastest reduction in the loss function are selected. The multimodal inspection features are then normalized to eliminate the effects of varying feature dimensions and value ranges, ensuring the stability and accuracy of model training. The normalized multimodal inspection features are then input into the trained gradient decision tree model. The model then performs node splitting on each sample feature, thereby obtaining a probability distribution of anomaly levels for the multimodal inspection features. Finally, the probability distribution of anomaly levels is mapped to the levee anomaly levels within the levee inspection area. The anomaly level with the highest probability is generally selected as the final prediction. The levee digital twin model is then updated based on the final prediction results—the levee anomaly levels within each levee inspection area. This visualization facilitates subsequent maintenance work on the target levee by levee staff.

[0035] In one embodiment, the historical inspection information includes historical inspection areas and historical abnormality levels. Generating a water conservancy inspection strategy based on the historical inspection information and the dam data twin model includes the following steps: Verify the location information of the preset dam inspection areas and historical inspection areas, and allocate regional inspection time for all dam inspection areas based on the inspection location verification results and historical abnormality levels; Locate the dam obstacle area of ​​the target dam based on the dam data twin model; Generate an initial inspection route based on the location information of all inspection areas and using the shortest path algorithm; The initial inspection route is modified using the dam obstacle area as a constraint to obtain the water conservancy inspection route; Integrate water conservancy inspection routes and regional inspection times to generate water conservancy inspection strategies.

[0036] In this embodiment, the spatial overlay function of the levee digital twin model is utilized to visualize the geographic boundaries of the levee inspection area and the historical inspection area using different color layers. For example, the levee inspection area is represented by a blue planar layer, while the historical inspection area is covered by a semi-transparent red planar layer. Areas of complete overlap, partial overlap, or complete non-overlap are marked, and the area percentage of the overlapping area is calculated. Coordinate comparison is performed on non-overlapping areas, for example, by calculating the Euclidean distance between two points to determine whether the positional deviation is within the allowable error range, such as within 5 to 10 meters. Regional inspection time is then allocated to each levee inspection area based on the historical anomaly level corresponding to the historical inspection area. Higher historical anomaly levels are associated with longer inspection times and a higher inspection frequency.

[0037] The dam data twin model is constructed based on the integration of BIM (Building Information Modeling) and GIS. It includes a 3D geometric model of the target dam, including structures such as the dam body, slope protection, and gates; topographic data such as the slope and depth of water accumulation pits; a sensor network for monitoring water levels and water quality, including water level and water quality sensors; and historical operating data such as flood frequency and material degradation. Therefore, the dam data twin model can directly identify areas difficult for inspection robots to pass through, known as dam obstacle zones. These include steep slopes with a gradient greater than 30°, water accumulation pits greater than 50 cm deep, and areas designated as no-go zones with a radius of 50 meters in the dam data twin model due to landslide risks. Furthermore, the inspection robot is equipped with millimeter-wave radar and visual cameras, enabling real-time identification of obstacles along the water conservancy inspection route, such as temporarily stored construction materials.

[0038] Next, a shortest path algorithm is used to generate an initial inspection route based on the location information of all regional inspection areas. Common shortest path algorithms include the Dijkstra algorithm and the Floyd-Warshall algorithm. Taking the Dijkstra algorithm as an example, its key feature is that it starts from a starting point and uses a greedy algorithm strategy, traversing the adjacent nodes of the nearest unvisited vertex each time until it reaches the end point. Specifically, all regional inspection areas are first abstracted as nodes in a graph, labeled with unique IDs (such as inspection area A, inspection area B, etc.) and geographic coordinates. For example, the dam management station is set as the starting node S, area inspection AF as the intermediate node, and the robot management station as the end node T. Then, based on the dam data twin model, actual traversable routes (dam roads, inspection paths) are extracted as edges, excluding impassable paths such as swamps and unopened areas. The directionality of edges is clarified (such as one-way construction sections). The straight-line distance between the center points of all regional inspection areas is used as the edge weight. For example, the Euclidean distance from center point A to center point B is 800 meters. Then, we construct an unvisited node set, which contains all the nodes of the graph. We mark the starting node S as visited and use it as the target node W. Then, we traverse all adjacent edges of the target node W, such as W to A and W to B, and select the intermediate node closest to the target node W. We use this intermediate node as the target node W and mark this intermediate node as visited. We repeat the above steps until the unvisited node set is empty, that is, all nodes are marked as visited, and the initial inspection route can be obtained.

[0039] Next, the initial inspection route is verified to determine whether it passes through the dam obstruction area. If not, the initial inspection route is used directly as the water conservancy inspection route. If it does, the initial inspection route is generated using location information from algorithms such as the Dijkstra algorithm and the Floyd-Warshall algorithm. A commonly used shortest path algorithm is used to replan the shortest path between nodes that pass through the dam obstruction area to obtain the water conservancy inspection route. The water conservancy inspection route and regional inspection time are combined to form a water conservancy inspection strategy.

[0040] In one embodiment, the multimodal inspection information includes dam image information, dam infrared information, and dam acoustic wave information.

[0041] In this embodiment, the inspection robot is equipped with infrared cameras, millimeter-wave radars, and visual cameras. These devices collect image information, infrared information, and acoustic wave information from the target dam. The image information is used to identify cracks on the water-facing and water-repelling surfaces of the target dam. The infrared information is used to locate leaks and detect structural thermal defects. Leakage location refers to locating the location of leaks within the target dam. This is due to abnormal surface temperatures caused by water infiltration. For example, in winter, the temperature of the leaking area is higher than that of the surrounding dry areas. Structural thermal defect detection involves detecting structural cracks or voids in the target dam. This is due to different heat dissipation rates in the concrete cracks or voids, resulting in a temperature gradient on the surface. For example, void areas cool more rapidly at night, appearing as cool patches. Acoustic wave information is used to detect hidden defects such as cracks and loose aggregate within the target dam. For example, crack width can be determined by echo attenuation, while defect depth can be determined by echo delay.

[0042] In one embodiment, extracting multimodal inspection features of a dam inspection area based on multimodal inspection information includes the following steps: The ZHANG iterative refinement algorithm is used to iteratively refine the dam image information, and the crack size characteristics of the dam image information are extracted based on the iterative refinement results. The dam infrared information is converted into dam temperature information according to the inspection equipment parameters acquired in advance, and the dam temperature characteristics of the dam inspection area are extracted according to the dam temperature information; The acoustic time-frequency characteristics of the dam acoustic information are extracted based on Fourier transform; The crack size characteristics, embankment temperature characteristics and acoustic wave time-frequency characteristics are integrated into the multimodal inspection characteristics of the embankment inspection area.

[0043] In this embodiment, the dam image information is first subjected to grayscale processing, filtering and noise reduction, edge detection, and morphological opening operations to obtain a target crack image. The target crack image is then iteratively refined using the ZHANG iterative refinement algorithm to obtain a full crack skeleton and a partial crack skeleton for the target crack image. The ZHANG iterative refinement algorithm is an algorithm for extracting skeletons from binary images. It iteratively deletes boundary points that meet specific conditions to obtain a single-pixel-wide skeleton. The ZHANG iterative refinement algorithm consists of two iterative steps. Each iterative refinement step deletes pixels in the target crack image that meet the first and second iterative refinement formulas. Through this iterative refinement process, a full crack skeleton and a partial crack skeleton for the target crack image are obtained. A full crack skeleton is a skeleton structure in the target crack image that only retains the crack centerline and has a width of one pixel. A partial crack skeleton, on the other hand, contains some non-skeleton pixels in addition to the full crack skeleton. Using the least squares method, non-skeleton pixels are fitted to a tangent line. An orthogonal line is drawn to this tangent line. The intersection of this orthogonal line with the crack edge is the point of maximum width, and the pixel distance between the two intersections is the crack pixel width. Next, the sum of the Euclidean distances between all adjacent pixels in the full crack skeleton is calculated to obtain the crack pixel length of the dam crack image. By integrating the crack pixel width and crack pixel length, the crack size characteristics of the dam image information are obtained.

[0044] The infrared camera on the inspection robot is calibrated using a blackbody furnace (e.g., a standard blackbody at 30°C, 50°C, or 70°C). The DN values ​​at different temperatures are recorded, and a linear relationship between the infrared camera's grayscale value (DN value) and radiance is fitted. Based on this fitted linear relationship, the dam's infrared information is converted into dam radiance. Then, based on the detection wavelength parameters specified in the infrared camera's technical documentation and using Planck's law, the dam temperature information of the inspection area is inverted. The same method is used to obtain the dam temperature information of all inspection points within the inspection area. The temperature sum of all inspection points is calculated, and the regional temperature mean of the inspection area is calculated based on the temperature sum. The maximum and minimum temperatures of all inspection points are selected as the regional temperature extremes of the inspection area. The regional temperature mean and regional temperature extremes are integrated to obtain the dam temperature characteristics of the inspection area.

[0045] To address the non-stationary nature of dam acoustic signals (such as ultrasonic flaw detection echoes and underwater sonar reflections) in dam acoustic wave information, wavelet decomposition is performed using a wavelet basis (such as db4 and sym8) with compact support and good time-frequency localization. This wavelet decomposition removes noise from the dam acoustic signal, yielding a noise-free wave signal. Starting from the starting point of the noise-free wave signal, a sliding window with a step size S (S = L × (1-overlap ratio)) is then applied to the noise-free wave signal within each window to yield a local acoustic wave signal. The local signal mean, local signal variance, and local signal peak are then calculated and integrated into the local time domain features of the local acoustic wave signal. The local frequency domain features of the local acoustic wave signal are calculated using Fourier transform. These local frequency domain features include the local spectral amplitude, local spectral phase, and local spectral frequency. The local time-domain and frequency-domain features are integrated to generate the acoustic time-frequency characteristics of the levee acoustic information. This step fuses the time-domain and frequency-domain information to form a multidimensional feature vector, providing comprehensive data support for subsequent levee anomaly identification. Finally, the crack size characteristics, levee temperature characteristics, and acoustic time-frequency characteristics are integrated to form a multimodal inspection feature for the levee inspection area.

[0046] In one embodiment, iteratively refining dam image information using the ZHANG iterative refinement algorithm and extracting crack size features of the dam image information based on the iterative refinement results includes the following steps: Preprocessing dam image information; An edge detection algorithm is used to detect edge point sets of the pre-processed dam image information, and based on the edge point sets, the dam crack image in the dam image information is separated from the dam image background; The morphological optimization of the dam crack image is completed through morphological opening operation to obtain a complete crack image; Traverse all connected domains in the complete crack image, calculate the total pixel value of all connected domains, and eliminate the connected domains whose total pixel value is less than a preset pixel total value threshold to obtain the target crack image; The ZHANG iterative refinement algorithm is used to perform iterative refinement of the target crack image, and the full crack skeleton and the partial crack skeleton of the target crack image are obtained. Combine the full crack skeleton and the incomplete crack skeleton to extract the crack salient point set of the target crack image; The least square method is used to fit the crack point set, and the crack pixel width of the dam crack image is calculated based on the fitting results; The sum of the Euclidean distances between all adjacent pixels in the full crack skeleton is calculated to obtain the crack pixel length of the dam crack image; The crack size characteristics of the embankment image information are obtained by integrating the crack pixel width and crack pixel length.

[0047] In this embodiment, the preprocessing steps include: first, converting the levee image information into a grayscale image to reduce the data dimension. Next, a median filter algorithm (e.g., a 3×3 window) is used to remove salt-and-pepper noise. This algorithm effectively preserves edges while eliminating isolated noise points by sorting pixel values ​​within the window and taking the median. After preprocessing, an edge detection algorithm, such as the Canny edge detection algorithm, is used to smooth the preprocessed levee image information using a Gaussian filter, calculate the gradient magnitude and direction, and refine the edges using non-maximum suppression. Finally, a dual-thresholding method (a combination of high and low thresholds) is used to extract a set of continuous edge points from the levee image information. A closed contour is constructed based on this set of edge points. The interior of the contour is marked as the levee crack image, and the exterior is marked as the levee image background. The levee crack image is then cut out from the levee image information to separate it from the background. Next, a morphological opening operation is performed to remove fine noise from the levee crack image, such as burrs and small protrusions at the crack edges. A morphological opening operation refers to the morphological operation of erosion followed by dilation. Next, all connected domains in the complete crack image are traversed, and the total number of pixels in each connected domain is counted to obtain its total pixel value. A pixel value threshold is set, for example, 50 pixels. Connected domains with a pixel value less than the threshold are identified as pseudo-cracks (such as gravel and stains) and removed, resulting in the final target crack image.

[0048] The ZHANG iterative refinement algorithm is then used to iteratively refine the target crack image, obtaining both the full crack skeleton and the partial crack skeleton. The ZHANG iterative refinement algorithm is an algorithm for extracting skeletons from binary images. It iteratively removes boundary points that meet specific conditions to obtain a single-pixel-width skeleton. The ZHANG iterative refinement algorithm consists of two iterative steps. Each iterative refinement step removes pixels in the target crack image that meet the first and second iterative refinement formulas. The first iterative refinement formula is as follows:

[0049] The second iterative refinement formula is as follows:

[0050]

[0051]

[0052] in, (i=1,2,…,9) are the 9 pixels in the convolution window, for The pixel value at is the number of times two consecutive pixel values ​​in the pixel neighborhood are 0 and 1, This means that both and ,Right now To express the meaning of "and", it is used to connect multiple conditions.

[0053] The process of obtaining full and partial crack skeletons is also the process of crack skeletonization, which can be understood as the process of edge corrosion and peeling of the crack. Each iteration deletes pixels on the edge that meet the deletion criteria, and the loop stops when all pixels no longer meet the deletion criteria. When the crack peeling cycle reaches its final stage, the pixel position that still meets the refinement criteria is the location of the maximum crack width. The entire process requires two skeletonization operations. The first skeletonization is a full skeletonization, that is, obtaining a full crack skeleton. The crack object is iteratively refined until no pixels meet the deletion criteria. The second skeletonization has one fewer iterative refinement than the first. The second skeletonization obtains the crack skeleton and some non-skeleton pixels, which are used as the crack salient point set of the target crack image.

[0054] The least squares method is then used to fit the salient crack points to a tangent line. An orthogonal line to this tangent is drawn. The intersection of this orthogonal line with the crack edge is the point of maximum width, meaning the pixel distance between the two intersections is the crack pixel width. The core steps of the least squares method for fitting tangent lines include establishing a linear model, constructing a sum-of-squares error function, taking partial derivatives, and solving a system of equations to obtain parameter estimates. These steps complete the fitting of the salient crack point set. Furthermore, since the full crack skeleton is a one-pixel skeleton structure that only retains the crack centerline in the target crack image, the pixel length of the crack in the dam crack image can be directly calculated by summing the Euclidean distances between all adjacent pixels in the full crack skeleton. By integrating the crack pixel width and length, the crack size characteristics of the dam image information are obtained.

[0055] In one embodiment, converting dam infrared information into dam temperature information based on pre-acquired inspection equipment parameters, and extracting dam temperature characteristics of the dam inspection area based on the dam temperature information includes the following steps: Complete the radiometric brightness mapping of the inspection equipment parameters acquired in advance, and convert the dam infrared information into the dam radiometric brightness according to the radiometric brightness mapping results; The dam temperature information of the dam inspection area is inverted based on the dam radiation brightness and Planck's law; Calculate the regional temperature mean of the dam inspection area based on the dam temperature information, and extract the regional temperature extreme value of the dam inspection area; The regional temperature mean and regional temperature extreme values ​​are integrated to obtain the dam temperature characteristics of the dam inspection area.

[0056] In this embodiment, the inspection equipment parameter refers to the infrared camera grayscale value (DN value). A mapping relationship between the infrared camera grayscale value (DN value) and radiant brightness is then established. A blackbody furnace (e.g., a standard blackbody at 30°C, 50°C, or 70°C) can be used to calibrate the infrared camera on the inspection robot. The DN values ​​at different temperatures are recorded, and a linear relationship between the infrared camera grayscale value (DN value) and radiant brightness is fitted. Based on the fitted linear relationship, the dam infrared information is converted into dam radiant brightness. Then, based on the detection wavelength parameters specified in the infrared camera's technical documentation, such as 8-14μm, Planck's law is used to invert the dam temperature information in the inspection area. The mathematical expression of Planck's law is as follows:

[0057] in, is the detection wavelength parameter, is the dam radiance, and is the radiation constant.

[0058] The same method is used to obtain the dam temperature information of all inspection points in the dam inspection area. All inspection points are traversed, and the sum of the temperatures of all inspection points is calculated. The regional temperature mean of the dam inspection area is calculated based on the sum of the temperatures, and the maximum temperature and minimum temperature of all inspection points are selected as the regional temperature extremes of the dam inspection area. The regional temperature mean and the regional temperature extremes are integrated to obtain the dam temperature characteristics of the dam inspection area.

[0059] In one embodiment, extracting the acoustic wave time-frequency characteristics of the dam acoustic wave information based on Fourier transform includes the following steps: The wavelet transform algorithm is used to remove the noise signal of the dam acoustic wave information and obtain a noise-free wave signal; The noise-free wave signal is decomposed into multiple local sound wave signals using the sliding window method; For any local sound wave signal, complete the time domain analysis of all local sound wave signals to obtain the local time domain features of the local sound wave signal, the local time domain features including the local signal mean, local signal variance, and local signal peak value; The local sound wave signal is converted into a time-frequency signal by Fourier transform to obtain a local frequency domain signal, and the local frequency domain features of the local frequency domain signal are extracted. The local frequency domain features include local spectrum amplitude, local spectrum phase and local spectrum frequency; The local time domain features and local frequency domain features of all local acoustic wave signals are integrated to obtain the acoustic wave time-frequency features of the dam acoustic wave information.

[0060] In this embodiment, to address the non-stationary characteristics of dam acoustic signals (such as ultrasonic flaw detection echoes and underwater sonar reflections) in dam acoustic wave information, a wavelet basis with compact support and good time-frequency localization (such as DB4 and SYM8) is selected. For example, the high-frequency components of ultrasonic echo signals are primarily concentrated between 100kHz and 500kHz, and the DB6 wavelet basis effectively matches this frequency band. The wavelet basis is then used to perform a multi-layer wavelet decomposition on the dam acoustic signal, yielding low-frequency approximate components and high-frequency detail components at each layer. For example, a 1MHz sampled ultrasonic signal is decomposed into five layers, with the low-frequency component A5 corresponding to 0-31.25kHz and the high-frequency components D1-D5 corresponding to different frequency bands between 31.25kHz and 1MHz, respectively. A soft or hard thresholding function is applied to each layer of high-frequency detail components, either setting coefficients below the threshold to zero (hard thresholding) or shrinking them (soft thresholding), retaining only coefficients above the threshold. An inverse wavelet transform is performed on the processed high-frequency and low-frequency components to reconstruct a noise-free wave signal. This process effectively preserves the high-frequency characteristics of the defect echo (such as the high-frequency components of crack reflections) while suppressing white noise. Furthermore, the denoising effect can be evaluated by calculating the signal-to-noise ratio (SNR) and mean square error (MSE) of the dam acoustic wave signal before and after denoising. For example, the original dam acoustic wave signal had an SNR of 12dB. After denoising, the SNR increased to 28dB, and the MSE decreased from 0.8 to 0.15, indicating that the noise was significantly reduced while the signal characteristics were fully preserved.

[0061] Next, the window length L is set based on the required temporal resolution of the noise-free signal. The window overlap ratio can be set to 50%-75% of the window length to avoid edge effects. Furthermore, to reduce spectral leakage, the noise-free signal can be windowed using a Hanning or Hamming window. Starting from the starting point of the noise-free signal, a sliding window is applied with a step size S (S = L × (1-overlap ratio)). The noise-free signal within each window is windowed to obtain a localized acoustic wave signal. For example, a 100ms noise-free signal can be decomposed into approximately 196 localized acoustic wave signals using a window length of 1024 and a 50% overlap ratio.

[0062] For each local sound wave signal, its local signal mean, local signal variance, and local signal peak are calculated. The local time domain characteristics intuitively reflect the changing characteristics of the signal in the time dimension. Among them, the local signal mean μ reflects the overall level of the signal, and the calculation formula is , where N is the number of sampling points, is the value of the i-th sampling point. Local signal variance Describes the degree of fluctuation of the signal around the mean, and the calculation formula is The local signal peak value K describes the sharpness of the signal distribution, and the calculation formula is: , where E represents the mathematical expectation. These local time-domain characteristics reflect the overall shape and variation characteristics of the local acoustic wave signal, helping to identify cracks within the target dam. A fast Fourier transform (FFT) is then performed on each local acoustic wave signal to obtain a local frequency domain signal. The local spectral amplitude, local spectral phase, and local spectral frequency of the local frequency domain signal are then calculated. The formula for the fast Fourier transform (FFT) algorithm is as follows:

[0063] in, is the local frequency domain signal frequency components, It is the local frequency domain signal (time domain signal) A point in time, is a negative exponential function, representing sine and cosine waves of different frequencies. is the number of signal samples, that is, the number of local frequency domain signals. The local spectrum amplitude represents the amplitude of the signal at each frequency component, usually in absolute value. The local spectrum phase represents the phase information of the signal at each frequency component, describing the phase offset of the signal waveform at that frequency. The local spectrum frequency refers to the individual frequency components contained in the signal and is related to the number of sampling points and sampling frequency of the fast Fourier transform.

[0064] The acoustic time-frequency characteristics of the dam acoustic wave information are obtained by integrating local time-domain features and local frequency-domain features. This step fuses the time-domain and frequency-domain information to form a multidimensional feature vector, providing comprehensive data support for subsequent dam anomaly identification. During the integration process, the scale and importance of different features need to be considered. Feature normalization methods, such as z-score normalization, can be used to eliminate dimensional differences between different features. This comprehensive feature not only incorporates the statistical characteristics and frequency information of the signal but also preserves the potential correlation between the time-domain and frequency-domain features. Through this integration, the characteristics of the local acoustic wave signal can be more comprehensively described, improving the accuracy and reliability of dam anomaly identification and providing strong data support for subsequent analysis of the dam anomaly level.

[0065] In one embodiment, the multimodal inspection feature classification is completed by forward propagation of the gradient decision tree model, and the dam anomaly level of the dam inspection area is obtained, which includes the following steps: Build a sample data set based on historical inspection information and use it to train a gradient decision tree model; Normalize multimodal inspection features; The multimodal inspection features are input into the gradient decision tree model that has completed model training. The gradient decision tree model is used to perform node splitting judgment on the multimodal inspection features, and a predicted contribution value is generated based on the node splitting judgment result. The weighted summation of all prediction contribution values ​​is completed through multiple decision trees in the gradient decision tree model to obtain the abnormal level probability distribution of multimodal inspection features; The abnormality level probability distribution is mapped to the dam abnormality level in the dam inspection area through the gradient decision tree model.

[0066] In this embodiment, the previously labeled historical inspection information is used as a sample dataset. This sample dataset is then divided according to a specific ratio (e.g., 7:2:1) to form a training set, a validation set, and a test set. The basic parameters of the gradient decision tree model are then set, including the number of trees (e.g., 100-200) for subsequent tuning using the validation set, the maximum depth (e.g., 5-8 layers to prevent model overfitting), and the learning rate (e.g., 0.1-0.3) to control the contribution weight of each decision tree. The gradient decision tree model is then trained on the training set, using the multimodal inspection features of the training set as input and the historical anomaly level labels as output. Multiple decision trees are iteratively constructed using gradient boosting. Each decision tree is constructed based on the residual of the previous tree. The splitting criteria for each node are determined by minimizing a loss function (e.g., mean squared error, cross entropy, etc.). The features and splitting thresholds that result in the fastest reduction of the loss function are selected. During model training, the basic model parameters can be optimized using a grid search combined with cross-validation to prevent overfitting and improve the model's generalization capability. The multimodal inspection features are then normalized to eliminate the impact of different feature dimensions and value ranges, ensuring the stability and accuracy of model training. The minimum-maximum normalization method can be used to linearly transform the eigenvalues ​​corresponding to the multimodal inspection features to the interval [0, 1]. The normalized multimodal inspection features are then input into the trained gradient decision tree model, which then performs a node splitting decision based on the features of each sample. Each decision tree begins at the root node and, based on the current node's splitting criteria (e.g., whether the crack length exceeds a preset length threshold), assigns samples to the left or right subtree until a leaf node is reached. Each leaf node is assigned a predicted contribution value, which is calculated based on the anomaly level labels of the samples at that leaf node during training. For example, if the majority of samples at a leaf node are classified as severely anomaly, the predicted contribution value for that leaf node will be biased towards the probability of a severe anomaly. The weighted summation of all predicted contributions from multiple decision trees in the gradient decision tree model is then performed. The predicted contribution of each decision tree is weighted according to its weight in the model. The weight is typically correlated with the tree's training error, with smaller errors increasing the weight. This weighted summation yields a probability distribution of abnormality levels for the multimodal inspection features. For example, the probability of a level 1 abnormality is 0.2, the probability of a level 2 abnormality is 0.5, and the probability of a level 3 abnormality is 0.3. Level 1 abnormalities are the highest abnormality level, followed by level 2 abnormalities, and level 3 abnormalities are the lowest. This distribution reflects the likelihood of the multimodal inspection features belonging to different abnormality levels. Finally, the abnormality level probability distribution is mapped to the dam abnormality level for the dam inspection area. The abnormality level with the highest probability is generally selected as the final prediction. For example, if the probability of a level 2 abnormality is the highest, the dam abnormality level for that inspection area is determined to be level 2.The dam digital twin model is updated based on the final prediction results, that is, the dam anomaly level in each dam inspection area, and is visualized through the dam digital twin model to facilitate subsequent dam maintenance on the target dam by dam staff.

[0067] The present application also provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores instructions, which are used to enable the machine to execute the water conservancy inspection robot inspection method based on multimodal data fusion according to any of the above items.

[0068] The present application also provides a water conservancy inspection robot inspection system based on multimodal data fusion, which is characterized by including: a memory configured to store instructions; and The processor is configured to call instructions from the memory and implement the water conservancy inspection robot inspection method based on multimodal data fusion according to any one of the above when executing the instructions.

[0069] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0070] Among them, the memory can be an internal storage unit of a computer device, such as a hard disk or memory of a computer device, or an external storage device of a computer device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD) or flash memory card (FC) equipped on the computer device. In addition, the memory can also be a combination of an internal storage unit and an external storage device of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or is to be output. This application does not impose any restrictions on this.

[0071] An embodiment of the present application also provides a machine-readable storage medium, which stores instructions for enabling a machine to execute the above-mentioned water conservancy inspection robot inspection method based on multimodal data fusion.

[0072] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0077] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0078] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0080] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A water conservancy inspection robot inspection method and system method based on multimodal data fusion, characterized in that: The method comprises the following steps: Obtain historical inspection information of the target dam through a pre-built dam data twin model; Generating a water conservancy inspection strategy based on the historical inspection information and the dam data twin model; Controlling the inspection robot to go to each dam inspection area according to the water conservancy inspection route in the water conservancy inspection strategy to collect multimodal inspection information of the target dam; For any of the dam inspection areas, extracting multimodal inspection features of the dam inspection area based on the multimodal inspection information; The multimodal inspection features are graded by forward propagation of a gradient decision tree model to obtain the dam anomaly grade of the dam inspection area.

2. The method according to claim 1, characterized in that The historical inspection information includes historical inspection areas and historical abnormality levels. Generating a water conservancy inspection strategy based on the historical inspection information and the dam data twin model includes the following steps: Verify the location information of the preset dam inspection area and the historical inspection area, and allocate regional inspection time for all the dam inspection areas based on the inspection location verification result and the historical abnormality level; Locating the dam obstacle area of ​​the target dam based on the dam data twin model; Generate an initial inspection route based on the location information of all the inspection areas and using a shortest path algorithm; Using the dam obstacle area as a constraint condition, the initial inspection route is modified to obtain a water conservancy inspection route; The water conservancy inspection route and the regional inspection time are integrated to generate a water conservancy inspection strategy.

3. The method according to claim 1, characterized in that The multimodal inspection information includes dam image information, dam infrared information and dam sound wave information.

4. The method according to claim 3, characterized in that The step of extracting the multimodal inspection features of the dam inspection area based on the multimodal inspection information comprises the following steps: Iteratively refining the dam image information using the ZHANG iterative refinement algorithm, and extracting crack size features of the dam image information based on the iterative refinement result; converting the dam infrared information into dam temperature information according to pre-acquired inspection equipment parameters, and extracting dam temperature characteristics of the dam inspection area according to the dam temperature information; Extracting the acoustic wave time-frequency characteristics of the dam acoustic wave information based on Fourier transform; The crack size characteristics, the dam temperature characteristics and the acoustic wave time-frequency characteristics are integrated into multimodal inspection characteristics of the dam inspection area.

5. The method according to claim 4, characterized in that The iterative refinement of the dam image information using the ZHANG iterative refinement algorithm and the extraction of crack size features of the dam image information according to the iterative refinement result include the following steps: pre-processing the dam image information; detecting an edge point set of the pre-processed dam image information using an edge detection algorithm, and separating the dam crack image from the dam image background in the dam image information based on the edge point set; completing morphological optimization of the dam crack image through morphological opening operation to obtain a complete crack image; Traversing all connected domains in the complete crack image, calculating the total pixel values ​​of all connected domains, and eliminating connected domains whose total pixel values ​​are less than a preset total pixel value threshold to obtain a target crack image; Iteratively thinning the target crack image using the ZHANG iterative thinning algorithm to obtain a full crack skeleton and a partial crack skeleton of the target crack image; Extracting a crack salient point set of the target crack image by combining the full crack skeleton and the incomplete crack skeleton; Fitting the crack salient point set using the least square method, and calculating the crack pixel width of the dam crack image according to the fitting result; Calculating the sum of the Euclidean distances between all adjacent pixels in the full crack skeleton to obtain the crack pixel length of the dam crack image; The crack pixel width and the crack pixel length are integrated to obtain the crack size feature of the dam image information.

6. The method according to claim 4, characterized in that The converting of the dam infrared information into dam temperature information according to the inspection equipment parameters acquired in advance, and extracting the dam temperature characteristics of the dam inspection area according to the dam temperature information comprises the following steps: Complete the radiometric brightness mapping of the inspection equipment parameters acquired in advance, and convert the infrared information of the dam into the radiometric brightness of the dam according to the radiometric brightness mapping result; Inversely calculate the dam temperature information of the dam inspection area according to the dam radiation brightness and based on Planck's law; Calculating a regional temperature mean of the dam inspection area based on the dam temperature information, and extracting a regional temperature extreme value of the dam inspection area; The regional temperature mean and the regional temperature extreme are integrated to obtain the dam temperature characteristics of the dam inspection area.

7. The method according to claim 4, characterized in that The method of extracting the acoustic wave time-frequency characteristics of the dam acoustic wave information based on Fourier transform comprises the following steps: Using a wavelet transform algorithm to remove noise signals from the dam acoustic wave information to obtain a noise-free wave signal; Decomposing the noise-free wave signal into a plurality of local sound wave signals using a sliding window method; For any of the local sound wave signals, complete time domain analysis of all local sound wave signals to obtain local time domain features of the local sound wave signals, wherein the local time domain features include a local signal mean, a local signal variance, and a local signal peak value; Performing time-frequency conversion of the local sound wave signal by Fourier transform to obtain a local frequency domain signal, and extracting local frequency domain features of the local frequency domain signal, wherein the local frequency domain features include local spectrum amplitude, local spectrum phase and local spectrum frequency; The local time domain features and local frequency domain features of all the local acoustic wave signals are integrated to obtain the acoustic wave time-frequency features of the dam acoustic wave information.

8. The method according to claim 1, characterized in that The step of completing the level classification of the multimodal inspection features by forward propagation of the gradient decision tree model and obtaining the dam anomaly level of the dam inspection area includes the following steps: Constructing a sample data set based on historical inspection information, and using the sample data set to train a gradient decision tree model; Normalizing the multimodal inspection features; Inputting the multimodal inspection features into the gradient decision tree model that has completed model training, performing node splitting judgment on the multimodal inspection features through the gradient decision tree model, and generating a predicted contribution value based on the node splitting judgment result; A weighted summation process of all the predicted contribution values ​​is completed through multiple decision trees in the gradient decision tree model to obtain an abnormality level probability distribution of the multimodal inspection feature; The abnormality level probability distribution is mapped to the dam abnormality level of the dam inspection area through a gradient decision tree model.

9. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions for enabling a machine to execute the water conservancy inspection robot inspection method based on multimodal data fusion according to any one of claims 1 to 8.

10. A water conservancy inspection robot inspection system based on multimodal data fusion, characterized in that: include: a memory configured to store instructions; as well as The processor is configured to call the instructions from the memory and implement the water conservancy inspection robot inspection method based on multimodal data fusion according to any one of claims 1 to 8 when executing the instructions.

Images