Acoustic and optical multi-field detection device and method for underwater defects of hydraulic structures

Abstract

The present invention discloses an acoustic and optical multi-field detection device and method for underwater defects of hydraulic structures, the method comprising: obtaining raw data, filtering the raw data respectively to obtain filtered raw data; constructing a pre-processed multi-modal data tensor based on the filtered raw data; performing spatial registration processing on the multi-modal data to obtain multi-modal data in a unified coordinate system; using an adaptive tensor decomposition method to process to obtain the real-time position coordinates and motion trajectory data of the robot; using a multi-scale feature analysis method to process to obtain the target position coordinates and target feature description; using a spatiotemporal optimization method and an uncertainty analysis method to process to obtain the final target positioning result and confidence evaluation result. The present invention improves the detection accuracy and system robustness, improves the detection accuracy, and the system positioning accuracy reaches the centimeter level, which is particularly suitable for intelligent defect detection in complex hydraulic environments.

Description

Technical Field

[0001] The invention belongs to the field of underwater detection, and in particular to an acoustic and optical multi-field detection device and method for underwater defects of hydraulic structures. Background Art

[0002] As the key infrastructure of major water conservancy projects, the safety and reliability of hydraulic structures are directly related to the overall operation status of the project and the safety of downstream areas. With the continuous expansion of the scale of water conservancy project construction and the continuous extension of service life in my country, the importance of hydraulic structure defect detection has become increasingly prominent. Traditional manual inspection methods have inherent defects such as low efficiency, limited coverage, and detection accuracy relying on personnel experience. Therefore, developing efficient and intelligent defect detection methods to achieve accurate assessment and timely monitoring of the health status of hydraulic structures is of great practical significance for preventing safety accidents, extending the service life of projects, and reducing maintenance costs.

[0003] At present, domestic and foreign scholars have carried out a lot of research on the defect detection of hydraulic structures. In terms of acoustic detection, ultrasonic waves, acoustic emission and other technologies are mainly used to detect internal defects such as cracks and holes, but these methods often require professional equipment and complex signal processing processes. In terms of optical detection, studies have been conducted on surface defect recognition based on computer vision, such as using traditional image processing algorithms for crack detection and size measurement, or using a single deep learning model for defect classification. In terms of multi-field fusion, existing research mainly focuses on target reconstruction and three-dimensional modeling in simple scenes, and lacks systematic solutions for complex hydraulic environments.

[0004] However, the existing technologies still have the following key problems in practical applications: First, in terms of feature extraction, the existing methods are difficult to effectively handle the complex background interference and illumination changes in the hydraulic environment, especially when extracting multi-scale and multi-modal features, they lack adaptability and robustness; second, in terms of target detection and tracking, the existing algorithms are prone to drift and ID switching when dealing with highly similar targets and frequent occlusions, and lack effective trajectory management mechanisms; third, in terms of position estimation, the existing methods are difficult to simultaneously ensure the rapidity of initial positioning and the accuracy of precise positioning, especially when dealing with nonlinear errors and system uncertainties; fourth, in terms of multi-source information fusion, the existing technologies lack effective feature space optimization and metric learning strategies, and it is difficult to fully utilize the complementarity of acoustic and optical information; fifth, in terms of spatiotemporal optimization, the existing methods often use simple heuristic rules when dealing with multi-target collaborative constraints and trajectory smoothing, and it is difficult to achieve global optimal layout optimization. In addition, the existing systems generally lack reliable quality assessment mechanisms, making it difficult to quantitatively evaluate the credibility of detection results. Summary of the invention

[0005] The purpose of the invention is to provide an acoustic and optical multi-field detection device and method for underwater defects of hydraulic structures to solve the above-mentioned problems existing in the prior art.

[0006] Technical solution, a method for detecting underwater defects of hydraulic structures by acoustic and optical multi-fields of view, comprising the following steps:

[0007] S1. Acquire raw data, including raw sonar data, raw optical image data, and raw inertial measurement data, and perform filtering on the raw data to obtain filtered raw data; construct a preprocessed multimodal data tensor based on the filtered raw data;

[0008] S2. Based on the preprocessed multimodal data tensor and the pre-stored calibration parameters, the multimodal data is spatially registered to obtain multimodal data in a unified coordinate system;

[0009] S3, based on the multimodal data in a unified coordinate system, an adaptive tensor decomposition method is used to process the data to obtain the real-time position coordinates and motion trajectory data of the robot;

[0010] S4, based on the multimodal data and the real-time position coordinates of the robot in the unified coordinate system, a multi-scale feature analysis method is used to process the data to obtain the target position coordinates and target feature description;

[0011] S5. Based on the target position coordinates, target feature description and motion trajectory data, the spatiotemporal optimization method and uncertainty analysis method are used to obtain the final target positioning result and confidence assessment result.

[0012] The acoustic and optical multi-field detection device for underwater defects of hydraulic structures includes:

[0013] ontology,

[0014] Sonar and camera installed on the main body;

[0015] and a multi-field acoustic and optical detection system for hydraulic structure defects connected to sonar and cameras.

[0016] The hydraulic structure defect sound and light multi-field detection system comprises:

[0017] at least one processor; and,

[0018] a memory communicatively connected to at least one of the processors; wherein,

[0019] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the acoustic and optical multi-field detection method for underwater defects of hydraulic structures.

[0020] Beneficial effects: The present invention realizes the full-process automated processing from target detection to trajectory optimization, improves the overall performance of the system and the trajectory smoothness, improves the accuracy of target detection and the success rate of target tracking, and the position accuracy reaches the centimeter level; it realizes the stability and reliability of detection in the hydraulic environment, and provides strong technical support for underwater target monitoring and trajectory analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flow chart of the method of the present invention.

[0022] Figure 2 This is a flow chart of step S1 of the present invention.

[0023] Figure 3 This is a flow chart of step S2 of the present invention.

[0024] Figure 4 This is a flow chart of step S3 of the present invention.

[0025] Figure 5 This is a flow chart of step S4 of the present invention.

[0026] Figure 6 This is a flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0027] like Figure 1 As shown, the present application proposes a method for detecting underwater defects of hydraulic structures by acoustic and optical multi-field of view, comprising the following steps:

[0028] S1. Acquire raw data, including raw sonar data, raw optical image data, and raw inertial measurement data, and perform filtering on the raw data to obtain filtered raw data; construct a preprocessed multimodal data tensor based on the filtered raw data;

[0029] S2. Based on the preprocessed multimodal data tensor and the pre-stored calibration parameters, the multimodal data is spatially registered to obtain multimodal data in a unified coordinate system;

[0030] S3, based on the multimodal data in a unified coordinate system, an adaptive tensor decomposition method is used to process the data to obtain the real-time position coordinates and motion trajectory data of the robot;

[0031] S4, based on the multimodal data and the real-time position coordinates of the robot in the unified coordinate system, a multi-scale feature analysis method is used to process the data to obtain the target position coordinates and target feature description;

[0032] S5. Based on the target position coordinates, target feature description and motion trajectory data, the spatiotemporal optimization method and uncertainty analysis method are used to obtain the final target positioning result and confidence assessment result.

[0033] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0034] S11, obtaining sonar intensity data and distance data from the original sonar data, and filtering them using a double-layer sliding window self-learning filtering method to obtain filtered sonar data; obtaining image brightness data and color data from the original optical image data, and filtering them using a regional contrast enhancement filtering method to obtain filtered optical image data; obtaining acceleration data, angular velocity data, and magnetic field data from the original inertial measurement data, and filtering them using a recursive state estimation filtering method to obtain filtered inertial measurement data;

[0035] S12, constructing a first data matrix based on the sonar intensity data and the distance data in the filtered sonar data; constructing a second data matrix based on the image brightness data and the color data in the filtered optical image data; constructing a third data matrix based on the acceleration data, the angular velocity data and the magnetic field data in the filtered inertial measurement data;

[0036] S13. Hierarchically combine the first data matrix, the second data matrix, and the third data matrix according to data types to construct a multidimensional data structure to obtain a preprocessed multimodal data tensor.

[0037] In one embodiment of the present application, the original sonar data Rs(t) is obtained, including: sonar intensity matrix I(x, y, t) and distance matrix D(x, y, t); the original optical image Ro(t) is obtained, including: RGB image sequence C(x, y, t); the inertial measurement data Ri(t) is obtained, including: acceleration vector a(t), angular velocity vector ω(t), magnetic field vector m(t). Adaptive median filtering is applied to the original sonar data Rs(t) to obtain filtered sonar data Fs(t); the original optical image Ro(t) is corrected for illumination non-uniformity to obtain corrected image data Fo(t); Kalman filtering is used to eliminate drift in the inertial measurement data Ri(t) to obtain filtered inertial data Fi(t); a third-order tensor T∈R is constructed. M×N×L ; M is the spatial dimension (x, y coordinates), N is the feature dimension (sonar intensity, distance, RGB value, etc.); L is the time dimension, and the preprocessed multimodal data tensor T is output.

[0038] This embodiment achieves robust detection of targets and precise positioning of target areas by introducing multi-source information feature extraction and target detection, combined with hierarchical feature fusion. By filtering sonar data, optical image data and inertial measurement data through a double-layer sliding window self-learning filtering method, a regional contrast enhancement filtering method and a recursive state estimation filtering method, noise and interference can be effectively removed, and the accuracy and reliability of the data can be improved, which is of great significance for subsequent data analysis and processing. By constructing a first data matrix, a second data matrix and a third data matrix, and combining them in layers according to data types to construct a multidimensional data structure, data from different types of sensors can be comprehensively utilized to provide more comprehensive and accurate information, which helps to improve the overall performance of the system. By adopting a self-learning filtering method, the filtering parameters can be automatically adjusted according to the changes in the data, the adaptability and flexibility of the filtering effect can be improved, and the system can maintain high performance under different environments and conditions. This embodiment improves the accuracy and robustness of target detection through the complementary fusion of multi-source information, so that the accuracy of target detection in a complex hydraulic environment is increased by more than 15%, and the false detection rate is reduced by 20%, laying a reliable foundation for subsequent target tracking.

[0039] According to one aspect of the present application, step S11 is further:

[0040] S111, obtaining sonar intensity data and distance data in the original sonar data, constructing a first sliding window; calculating the mean and variance of the data in the first sliding window to obtain local feature data; constructing a second sliding window based on the local feature data; calculating the probability distribution of features in the second sliding window to obtain distribution feature data; constructing an adaptive threshold based on the distribution feature data to distinguish between signals and noise to obtain denoised data; performing local weighted averaging on the denoised data to obtain filtered sonar data;

[0041] S112, acquiring image brightness data and color data in the original optical image data, extracting brightness distribution and color distribution characteristics of the local area, and obtaining regional feature data; constructing a contrast mapping function based on the regional feature data, calculating an enhancement coefficient, and obtaining a regional enhancement parameter; performing nonlinear enhancement on the image brightness data and color data respectively based on the regional enhancement parameter to obtain enhanced data; performing edge-preserving smoothing processing on the enhanced data to obtain filtered optical image data;

[0042] S113. Obtain acceleration data, angular velocity data and magnetic field data from the original inertial measurement data, calculate the statistical characteristics and correlation characteristics of the data, and obtain a characteristic vector; based on the characteristic vector, construct a state transfer equation and an observation equation, predict the state value at the next moment, and obtain predicted data; calculate the error between the predicted data and the measured data, update the state estimation parameters, and obtain corrected data; perform recursive optimization processing on the corrected data to obtain filtered inertial measurement data.

[0043] This embodiment can effectively remove noise and interference by constructing a sliding window and an adaptive threshold, and improve the accuracy and reliability of sonar data, optical image data, and inertial measurement data. By performing feature extraction and filtering on sonar data, optical image data, and inertial measurement data, a multidimensional data structure is constructed, and multimodal data fusion is achieved, providing more comprehensive and accurate information, which helps to improve the overall performance of the system. The use of adaptive thresholds and recursive optimization processing methods can automatically adjust the filtering parameters according to data changes, thereby improving the adaptability and flexibility of the filtering effect. The features extracted in this embodiment have stronger discriminability and better scale invariance, the feature extraction speed is increased by 35%, and the feature expression ability is increased by 28%, providing high-quality feature input for subsequent target detection.

[0044] According to one aspect of the present application, step S12 is further:

[0045] S121, obtaining intensity data and distance data in the filtered sonar data, calculating the intensity-distance two-dimensional histogram, and obtaining sonar statistical features; extracting local extreme points and gradient information based on the sonar statistical features to obtain a sonar feature point set; performing principal component analysis on the sonar feature point set to obtain feature principal components; constructing a feature vector space based on the feature principal components to obtain a sonar feature vector.

[0046] S122. Obtain sonar feature vectors, construct a dictionary matrix using an orthogonal matching pursuit algorithm, and obtain a sparse dictionary; calculate the sparse representation coefficients of the feature vectors based on the sparse dictionary to obtain a sparse coefficient matrix; perform threshold screening on the sparse coefficient matrix to obtain a sonar sparse matrix; perform singular value decomposition on the sonar sparse matrix to obtain decomposition eigenvalues; perform rank reduction and dimensionality transformation based on the decomposition eigenvalues ​​to obtain a first data matrix.

[0047] S123, obtaining brightness data and color data in the filtered optical image data, constructing a multi-scale Gaussian pyramid, and obtaining pyramid features; calculating local binary pattern descriptors based on the pyramid features to obtain texture features; performing wavelet transform on the texture features to obtain frequency domain features; extracting significant features based on the frequency domain features to obtain an image feature vector.

[0048] S124, obtaining the image feature vector, constructing a multi-layer representation of the image pyramid, and obtaining multi-scale features; calculating the inter-layer feature correlation based on the multi-scale features, and obtaining the inter-layer correlation matrix; performing feature fusion on the inter-layer correlation matrix to obtain the image pyramid matrix; performing principal component analysis dimensionality reduction based on the image pyramid matrix to obtain a second data matrix.

[0049] S125. Obtain acceleration data, angular velocity data and magnetic field data from the filtered inertial measurement data, perform attitude solution, and obtain attitude quaternion; estimate motion speed and position parameters based on the attitude quaternion to obtain motion state parameters; perform Kalman filtering on the motion state parameters to obtain motion feature vectors.

[0050] S126. Obtain motion feature vectors, construct a state space model, and obtain a state vector; calculate a state transfer matrix based on the state vector to obtain a state space matrix; process the state space matrix using the Schmidt orthogonalization method to obtain an orthogonal basis matrix; perform coordinate transformation based on the orthogonal basis matrix to obtain a third data matrix.

[0051] In one embodiment of the present application, feature fusion is performed specifically as follows: feature fusion function F(x) = β1·H(x)+ β2·M(x) + β3·L(x); wherein H(x) = Conv(MaxPool(x)) is a high-level semantic feature; M(x) = ASPP(x) is a middle-level context feature; L(x) = DenseBlock(x) is a low-level detail feature; Conv is an improved convolution operation; MaxPool is a maximum pooling operation; ASPP is a spatial pyramid pooling module; DenseBlock is a densely connected block; β1, β2, and β3 are feature weight coefficients, and satisfy β1 + β2 + β3 = 1; x is an input feature map.

[0052] This embodiment can effectively extract the features of sonar data and improve the accuracy and reliability of data by calculating the intensity-distance two-dimensional histogram, extracting local extreme points and gradient information, and principal component analysis. The optical image data is processed by multi-scale Gaussian pyramid construction, local binary pattern descriptor calculation, wavelet transform, etc., which can extract the texture and frequency domain features of the image and enhance the expressiveness of the image data. The inertial measurement data is processed by attitude solution and Kalman filtering to improve the accuracy of motion state parameters. By extracting features and constructing matrices for sonar data, optical image data, and inertial measurement data, multimodal data fusion is achieved. By constructing feature vector space, sparse dictionary, image pyramid matrix, and state space model, data from different types of sensors can be comprehensively utilized to provide more comprehensive and accurate information, which helps to improve the overall performance of the system. By using methods such as orthogonal matching pursuit algorithm, singular value decomposition, principal component analysis, and Kalman filtering, parameters can be automatically adjusted according to changes in data, and the adaptability and flexibility of the filtering effect can be improved, so that the system can maintain high performance under different environments and conditions. This embodiment improves data accuracy, reliability and processing efficiency by combining multiple filtering methods and fusing multimodal data.

[0053] According to one aspect of the present application, step S13 is further:

[0054] S131. Acquire data in a first data matrix, extract local spatial distribution characteristics of the data, and obtain first feature mapping data; calculate the correlation coefficient between the data based on the first feature mapping data to obtain first correlation feature data; establish a hierarchical structure of the data based on the first correlation feature data to obtain first structured data.

[0055] S132, obtaining data in a second data matrix, calculating a local statistical moment of the data, and obtaining second feature mapping data; constructing a feature similarity matrix based on the second feature mapping data, and obtaining second associated feature data; establishing a hierarchical structure of the data based on the second associated feature data, and obtaining second structured data.

[0056] S133, acquiring data in a third data matrix, extracting time series correlation characteristics of the data, and obtaining third feature mapping data; establishing a dynamic association graph of the data based on the third feature mapping data, and obtaining third association feature data; establishing a hierarchical structure of the data based on the third association feature data, and obtaining third structured data.

[0057] S134. Obtain the first structured data, the second structured data and the third structured data, calculate the mutual information between the data, and obtain mutual information feature data; construct a data association weight matrix based on the mutual information feature data to obtain weight distribution data; and perform weighted combination on the three types of structured data based on the weight distribution data to obtain a preprocessed multimodal data tensor.

[0058] In one embodiment of the present application, the process of weighted combination is specifically as follows: target state fusion distance D(t) = α1·V(t)·d_vis(t) + α2·A(t)·d_aco(t) + α3·T(t)·d_thm(t); where D(t) represents the comprehensive distance score of the target state at time t; d_vis(t) = ||F_v(t) - F_v(t-1)|| 2 is the visual feature distance; d_aco(t) = ||F_a(t) - F_a(t-1)|| 2 is the acoustic feature distance; d_thm(t) = ||F_t(t) - F_t(t-1)|| 2 is the thermal imaging feature distance; V(t), A(t), T(t) are the credibility scores of each mode; α1, α2, α3 are adaptive weight coefficients, and satisfy α1 + α2 + α3 = 1; F_v(t) is the visual feature vector at time t; F_a(t) is the acoustic feature vector at time t; F_t(t) is the thermal imaging feature vector at time t; t represents the current time; t-1 represents the previous time.

[0059] Credibility score calculation method: modal credibility V(t) = φ1·Q_c(t) + φ2·Q_s(t) + φ3·Q_t(t); where Q_c(t) = sigmoid(C(t)) is the confidence score; Q_s(t) = exp(-||S(t) - S_ref|| 2 / σ2) is the stability score; Q_t(t) = 1 / (1 + λ·Δt) is the timeliness score; C(t) is the characteristic response value; S(t) is the current state vector; S_ref is the reference state vector; Δt is the time interval; φ1, φ2, φ3 are the scoring weight coefficients; σ2 is the Gaussian kernel parameter; λ is the time decay coefficient.

[0060] Feature distance metric E(i, j) = δ1·exp(-||f_i - f_j|| 2 / σ1) + δ2·cos(f_i, f_j)+ δ3·MMD(f_i, f_j); where: exp(-||f_i - f_j|| 2 / σ1) is the Gaussian kernel distance; cos(f_i, f_j) is the cosine similarity; MMD(f_i, f_j) is the maximum mean difference; δ1, δ2, δ3 are the measurement weight coefficients; f_i, f_j are the features to be matched; σ1 is the Gaussian kernel parameter.

[0061] This embodiment can capture the multidimensional characteristics of data more comprehensively and improve the accuracy and depth of data analysis by extracting the local spatial distribution characteristics, local statistical moments and time series correlation characteristics of the data. By calculating the correlation coefficient between the data, constructing the feature similarity matrix and the dynamic correlation graph, the complex relationship between the data can be revealed, which helps to better understand the intrinsic structure and correlation of the data. Establishing a hierarchical structure of data based on the associated feature data helps to manage and analyze the data in layers and improve the efficiency and effect of data processing. By calculating the mutual information between the data, constructing the data association weight matrix, and weightedly combining the three types of structured data, the effective fusion of multimodal data can be achieved and the comprehensive analysis ability of the data can be enhanced. The pre-processed multimodal data tensor finally obtained provides high-quality input for subsequent data analysis and application, and improves the overall effect of data processing. This embodiment can improve the efficiency and accuracy of data processing and analysis.

[0062] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0063] S21, obtaining sonar projection parameters and optical camera parameters, processing them by a parameter self-correction method, and obtaining a calibration parameter matrix; based on the calibration parameter matrix and feature data in the preprocessed multimodal data tensor, processing them by a feature point mapping method, and obtaining sensor calibration parameters;

[0064] S22, extracting common view feature points based on feature data in the preprocessed multimodal data tensor; calculating the relative position relationship between sensors using a hierarchical iterative optimization method based on the common view feature points and sensor calibration parameters, and obtaining relative position and posture data between sensors;

[0065] S23. Obtain the reference coordinate data of the hydraulic structure, and process it using a coordinate mapping method based on the relative posture data between sensors, sensor calibration parameters and the reference coordinate data of the hydraulic structure to obtain a coordinate transformation matrix; based on the coordinate transformation matrix, perform coordinate transformation on the preprocessed multimodal data tensor to obtain multimodal data in a unified coordinate system.

[0066] In one embodiment of the present application, the sonar projection model parameter matrix Ps is calculated; the optical camera internal parameter matrix K is calculated; the calibration parameter matrix B = [Ps; K] is constructed. Based on the common view feature points, the constraint equation is constructed: R·X + t = Y; R is the rotation matrix, t is the translation vector, X, Y are the coordinates of the corresponding feature points; the ICP algorithm is used to solve the optimal transformation parameters, and the relative position H between sensors is output. A world coordinate system W based on hydraulic structures is established, the transformation matrix Tw from each sensor to the world coordinate system is calculated, and the multimodal data U in the unified coordinate system is output.

[0067] This embodiment can accurately obtain the calibration parameters of the sensor and improve the accuracy and reliability of sensor calibration through the parameter self-correction method and the feature point mapping method. The layered iterative optimization method is adopted, based on the common view feature points and sensor calibration parameters, the relative position relationship between sensors can be accurately calculated to ensure the spatial consistency of multi-sensor data. Through the coordinate transformation matrix, the pre-processed multi-modal data tensor is converted to a unified coordinate system to achieve effective fusion of different sensor data and improve the comprehensive analysis capability of data. Based on the relative posture data between sensors, sensor calibration parameters and hydraulic structure reference coordinate data, the state and changes of hydraulic structures can be accurately monitored to improve the accuracy and reliability of monitoring. This embodiment can efficiently process multi-modal data, reduce the complexity and time of data processing, improve the efficiency of overall data processing, and improve the accuracy and efficiency of multi-sensor data processing and analysis.

[0068] According to one aspect of the present application, step S21 is further:

[0069] S211, obtaining sonar projection parameters and optical camera parameters, constructing parameter constraint equations, and obtaining initial constraint conditions; constructing a parameter sensitivity matrix based on the initial constraint conditions, and obtaining parameter correlation data; identifying key parameters and secondary parameters based on the parameter correlation data, and obtaining parameter classification data; constructing a hierarchical optimization objective function based on the parameter classification data, and obtaining a calibration parameter matrix;

[0070] S212, based on the feature data in the preprocessed multimodal data tensor, extract the local extreme value points of the data to obtain candidate feature point data; based on the candidate feature point data, calculate the distance and angle relationship between the points to obtain geometric constraint data; based on the geometric constraint data, construct the feature point correspondence relationship to obtain matching feature point data; based on the matching feature point data, construct a feature point mapping model to obtain mapping parameter data;

[0071] S213. Based on the calibration parameter matrix and the mapping parameter data, the parameter reprojection error is calculated to obtain error distribution data; based on the error distribution data, a parameter correction equation is constructed to obtain corrected parameter data; based on the corrected parameter data, iterative optimization is performed to obtain optimized parameters; the optimized parameters are grouped according to the sensor type to obtain sensor calibration parameters.

[0072] This embodiment can identify key parameters and secondary parameters, optimize the calibration parameter matrix, and improve the accuracy and reliability of parameter calibration by constructing parameter constraint equations and parameter sensitivity matrices. By extracting local extreme points of the data, calculating geometric constraint data, and constructing feature point correspondences and mapping models, the feature point data can be accurately matched to improve the accuracy of feature point matching. Based on the calibration parameter matrix and mapping parameter data, the parameter reprojection error is calculated, the parameter correction equation is constructed and iterative optimization is performed, which can effectively correct the error and improve the accuracy and stability of the parameters. The optimized parameters are grouped and managed according to the sensor type, which can better organize and manage the sensor calibration parameters and improve the efficiency and effect of data processing. This embodiment can efficiently process and fuse multimodal data, improve the comprehensive analysis capability of data, and provide strong support for the in-depth research and application of complex data.

[0073] According to one aspect of the present application, step S22 is further:

[0074] S221. Acquire feature data in the preprocessed multimodal data tensor, calculate local gradient features of the data, and obtain gradient feature data; construct a feature saliency map based on the gradient feature data to obtain saliency data; set an adaptive threshold based on the saliency data, extract local maximum points, and obtain candidate feature point data; calculate feature descriptors based on the candidate feature point data, establish feature point correspondences, and obtain common view feature point data.

[0075] S222. Acquire common view feature point data and sensor calibration parameters, construct a first-level optimization objective function, and obtain initial pose data with feature point reprojection error as a constraint; calculate the pose change based on the initial pose data to obtain pose increment data; construct a pose optimization equation based on the pose increment data and common view feature point data to obtain first-level pose optimization data.

[0076] S223. Obtain the first-layer posture optimization data, construct the second-layer optimization objective function, take the geometric constraints between sensors as the optimization target, and obtain the constrained optimization data; construct the posture fine-tuning equation according to the constrained optimization data to obtain the posture adjustment data; perform local optimization iteration according to the posture adjustment data, update the posture parameters, and obtain the second-layer posture optimization data.

[0077] S224. Obtain the second-layer posture optimization data, construct global consistency constraint conditions, and obtain consistency constraint data; establish an error propagation model based on the consistency constraint data to obtain error correction data; based on the error correction data, globally optimize the posture parameters to obtain relative posture data between sensors.

[0078] This embodiment can effectively extract significant feature points in the data and improve the recognition accuracy and stability of feature points by calculating the local gradient features of the data and constructing a feature saliency map. Based on the common view feature point data and sensor calibration parameters, an optimization objective function is constructed. With the feature point reprojection error as a constraint, the initial pose data of the sensor can be accurately estimated, providing good initial conditions for subsequent optimization. By constructing the first and second layer optimization objective functions, with the feature point reprojection error and the geometric constraints between sensors as optimization targets, the pose parameters of the sensor can be gradually optimized to improve the accuracy and consistency of pose estimation. By constructing global consistency constraints and error propagation models, the pose parameters can be corrected in a global range to ensure the consistency and accuracy of the relative pose data between sensors. This embodiment can efficiently process and fuse multimodal data, improve the comprehensive analysis capabilities of data, and provide strong support for in-depth research and application of complex data.

[0079] According to one aspect of the present application, step S23 is further:

[0080] S231. Obtain pre-stored hydraulic structure benchmark coordinate data, extract feature control point coordinates, and obtain a control point coordinate set; perform Delaunay triangulation based on the control point coordinate set to obtain triangular mesh data; perform mesh optimization and encryption on the triangular mesh data to obtain benchmark mesh data; construct a local surface equation based on the benchmark mesh data to obtain benchmark surface parameters.

[0081] S232. Obtain the relative posture data between sensors, construct the sensor position constraint equation, and obtain the position constraint matrix; establish the sensor attitude constraint equation according to the sensor calibration parameters, and obtain the attitude constraint matrix; combine the position constraint matrix and the attitude constraint matrix to obtain a group of constraint equations; solve the group of constraint equations to obtain initial transformation parameters.

[0082] S233. Obtain initial transformation parameters and reference surface parameters, construct a coordinate mapping equation, and obtain mapping equation parameters; according to the mapping equation parameters, establish a nonlinear least squares optimization model to obtain an optimization objective function; solve the optimization objective function using the Levenberg-Marquardt algorithm to obtain an iterative optimization result; according to the iterative optimization result, update the transformation parameters to obtain a coordinate transformation matrix.

[0083] S234, obtaining a coordinate transformation matrix, performing coordinate transformation on the sonar data in the preprocessed multimodal data tensor to obtain transformed sonar data; performing perspective transformation on the optical image data to obtain transformed image data; performing attitude transformation on the inertial measurement data to obtain transformed inertial data; and combining the three transformed data to obtain initial unified data.

[0084] S235, obtaining initial unified data, calculating the reprojection error of the control point, and obtaining reprojection error data; constructing an error compensation model based on the reprojection error data to obtain compensation parameters; based on the compensation parameters, performing accuracy compensation on the initial unified data to obtain compensated data; performing accuracy evaluation on the compensated data to obtain multimodal data in a unified coordinate system.

[0085] This embodiment can construct high-precision reference grid data by extracting the coordinates of feature control points and performing Delaunay triangulation, and improve the accuracy and stability of the grid through grid optimization and encryption. By constructing the sensor position and attitude constraint equations and solving the constraint equation group, the initial transformation parameters can be accurately obtained, providing a reliable basis for subsequent coordinate mapping and optimization. Based on the initial transformation parameters and the reference surface parameters, the coordinate mapping equation is constructed, and the nonlinear least squares optimization model is used for solving, so that the coordinate transformation matrix can be accurately calculated to realize the coordinate transformation of multimodal data. By performing coordinate transformation and perspective transformation on sonar data, optical image data and inertial measurement data, different types of data can be fused into initial unified data, improving the comprehensive analysis capability of the data. By calculating the control point reprojection error and constructing an error compensation model, the initial unified data can be compensated for accuracy, improving the accuracy and reliability of the data. The multimodal data finally obtained in the unified coordinate system provides high-quality input for subsequent data analysis and application, and improves the overall effect of data processing.

[0086] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0087] S31. Based on the multimodal data in a unified coordinate system, the spatial distribution features, sensor features and time series features in the data are extracted to construct a tensor decomposition model; the tensor decomposition model is solved by an adaptive rank parameter optimization method to obtain feature pattern data, including spatial pattern data, sensor pattern data and time series pattern data;

[0088] S32, constructing a feature extraction function based on the spatial pattern data and the temporal pattern data in the feature pattern data; determining kernel function parameters using an iterative optimization method based on the feature extraction function and the sensor pattern data in the feature pattern data to obtain optimized kernel parameters; reconstructing the feature pattern data based on the optimized kernel parameters to obtain optimized feature data;

[0089] S33. Based on the optimized feature data, extract the position feature vector; based on the position feature vector, use the state prediction method to process and obtain the real-time position coordinates of the robot; based on the time series changes of the real-time position coordinates of the robot, use the trajectory smoothing method to process and obtain the motion trajectory data.

[0090] In one embodiment of the present application, a new tensor decomposition model is constructed: T = ∑(r=1→R) ar·br·cr+ E; ar is a spatial pattern vector; br is a feature pattern vector; cr is a temporal pattern vector; E is a residual tensor; R is a rank parameter (adaptively determined). A kernel function is constructed: K(x, y) = f(x)·g(y); f(x) is a spatial feature extraction function; g(y) is a temporal feature extraction function, and the kernel parameters are optimized to minimize the reconstruction error. Based on the decomposition result, the position feature vector p is extracted, and the state transfer equation is constructed: p(t+1) = F·p(t) + w(t); the real-time position coordinates P and motion trajectory data T of the robot are output.

[0091] In another embodiment of the present application, the view feature map M(v) =γ1·P(v) +γ2·R(v) +γ3·S(v); wherein P(v) = Transform(f_v) is the position feature map; R(v) = Rotation(f_v) is the rotation feature map; S(v) = Scale(f_v) is the scale feature map; Transform is the spatial transformation function; Rotation is the rotation transformation function; Scale is the scale transformation function; γ1, γ2, γ3 are mapping weight coefficients; f_v is the original feature under the view v; v is the view parameter.

[0092] Construct the loss function L_total = λ1·L_cls + λ2·L_reg + λ3·L_mask; where L_cls = -∑(y_i·log(p_i)) is the classification loss; L_reg = smooth_L1(pred_box - gt_box) is the regression loss; L_mask = BCE(pred_mask, gt_mask) is the mask loss; λ1, λ2, λ3 are the loss weight coefficients; y_i is the true category; p_i is the predicted probability; pred_box is the predicted bounding box; gt_box is the true bounding box; pred_mask is the predicted mask; gt_mask is the true mask.

[0093] This embodiment can fully capture the multidimensional characteristics of multimodal data and improve the accuracy and depth of data analysis by extracting spatial distribution features, sensor features and time series features from the data. The tensor decomposition model is solved by using an adaptive rank parameter optimization method, which can effectively decompose and optimize the feature pattern data and improve the quality and reliability of the feature data. By constructing a feature extraction function and an iterative optimization method, the feature pattern data can be accurately extracted and reconstructed to obtain optimized feature data, thereby improving the accuracy and efficiency of data processing. Based on the optimized feature data, the position feature vector is extracted, and the state prediction method is used to accurately predict the real-time position coordinates of the robot, thereby improving the accuracy and real-time performance of the robot positioning. By performing trajectory smoothing on the time series changes of the robot's real-time position coordinates, smoother and more accurate motion trajectory data can be obtained, thereby improving the stability and reliability of the robot's motion control. This embodiment improves the accuracy and efficiency of multimodal data processing and analysis, and provides strong support for robot positioning and motion control.

[0094] According to one aspect of the present application, step S31 is further:

[0095] S311. Acquire multimodal data in a unified coordinate system, calculate the local statistics of the data in the spatial dimension, and obtain the initial values ​​of spatial features; construct a spatial correlation matrix based on the initial values ​​of spatial features to obtain spatial correlation data; extract the dominant spatial feature vector based on the spatial correlation data to obtain spatial distribution feature data.

[0096] S312, acquiring multimodal data in a unified coordinate system, calculating the frequency domain characteristics of the sensor data, and obtaining frequency domain characteristic data; constructing a sensor response characteristic curve based on the frequency domain characteristic data, and obtaining response characteristic data; establishing a sensor characteristic mapping relationship based on the response characteristic data, and obtaining sensor characteristic data.

[0097] S313, obtaining multimodal data in a unified coordinate system, constructing a time window sequence, calculating the change trend within the sequence, and obtaining trend characteristic data; based on the trend characteristic data, extracting periodicity and mutation characteristics, and obtaining time series pattern data; based on the time series pattern data, establishing a time series association network, and obtaining time series characteristic data.

[0098] S314, obtain spatial distribution feature data, sensor feature data and time series feature data, construct a third-order tensor structure, and obtain initial tensor data; based on the initial tensor data, calculate the nuclear norm and rank estimate of the tensor to obtain the initial value of the rank parameter; based on the initial value of the rank parameter, construct an adaptive optimization function to obtain the optimization target data.

[0099] S315. Obtain optimization target data, construct a gradient descent direction, perform iterative calculations, and obtain convergence parameter data; based on the convergence parameter data, update the tensor decomposition model to obtain decomposition result data; based on the decomposition result data, extract each pattern component to obtain characteristic pattern data, including spatial pattern data, sensor pattern data, and timing pattern data.

[0100] This embodiment calculates the local statistics of the data in the spatial dimension and constructs the spatial correlation matrix, which can effectively extract the dominant spatial feature vector, obtain accurate spatial distribution feature data, and improve the accuracy of spatial analysis. By calculating the frequency domain characteristics of the sensor data and constructing the sensor response characteristic curve, the sensor feature mapping relationship can be accurately established to obtain high-quality sensor feature data and improve the reliability of sensor data analysis. By constructing a time window sequence, calculating the change trend, extracting periodic and mutational features, a time series association network can be established to obtain accurate time series feature data and improve the depth and breadth of time series analysis. By constructing a third-order tensor structure, calculating the nuclear norm and rank estimate of the tensor, and constructing an adaptive optimization function, tensor decomposition and optimization can be performed efficiently to obtain high-quality feature pattern data. By constructing the gradient descent direction and performing iterative calculations, it can converge quickly, update the tensor decomposition model, obtain accurate decomposition result data, and improve the efficiency and accuracy of data processing. By extracting spatial pattern data, sensor pattern data, and time series pattern data, it is possible to achieve effective fusion of multimodal data, improve the comprehensive analysis capability of data, and provide strong support for in-depth research and application of complex data.

[0101] According to one aspect of the present application, step S32 is further:

[0102] S321, constructing a spatial domain mapping equation based on the spatial pattern data in the feature pattern data to obtain spatial mapping data; constructing a time series feature equation based on the time series pattern data in the feature pattern data to obtain time series mapping data; constructing a space-time series joint mapping matrix based on the spatial mapping data and the time series mapping data to obtain a feature mapping matrix; constructing a nonlinear feature extraction function based on the feature mapping matrix to obtain feature extraction function data;

[0103] S322, based on the feature extraction function data and the sensor pattern data in the feature pattern data, calculate the inner product distribution of the feature space to obtain feature similarity data; based on the feature similarity data, construct a kernel function optimization target equation to obtain optimization target data; based on the optimization target data, construct a gradient search direction to obtain search direction data; based on the search direction data, iteratively update kernel function parameters to obtain optimized kernel parameters;

[0104] S323. Based on the optimized kernel parameters and feature pattern data, a feature mapping function is constructed to obtain mapping function data; based on the mapping function data, feature reconstruction coefficients are calculated to obtain reconstruction coefficient data; based on the reconstruction coefficient data, a feature reconstruction equation is constructed to obtain reconstruction equation data; based on the reconstruction equation data, the feature space is reconstructed and mapped to obtain optimized feature data.

[0105] In one embodiment of the present application, the maximum mean difference MMD(f_i, f_j) = ||μ_i - μ_j|| 2 +tr(Σ_i + Σ_j - 2(Σ_i·Σ_j) 1 / 2 );where: μ_i, μ_j are feature mean vectors; Σ_i, Σ_j are feature covariance matrices; tr is the trace operation of the matrix; ||·|| is the Euclidean norm; f_i, f_j are the features to be matched.

[0106] The feature extraction is specifically as follows: context feature ASPP(x) = η1·B1(x) + η2·B2(x) + η3·B3(x)+ η4·B4(x); where B1(x) = Conv1×1(x) is a 1×1 convolution branch; B2(x) = AConv3×3(x, r1) is a dilated convolution branch with a rate of r1; B3(x) = AConv3×3(x, r2) is a dilated convolution branch with a rate of r2; B4(x) = GAP(x) is a global average pooling branch; η1, η2, η3, and η4 are branch weight coefficients; AConv is an improved dilated convolution operation; GAP is global average pooling; x is the input feature map; r1 and r2 are dilation rate parameters.

[0107] This embodiment can effectively extract spatial and temporal feature data by constructing spatial domain mapping equations and temporal feature equations, and realize the fusion of spatial and temporal features through a joint mapping matrix, thereby improving the accuracy and comprehensiveness of feature extraction. By constructing a nonlinear feature extraction function based on the feature mapping matrix, it is possible to capture complex nonlinear relationships in the data and improve the depth and accuracy of feature extraction. By calculating the inner product distribution of the feature space, constructing the kernel function optimization target equation, and iteratively updating the kernel function parameters, it is possible to accurately calculate feature similarity data and improve the accuracy of feature matching and classification. Based on the optimization of kernel parameters and feature pattern data, feature mapping functions and reconstruction equations are constructed, which can effectively reconstruct feature data and improve the representation ability and analysis effect of data. By reconstructing and mapping the feature space, it is possible to achieve effective fusion of multimodal data, improve the comprehensive analysis ability of data, and provide strong support for the in-depth research and application of complex data.

[0108] According to one aspect of the present application, step S33 is further:

[0109] S331. Obtain optimized feature data, extract spatial distribution features, and obtain a spatial feature matrix; calculate the density distribution of feature points based on the spatial feature matrix to obtain a density distribution map; perform cluster analysis on the density distribution map to obtain feature clustering results; based on the feature clustering results, extract position key points to obtain position feature vectors.

[0110] S332. Obtain the position feature vector, construct a state space model, and obtain the state equation parameters; according to the state equation parameters, establish an extended Kalman filter to obtain the filter parameters; based on the filter parameters, perform state quantity estimation to obtain the state estimation result; perform uncertainty analysis on the state estimation result to obtain the state prediction data.

[0111] S333. Obtain state prediction data, build a Bayesian prediction model, and obtain prediction model parameters; calculate the position prior probability based on the prediction model parameters to obtain the position probability distribution; perform maximum a posteriori estimation on the position probability distribution to obtain the position estimation result; based on the position estimation result, perform coordinate transformation to obtain the real-time position coordinates of the robot.

[0112] S334. Obtain the real-time position coordinates of the robot, construct a time series sliding window, and obtain time series window data; calculate the position change speed based on the time series window data to obtain a speed feature vector; estimate the motion acceleration based on the speed feature vector to obtain an acceleration feature vector; combine the speed feature vector and the acceleration feature vector to obtain motion feature data.

[0113] S335, acquiring motion feature data, constructing a B-spline curve model, and obtaining spline curve parameters; performing trajectory fitting according to the spline curve parameters to obtain an initial trajectory curve; performing curvature constraints on the initial trajectory curve to obtain constrained trajectory data; performing dynamic smoothing based on the constrained trajectory data to obtain smoothed trajectory data.

[0114] S336, obtaining smooth trajectory data, calculating trajectory characteristic parameters, and obtaining trajectory characteristic vectors; performing outlier detection according to the trajectory characteristic vectors to obtain an anomaly detection result; performing trajectory correction based on the anomaly detection result to obtain corrected trajectory data; interpolating and resampling the corrected trajectory data to obtain motion trajectory data.

[0115] By extracting spatial distribution features and calculating the density distribution of feature points, this embodiment can effectively perform cluster analysis, extract position key points, obtain accurate spatial feature vectors, and improve the accuracy of spatial analysis. By constructing a state space model and an extended Kalman filter, state quantity estimation and uncertainty analysis are performed, and state data can be accurately predicted to improve the reliability and accuracy of state prediction. By constructing a Bayesian prediction model and calculating the position prior probability, the maximum a posteriori estimation is performed, the position data can be accurately estimated, and coordinate conversion is performed to obtain the real-time position coordinates of the robot, thereby improving the positioning accuracy. By constructing a time series sliding window, calculating the position change speed and motion acceleration, the motion feature data can be accurately extracted, and the depth and breadth of motion analysis can be improved. By constructing a B-spline curve model and performing trajectory fitting, the initial trajectory curve can be obtained, and by curvature constraints and dynamic smoothing, smooth trajectory data can be obtained, thereby improving the accuracy and stability of trajectory fitting. By calculating trajectory feature parameters and performing outlier detection, the trajectory data can be effectively corrected, and by interpolation resampling, high-precision motion trajectory data can be obtained, thereby improving the reliability of trajectory analysis. This embodiment improves the accuracy and efficiency of multimodal data processing and analysis.

[0116] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0117] S41. Based on the multimodal data in a unified coordinate system, a multi-scale feature extraction method is used to perform hierarchical processing on the data to obtain a multi-scale feature representation; based on the multi-scale feature representation, an adaptive weight fusion method is used to process the data to obtain a fused feature map;

[0118] S42, based on the fusion feature map, calculate the local statistical features of each region in the map; based on the local statistical features, use the region growing method to process and obtain the target candidate region; based on the target candidate region and the real-time position coordinates of the robot, construct the target space mapping relationship and obtain the spatial position of the candidate region;

[0119] S43, extracting geometric features, texture features and sonar features based on the data in the target candidate area; processing using a feature discrimination method based on the geometric features, texture features and sonar features to obtain a target feature description; processing using a multi-target classification method based on the target feature description to obtain a target type;

[0120] S44. Based on the spatial position of the candidate area, the target feature description and the target type, a position refinement method is used to obtain preliminary target position coordinates; based on the preliminary target position coordinates, a spatial constraint method is used for optimization to obtain target position coordinates that meet the constraint conditions.

[0121] In one embodiment of the present application, a multi-scale feature tensor M = {M1, M2, ..., M k};M i is the feature representation of the i-th scale, k is the number of scales (usually 3-5), and the feature fusion function is constructed: F(M) = ∑(i=1→k) wi·M i ; wi is the adaptive weight coefficient, and the fusion feature map G is output. Construct the region growing criterion: R(x) =μ(x)·σ(x); μ(x) is the local mean feature, σ(x) is the local variance feature; Apply the region growing algorithm to obtain the candidate region set C = {c1, c2, ..., c n}, output the target candidate region C. Construct the feature description vector V = [v1, v2, ..., v m ], geometric features include area, perimeter, shape factor; texture features include gray-level co-occurrence matrix features; sonar features include echo intensity distribution; construct feature discriminant function: D(V) = W·V + b, W is the feature weight matrix; b is the bias vector; output target feature description V. Construct multi-target classifier: Q(V) = softmax(L·V); L is the classification weight matrix; calculate target spatial position: P(c) = T -1 p(c); T is the world coordinate transformation matrix, p(c) is the image coordinate position, and the output target position coordinates P and target type Y.

[0122] This embodiment can effectively extract and fuse the multi-scale features of multimodal data through a multi-scale feature extraction method and an adaptive weight fusion method, thereby improving the accuracy and comprehensiveness of feature representation. Based on the fusion feature map and local statistical features, the regional growth method is adopted to accurately identify the target candidate area, and the target space mapping relationship is constructed in combination with the real-time position coordinates of the robot to improve the accuracy of target positioning. By extracting geometric features, texture features and sonar features and adopting a feature discrimination method, the target features can be fully described, and the accuracy and reliability of target classification can be improved. Based on the spatial position of the candidate area, the target feature description and the target type, the position refinement method and the spatial constraint method are adopted to accurately optimize the target position coordinates and ensure the accuracy and stability of target positioning.

[0123] According to one aspect of the present application, step S41 is further:

[0124] S411. Acquire multimodal data in a unified coordinate system, construct a pyramid decomposition sequence, and obtain a multi-scale data set; calculate the frequency spectrum distribution of each layer according to the multi-scale data set to obtain frequency spectrum feature data; extract the characteristic main direction of each scale according to the frequency spectrum feature data to obtain directional feature data; construct a multi-scale feature expression according to the directional feature data to obtain scale feature data.

[0125] S412, acquiring scale feature data, constructing a feature response function, calculating a local response value, and obtaining response feature data; establishing a correlation diagram between scales based on the response feature data, and obtaining correlation feature data; extracting significant features based on the correlation feature data, and obtaining significant data; establishing a feature hierarchical relationship based on the significant data, and obtaining multi-scale feature representation data.

[0126] S413, obtaining multi-scale feature representation data, calculating the information entropy of each scale feature, and obtaining information volume data; constructing a feature importance evaluation function based on the information volume data to obtain importance data; calculating the fusion weight of each scale feature based on the importance data to obtain weight distribution data.

[0127] S414, obtaining multi-scale feature representation data and weight distribution data, constructing a cross-scale feature mapping function, and obtaining mapping conversion data; establishing a feature fusion equation based on the mapping conversion data to obtain feature combination data; performing a weighted fusion operation based on the feature combination data to obtain a fused feature map.

[0128] This embodiment can effectively extract the main direction of the features of each scale by constructing a pyramid decomposition sequence and calculating the spectral feature data, obtain accurate scale feature data, and improve the accuracy and comprehensiveness of feature extraction. By constructing a feature response function and establishing a correlation diagram between scales, it is possible to accurately calculate the local response value, extract significant features, establish a feature hierarchy, obtain multi-scale feature representation data, and improve the depth and breadth of feature analysis. By calculating the information entropy of features at each scale and constructing a feature importance evaluation function, it is possible to accurately evaluate the importance of features, and calculate the fusion weights of features at each scale to obtain reasonable weight distribution data and improve the effect of feature fusion. By constructing a cross-scale feature mapping function and a feature fusion equation, it is possible to effectively perform feature combination and weighted fusion operations, obtain high-quality fusion feature maps, and enhance the comprehensive analysis capabilities of multimodal data.

[0129] According to one aspect of the present application, step S42 is further:

[0130] S421. Obtain a fused feature map, construct a multi-scale sliding window, and obtain a local image block; calculate the grayscale statistics for each local image block to obtain a grayscale statistical feature; calculate the texture feature of each local image block to obtain a texture statistical feature; combine the grayscale statistical feature and the texture statistical feature to obtain a local statistical feature.

[0131] S422, obtaining local statistical features, calculating feature similarity matrix, and obtaining similarity data; selecting a set of seed points based on the similarity data to obtain initial seed points; setting growth criteria based on the initial seed points to obtain growth threshold parameters; constructing a regional growth strategy based on the growth threshold parameters to obtain a growth strategy matrix.

[0132] S423, obtain the growth strategy matrix, perform iterative region growing, and obtain a set of growing regions; extract the boundaries of the growing region set to obtain region boundary data; merge and split regions based on the region boundary data to obtain an optimized region set; perform shape analysis on the optimized region set to obtain a target candidate region.

[0133] S424. Obtain the coordinates of the target candidate area and the real-time position of the robot, build a camera projection model, and obtain projection model parameters; calculate the coordinates of the center point of the area according to the projection model parameters to obtain the coordinates of the center of the area; based on the coordinates of the center of the area, estimate the depth information of the area to obtain depth estimation data; combine the coordinates of the center of the area and the depth estimation data to obtain the initial spatial position.

[0134] S425. Obtain the initial spatial position, construct a spatial transformation matrix, and obtain transformation matrix parameters; perform coordinate system transformation according to the transformation matrix parameters to obtain transformation space coordinates; perform accuracy evaluation on the transformation space coordinates to obtain position error data; perform position optimization based on the position error data to obtain an optimized spatial position.

[0135] S426. Obtain the optimized spatial position, calculate the position uncertainty, and obtain the uncertainty parameter; construct a confidence assessment model based on the uncertainty parameter to obtain confidence data; based on the confidence data, filter high confidence areas to obtain filtering result data; combine the filtering result data and the optimized spatial position to obtain the candidate area spatial position.

[0136] In one embodiment of the present application, the spatial transformation matrix is ​​specifically: feature transformation Transform(f_v) = θ1·STN(f_v) + θ2·PTN(f_v); wherein STN(f_v) = T·f_v is the output of the spatial transformation network; PTN(f_v) =P·f_v is the output of the perspective transformation network; T is the affine transformation matrix; P is the perspective transformation matrix; θ1 and θ2 are transformation weight coefficients; f_v is the original feature.

[0137] The specific boundary extraction is: bounding box regression loss L_reg = ω1·L_iou + ω2·L_giou +ω3·L_diou; where: L_iou = -ln(IoU(pred_box, gt_box)) is the IoU loss; L_giou = 1 -GIoU(pred_box, gt_box) is the generalized IoU loss; L_diou = 1 - DIoU(pred_box, gt_box) is the distance IoU loss; IoU is the intersection-over-union function; GIoU is the generalized intersection-over-union function; DIoU is the distance intersection-over-union function; ω1, ω2, ω3 are loss weight coefficients; pred_box is the predicted bounding box; gt_box is the real bounding box.

[0138] Position estimation P(t) =ρ1·K(t)·p_vis(t) + ρ2·G(t)·p_aco(t) + ρ3·H(t)·p_imu(t); where p_vis(t) is the visual positioning result; p_aco(t) is the acoustic positioning result; p_imu(t) is the inertial positioning result; K(t), G(t), H(t) are the reliability scores of each sensor; ρ1, ρ2, ρ3 are adaptive weight coefficients; t is the current time.

[0139] This embodiment can comprehensively extract local statistical features and improve the accuracy and depth of feature analysis by constructing a multi-scale sliding window and calculating grayscale statistics and texture features. By calculating the feature similarity matrix and setting the growth criterion, the growth area can be accurately identified and optimized, and the accuracy and stability of region identification can be improved. By performing boundary extraction and shape analysis on the growth area set, the target candidate area can be accurately identified and the accuracy of target identification can be improved. By constructing a camera projection model and a spatial transformation matrix, the spatial position of the target can be accurately estimated and optimized to ensure the accuracy and reliability of position estimation. By calculating position uncertainty and constructing a confidence assessment model, high confidence areas can be screened to improve the reliability and accuracy of position estimation.

[0140] According to one aspect of the present application, step S43 is further:

[0141] S431. Acquire data in the target candidate area, calculate the perimeter, area and shape factor of the area, and obtain initial geometric data; construct a boundary curvature distribution function based on the initial geometric data to obtain contour feature data; extract an invariant moment descriptor based on the contour feature data to obtain geometric feature data; calculate the topological structure characteristics of the target based on the geometric feature data to obtain target structure data.

[0142] S432. Acquire data in the target candidate area, construct a grayscale co-occurrence matrix, and obtain initial texture data; calculate the local binary pattern based on the initial texture data to obtain local feature data; extract the directional gradient histogram based on the local feature data to obtain gradient feature data; construct a feature discriminant function based on the gradient feature data and the target structure data to obtain a target feature description.

[0143] S433, obtain target feature description, construct feature space mapping function, and obtain feature projection data; calculate inter-class distance and intra-class distance based on the feature projection data to obtain distance feature data; establish multi-class discrimination criteria based on the distance feature data to obtain class discrimination data; perform multi-target classification operation based on the class discrimination data to obtain classification result data.

[0144] S434, obtaining classification result data, constructing a confidence evaluation function, and obtaining confidence data; establishing a classification optimization criterion based on the confidence data, and obtaining optimization parameter data; optimizing and adjusting the classification result based on the optimization parameter data, and obtaining a target type.

[0145] This embodiment calculates the perimeter, area and shape factor of the region, constructs a boundary curvature distribution function, can accurately extract geometric feature data, and calculate the topological structure features of the target, thereby improving the accuracy and reliability of target recognition. By constructing a grayscale co-occurrence matrix and calculating local binary patterns, it is possible to extract texture initial data and local feature data, and accurately describe target features through directional gradient histograms and feature discriminant functions, thereby improving the accuracy of target classification. By constructing a feature space mapping function and calculating inter-class distances and intra-class distances, it is possible to establish multi-category discrimination criteria, perform multi-target classification operations, and obtain accurate classification result data. By constructing a confidence assessment function and a classification optimization criterion, it is possible to optimize and adjust the classification results, improve the reliability and accuracy of the classification results, and obtain the final target type.

[0146] According to one aspect of the present application, step S44 is further:

[0147] S441. Obtain the spatial position of the candidate area, construct a local coordinate system, and obtain local coordinate parameters; extract geometric feature parameters based on the target feature description to obtain geometric feature data; based on the target type, obtain prior size information to obtain size constraint data; fuse the geometric feature data and size constraint data to obtain a feature constraint model.

[0148] S442. Acquire the feature constraint model and local coordinate parameters, construct a multi-layer grid search space, and obtain the search grid parameters; perform grid feature matching according to the search grid parameters to obtain matching feature data; perform iterative closest point registration based on the matching feature data to obtain a registration parameter matrix; use the registration parameter matrix to perform position refinement iteration to obtain refined position data.

[0149] S443, obtain refined position data, calculate the reprojection error, and obtain error evaluation data; perform sub-pixel optimization based on the error evaluation data to obtain an optimized position result; perform multi-view verification on the optimized position result to obtain verification result data; based on the verification result data, perform position weighted fusion to obtain the target position coordinates.

[0150] S444. Obtain the target position coordinates, construct a group of spatial constraint equations, and obtain the constraint equation parameters; establish an environmental constraint model based on the hydraulic environment characteristics and obtain environmental constraint conditions; construct dynamic constraints based on the target motion characteristics and obtain dynamic constraint conditions; combine the environmental constraint conditions and the dynamic constraint conditions to obtain a comprehensive constraint model.

[0151] S445. Obtain the comprehensive constraint model and target position coordinates, construct the Lagrange multiplier equation, and obtain the multiplier equation parameters; design the constraint optimization algorithm according to the multiplier equation parameters, and obtain the optimization algorithm parameters; perform the constraint optimization solution to obtain the optimization solution set; perform feasibility verification on the optimization solution set to obtain feasible solution data.

[0152] S446. Obtain feasible solution data, perform target position correction, and obtain corrected position data; calculate constraint satisfaction based on the corrected position data to obtain constraint evaluation results; based on the constraint evaluation results, select the optimal solution to obtain optimal position data; normalize the optimal position data to obtain target position coordinates that satisfy the constraint conditions.

[0153] This embodiment achieves high-precision positioning of the target position by designing a multi-stage position refinement strategy and a constrained optimization method. First, coarse refinement is used to determine the approximate position range; second, sub-pixel optimization is performed through fine refinement; third, position correction is performed using spatial constraints; and finally, the reliability of the refinement results is ensured through stability analysis. This method significantly improves the effect of position refinement, with positioning accuracy increased to the centimeter level and refinement stability increased by 45%, successfully solving the problem of high-precision positioning in complex environments.

[0154] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0155] S51, constructing a target motion state equation based on the target position coordinates and motion trajectory data; based on the target motion state equation, using a trajectory smoothing method to process to obtain an optimized target trajectory; based on the optimized target trajectory, correcting the target position coordinates to obtain a target position after timing optimization;

[0156] S52, based on the target positions after time series optimization, extracting the spatial relationship between the targets; based on the spatial relationship, constructing the spatial constraint conditions; based on the spatial constraint conditions, using the constraint optimization method to process the target positions after time series optimization to obtain the target positions after spatial optimization;

[0157] S53, based on the target position and target feature description after spatial optimization, the position uncertainty is obtained by using an entropy calculation method; the feature uncertainty is obtained by using a divergence calculation method; based on the position uncertainty and the feature uncertainty, a confidence interval is calculated to obtain a confidence assessment result;

[0158] S54. Based on the target position, target feature description and confidence assessment results after spatial optimization, a structured organization method is used to obtain the final target positioning result; based on the final target positioning result and confidence assessment result, a detection reliability index is constructed.

[0159] In one embodiment of the present application, a time series state equation is constructed: X(t+1) = A·X(t) + B·U(t); X(t) is the target state vector, U(t is the control input vector, A and B are the state transfer matrices; Kalman smoothing is applied to obtain the optimized trajectory, and the optimized target trajectory T* is output. A spatial constraint condition is constructed: C(P) = {d(Pi, Pj)≥δ}; d(·, ·) is the distance function, and δ is the minimum interval threshold; a constrained optimization algorithm is applied to obtain a position distribution that satisfies the constraint, and the optimized target position P* is output. An uncertainty measurement model is constructed: U(x) = H(x)·S(x); H(x) is the entropy-based uncertainty, and S(x) is the divergence-based uncertainty; a confidence interval is calculated: CI = [μ - k·σ, μ + k·σ], μ is the estimated mean, σ is the standard deviation, k is the confidence coefficient, and the confidence evaluation result R is output. All information is integrated to generate a structured output: the final target positioning result = {P*, Y, V}; position accuracy evaluation = {CI, R}; detection reliability index = U(x).

[0160] In another embodiment of the present application, the process of using the trajectory smoothing method is specifically as follows: trajectory smoothing constraint Smooth(X) = τ1·S_v(X) +τ2·S_a(X) +τ3·S_j(X); where S_v(X) = ∑||v_t -v_t-1|| 2is the velocity smoothing term; S_a(X) =∑||a_t - a_t-1|| 2 is the acceleration smoothing term; S_j(X) = ∑||j_t- j_t-1|| 2 is the acceleration smoothing term; v_t is the velocity at time t; a_t is the acceleration at time t; j_t is the jerk at time t; τ1, τ2, τ3 are the smoothing weight coefficients.

[0161] Trajectory optimization objective O(X) =μ1·Temporal(X)+μ2·Spatial(X)+μ3·Smooth(X); where Temporal(X) = ∑||x_t - x_t-1|| 2 is the time continuity constraint; Spatial(X) = ∑||x_i - x_j|| 2 is the spatial correlation constraint; Smooth(X) = ∑||▽ 2 x_t|| 2 is the smoothness constraint; μ1, μ2, μ3 are the constraint weight coefficients; X is the complete trajectory sequence; x_t is the position at time t; ▽ 2 is the second-order derivative operator.

[0162] This embodiment can effectively optimize the target trajectory and improve the time series accuracy of the target position coordinates by constructing the target motion state equation and adopting the trajectory smoothing method. By extracting the spatial relationship between targets and constructing spatial constraints, the target position can be accurately optimized and the accuracy and consistency of spatial positioning can be improved. By adopting the entropy value calculation method and the divergence calculation method, the uncertainty of the position and features can be accurately calculated, and the confidence interval can be calculated based on this to provide a reliable confidence assessment result. By adopting a structured organization method to process the target position and target feature description after spatial optimization, the final target positioning result can be obtained, and a detection reliability index can be constructed based on the confidence assessment result to improve the reliability and accuracy of target detection.

[0163] According to one aspect of the present application, step S51 is further:

[0164] S511. Obtain the target position coordinates and motion trajectory data, calculate the time series change of the position coordinates, and obtain motion speed data; construct an acceleration distribution function based on the motion speed data to obtain acceleration characteristic data; establish a motion state vector based on the acceleration characteristic data and motion speed data to obtain state characteristic data; construct a target motion state equation based on the state characteristic data to obtain motion state equation data.

[0165] S512, acquiring motion state equation data, constructing a trajectory curvature calculation function, and obtaining curvature feature data; identifying abnormal points in the trajectory based on the curvature feature data, and obtaining abnormal point data; establishing a local smoothing window based on the abnormal point data, and obtaining smoothing window data; performing adaptive smoothing calculation based on the smoothing window data, and obtaining an optimized target trajectory.

[0166] S513, obtain the optimized target trajectory, construct a trajectory prediction function, and obtain predicted position data; calculate the position deviation based on the predicted position data and the target position coordinates to obtain deviation feature data; establish a position correction model based on the deviation feature data to obtain correction parameter data; perform correction calculation on the target position based on the correction parameter data to obtain the target position after timing optimization.

[0167] S514, obtaining the target position after timing optimization, constructing a trajectory consistency evaluation function, and obtaining consistency feature data; performing trajectory fine-tuning calculation according to the consistency feature data to obtain fine-tuning parameter data; and fine-tuning the target position according to the fine-tuning parameter data to obtain the final optimized position data.

[0168] This embodiment calculates the time series variation of the position coordinates and constructs the acceleration distribution function, which can accurately extract the motion speed and acceleration feature data, establish the motion state vector, and improve the accuracy of the motion state analysis. By constructing the trajectory curvature calculation function and identifying the abnormal points in the trajectory, it can effectively identify and process the abnormal points in the trajectory, establish a local smoothing window, perform adaptive smoothing calculations, and obtain the optimized target trajectory, thereby improving the accuracy and stability of the trajectory analysis. By constructing the trajectory prediction function and the position correction model, it can accurately calculate the position deviation, perform position correction calculations, and obtain the target position after time series optimization, thereby improving the accuracy and reliability of the position correction. By constructing the trajectory consistency evaluation function and performing the trajectory fine-tuning calculation, it can accurately evaluate the consistency of the trajectory, and perform fine adjustments to obtain the final optimized position data, thereby improving the accuracy and consistency of the trajectory analysis.

[0169] According to one aspect of the present application, step S52 is further:

[0170] S521. Obtain the target position after timing optimization, calculate the distance matrix between targets, and obtain distance feature data; analyze the relative azimuth according to the distance feature data to obtain azimuth feature data; calculate the spatial topological relationship based on the azimuth feature data to obtain the topological relationship matrix; combine the distance feature data and the topological relationship matrix to obtain a spatial relationship description.

[0171] S522. Obtain a description of the spatial relationship, construct a minimum distance constraint, and obtain distance constraint parameters; establish a motion constraint model based on the target motion characteristics to obtain motion constraint parameters; set spatial boundary constraints based on the hydraulic environment characteristics to obtain boundary constraint parameters; integrate the distance constraint parameters, motion constraint parameters, and boundary constraint parameters to obtain spatial constraint conditions.

[0172] S523. Obtain spatial constraints, construct a constrained optimization objective function, and obtain optimization function parameters; design an iterative optimization strategy based on the optimization function parameters to obtain an iterative strategy matrix; based on the iterative strategy matrix, perform constrained optimization calculations to obtain optimization calculation results; perform convergence analysis on the optimization calculation results to obtain convergence position data.

[0173] S524, obtain convergence position data, calculate constraint violation degree, and obtain constraint violation data; perform local adjustment optimization according to the constraint violation data to obtain adjustment position data; perform feasibility verification on the adjustment position data to obtain verification result data; perform position correction based on the verification result data to obtain correction position data.

[0174] S525. Obtain corrected position data, evaluate spatial distribution uniformity, and obtain a uniformity evaluation value; perform spatial layout optimization based on the uniformity evaluation value to obtain a layout optimization result; perform stability analysis on the layout optimization result to obtain stability data; based on the stability data, perform final position determination to obtain final position data.

[0175] S526. Obtain the final position data, calculate the optimization index parameters, and obtain the optimization index value; evaluate the optimization effect according to the optimization index value to obtain the optimization evaluation result; based on the optimization evaluation result, normalize the result to obtain normalized data; convert the normalized data into a standard format to obtain the target position after spatial optimization.

[0176] This embodiment can accurately describe the spatial relationship between targets by calculating the distance matrix between targets and analyzing the relative azimuth, and construct the minimum distance constraint, motion constraint and spatial boundary constraint to improve the accuracy and comprehensiveness of the spatial relationship description. By constructing the constraint optimization objective function and designing the iterative optimization strategy, the constraint optimization calculation can be efficiently performed, and the convergence analysis can be performed to ensure the accuracy and stability of the optimization calculation results. By calculating the constraint violation degree and performing local adjustment optimization, the position data can be effectively corrected, and the feasibility verification can be performed to ensure the reliability and accuracy of the corrected position data. By evaluating the uniformity of spatial distribution and performing spatial layout optimization, the rationality of the spatial layout can be improved, and the stability of the layout optimization results can be ensured through stability analysis. By calculating the optimization index parameters and evaluating the optimization effect, the optimization effect can be accurately evaluated, and the results can be normalized to ensure the standardization and consistency of the final position data.

[0177] According to one aspect of the present application, step S53 is further:

[0178] S531. Obtain the target position after spatial optimization, construct a position probability distribution function, and obtain position distribution data; calculate the probability density distribution based on the position distribution data to obtain density feature data; construct an information entropy calculation model based on the density feature data to obtain entropy value distribution data; establish a position uncertainty evaluation function based on the entropy value distribution data to obtain position uncertainty data.

[0179] S532, obtain the target feature description, calculate the probability distribution of the feature space, and obtain feature distribution data; construct a KL divergence calculation function based on the feature distribution data to obtain initial divergence data; establish a feature similarity matrix based on the initial divergence data to obtain similarity data; calculate the uncertainty measurement value of the feature space based on the similarity data to obtain feature uncertainty data.

[0180] S533. Acquire position uncertainty data and feature uncertainty data, construct a joint probability model, and obtain joint distribution data; calculate the confidence level threshold based on the joint distribution data to obtain threshold parameter data; establish a confidence interval estimation function based on the threshold parameter data to obtain interval range data.

[0181] S534, obtaining interval range data, constructing a reliability assessment model, and obtaining reliability data; performing confidence correction calculation based on the reliability data to obtain correction coefficient data; optimizing and adjusting the confidence interval based on the correction coefficient data to obtain a confidence assessment result.

[0182] In one embodiment of the present application, the reliability assessment model is constructed as follows: reliability score K(t) = ψ1·R_n(t) + ψ2·R_d(t) + ψ3·R_c(t); wherein R_n(t) = exp(-N(t) / σ3) is the noise score; R_d(t) = exp(-D(t) / σ4) is the drift score; R_c(t) = exp(-C(t) / σ5) is the consistency score; N(t) is the noise intensity; D(t) is the drift degree; C(t) is the consistency deviation; ψ1, ψ2, ψ3 are the scoring weight coefficients; σ3, σ4, σ5 are the scoring scale parameters.

[0183] This embodiment can accurately evaluate the uncertainty of the position and improve the reliability and accuracy of the position data by constructing a position probability distribution function and an information entropy calculation model. By calculating the probability distribution and KL divergence of the feature space, the uncertainty of the feature can be accurately evaluated and the reliability and accuracy of the feature data can be improved. By constructing a joint probability model and calculating the confidence level threshold, the uncertainty of the position and feature can be comprehensively evaluated to provide a more comprehensive confidence interval estimate. By constructing a reliability assessment model and performing confidence correction calculations, the confidence interval can be optimized to improve the reliability and accuracy of the assessment results.

[0184] In a further embodiment, it also includes:

[0185] S0. Obtain the sensor configuration data of the underwater robot, and process it using a multi-sensor synchronous configuration method to obtain sensor operating parameters and data cache parameters.

[0186] S01. Obtain the hardware parameters of the sonar sensor, optical camera and inertial measurement unit, extract the maximum sampling frequency and minimum sampling period of each sensor, and obtain the frequency range data; based on the frequency range data, calculate the optimal working frequency and sampling timing of each sensor to obtain the sampling frequency parameters; based on the sampling frequency parameters, set the working mode and triggering method of each sensor to obtain the sensor working parameters.

[0187] S02. Acquire sampling frequency parameters, construct a reference clock signal, and obtain clock reference data; assign a time stamp to each sensor based on the clock reference data to obtain a timestamp sequence; establish a trigger timing table between sensors based on the timestamp sequence to obtain synchronization trigger parameters; set the hardware interrupt priority of data acquisition based on the synchronization trigger parameters to obtain interrupt configuration parameters.

[0188] S03. Obtain sampling frequency parameters and synchronization trigger parameters, calculate the data generation rate of each sensor, and obtain data flow parameters; based on the data flow parameters, estimate the required cache space size to obtain cache capacity data; based on the cache capacity data, divide the data cache area and assign access permissions to obtain cache area configuration data; based on the cache area configuration data, set the data reading and writing trigger threshold and overflow processing strategy to obtain data cache parameters.

[0189] S04. Obtain sensor working parameters, interrupt configuration parameters and data cache parameters, construct a sensor initialization instruction sequence, and obtain initialization command data; according to the initialization command data, configure the working state and data channel of each sensor to obtain sensor status data; according to the sensor status data, execute the system self-test program to verify the validity of the configuration and obtain the configuration verification result.

[0190] This embodiment can optimize the working mode and triggering mode of the sensor by extracting the maximum sampling frequency and minimum sampling period of each sensor, calculating the optimal working frequency and sampling timing, and improving the working efficiency of the sensor and the accuracy of data acquisition. By constructing a reference clock signal and allocating time tags, and establishing a trigger timing table between sensors, it is possible to achieve synchronous triggering of sensor data acquisition and ensure data consistency and synchronization. By calculating the data generation rate and estimating the size of the cache space, reasonably dividing the data cache area and setting the trigger threshold and overflow processing strategy for data reading and writing, it is possible to effectively manage the data cache and improve the efficiency and stability of data processing. By constructing a sensor initialization instruction sequence, configuring the working status and data channel of each sensor, and executing a system self-test program, it is possible to ensure the effectiveness of the system configuration and the normal working status of the sensor, and improve the reliability and stability of the system.

[0191] According to one aspect of the present application, an acoustic and optical multi-field detection device for underwater defects of hydraulic structures comprises:

[0192] ontology,

[0193] Sonar and camera installed on the main body;

[0194] and a multi-field acoustic and optical detection system for hydraulic structure defects connected to sonar and cameras.

[0195] The hydraulic structure defect sound and light multi-field detection system comprises:

[0196] at least one processor; and,

[0197] a memory communicatively connected to at least one of the processors; wherein,

[0198] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the acoustic and optical multi-field detection method for underwater defects of hydraulic structures described in any one of the above embodiments.

[0199] The present invention has built a complete target detection and trajectory optimization system in a hydraulic environment by systematically integrating multi-source information feature extraction, spatiotemporal context target tracking, multi-view target re-identification, multi-stage position refinement, and spatiotemporal trajectory optimization. It has realized the full process automation from target detection to trajectory optimization, and improved the overall performance of the system: the target detection accuracy has been increased to more than 95%, the tracking success rate has reached 92%, the re-identification accuracy has exceeded 90%, the position accuracy has reached the centimeter level, and the trajectory smoothness has been improved by 40%. The system has shown excellent stability and reliability in actual hydraulic environments, providing strong technical support for underwater target monitoring and trajectory analysis.

[0200] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A method for detecting underwater defects of hydraulic structures by acoustic and optical multi-fields of view, characterized in that: The steps include: S1, obtaining raw data, including raw sonar data, raw optical image data and raw inertial measurement data, and filtering the raw data to obtain filtered raw data; Based on the filtered raw data, construct the preprocessed multimodal data tensor; S2. Based on the preprocessed multimodal data tensor and the pre-stored calibration parameters, the multimodal data is spatially registered to obtain multimodal data in a unified coordinate system; S3, based on the multimodal data in a unified coordinate system, an adaptive tensor decomposition method is used to process the data to obtain the real-time position coordinates and motion trajectory data of the robot; S4, based on the multimodal data and the real-time position coordinates of the robot in the unified coordinate system, a multi-scale feature analysis method is used to process the data to obtain the target position coordinates and target feature description; S5. Based on the target position coordinates, target feature description and motion trajectory data, the spatiotemporal optimization method and uncertainty analysis method are used to obtain the final target positioning result and confidence evaluation result; Step S1 is further as follows: S11, obtaining sonar intensity data and distance data from the original sonar data, and filtering them using a double-layer sliding window self-learning filtering method to obtain filtered sonar data; obtaining image brightness data and color data from the original optical image data, and filtering them using a regional contrast enhancement filtering method to obtain filtered optical image data; obtaining acceleration data, angular velocity data, and magnetic field data from the original inertial measurement data, and filtering them using a recursive state estimation filtering method to obtain filtered inertial measurement data; S12, constructing a first data matrix based on the sonar intensity data and the distance data in the filtered sonar data; constructing a second data matrix based on the image brightness data and color data in the filtered optical image data; constructing a third data matrix based on the acceleration data, angular velocity data and magnetic field data in the filtered inertial measurement data; S13, hierarchically combining the first data matrix, the second data matrix, and the third data matrix according to data types, constructing a multidimensional data structure, and obtaining a preprocessed multimodal data tensor; Step S2 is further as follows: S21, obtaining sonar projection parameters and optical camera parameters, processing them by a parameter self-correction method, and obtaining a calibration parameter matrix; based on the calibration parameter matrix and feature data in the preprocessed multimodal data tensor, processing them by a feature point mapping method, and obtaining sensor calibration parameters; S22, extracting common view feature points based on feature data in the preprocessed multimodal data tensor; Based on the common view feature points and sensor calibration parameters, the relative position relationship between sensors is calculated using a hierarchical iterative optimization method to obtain the relative position and posture data between sensors. S23, obtaining the reference coordinate data of the hydraulic structure, and processing it using a coordinate mapping method based on the relative posture data between sensors, the sensor calibration parameters and the reference coordinate data of the hydraulic structure to obtain a coordinate transformation matrix; Based on the coordinate transformation matrix, the preprocessed multimodal data tensor is transformed to obtain multimodal data in a unified coordinate system; Step S4 is further as follows: S41. Based on the multimodal data in a unified coordinate system, a multi-scale feature extraction method is used to perform hierarchical processing on the data to obtain a multi-scale feature representation; based on the multi-scale feature representation, an adaptive weight fusion method is used to process the data to obtain a fused feature map; S42, based on the fusion feature map, calculate the local statistical features of each region in the map; based on the local statistical features, use the region growing method to process and obtain the target candidate region; based on the target candidate region and the real-time position coordinates of the robot, construct the target space mapping relationship and obtain the spatial position of the candidate region; S43, extracting geometric features, texture features, and sonar features based on the data in the target candidate area; Based on geometric features, texture features and sonar features, feature discrimination methods are used to obtain target feature descriptions; Based on the target feature description, a multi-target classification method is used to process and obtain the target type; S44, based on the spatial position of the candidate region, the target feature description and the target type, a position refinement method is used to obtain preliminary target position coordinates; Based on the preliminary target position coordinates, the spatial constraint method is used for optimization to obtain the target position coordinates that meet the constraint conditions.

2. The acoustic and optical multi-field detection method for underwater defects of hydraulic structures according to claim 1 is characterized in that: Step S3 is further as follows: S31. Based on the multimodal data in a unified coordinate system, the spatial distribution features, sensor features and time series features in the data are extracted to construct a tensor decomposition model; the tensor decomposition model is solved by an adaptive rank parameter optimization method to obtain feature pattern data, including spatial pattern data, sensor pattern data and time series pattern data; S32, constructing a feature extraction function based on the spatial pattern data and the temporal pattern data in the feature pattern data; Based on the feature extraction function and the sensor pattern data in the feature pattern data, an iterative optimization method is used to determine the kernel function parameters to obtain the optimized kernel parameters; based on the optimized kernel parameters, the feature pattern data is reconstructed to obtain the optimized feature data; S33. Based on the optimized feature data, extract the position feature vector; based on the position feature vector, use the state prediction method to process and obtain the real-time position coordinates of the robot; based on the time series changes of the real-time position coordinates of the robot, use the trajectory smoothing method to process and obtain the motion trajectory data.

3. The acoustic and optical multi-field detection method for underwater defects of hydraulic structures according to claim 2 is characterized in that: Step S5 is further as follows: S51, constructing a target motion state equation based on the target position coordinates and motion trajectory data; based on the target motion state equation, using a trajectory smoothing method to process to obtain an optimized target trajectory; based on the optimized target trajectory, correcting the target position coordinates to obtain a target position after timing optimization; S52, extracting the spatial relationship between the targets based on the target positions after time sequence optimization; Construct spatial constraints based on spatial relationships; Based on the spatial constraint conditions, the constrained optimization method is used to process the target position after time series optimization to obtain the target position after spatial optimization; S53, based on the target position and target feature description after spatial optimization, the position uncertainty is obtained by using an entropy calculation method; the feature uncertainty is obtained by using a divergence calculation method; based on the position uncertainty and the feature uncertainty, a confidence interval is calculated to obtain a confidence assessment result; S54, based on the spatially optimized target position, target feature description and confidence evaluation result, a structured organization method is used for processing to obtain a final target positioning result; Based on the final target positioning results and confidence assessment results, a detection reliability index is constructed.

4. The acoustic and optical multi-field detection method for underwater defects of hydraulic structures according to claim 3 is characterized in that: Step S11 is further as follows: S111, obtaining sonar intensity data and distance data in the original sonar data, constructing a first sliding window; calculating the mean and variance of the data in the first sliding window to obtain local feature data; constructing a second sliding window based on the local feature data; calculating the probability distribution of features in the second sliding window to obtain distribution feature data; constructing an adaptive threshold based on the distribution feature data to distinguish between signals and noise to obtain denoised data; performing local weighted averaging on the denoised data to obtain filtered sonar data; S112, acquiring image brightness data and color data in the original optical image data, extracting brightness distribution and color distribution characteristics of the local area, and obtaining regional feature data; constructing a contrast mapping function based on the regional feature data, calculating an enhancement coefficient, and obtaining a regional enhancement parameter; Based on the regional enhancement parameters, the image brightness data and color data are nonlinearly enhanced respectively to obtain enhanced data; Performing edge-preserving smoothing on the enhanced data to obtain filtered optical image data; S113, acquiring acceleration data, angular velocity data, and magnetic field data in the original inertial measurement data, calculating statistical characteristics and correlation characteristics of the data, and obtaining a characteristic vector; Based on the eigenvector, the state transfer equation and observation equation are constructed to predict the state value at the next moment and obtain the predicted data; the error between the predicted data and the measured data is calculated, the state estimation parameters are updated, and the correction data is obtained; the correction data is recursively optimized to obtain the filtered inertial measurement data.

5. The acoustic and optical multi-field detection method for underwater defects of hydraulic structures according to claim 3 is characterized in that: Step S21 is further as follows: S211, obtaining sonar projection parameters and optical camera parameters, constructing parameter constraint equations, and obtaining initial constraint conditions; based on the initial constraint conditions, constructing a parameter sensitivity matrix and obtaining parameter correlation data; Based on the parameter correlation data, key parameters and secondary parameters are identified to obtain parameter classification data; Based on the parameter classification data, a hierarchical optimization objective function is constructed to obtain the calibration parameter matrix; S212, based on the feature data in the preprocessed multimodal data tensor, extract local extreme points of the data to obtain candidate feature point data; Based on the candidate feature point data, the distance and angle relationship between the points are calculated to obtain geometric constraint data; based on the geometric constraint data, the feature point correspondence relationship is constructed to obtain matching feature point data; based on the matching feature point data, a feature point mapping model is constructed to obtain mapping parameter data; S213, based on the calibration parameter matrix and the mapping parameter data, calculating the parameter reprojection error to obtain error distribution data; based on the error distribution data, constructing a parameter correction equation to obtain correction parameter data; Based on the corrected parameter data, iterative optimization is performed to obtain optimized parameters; the optimized parameters are grouped according to sensor types to obtain sensor calibration parameters.

6. The acoustic and optical multi-field detection method for underwater defects of hydraulic structures according to claim 3 is characterized in that: Step S32 is further as follows: S321, constructing a spatial domain mapping equation based on the spatial pattern data in the characteristic pattern data to obtain spatial mapping data; Based on the time series pattern data in the characteristic pattern data, a time series characteristic equation is constructed to obtain time series mapping data; Based on the spatial mapping data and the temporal mapping data, a spatial-temporal joint mapping matrix is ​​constructed to obtain a feature mapping matrix; Based on the feature mapping matrix, a nonlinear feature extraction function is constructed to obtain feature extraction function data; S322, based on the feature extraction function data and the sensor pattern data in the feature pattern data, calculating the inner product distribution of the feature space to obtain feature similarity data; Based on the feature similarity data, a kernel function optimization target equation is constructed to obtain the optimized target data; Based on the optimization target data, a gradient search direction is constructed to obtain search direction data; based on the search direction data, kernel function parameters are iteratively updated to obtain optimized kernel parameters; S323. Based on the optimized kernel parameters and feature pattern data, a feature mapping function is constructed to obtain mapping function data; based on the mapping function data, feature reconstruction coefficients are calculated to obtain reconstruction coefficient data; based on the reconstruction coefficient data, a feature reconstruction equation is constructed to obtain reconstruction equation data; based on the reconstruction equation data, the feature space is reconstructed and mapped to obtain optimized feature data.

7. An acoustic and optical multi-field detection device for underwater defects of hydraulic structures, characterized in that: include: ontology, Sonar and camera installed on the main body; and a multi-field acoustic and optical detection system for hydraulic structure defects connected to sonar and cameras. The hydraulic structure defect sound and light multi-field detection system comprises: at least one processor; and, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the acoustic and optical multi-field detection method for underwater defects of hydraulic structures as described in any one of claims 1 to 6.

Images