Multi-modal defect detection and three-dimensional reconstruction interaction method based on virtual reality

Abstract

The invention discloses a multi-modal defect detection and three-dimensional reconstruction interaction method based on virtual reality, and relates to the field of structural health monitoring of industrial equipment and infrastructures. According to the technical scheme, the method comprises the steps of obtaining sensor data of target industrial equipment or infrastructure; adaptive uncertainty correction and dynamic weighting are carried out on the sensor data to realize data fusion processing, and fused joint feature vector data are obtained; performing point cloud reconstruction algorithm processing on the fused joint feature vector data to obtain three-dimensional defect model data; performing coordinate transformation processing on the three-dimensional defect model data to obtain three-dimensional defect model data in the VR system; performing multi-level iteration processing on the three-dimensional defect model data in the VR system to obtain defect probability distribution data; and performing optimization interaction mapping processing to generate dynamic interaction function data, and completing multi-modal defect detection and three-dimensional reconstruction interaction based on virtual reality in the form of a mixed reality feedback mechanism.

Description

Technical Field

[0001] The present invention relates to the field of structural health monitoring of industrial equipment and infrastructure, and specifically to a multimodal defect detection and three-dimensional interaction method for structures such as storage tanks, pipelines, and bridges based on virtual reality (VR). The method is particularly suitable for real-time visual detection and decision support in high-risk scenarios (such as chemical tank corrosion, bridge cracks, and pipeline component fatigue damage). Background Art

[0002] In the fields of petrochemicals, electricity, transportation, etc., defects (such as corrosion and cracks) in key infrastructure such as storage tanks, pipelines, and bridges may cause major safety accidents. However, traditional defect detection technology has significant limitations. Currently, most detection methods rely mainly on a single type of sensor data, such as ultrasonic, X-ray or eddy current detection. Although these single sensors can obtain specific types of defect information, due to the single detection perspective and principle, it is difficult to fully capture the multi-angle and multi-dimensional characteristics of structural defects. For example, for defects with complex geometric shapes or deep buried in the material, a single sensor may not be able to accurately identify them due to detection blind spots or signal attenuation.

[0003] Furthermore, traditional inspection systems lack intuitive visual interaction platforms, forcing inspectors to rely on complex two-dimensional images or data reports for defect analysis. This not only increases cognitive burden but also limits the ability to interact and make intelligent decisions in real time. The inefficiency of traditional methods is particularly prominent when faced with urgent inspection tasks or large-scale structural assessments.

[0004] It is worth noting that the application of virtual reality (VR) technology in the field of defect detection is still in its infancy, and its multi-dimensional and immersive interactive capabilities have not yet been fully explored. Therefore, exploring the deep integration of VR technology and defect detection and building an intelligent and visual inspection platform has become a technical challenge that needs to be solved urgently. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal defect detection and three-dimensional reconstruction interactive method based on virtual reality to solve the problem that existing defect detection mainly relies on single sensor data, is difficult to fully capture multi-angle information of structural defects, lacks an intuitive visualization platform, and is difficult to meet the needs of real-time interaction and intelligent decision-making support of inspection personnel.

[0006] The technical solution of the present invention to solve the above technical problems is as follows:

[0007] A multimodal defect detection and three-dimensional reconstruction interactive method based on virtual reality includes the following steps:

[0008] S1. Obtain data from ultrasonic sensors, infrared sensors, and visual sensors;

[0009] S2. Perform adaptive uncertainty correction and dynamic weighting on the ultrasonic sensor, infrared sensor, and visual sensor data to achieve data fusion processing and obtain fused joint feature vector data;

[0010] S3, performing point cloud reconstruction algorithm processing on the fused joint feature vector data to obtain three-dimensional defect model data;

[0011] S4, performing coordinate transformation processing on the three-dimensional defect model data to obtain the three-dimensional defect model data in the VR system;

[0012] S5. Perform multi-level iterative processing of the deep neural network and graph neural network on the three-dimensional defect model data in the VR system to obtain defect probability distribution data;

[0013] S6. Optimize the interactive mapping process on the joint feature vector data of step S2 to obtain dynamic interactive function data. This data completes the multimodal defect detection and three-dimensional reconstruction interaction based on virtual reality through the form of a mixed reality feedback mechanism.

[0014] Furthermore, in step S2, it is assumed that the system collects N sensor data and extracts their respective feature vectors f i (where i = 1, 2, ..., N),

[0015] The fused joint feature vector data is:

[0016]

[0017] Among them, f i is the feature vector extracted by the i-th sensor, F fusion is the multimodal feature after fusion; w i is the dynamic weight of the i-th sensor data, and N is the total number of sensors in the system.

[0018] Furthermore, the dynamic weight w of the i-th sensor data i The expression is:

[0019]

[0020] Among them, w i is the dynamic weight of the i-th sensor data, σ i is the uncertainty index of the i-th sensor data, α j is the uncertainty index of the j-th sensor data, α i is the adjustment parameter for regulating the influence of the uncertainty of the i-th sensor on the weight, σ j is the adjustment parameter for regulating the influence of the uncertainty of the jth sensor on the weight, N is the total number of sensors in the system, and i is the i-th sensor in the system;

[0021] The expression of adaptive uncertainty correction U is:

[0022]

[0023] Among them, U is the overall uncertainty, reflecting the comprehensive degree of noise of all sensor data, σ i is the uncertainty index of the i-th sensor data, and N is the total number of sensors in the system.

[0024] Furthermore, in step S3, the three-dimensional defect model expression is:

[0025] M 3D =R(F fusion )

[0026] Where R(·) is the reconstruction function, which usually relies on voxelization or grid reconstruction algorithms, and M 3D The generated three-dimensional defect model point cloud data.

[0027] Furthermore, the model data finally mapped to the VR system in step S4 includes nonlinear mapping optimization from the fusion features to the VR model:

[0028] min θ ||M VR -g θ (F fusion )|| 2

[0029] Among them, F fusion To fuse the feature vector, the feature information extracted by multiple sensors is integrated. θ is a nonlinear mapping function with parameter θ, which is used to map the fusion feature into a VR model. θ is the mapping function g θ The parameter set M needs to be learned through the optimization process. VR The three-dimensional model presented in the VR environment represents the spatial structure of the defect. 2 It is the square norm of a vector or matrix, and is usually used to measure the difference between two objects.

[0030] Furthermore, in step S4, the three-dimensional defect model data undergoes a coordinate transformation process, including a linear transformation from the original coordinates of the three-dimensional defect model data to the virtual reality VR coordinates, which is expressed as:

[0031] x VR =R·x orig +t

[0032] Among them, x origis the three-dimensional coordinate in the original model, R is the rotation matrix used to adjust the model direction, t is the translation vector used for positioning, x VR The coordinates in the VR coordinate system after mapping.

[0033] Furthermore, in step S5, the expression of the defect probability distribution data is:

[0034]

[0035] Among them, F(·) is the fusion function, which is the weighted average or other voting method, F fusion is the multimodal feature after fusion, P (l) (D|F fusion ) is the conditional probability estimate of the defect D at the l-th iteration, l is the iteration layer index, l=1,…,L.

[0036] Furthermore, the expression for multi-level iteration of the graph neural network GNN is:

[0037]

[0038] Among them, P (l) (D|F fusion ) is the conditional probability estimate of defect D at the lth layer iteration;

[0039] Z is the normalization factor to ensure probability normalization, l is the iteration layer index, l=1,…,L, L is the total number of iteration layers, F fusion is the multimodal feature after fusion;

[0040] The expression of the defect probability distribution of the preliminary output is:

[0041]

[0042] Among them, φ(·) is the nonlinear activation function, W1, W2 are the weight matrices of the fully connected layer, b1, b2 are the bias terms, and F fusion is the multimodal feature after fusion, Preliminary output of defect probability distribution.

[0043] Furthermore, in step S6, the expression for optimizing the interaction mapping is:

[0044]

[0045] Among them, g θ″ (·) is a mapping function with parameter θ″, which is used to map the fusion features and user input into the optimized VR interaction output, F fusion is the fused multimodal feature, is the representation of the current state of the model, and I int(t) is the output signal of the interactive interface at time t, which is used to drive the VR display, u(t) is the input vector of the detection person at time t, L int is the interaction error loss function, which measures the gap between the system output and the expectation, and T is the total duration of the interaction process.

[0046] Furthermore, the expression of dynamic interaction function data is:

[0047] I int (t) = h(F fusion ,u(t))

[0048] Among them, I int (t) is the output signal of the interactive interface at time t, which is used to drive the VR display. h(·,·) is the interactive mapping function that integrates the system state and user input information. fusion is the fused multimodal feature, is the representation of the current state of the model, and u(t) is the input vector of the detected person at time t.

[0049] The present invention has the following beneficial effects:

[0050] The present invention utilizes an adaptive uncertainty correction mechanism to achieve precise fusion of multi-source sensor data, effectively improving feature robustness and detection accuracy; based on point cloud reconstruction and coordinate transformation, it realizes high-precision three-dimensional reconstruction of the defect model and seamlessly maps it to the VR system for intuitive visualization; it integrates deep neural networks and graph neural networks to obtain stable and accurate defect probability estimation through multi-layer iterative updates; it constructs a time-continuously optimized interactive mapping function to achieve real-time, multimodal, closed-loop feedback interaction between inspection personnel and the system, greatly improving user experience and on-site operation efficiency.

[0051] By closely combining virtual reality with multimodal defect detection, the present invention not only achieves efficient defect detection, three-dimensional reconstruction and real-time visualization, but also constructs a complete interactive feedback closed loop, providing intuitive and dynamic decision-making support for inspectors, and has significant industrial application prospects and technological competitive advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is an exemplary flow chart of a multimodal defect detection and three-dimensional reconstruction interactive method based on virtual reality according to some embodiments of this specification;

[0053] Figure 2 Generate results for the defect heat map of the present invention; DETAILED DESCRIPTION

[0054] The following is a clear and complete description of the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] Please refer to Figure 1-2 The present invention provides a multimodal defect detection and three-dimensional reconstruction interactive method based on virtual reality, comprising the following steps:

[0056] S1. Obtaining data from ultrasonic sensors, infrared sensors, and visual sensors. In the implementation of the present invention, ultrasonic sensors, infrared sensors, and visual sensors are first used to collect data from industrial equipment. These sensors can capture different physical properties of the equipment surface, such as ultrasonic sensors detecting defects inside the material, infrared sensors identifying areas with abnormal temperature, and visual sensors capturing visible defects such as surface cracks and corrosion.

[0057] S2. Perform adaptive uncertainty correction and dynamic weighting on the ultrasonic sensor, infrared sensor, and visual sensor data to achieve data fusion processing, obtaining fused joint feature vector data. To fully utilize these sensor data, the present invention uses adaptive uncertainty correction and dynamic weighting to achieve data fusion. The specific steps are as follows:

[0058] Feature extraction: Extract feature vectors from the data collected by each sensor. These feature vectors can reflect the defect information detected by each sensor. Assume that the system collects N sensor data and extracts each feature vector f i (where i = 1, 2, ..., N),

[0059] Data fusion: According to the calculated weight w i , the feature vector f of each sensor i Perform weighted summation to obtain the fused multimodal feature F fusion , the fused feature vector will be used as the input of subsequent steps.

[0060] The fused joint feature vector data is:

[0061]

[0062] Among them, f i is the feature vector extracted by the i-th sensor, F fusion is the multimodal feature after fusion, w i is the dynamic weight of the i-th sensor data, and N is the total number of sensors in the system.

[0063] Sensor weight: Use dynamic weighting formula to calculate the weight w of each sensor i The size of the weight is inversely proportional to the uncertainty of the sensor data, that is, the smaller the uncertainty, the larger the weight.

[0064] The expression of each sensor weight is:

[0065]

[0066] Among them, w i is the dynamic weight of the i-th sensor data, σ i is the uncertainty index of the i-th sensor data, α j is the uncertainty index of the j-th sensor data, α i is the adjustment parameter for regulating the influence of the uncertainty of the i-th sensor on the weight, σ j is the adjustment parameter for regulating the influence of the uncertainty of the j-th sensor on the weight, N is the total number of sensors in the system, and i is the i-th sensor in the system.

[0067] By dynamic weighting, it can be ensured that sensors with high data quality play a greater role in the fusion process.

[0068] Uncertainty calculation: Calculate the uncertainty index U for each sensor data point. The uncertainty index reflects the noise level and reliability of the sensor data. Uncertainty can be calculated using statistical methods (such as variance and standard deviation) or uncertainty estimation models based on machine learning.

[0069] The expression defining global uncertainty is:

[0070]

[0071] Among them, U is the overall uncertainty, reflecting the comprehensive degree of noise of all sensor data, σ i is the uncertainty index of the i-th sensor data, and N is the total number of sensors in the system.

[0072] S3, performing point cloud reconstruction algorithm processing on the fused joint feature vector data to obtain three-dimensional defect model data;

[0073] 3D defect model reconstruction: based on fusion feature F fusion , a point cloud reconstruction algorithm is used to generate a 3D defect model. Point cloud reconstruction algorithms usually rely on voxelization or mesh reconstruction algorithms to map feature vectors into 3D space and connect adjacent points to form a continuous surface. Let the reconstruction function be R(·) and the input be the fusion feature F fusion , the output is the generated three-dimensional defect model point cloud data model.

[0074] The three-dimensional defect model expression is:

[0075] M 3D =R(F fusion )

[0076] Where R(·) is the reconstruction function, which usually relies on voxelization or grid reconstruction algorithms, and M 3D Point cloud data of the generated three-dimensional defect model;

[0077] S4, coordinate transformation processing is performed on the three-dimensional defect model data to obtain the three-dimensional defect model data in the VR system; VR coordinate mapping: In order to realize the intuitive visualization of the three-dimensional defect model in the VR system, the model needs to be mapped to the VR coordinate system. Through coordinate transformation, the three-dimensional coordinate x in the original model is converted to orig Convert to coordinate x in VR coordinate system VR The coordinate transformation includes two steps: rotation and translation, which are respectively realized by the rotation matrix R and the translation vector t. Let the three-dimensional coordinates of the original model be x orig , the coordinate in the VR coordinate system after mapping is x VR .

[0078] The expression of the three-dimensional defect model data in the VR system is:

[0079] x VR =R·x orig +t

[0080] Among them, x orig is the three-dimensional coordinate in the original model, R is the rotation matrix used to adjust the model direction, t is the translation vector used for positioning, x VR is the coordinate in the VR coordinate system after mapping;

[0081] Nonlinear mapping optimization: In order to improve the accuracy of VR mapping, a nonlinear mapping function is used for optimization. The nonlinear mapping function is trained through a neural network or other optimization algorithms to minimize the mapping error (such as the Euclidean norm). Let the nonlinear mapping function be g θ (·), the parameter is θ, and the output is the model M that is finally mapped to the VR system VR , then the nonlinear mapping function can be expressed as g θ (·) Optimize VR mapping accuracy.

[0082] Nonlinear mapping function g θ (·) The expression for optimizing VR mapping accuracy is:

[0083] min θ ||M VR -g θ (F fusion )|| 2

[0084] Among them, g θ (·) is a nonlinear mapping function with parameter θ, M VR is the model finally mapped to the VR system, ‖·‖ is the Euclidean norm, which is used to measure the mapping error.

[0085] S5. Perform multi-level iterative processing of deep neural network and graph neural network on the three-dimensional defect model data in the VR system to obtain defect probability distribution data;

[0086] The expression of defect probability distribution is:

[0087]

[0088] Among them, F(·) is the fusion function, which is the weighted average or other voting method, F fusion is the multimodal feature after fusion, P (l) (D|F fusion ) is the conditional probability estimate of the defect D at the l-th iteration, l is the iteration layer index, l=1,…,L.

[0089] In order to improve the prediction accuracy, a graph neural network (GNN) is used for multi-layer iterative update. The expression of multi-level iteration of the graph neural network GNN is:

[0090]

[0091] Among them, P (l) (D|F fusion ) is the conditional probability estimate of defect D at the lth layer iteration;

[0092] Z is the normalization factor to ensure probability normalization, l is the iteration layer index, l=1,…,L, L is the total number of iteration layers, F fusion is the multimodal feature after fusion;

[0093] Deep defect determination and iterative optimization: Based on the three-dimensional defect model in the VR system, through multi-level iterative updates of deep neural networks and graph neural networks, the defect probability is output to provide a basis for defect classification and positioning.

[0094] The expression of the defect probability distribution of the preliminary output is:

[0095]

[0096] Among them, φ(·) is the nonlinear activation function, W1, W2 are the weight matrices of the fully connected layer, b1, b2 are the bias terms, and F fusion is the multimodal feature after fusion, is the initial output defect probability distribution.

[0097] S6. Optimize the interactive mapping process on the joint feature vector data of step S2 to obtain dynamic interactive function data. This data completes the multimodal defect detection and three-dimensional reconstruction interaction based on virtual reality through the form of a mixed reality feedback mechanism.

[0098] VR interaction: Inspectors can enter a virtual environment by wearing a VR headset and interact with the system through natural interaction methods such as gestures and voice. The system adjusts the display content and interaction effects in real time based on the inspector's input, allowing inspectors to intuitively see the three-dimensional form of defects and the judgment results.

[0099] Mixed Reality Feedback: Using mixed reality technology, the system can integrate virtual defect models with the real inspection environment, allowing inspectors to see the virtual defect models within the real environment. This helps inspectors more accurately determine the location and shape of defects. The system also allows inspectors to annotate and adjust the virtual models, achieving closed-loop feedback and dynamic adaptive updates.

[0100] Dynamic interaction function: To achieve the time sequence mapping between the detection personnel and the model display, the present invention uses a dynamic interaction function to describe this mapping relationship. Let the output signal of the interactive interface be I int (t), used to drive VR display; the interaction mapping function is h(·,·), which integrates system state and user input information; the fused multimodal feature is F fusion , which represents the current state of the model; the input vector of the detector at time t is u(t) (such as gestures, voice commands, eye movements).

[0101] Then the expression of dynamic interaction function data is:

[0102] I int (t) = h(F fusion ,u(t))

[0103] Among them, I int (t) is the output signal of the interactive interface at time t, which is used to drive the VR display. h(·,·) is the interactive mapping function that integrates the system state and user input information. fusion is the fused multimodal feature, which represents the current state of the model, and u(t) is the input vector of the detection person at time t (such as gestures, voice commands, eye movement trajectory).

[0104] Time-continuous optimization: To optimize the interactive mapping, a time-continuous optimization objective is constructed. Let the mapping function be g θ″ (·), the parameter is θ″, which is used to map the fusion features and user input into the optimized VR interaction output; the interaction error loss function is L int, which measures the gap between the system output and the expectation. The total duration of the interaction process is T, and the time-continuous optimization objective can be expressed as:

[0105]

[0106] Among them, g θ″ (·) is a mapping function with parameter θ″, which is used to map the fusion features and user input into the optimized VR interaction output, F fusion is the fused multimodal feature, is the representation of the current state of the model, and I int (t) is the output signal of the interactive interface at time t, which is used to drive the VR display, u(t) is the input vector of the detection person at time t (such as gestures, voice commands, eye movement trajectory), L int is the interaction error loss function, which measures the gap between the system output and the expectation, and T is the total duration of the interaction process.

[0107] In the detection of a certain industrial storage tank, the present invention successfully applied the above technical solution. In the detection of a certain industrial storage tank, the system collects data through ultrasound, infrared and camera, and generates feature F after uncertainty assessment and dynamic weighting. fusion . Use the point cloud reconstruction function R(·) to construct a preliminary three-dimensional defect model M 3D , by mapping the rotation matrix R and the translation vector t to the VR coordinate system to obtain x VR After the tester wears the VR headset, he or she uses natural gestures and voice input (to form u(t), and the system adjusts the display interface I in real time through the mapping function h(·,·) int (t), and optimize g by iterative θ″ (·) Continuously update the interactive effect. Deep neural networks and graph neural networks continuously optimize the defect judgment output and P(D|F fusion ), ultimately generating accurate defect detection reports and repair suggestions.

[0108] Wearing a VR headset and entering a virtual environment, inspectors can clearly visualize the shape and distribution of defects on the tank's surface. Using gestures and voice input, inspectors annotate and adjust the virtual model, updating the results in real time based on the system's feedback. Ultimately, the system generates accurate defect detection reports and remediation recommendations, providing strong support for tank maintenance and repair.

[0109] During its application, this invention has demonstrated significant advantages. First, through multimodal data fusion and 3D reconstruction technology, it achieves high-precision detection of industrial equipment defects. Second, through VR interaction and mixed reality feedback mechanisms, it improves the interactive decision-making efficiency and work experience of inspectors. Finally, this invention also has good scalability and adaptability, and can be applied to more fields and scenarios.

[0110] In summary, this paper proposes a multimodal defect detection and 3D reconstruction interactive method based on virtual reality. This method successfully addresses the challenges of traditional defect detection methods by integrating multiple sensor data, implementing 3D reconstruction and VR coordinate mapping, performing deep defect determination and iterative optimization, and introducing VR interaction and mixed reality feedback mechanisms. In the future, as the technology continues to develop and its applications expand, this paper is expected to play a significant role in even more areas.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A virtual reality-based multimodal defect detection and 3D reconstruction interactive method, applied to structural health monitoring of storage tanks, pipelines, bridges or aerospace components, characterized by: The following steps are involved: S1. Acquire ultrasonic sensor, infrared sensor, and visual sensor data of target industrial equipment or infrastructure; S2. Performing adaptive uncertainty correction and dynamic weighting on the data of the ultrasonic sensor, infrared sensor, and visual sensor to achieve data fusion processing to obtain fused joint feature vector data; S3, performing point cloud reconstruction algorithm processing on the fused joint feature vector data to obtain three-dimensional defect model data; S4, performing coordinate transformation processing on the three-dimensional defect model data to obtain model data that is finally mapped to the VR system; S5. Perform multi-level iterative processing of deep neural network and graph neural network on the three-dimensional defect model data in the VR system to obtain defect probability distribution data; S6. Optimize the interactive mapping process on the joint feature vector data of step S2 to generate dynamic interactive function data for the target device or infrastructure. This data completes the multimodal defect detection and three-dimensional reconstruction interaction based on virtual reality through the form of a mixed reality feedback mechanism.

2. The method for interactive multimodal defect detection and three-dimensional reconstruction based on virtual reality according to claim 1, characterized in that: In step S2, Assume that the system collects N sensor data and extracts their respective feature vectors f i (where i = 1, 2, ..., N), The fused joint feature vector data is: Among them, f i is the feature vector extracted by the i-th sensor, F fusion is the multimodal feature after fusion; w i is the dynamic weight of the i-th sensor data, and N is the total number of sensors in the system.

3. The method for interactive multimodal defect detection and three-dimensional reconstruction based on virtual reality according to claim 2, characterized in that: The dynamic weight w of the i-th sensor data i The expression is: Among them, w i is the dynamic weight of the i-th sensor data, σ i is the uncertainty index of the i-th sensor data, α j is the uncertainty index of the j-th sensor data, α i is the adjustment parameter for regulating the influence of the uncertainty of the i-th sensor on the weight, σ j is the adjustment parameter for regulating the influence of the uncertainty of the jth sensor on the weight, N is the total number of sensors in the system, and i is the i-th sensor in the system; The expression of adaptive uncertainty correction U is: Among them, U is the overall uncertainty, reflecting the comprehensive degree of noise of all sensor data, σ i is the uncertainty index of the i-th sensor data, and N is the total number of sensors in the system.

4. The method for interactive multimodal defect detection and three-dimensional reconstruction based on virtual reality according to claim 1, characterized in that: In step S3, the three-dimensional defect model expression is: M 3D =R(F fusion ) Where R(·) is the reconstruction function, which usually relies on voxelization or grid reconstruction algorithms, and M 3D The generated three-dimensional defect model point cloud data.

5. The method for interactive multimodal defect detection and three-dimensional reconstruction based on virtual reality according to claim 1, characterized in that: The model data finally mapped to the VR system in step S4 includes nonlinear mapping optimization from fusion features to VR model, which is expressed as: life θ ||M VR -g θ (F fusion )|| 2 Among them, F fusion To fuse the feature vector, the feature information extracted by multiple sensors is integrated. θ is a nonlinear mapping function with parameter θ, which is used to map the fusion feature into a VR model. θ is the mapping function g θ The parameter set M needs to be learned through the optimization process. VR For the three dimensional model, representing the spatial structure of defects, |·| 2 It is the square norm of a vector or matrix, and is usually used to measure the difference between two objects.

6. The method for interactive multimodal defect detection and three-dimensional reconstruction based on virtual reality according to claim 1, characterized in that: In step S4, the three-dimensional defect model data undergoes a coordinate transformation process, including a linear transformation from the original coordinates of the three-dimensional defect model data to the virtual reality VR coordinates, which is expressed as: x VR =R·x orig +t Among them, x orig is the three-dimensional coordinate in the original model, R is the rotation matrix used to adjust the model direction, t is the translation vector used for positioning, x VR The coordinates in the VR coordinate system after mapping.

7. The method of multimodal defect detection and three-dimensional reconstruction interaction based on virtual reality according to claim 1, characterized in that: In step S5, the expression of the defect probability distribution data is: Among them, F(·) is the fusion function, which is the weighted average or other voting method, F fusion is the multimodal feature after fusion, P (l) (D|F fusion ) is the conditional probability estimate of the defect D at the l-th iteration, l is the iteration layer index, l=1,…,L.

8. The method for interactive multimodal defect detection and three-dimensional reconstruction based on virtual reality according to claim 6 or 7, characterized in that: The expression for multi-level iteration of graph neural network GNN is: Among them, P (l) (D|F fusion ) is the conditional probability estimate of defect D at the lth layer iteration; Z is the normalization factor to ensure probability normalization, l is the iteration layer index, l=1,…,L, L is the total number of iteration layers, F fusion is the multimodal feature after fusion; The expression of the defect probability distribution of the preliminary output is: Among them, φ(·) is the nonlinear activation function, W1, W2 are the weight matrices of the fully connected layer, b1, b2 are the bias terms, and F fusion is the multimodal feature after fusion, Preliminary output of defect probability distribution.

9. The method for interactive multimodal defect detection and three-dimensional reconstruction based on virtual reality according to claim 1, characterized in that: In step S6, the expression for optimizing the interaction mapping is: Among them, g θ″ (·) is a mapping function with parameter θ″, which is used to map the fusion features and user input into the optimized VR interaction output, F fusion is the fused multimodal feature, is the representation of the current state of the model, and I int (t) is the output signal of the interactive interface at time t, which is used to drive the VR display, u(t) is the input vector of the detection person at time t, L int is the interaction error loss function, which measures the gap between the system output and the expectation, and T is the total duration of the interaction process.

10. The method for interactive multimodal defect detection and three-dimensional reconstruction based on virtual reality according to claim 8, characterized in that: The expression of dynamic interaction function data is: I int (t)=h(F fusion ,u(t)) Among them, I int (t) is the output signal of the interactive interface at time t, which is used to drive the VR display. h(·,·) is the interactive mapping function that integrates the system state and user input information. fusion is the fused multimodal feature, is the representation of the current state of the model, and u(t) is the input vector of the detected person at time t.

Images