Structural test monitoring and early warning system and method based on digital twin and augmented reality

Abstract

The present invention discloses a structural test monitoring and early warning system and method based on digital twin and augmented reality. The structural test monitoring and early warning system includes a machine vision system, a test control system, a digital twin simulation system, an augmented reality server, and augmented reality glasses. The present invention uses a machine vision system and augmented reality glasses to collaboratively complete the construction of augmented reality coordinates and the estimation of the test piece's posture contour during the simultaneous movement of the observer and the test piece. The present invention uses a test monitoring system and a digital twin system in combination to analyze the full-field response and potential failure areas of the test piece in real time. The augmented reality server integrates real scenes with numerical analysis and test data, and the augmented reality glasses are used to superimpose data and real scenes to achieve human-computer interaction. This is of great significance for assisting test personnel in monitoring and early warning large-scale structural test processes, improving test efficiency and safety, and intuitively understanding test processes and phenomena.

Description

Technical Field

[0001] The present invention relates to a system and method for full-field monitoring and early warning of structures, and specifically to a system and method for monitoring the test process of large-scale structures and early warning of test operations based on digital twins and augmented reality. Background Art

[0002] In fields such as aerospace and rail transportation, in order to verify the reliability of structural product designs, it is often necessary to conduct mechanical tests or thermal assessment tests on the structures, such as full-scale static tests on aircraft wings, combined thermal tests on high-speed aircraft cabins, and thermal assessment tests on thermal protection structures. However, the testing of large-scale structural products is often very difficult, involving a large amount of loading design, multi-load linkage, etc., with a long test preparation cycle and high product costs. Especially for new product designs, due to insufficient understanding of structural characteristics, on the one hand, unexpected failures may occur during testing, resulting in losses of structural products and test equipment, and on the other hand, some key test phenomena may be overlooked during the test. Therefore, it is very necessary to enhance real-time and comprehensive monitoring of the test process of large-scale structural products.

[0003] Traditionally, structural test monitoring required simultaneous monitoring of multiple instruments and sensors. This inherently lags due to the need to track multiple data points and determine their correspondence with the test specimen. Furthermore, the limited number of sensors makes comprehensive monitoring of the test process difficult. This not only poses safety risks during the test, but also hinders the tester's ability to gain a comprehensive understanding of the test process, complicating subsequent test analysis. Currently, augmented reality (AR) technology is being applied to monitor the operating status of various types of mechanical equipment for operational warnings and maintenance assistance. However, unlike these applications, AR monitoring and early warning in structural testing faces the following challenges: 1) difficulty in overlaying virtual and real scenes. This can lead to numerous scene changes during the test due to changes in the test object and loading method, significant deformation of the virtual scene overlay object (i.e., the structural test specimen), and frequent movement of the tester (i.e., the observer). 2) The limited number of measurement points installed on the structure cannot provide comprehensive information about the test specimen, potentially overlooking unforeseen potential failure areas, making it difficult to directly utilize for comprehensive structural monitoring and early warning. Summary of the Invention

[0004] The purpose of the present invention is to provide a structural test monitoring and early warning system and method based on digital twins and augmented reality. The system and method can provide test personnel with a combination of virtual and real test processes and full-field response information of test pieces during the structural product test process, assisting in test process monitoring and early warning.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A structural test monitoring and early warning system based on digital twin and augmented reality includes a machine vision system, a test control system, a digital twin simulation system, an augmented reality server, and augmented reality glasses, wherein:

[0007] The machine vision system is used to complete image acquisition of real test scenes and test structural parts;

[0008] The test control system is used to complete the test load loading and sensor data collection installed on the test piece;

[0009] The digital twin simulation module is used to obtain test load and sensor data from the test control system and complete the online simulation of the structural response of the test piece under the test load;

[0010] The augmented reality server is used to integrate data from the machine vision system, the test control system, and the digital twin simulation system to construct the real scene coordinates and the deformed model of the test piece, and to superimpose the test piece model and the test piece response data to construct a virtual scene;

[0011] The augmented reality glasses are used to obtain data from the augmented reality server and complete scene rendering and output images to complete human-computer interaction.

[0012] A structural test monitoring and early warning method based on digital twin and augmented reality is implemented using the above system, comprising the following steps:

[0013] Step 1) Using a machine vision system to acquire an image of the actual test scene, the deformed contour of the test structure is measured and transmitted to an augmented reality server. The augmented reality server then constructs the coordinates corresponding to the actual test scene and the contour coordinates of the test structure.

[0014] Step 2) Load the test structure through the test control system, synchronously obtain sensor data during the test, drive the digital twin system to perform online simulation calculations, obtain the full-field temperature, deformation, stress, and strain response data of the test structure during the test, and mark the hot spots of the test piece according to the failure criteria;

[0015] Step 3) The augmented reality server receives the load and sensor data from the test control system, the full-field strain and stress response data from the digital twin system, and the hot spot area information of the test structure, and superimposes them on the deformed surface of the test structure to construct a virtual scene;

[0016] Step 4) The tester uses augmented reality glasses to observe the test structure. The augmented reality glasses acquire an image of the actual test scene in the background and establish the observer's coordinates in real time. The glasses then send the coordinate information and data requirements to the augmented reality server. The data returned by the augmented reality server is then rendered as an image and superimposed on the surface of the actual test structure on the display screen.

[0017] Step 5) The test personnel monitor the test status and issue early warnings based on the virtual-real synchronization information.

[0018] In the present invention, a machine vision system and augmented reality glasses are used to collaboratively complete the augmented reality coordinate construction and test piece pose and morphology estimation when the observer and the test structure are moving simultaneously. The machine vision system is located in a fixed position and is used to complete the test area coordinate construction and test structure contour measurement, and the augmented reality glasses are used to establish the coordinates during the observer's movement.

[0019] In the present invention, the digital twin system performs simulation based on physical mechanisms, reconstructs the full-field response of the test structure to enhance the discrete sensor data obtained by the test control system. The test personnel can set the failure criteria of the test structure to realize real-time analysis and marking of hot spots, and assist the test personnel in visual early warning of the test.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] The present invention uses a machine vision system and augmented reality glasses to collaboratively complete the construction of augmented reality coordinates and the estimation of the test piece's posture contour during the simultaneous movement of the observer and the test piece. It uses a test monitoring system combined with a digital twin system to analyze the full-field response and potential failure areas of the test piece in real time. The augmented reality server integrates real scenes with numerical analysis and test data, and the augmented reality glasses are used to superimpose data and real scenes to achieve human-computer interaction. This is of great significance for assisting test personnel in monitoring and early warning of large-scale structural test processes, improving test efficiency and safety, and intuitively understanding test processes and phenomena. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a structural diagram of the structural test monitoring and early warning system based on digital twins and augmented reality. In the figure: ① machine vision system, ② test control system, ③ digital twin simulation system, ④ augmented reality server, ⑤ augmented reality glasses. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0024] The present invention provides a structural test monitoring and early warning system based on digital twin and augmented reality, such as Figure 1 As shown in the figure, the system consists of five subsystems: machine vision system, test control system, digital twin simulation system, augmented reality server, and augmented reality glasses. The above subsystems are integrated into an overall system, in which:

[0025] The machine vision system is used to complete image acquisition of real test scenes and test structural parts;

[0026] The test control system is used to complete the test load loading and sensor data collection installed on the test piece;

[0027] The digital twin simulation module is used to obtain test load and sensor data from the test control system and complete the online simulation of the structural response of the test piece under the test load;

[0028] The augmented reality server is used to integrate data from the machine vision system, the test control system, and the digital twin simulation system to construct the real scene coordinates and the deformed model of the test piece, and to superimpose the test piece model and the test piece response data to construct a virtual scene;

[0029] The augmented reality glasses are used to obtain data from the augmented reality server and complete scene rendering and output images to complete human-computer interaction.

[0030] In the present invention, the machine vision system is installed in a fixed position and completes two functions through multi-eye image acquisition and image identification on the test structure, namely: completing image acquisition of the test area and the test structure through two cameras, and performing coordinate identification of the test area and posture contour measurement of the test structure after deformation.

[0031] In the present invention, the digital twin simulation system runs on a high-performance computing platform, adopts ANSYS DiscoveryLive software and interface development, and establishes a two-way data connection with the test control system and the augmented reality server.

[0032] In the present invention, the digital twin simulation system uses the load and sensor data provided by the test control system to run a simulation of the test process while the test is in progress, and calculates the full-field temperature, deformation, stress, and strain response data of the test structure in real time, realizes real-time analysis and marking of potential failure hotspots of the test structure, and transmits the data to the augmented reality server, wherein: the simulation adopts a model based on physical mechanisms corresponding to the test conditions, and combines the sensor data provided by the test control system to perform real-time verification of the model, and uses the verified data to calculate the full-field temperature, deformation, stress, and strain response data of the test structure in real time, and reconstructs the full-field response of the test structure to enhance the discrete sensor data obtained by the test control system.

[0033] In the present invention, the augmented reality server is a real scene and data integration and processing unit, and uses Vuforia to build an augmented reality application for the test system, which is used to receive the 3D contour coordinates of the deformed test structure given by the machine vision system, receive the sensor data transmitted by the test control system, and the structural temperature, deformation, stress, and strain response data calculated by the digital twin simulation system, and integrate the data with the 3D contour of the test structure after deformation to construct a virtual scene. According to the requirements of the augmented reality glasses, the virtual scene is transformed to the observer coordinates and transmitted to the augmented reality glasses via a wireless network.

[0034] In the present invention, the augmented reality glasses use their own cameras to obtain real-scene image information, establish observer coordinates based on the camera angle and test area identification, read the test structure response information provided by the augmented reality server and perform image rendering, and superimpose it on the deformed test structure surface on the display screen, thereby assisting test personnel in performing test visualization warning.

[0035] A structural test monitoring and early warning method based on digital twin and augmented reality is implemented using the above system, comprising the following steps:

[0036] Step 1) Using a machine vision system to acquire an image of the actual test scene, the deformed contour of the test structure during the test is measured using image features such as speckle and light spot on the test structure, and the image is transmitted to an augmented reality server. The augmented reality server then constructs coordinates corresponding to the actual test scene and the contour coordinates of the test structure.

[0037] Step 2) Import the geometric model and material constitutive model of the test structure into the digital twin simulation system, set boundary conditions consistent with the test load, set the Mises strength criterion for metal structures, set the Tsai-Hill criterion for composite structures, set experimentally determined phenomenological criteria for other complex materials, and set a maximum temperature criterion for heat-resistant materials. Use finite element technology to simulate the test process, load the test piece through the test control system, and synchronously obtain sensor data during the test process to drive the digital twin system to perform online simulation calculations. Combined with the test structure deformation and local strain sensor measurement results provided by the test control system, the simulation results of the same location provided by the simulation model are compared. The material parameters in the digital twin simulation system are adjusted through optimization methods to ensure that the simulation results are consistent with the test. Using the corrected model, obtain the full-field temperature, deformation, stress, and strain response data of the test structure during the test. Mark the hot spots of the test structure according to the failure criterion, and extract the full-field response information and corresponding coordinate data of the test structure surface, and transmit them to the augmented reality server.

[0038] Step 3) Receive the load and sensor data from the test control system, the full-field strain and stress response data from the digital twin system, and the hot spot area information of the test structure through the augmented reality server, establish the correspondence between the simulation data, experimental test data and the coordinates of the actual test structure, and build a virtual scene.

[0039] Step 4) The test personnel use augmented reality glasses to observe the test structure. The augmented reality glasses obtain the image of the real test scene in the background and establish the observer's coordinates in real time. The coordinate information and data requirements are sent to the augmented reality server. The image is rendered based on the data returned by the augmented reality server and superimposed on the surface of the real test structure on the display screen.

[0040] Step 5) The test personnel monitor the test status and issue early warnings based on the virtual-real synchronization information.

Claims

1. A structural test monitoring and early warning system based on digital twin and augmented reality, characterized by The structural test monitoring and early warning system includes a machine vision system, a test control system, a digital twin simulation system, an augmented reality server, and augmented reality glasses, wherein: The machine vision system is used to complete image acquisition of real test scenes and test structural parts; The test control system is used to complete the test load loading and sensor data collection installed on the test piece; The digital twin simulation module is used to obtain test load and sensor data from the test control system and complete the online simulation of the structural response of the test piece under the test load; The augmented reality server is used to integrate data from the machine vision system, the test control system, and the digital twin simulation system to construct the real scene coordinates and the deformed model of the test piece, and to superimpose the test piece model and the test piece response data to construct a virtual scene; The augmented reality glasses are used to obtain data from the augmented reality server and complete scene rendering and output images to complete human-computer interaction; The digital twin simulation system uses the load and sensor data provided by the test control system to run a simulation of the test process while the test is in progress, and calculates the full-field temperature, deformation, stress, and strain response data of the test structure in real time. It realizes real-time analysis and marking of potential failure hotspots of the test structure, and transmits the data to the augmented reality server; The augmented reality server is a real-world scene and data integration and processing unit. It uses Vuforia to build an augmented reality application for the test system, which is used to receive the 3D contour coordinates of the deformed test structure provided by the machine vision system, receive sensor data transmitted by the test control system, and receive structural temperature, deformation, stress, and strain response data calculated by the digital twin simulation system. It integrates the data with the 3D contour of the deformed test structure to construct a virtual scene. The virtual scene is transformed to the observer coordinates according to the requirements of the augmented reality glasses and transmitted to the augmented reality glasses via a wireless network.

2. The structural test monitoring and early warning system based on digital twin and augmented reality according to claim 1 is characterized in that The machine vision system is installed in a fixed position and completes image acquisition of the test area and the test structure through two cameras, performs coordinate identification of the test area and posture contour measurement of the test structure after deformation.

3. The structural test monitoring and early warning system based on digital twin and augmented reality according to claim 1 is characterized in that The simulation uses a model based on physical mechanisms corresponding to the test conditions, and combines it with sensor data provided by the test control system to perform real-time verification of the model. The verified data is used to calculate the full-field temperature, deformation, stress, and strain response data of the test structure in real time, and the full-field response of the test structure is reconstructed to enhance the discrete sensor data obtained by the test control system.

4. The structural test monitoring and early warning system based on digital twin and augmented reality according to claim 1 is characterized in that The augmented reality glasses use their own cameras to obtain real-scene image information, establish observer coordinates based on the camera angle and test area identification, read the test structure response information provided by the augmented reality server and perform image rendering, and superimpose it on the deformed test structure surface on the display screen, thereby assisting test personnel in visual warning of the test.

5. A structural test monitoring and early warning method based on digital twin and augmented reality is implemented by using the structural test monitoring and early warning system according to any one of claims 1 to 4, characterized in that The structural test monitoring and early warning method comprises the following steps: Step 1) Acquire an image of the real test scene through a machine vision system, measure the deformed contour of the test structure during the test, and transmit it to an augmented reality server, which constructs coordinates corresponding to the real test scene and the contour coordinates of the test structure; Step 2) Loading the test structure through the test control system, synchronously acquiring sensor data during the test process, driving the digital twin system to perform online simulation calculations, acquiring full-field temperature, deformation, stress, and strain response data of the test structure during the test, and marking the hot spots of the test piece according to the failure criterion; Step 3) The augmented reality server receives the load and sensor data from the test control system, the full-field strain and stress response data from the digital twin system, and the hot spot area information of the test structure, and superimposes them on the surface of the deformed test structure to construct a virtual scene; Step 4) The test personnel use augmented reality glasses to observe the test structure. The augmented reality glasses obtain an image of the real test scene in the background and establish the observer's coordinates in real time. The augmented reality glasses send the coordinate information and data requirements to the augmented reality server. The augmented reality server then renders the image based on the data returned and overlays it on the surface of the real test structure on the display screen. Step 5) The test personnel monitor the test status and issue early warnings based on the virtual-real synchronization information.

Images