An intelligent operation and maintenance system for the whole life cycle of large steel structures

Abstract

The present invention relates to the technical field of operation and maintenance of large steel structures, and discloses an intelligent operation and maintenance system for the whole life cycle of large steel structures, including an intelligent recognition and diagnosis module for obtaining inspection images taken by inspection equipment during the inspection process and performing damage recognition and positioning based on a pre-trained damage recognition model; a digital twin model construction module for constructing a digital twin model and performing online synchronization based on strain, temperature or vibration distribution data obtained by a real-time data acquisition system; and a digital twin model visualization module for, in response to a visualization request, obtaining the current digital twin model and synchronizing the damage recognized based on the inspection images, as well as the information of the inspection equipment, welding equipment and repair equipment in the working state, to the digital twin model and performing visualization. On the basis of realizing the digital twin model, the present invention superimposes the damage and related operation and maintenance equipment, and can more intuitively remotely display the real-time state of the large steel structure to the operation and maintenance personnel.

Description

Technical Field

[0001] The present invention belongs to the technical field of operation and maintenance of large steel structures, and particularly relates to an intelligent operation and maintenance system for the whole life cycle of large steel structures. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] Large steel structures have the characteristics of irregular component shapes, complex connections among numerous members, and interlaced occlusion. It is difficult for a single automated inspection device to achieve full coverage of the field of view. For example, drones and ground inspection robots have difficulty taking into account the internal members with occlusion, especially because many steel structures have external enclosures and internal ceilings and are simply not applicable. Therefore, the inspection of large steel structures still mainly relies on manual methods, which require high-altitude operations and pose great risks. At the same time, manual inspections usually rely on the experience of operation and maintenance personnel. When they subjectively judge that there may be damage at a certain position, they then use inspection equipment to check, which is prone to missed inspections. Moreover, people cannot reach places with narrow spaces, so full coverage cannot be achieved. In addition, manual methods require high professional knowledge of operation and maintenance personnel, who need to be familiar with the action mechanism of various loads on steel structures and the applicability of different inspection tools. The above situations all pose challenges such as one-sidedness, staticness, passivity, and blindness to the operation and maintenance of complex large steel structures. Summary of the Invention

[0004] To overcome the deficiencies of the above-mentioned prior art, the present invention provides an intelligent operation and maintenance system for the whole life cycle of large steel structures. On the basis of realizing the online synchronization of the digital twin model, by superimposing the damage recognition and analysis results obtained from inspection images, the real-time state of the large steel structure can be more intuitively displayed to operation and maintenance personnel remotely.

[0005] To achieve the above object, one or more embodiments of the present invention provide an intelligent operation and maintenance system for the whole life cycle of large steel structures, including a real-time data acquisition system, inspection and repair equipment, and an integrated platform. Among them, the integrated platform is configured to include:

[0006] A collaborative control module for collaboratively controlling inspection equipment information, welding equipment, and repair equipment;

[0007] An equipment status monitoring module for obtaining the real-time positioning of inspection equipment, welding equipment, and repair equipment;

[0008] An intelligent identification and diagnosis module for obtaining inspection images taken by inspection equipment during inspection, and performing damage identification and positioning on the inspection images based on a pre-trained damage recognition model;

[0009] The digital twin model construction module is used to construct a digital twin model and perform online synchronization based on the strain, temperature, or vibration distribution data obtained by the real-time data acquisition system;

[0010] The digital twin model visualization module is used to respond to a visualization request, obtain the current digital twin model, and synchronize the damage recognized from the inspection images, as well as the inspection device information, welding device, and repair device in the working state, to the digital twin model and perform visualization.

[0011] In some embodiments, the inspection devices include drones and wall-climbing robots; the inspection path planning method based on the drones and wall-climbing robots includes:

[0012] Obtain the three-dimensional point cloud data of the large steel structure, and according to the flight height and field of view range of the given drone, determine the point cloud set that can fall within the field of view of the drone to obtain the inspection range of the drone; the point cloud set outside the inspection range of the drone is the inspection range of the wall-climbing robot;

[0013] Respectively plan the inspection paths of the drones and wall-climbing robots within the inspection ranges of the drones and wall-climbing robots.

[0014] In some embodiments, planning the inspection path of the drone within the inspection range of the drone includes:

[0015] Extract the outer layer data from the point cloud data within the inspection range of the drone and expand it by a set distance, and fit to obtain the inspection flight surface of the drone;

[0016] Cluster the steel structure point cloud data within the inspection range of the drone to obtain the point cloud data of multiple steel segments. According to the point cloud data of each steel segment, extract the center line to obtain the center line space network; perform three-dimensional grid division on the center line space network, and for each grid through which the center line passes, extract a detection view point;

[0017] Cluster all the intersection points according to the set distance threshold to obtain multiple intersection point clusters, and use the clustering center of each intersection point cluster as an inspection fixed-point position of the drone.

[0018] In some embodiments, planning the inspection path of the wall-climbing robot within the inspection range of the wall-climbing robot includes:

[0019] Taking the set position as the origin, construct a space coordinate system to obtain the coordinate information of the point cloud data; according to the inspection range of the wall-climbing robot, obtain the boundary coordinate information of the inspection range of the wall-climbing robot;

[0020] Divide the steel structure point cloud data within the boundary range into steel segments, project all steel segments onto a set two-dimensional plane, and calculate the projection matrix; extract detection viewpoints for each projected steel segment through grid sampling, and calculate the line-of-sight directions corresponding to all detection viewpoints based on the plane normal vector feature;

[0021] According to the projection matrix, restore all viewpoints and line-of-sight directions to the three-dimensional space, obtain the intersections of all line-of-sight directions and the nearest steel segments, and determine the steel segments where these intersections are located; these steel segments are recorded as the steel segments to be crawled, and these intersections are recorded as the inspection fixed-point positions of the wall-climbing robot.

[0022] In some embodiments, the digital twin model construction method includes:

[0023] Construct an in-service steel structure geometric model based on multi-viewpoint cloud data and design drawings;

[0024] Obtain multi-source heterogeneous data reflecting the health status of the steel structure;

[0025] Perform cross-modal fusion on the multi-source heterogeneous data to obtain cross-modal fusion features, specifically including:

[0026] Obtain the strain-temperature feature mapping and vibration feature mapping based on the multi-source heterogeneous data of the steel structure;

[0027] Calculate the characterization value of the information at any position in the strain-temperature feature mapping as a unary function;

[0028] Calculate the correlation between the information at any position in the strain-temperature feature mapping and the information at any position in the vibration feature mapping in the form of an embedded Gaussian function as a binary function;

[0029] Perform cross-modal fusion based on the unary function and the binary function to obtain cross-modal fusion features;

[0030] Combine the steel structure geometric model data and the cross-modal fusion feature data to construct a digital twin model corresponding to the steel structure geometric model;

[0031] Perform dynamic prediction and correction on the constructed digital twin model to obtain the corrected digital twin model.

[0032] In some embodiments, the formula for performing cross-modal fusion based on the unary function and the binary function to obtain cross-modal fusion features is:

[0033]

[0034] Among them, the unary function is used to calculate the characterization value of the feature mapping at the position of the information The binary function Calculate the feature map in the form of an embedded Gaussian function at the position information at and the feature map at the position information at correlation.

[0035] In some embodiments, the digital twin model corresponding to the steel structure geometric model is constructed by combining the steel structure geometric model data and the cross-modal fusion feature data, including:

[0036] Align the steel structure geometric model data and the cross-modal fusion feature data in the time dimension and the space dimension;

[0037] Train the deduction model based on the aligned steel structure geometric model data and the fusion feature data before and after the change of the steel structure parameters to obtain the trained deduction model;

[0038] Combine the multi-source heterogeneous data reflecting the health status of the steel structure monitored in real time and the finite element simulation analysis results, and use the trained deduction model to perform the mapping from the monitoring data to the three-dimensional model, that is, complete the construction of the digital twin model.

[0039] In some embodiments, the constructed digital twin model is dynamically predicted and corrected to obtain the corrected digital twin model. The correction is based on the influence differences of multi-source data on different nodes, and different weights are selected for multi-source data fusion to achieve the goal of minimizing the sum of the squares of the Euclidean distances between the fusion result and each data.

[0040] In some embodiments, the operation and maintenance system further includes a damage alarm module, which is used to send an alarm message when damage is identified. The alarm message includes the damage image and the position information of the steel section where the damage is located.

[0041] In some embodiments, the operation and maintenance system further includes an intelligent repair module, which is configured to control the repair robot and the welding robot to perform autonomous repair on the steel structure according to the positioning of the damage, as well as the damage type and the degree of damage.

[0042] In the above one or more technical solutions, the strain, temperature or vibration distribution data, and the inspection image data are respectively obtained. On the basis of realizing the online synchronization of the digital twin model, by superimposing the damage recognition results obtained from the inspection images, the real-time state of the large steel structure can be more intuitively remotely displayed to the operation and maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings forming a part of this invention are used to provide a further understanding of the invention. The schematic embodiments of the invention and their descriptions are used to explain the invention and shall not unduly limit the invention.

[0044] Figure 1 It is a functional architecture diagram of the intelligent operation and maintenance system for the whole life cycle of large steel structures in the embodiments of the present invention;

[0045] Figure 2 It is the overall flow chart of the path planning method in the embodiments of the present invention;

[0046] Figure 3 It is a schematic diagram of the principle of unmanned aerial vehicle path planning in the embodiments of the present invention;

[0047] Figure 4 It is a flow chart of the method for constructing a digital twin model of large steel structures integrating multi-source data in the embodiments of the present invention;

[0048] Figure 5 It is a schematic diagram of the real-time data acquisition system provided by the embodiments of the present invention;

[0049] Figure 6 It is a schematic diagram of cross-modal fusion features provided by the embodiments of the present invention. Detailed implementation manners

[0050] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not used to limit the protection scope of the present application.

[0051] In the description of the embodiments of the present application, the term "including" and its like terms should be understood as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on".

[0052] In view of the problems existing in the current operation and maintenance of large steel structures, one or more embodiments of the present invention provide an intelligent operation and maintenance system for the whole life cycle of large steel structures, including a real-time data acquisition system, inspection equipment, repair equipment, an integrated platform, and a client. The real-time data acquisition system, inspection equipment, and repair equipment are all connected to the integrated platform, and the client can establish a communication connection with the integrated platform to access the integrated platform.

[0053] The real-time data acquisition system uses a steel structure key parameter monitoring system based on distributed optical fiber sensing to collect real-time monitoring data. The system includes a pulsed light generator, a circulator, a wavelength division multiplexer, two photodetectors, a sensing optical fiber, and a computer. The computer sends control signals to the pulsed light generator and the photodetectors and simultaneously starts collecting signals through three channels. The pulsed light generator emits pulsed light, which reaches the sensing optical fiber disposed at a set position on the surface of the steel structure through the circulator and the wavelength division multiplexer. The backward Rayleigh reflected light is collected by the first photodetector through the wavelength division multiplexer and the circulator. The backward Brillouin reflected light includes Stokes light and anti-Stokes light. Due to different wavelengths, the Stokes light and the anti-Stokes light are respectively collected by the second photodetector through the wavelength division multiplexer. After the second photodetector collects the reflected light signal, it converts it into an electrical signal and transmits it to the main control computer for analysis, so as to obtain the strain, temperature or vibration distribution on the surface of the steel structure at the current moment, and obtain multi-source heterogeneous data reflecting the health status of the steel structure.

[0054] As an example, as Figure 5 shown, when the system is working normally, the main control computer sends control signals to the pulsed light generator and the photodetectors (APD1, APD2) and simultaneously through three channels (Channel 1, Channel 2, and Channel 3), and collects signals through a data acquisition card. The pulsed light generator emits infrared pulsed light with a central wavelength in the C band, which reaches the sensing optical fiber disposed at a set position on the surface of the steel structure through the circulator and the wavelength division multiplexer. Due to changes in external stress, temperature, or vibration, the scattering information of the pulsed light in the sensing optical fiber changes. This system mainly collects backward Rayleigh scattered light and two-way backward Brillouin scattered light. Among them, the backward Rayleigh reflected light is collected by the photodetector APD1 through the wavelength division multiplexer and the circulator. The Stokes light and the anti-Stokes light in the backward Brillouin reflected light are respectively collected by the photodetector APD2 through the wavelength division multiplexer. After the photodetector collects the reflected light signal, it converts it into an electrical signal and transmits it to the main control computer. After the main control computer performs signal preprocessing such as denoising and decoupling on the collected signals, it generates corresponding strain, temperature, or vibration curve graphs, and obtains the strain, temperature, or vibration distribution on the surface of the steel structure at the current moment.

[0055] When the system detects abnormal strain, temperature, or vibration at a certain position, the location where the steel structure damage occurs can be determined through signal decoupling analysis. Secondly, it can be determined whether the optical fiber currently disposed on the surface of the steel structure is broken or bent by analyzing the intensity of the scattered signal, ensuring the reliability of the acquired data.

[0056] The inspection device is configured to:

[0057] Perform inspections based on a set inspection path. During the inspection process, the fixed-point inspection images are transmitted back to the integrated platform.

[0058] To reduce the data transmission pressure, a preliminary defect discrimination model can be built into the inspection equipment to initially screen the captured images during the inspection process, exclude normal images, and only transmit the suspected defect images to the integrated platform. As an example, images of defect-free steel segments and defective steel segments are used to train a binary classification model based on the support vector machine (SVM), which only distinguishes between the presence and absence of defects. The images of the defective steel segments determined to be defective are used as suspected defective steel segment images and transmitted to the integrated platform.

[0059] As Figure 1 shown, the integrated platform is configured as follows:

[0060] A data storage subsystem, which is used to obtain the fixed-point inspection images at each point, as well as the strain, temperature, or vibration distribution of the steel structure surface at each moment, and store them. Based on this, for each fixed point of the inspection, inspection images throughout the life cycle can be obtained. It can be understood that in order to ensure the transparency and traceability of each link, technologies such as blockchain, access passwords, and digital digests can be used to combine the information throughout the life cycle in the cloud to build a full-chain information traceability mechanism, which helps to analyze the evolution mechanism of different types of damage to large steel structures and provides a data basis for building an inversion model for large steel structure damage identification.

[0061] An equipment control and management subsystem, which is configured to include:

[0062] An equipment information management module, which is used to manage the inspection equipment information, welding equipment, and repair equipment. The inspection equipment includes one or more wall-climbing robots and / or one or more drones. The welding equipment is, for example, a welding robot, and the repair equipment is, for example, a wall-climbing rust repair robot, which is not specifically limited here.

[0063] A collaborative control module, which is used to collaboratively control the inspection equipment information, welding equipment, and repair equipment;

[0064] An equipment status monitoring module, which is used to obtain the real-time positioning and real-time operating parameters of the inspection equipment, welding equipment, and repair equipment, and judge whether they are operating normally and whether they are operating along the specified path.

[0065] An inspection execution subsystem, which is configured to include:

[0066] An inspection path planning module, which is used to plan the inspection paths for one or more inspection equipment for large steel structures to obtain the inspection paths of each inspection equipment.

[0067] As a specific implementation method, the inspection equipment includes two types: drones and wall-climbing robots. An inspection path planning method of air-ground cooperation is adopted. As Figure 2 shown, it specifically includes:

[0068] Step 1: Obtain the 3D point cloud data of the large steel structure. According to the flight height and field of view range of the given drone, determine the point cloud set that can fall within the field of view of the drone to obtain the inspection range of the drone; the point cloud set outside the inspection range of the drone is the inspection range of the wall - climbing robot.

[0069] Step 2: Plan the inspection paths of the drone and the wall - climbing robot within the inspection ranges of the drone and the wall - climbing robot respectively.

[0070] In the above - mentioned Step 1, first, taking the set position as the origin, construct a spatial coordinate system to obtain the coordinate information of the point cloud data; as an example, the center point of the ground of the steel - structure building can be taken as the origin. The inspection range of the drone is mainly analyzed according to performance parameters such as the detection accuracy and monitoring range of the drone. The flight height of the drone is the maximum distance between the drone and the steel structure, which is related to the resolution of the vision system carried on it. Within the maximum flight height of the drone, the height at which a given - size defect can be clearly photographed is used as the standard. The field of view range is related to the field of view of the vision system and the degrees of freedom of the gimbal where it is located. When the parameters of the drone, its gimbal, and the vision system are determined, the flight height and the field of view range can be calculated.

[0071] In the above - mentioned Step 2, as Figure 3 shown, the specific steps for planning the inspection path of the drone within the inspection range of the drone include:

[0072] (1) According to the point cloud data within the inspection range of the drone, extract the outer - layer data and expand it by a set distance, and fit to obtain the inspection flight plane of the drone. The set distance is the distance between the drone and the steel structure during the set inspection of the drone. It can be understood that when taking the center point of the ground of the steel - structure building as the origin, on the same ray direction starting from the coordinate origin, the point with the farthest distance from the coordinate origin is recorded as the outer - layer data.

[0073] (2) Cluster the steel - structure point cloud data within the inspection range of the drone to obtain the point cloud data of multiple steel segments. According to the point cloud data of each steel segment, extract the center line to obtain the center - line spatial network; perform three - dimensional grid (cubic grid) division on the center - line spatial network. For each grid through which the center line passes, extract a detection view point.

[0074] (3) For each detection view point, calculate the intersection point when its distance from the inspection flight plane of the drone is the shortest, and record the direction from the detection view point to this intersection point as the line - of - sight direction.

[0075] (4) According to the set distance threshold, cluster all the intersection points to obtain multiple intersection - point clusters, and use the clustering center of each intersection - point cluster as a fixed inspection position of the drone. The drone takes pictures of the corresponding view points at this fixed position.

[0076] (5) Perform the shortest path planning based on all the inspection fixed-point positions of the UAV.

[0077] By extracting the center line within the inspection range and performing 3D grid division, a viewing point is extracted from each 3D grid, ensuring that when the UAV conducts inspections later, the field of view can cover all the steel structure segments within the entire inspection range, avoiding missing perspectives or missing defects.

[0078] After obtaining all the inspection fixed-point positions of the UAV, it is also necessary to organize them into a coherent path through path planning based on these inspection fixed-point positions. As an example, the ant colony algorithm can be used for path planning.

[0079] Since the wall-climbing robot works by attaching to the surface of the steel structure and has a limited detection field of view, in order to balance work efficiency and energy conservation, on the basis of ensuring that the field of view of the wall-climbing robot can cover the entire inspection range, the crawling path should be made as short as possible.

[0080] Based on this, the planning of the inspection path of the wall-climbing robot within its inspection range specifically includes:

[0081] (1) Cluster the point cloud data of the steel structure within the inspection range of the wall-climbing robot to obtain the point cloud data of multiple steel segments. According to the point cloud data of each steel segment, extract the center line to obtain the center line spatial network; perform 3D grid (cubic grid) division on the center line spatial network. For each grid through which the center line passes, extract a detection viewing point;

[0082] (2) For each detection viewing point, determine the multiple steel segments that can be connected without obstacles, that is, for each detection viewing point, a set of steel segments that can be visualized by it can be obtained. Analyze these sets of steel segments, and preferentially select the steel segment with the most visible viewing points, with the goal of minimizing the number of steel segments, to obtain the steel segment corresponding to each viewing point;

[0083] (3) These steel segments are recorded as the steel segments to be crawled, and the intersection points of the detection viewing points to the corresponding steel segments are recorded as the inspection fixed-point positions of the wall-climbing robot, which are used to take pictures of the viewing points on the corresponding visual lines.

[0084] (4) Perform the shortest path planning based on the steel segments to be crawled and the fixed-point inspection positions on each steel segment. It can be understood that the shortest path is a coherent path that covers all the steel segments to be crawled and the fixed-point inspection positions on each steel segment.

[0085] Since it is necessary to ensure that the wall - climbing robot passes through all the sections to be crawled and repeated passage through steel sections is allowed, the above - mentioned path - planning problem can be equivalent to the arc routing problem, that is, it is required to find a shortest path such that each steel section to be crawled in the steel structure is passed through at least once. As an example, the following method can be adopted: establish an undirected road network graph based on all the steel sections to be crawled, with the endpoints of each steel section as the nodes of the graph and the steel sections as the edges connecting the two nodes. The length of each steel section determines the weight of the edge, and use the existing solutions for solving problems under large - scale road networks for path planning.

[0086] Solve the path based on the above - mentioned method, and smoothly connect all the fixed - point inspection positions within each steel section to generate the final path.

[0087] Based on this, the division of the inspection ranges of the UAV and the wall - climbing robot and the planning of the inspection paths are realized. For large - scale steel structures, a single UAV and a single wall - climbing robot cannot meet the daily inspection requirements due to limitations such as their endurance. Multiple devices may need to cooperate to achieve this.

[0088] Based on this, for large - scale steel structures, it also includes:

[0089] Step 3: According to the performance parameters of the UAV and the wall - climbing robot, determine whether a single device can complete the inspection task. If not, further calculate the number of UAVs and wall - climbing robots required, and allocate the inspection paths.

[0090] By extracting viewpoints based on point - cloud data, the comprehensiveness of subsequent inspection viewpoints is guaranteed. Then, clustering is performed on the lines of sight to determine the fixed - point inspection positions during the inspection process of the inspection equipment. While ensuring comprehensive coverage of the inspection field of view, it enables the inspection equipment to capture a larger range at a single vertex inspection position, so that when photographing a dense area of steel sections, the inspection equipment can stay for a longer time.

[0091] The intelligent expert subsystem is configured to include:

[0092] The intelligent recognition and diagnosis module is used to perform damage recognition based on the inspection images and a pre - trained damage recognition model. As an example, the damage recognition model can be trained using an existing neural network model to achieve precise positioning of rust, feature identification and area calculation, bolt looseness recognition, and pixel - level extraction of weld cracking.

[0093] The damage alarm module is used to send an alarm message when damage is recognized. The alarm message includes the damage image and the position information of the steel section where the damage is located.

[0094] The intelligent repair module is configured to control the repair robot and the welding robot to perform autonomous repair on the steel structure according to the positioning of the damage, as well as the type and degree of the damage.

[0095] The visualization subsystem is configured to include:

[0096] The BIM model visualization module is used to visualize the BIM model of the large steel structure.

[0097] The digital twin model construction module is used for the online synchronization of the digital twin model.

[0098] The digital twin model visualization module obtains the damage identified from the inspection images, synchronizes the damage to the digital twin model according to the location and size of the damage, and visualizes it.

[0099] Among them, as Figure 4 shown, the construction method of the digital twin model includes the following steps:

[0100] Step 1: Construct an in-service steel structure geometric model based on multi-viewpoint cloud data and design drawings;

[0101] Step 2: Obtain multi-source heterogeneous data reflecting the health status of the steel structure;

[0102] Step 3: Perform cross-modal fusion on the multi-source heterogeneous data to obtain cross-modal fusion features;

[0103] Step 4: Combine the steel structure geometric model data and the cross-modal fusion feature data to construct a digital twin model corresponding to the steel structure geometric model;

[0104] Step 5: Perform dynamic prediction and correction on the constructed digital twin model to obtain a corrected digital twin model.

[0105] Among them, the specific content of Step 1 includes:

[0106] Construct a three-dimensional model of the in-service steel structure based on the obtained multi-viewpoint cloud data;

[0107] Construct a BIM design model of the in-service steel structure based on the design drawings;

[0108] Compare the three-dimensional model of the in-service steel structure with the BIM design model, perform the matching of the feature point matrix of the three-dimensional point cloud model of the steel member, and obtain the geometric three-dimensional model of the steel structure.

[0109] During on-site construction, aiming at problems such as geometric deviations and shape differences between the BIM design model and on-site steel members, by comparing the BIM design model with the three-dimensional reconstruction model of the steel structure scanned on-site, the quality monitoring of steel members can be realized. Those skilled in the art can understand that the above-mentioned three-dimensional model construction methods based on point cloud data and design drawings can be implemented using existing methods, and specific limitations are not made here.

[0110] After obtaining multi-source heterogeneous data reflecting the health status of the steel structure in step 2, data preprocessing is performed, including: denoising the multi-source heterogeneous data based on the constructed denoising autoencoder; normalizing the denoised multi-source heterogeneous data. The multi-source heterogeneous data is structured data mainly in the form of dynamic time-series data. By performing data cleaning operations such as deleting duplicate data and filling in missing data on the multi-source data, dynamic correction of existing incorrect data is completed. In view of the problem that the signals obtained by the system are accompanied by noise interference and the signal confidence is reduced, methods such as wavelet denoising and adaptive filtering are used to design a denoising autoencoder network to preprocess the collected data for denoising.

[0111] A large amount of information is contained in the multi-source heterogeneous data collected during the operation and maintenance of large steel structures. How to extract the key features from the heterogeneous multi-source information, filter out redundant information, and achieve the deep integration of multi-source data with the 3D model is a difficult problem that the current multi-source heterogeneous data fusion digital twin model urgently needs to solve. Step 3 adopts a feature extraction method based on a pre-trained model, enabling the network to pay more attention to useful feature information for dynamic data and improving its ability to suppress redundant information. Then, through feature fusion technology, different modal data are comprehensively processed, and a cross-modal attention mechanism is introduced to further perform feature interaction among multi-source heterogeneous data, enhancing the data fusion effect to provide more comprehensive, accurate, and reliable information. Step 3 specifically includes:

[0112] Input the distributed optical fiber strain, temperature, and vibration time-series data collected by the multi-source heterogeneous data monitoring platform into the pre-trained model to obtain the strain and temperature feature mappings and vibration feature mappings ; In this embodiment, the pre-trained model can use a deep neural network (such as a convolutional neural network, CNN, or a recurrent neural network, RNN) as the basic architecture, and learn the features from a large amount of labeled data through training, so as to effectively extract the features of the input data. Adopting the feature extraction method based on the pre-trained model enables the network to pay more attention to the useful feature information of the distributed optical fiber dynamic monitoring data and improves its ability to suppress redundant information.

[0113] Then, through feature fusion technology, different modal data are comprehensively processed, and a cross-modal attention mechanism is introduced to further perform feature interaction among multi-source heterogeneous data, enhancing the data fusion effect to provide more comprehensive, accurate, and reliable information.

[0114] Figure 6 FIG. is the structural diagram of the cross-modal cross-attention module. The two inputs of this module are the strain and temperature feature mappings extracted by the neural network and vibration feature mappings . The cross-modal cross-attention module can be defined as:

[0115]

[0116] Among them, is the result of normalizing and weighted summarizing the features at all positions and the features at position ; is the index of a certain position in ; is the index of a certain position in ; The unary function is used to calculate the representation value of the feature map at position The information is usually calculated in a linearly weighted manner as follows: , where is optimized as a weight parameter during model training and can be implemented using a 1×1 convolution operation in actual calculations.

[0117] In addition, in this embodiment, a binary function is used to calculate the correlation between the information at position in the feature map and the information at position in the feature map in the form of an embedded Gaussian function. The expression of the binary function is:

[0118]

[0119] where and are used as two embedding terms, and their calculation process is as follows:

[0120]

[0121]

[0122] where the weight matrices and are optimized using 1×1 convolution operations.

[0123] Finally, the cross-modal fusion feature output by the cross-modal cross-attention module is obtained by element-wise addition of the feature maps and :

[0124]

[0125] where As the weight matrix to be trained, it is also implemented by using 1×1 convolution operation.

[0126] Step 4 specifically includes the following steps:

[0127] First, to ensure accurate data mapping and consistency, methods such as resampling and time series alignment are used to complete the synchronous alignment of multi-source heterogeneous data and the 3D model in the time and space dimensions;

[0128] Based on the multi-view point cloud data and the multi-source heterogeneous data fusion features before and after reflecting the steel structure health data, the deduction model is trained to obtain a trained neural network deduction model;

[0129] It is expressed as:

[0130] ,

[0131] ,

[0132] where represents the geometric model of steel structure reconstruction, represents the data fusion feature before the change of key parameters of the steel structure, represents the data fusion feature after the change of key parameters of the steel structure at time t, represents the mapping relationship between the multi-source data monitoring field and the deformation field of the reconstruction model at time t, represents the neural network deduction model, represents the monitoring data at time t;

[0133] Using the multi-physical quantity information such as temperature, stress, and vibration collected in real time by the distributed optical fiber monitoring technology, combined with the finite element simulation analysis results, the trained deduction model is used to map and deduce the data to achieve the accurate mapping of the monitoring data to the 3D model, that is, to complete the construction of the digital twin model.

[0134] Compare the monitoring results with the mapping results under the current monitoring state, and verify the accuracy and credibility of the digital twin deduction model through the difference between the measured monitoring data and the mapping results.

[0135]

[0136] where represents the average verification result, represents the measured data, represents the model mapping result.

[0137] To further improve the accuracy of the digital twin model, a dynamic prediction correction method for the digital twin model based on multi-source data fusion is proposed. Considering the different impacts of multi-source data on different nodes, different weights are selected for multi-source data fusion to achieve the goal of minimizing the sum of the squares of the Euclidean distances between the fusion result and each data. The calculation formula is as follows:

[0138]

[0139] Among them, is the spatial context information of the th data source at time, is the weight of the th data source, which is set according to its influence on the steel structure, is the bias term of the th data source, which is used to adjust the fusion result and can be used as part of the network optimization parameters.

[0140] The twin model is corrected by combining the fused data with the simulation data of the finite element model to ensure a high degree of restoration of the digital twin model. A generative decoder is used to predict the distributed optical fiber monitoring data at the current time, and the multi-step prediction results are directly output; the specific steps are as follows:

[0141] Taking the collected steel structure monitoring data as the "start character" and setting all the data to be predicted to 0 as the decoder input data, the trained decoder predicts the decoder input data and puts the prediction results into the prediction content for output;

[0142] The decoding process of the entire decoder abandons the dynamic decoding process and instead uses a single forward process to decode the entire output sequence.

[0143] In addition, MSE is selected as the loss function during training, and the loss of the entire prediction content is calculated to obtain the final prediction result. The dimension of the output of the fully connected layer depends on the dimension of the variable to be predicted.

[0144] It should be noted that the original prediction method is point-by-point dynamic decoding, that is, the hidden layer state of the current step is calculated by inputting the hidden layer state of the previous step and the output of the previous step, and then the output data of the next step is predicted. This method has the disadvantages of a sharp decline in prediction speed and a sharp increase in loss as the monitoring time becomes longer. Using a generative decoder can speed up the prediction speed of the network and reduce the cumulative error propagation during inference.

[0145] Through the dynamic prediction mechanism, the digital twin model is ensured to reflect the state changes of the steel structure entity, and finally the digital twin model construction that is coordinated and linked with the steel structure entity in real time is realized. This enables the operation and maintenance personnel to intuitively understand the global state of the large steel structure. In addition, by superimposing the identification results of the apparent damage obtained by the inspection operation, the availability of the digital twin model is further improved. It helps to transform "passive maintenance" into "active maintenance" and then develop a new generation of intelligent and safe operation and maintenance system for large steel structures.

[0146] Although adopting specific order to describe each operation, this should be understood as requiring such operation to be performed in the specific order shown or in sequential order, or requiring all illustrated operations to be performed to obtain desired results. Under certain environment, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the application. Some features described in the context of separate embodiments can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in the mode of any suitable sub-combination.

[0147] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the scope of protection of the present application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are only exemplary forms.

Claims

1. An intelligent operation and maintenance system for the whole life cycle of a large-scale steel structure, characterized in that, It includes a real-time data acquisition system, inspection equipment, and an integrated platform. Among them, the integrated platform is configured to include: A collaborative control module for collaboratively controlling inspection equipment information, welding equipment, and repair equipment; An equipment status monitoring module for obtaining the real-time positioning of inspection equipment, welding equipment, and repair equipment; An intelligent recognition and diagnosis module for obtaining inspection images taken by the inspection equipment during the inspection process, and performing damage recognition and positioning on the inspection images based on a pre-trained damage recognition model; A digital twin model construction module for constructing a digital twin model and performing online synchronization based on strain, temperature, or vibration distribution data obtained by the real-time data acquisition system; A digital twin model visualization module for, in response to a visualization request, obtaining the current digital twin model, and synchronizing the damage recognized based on the inspection images, as well as the inspection equipment information, welding equipment, and repair equipment in the working state, to the digital twin model and performing visualization; The inspection equipment includes an unmanned aerial vehicle (UAV) and a wall-climbing robot; the inspection path planning method includes: Obtaining three-dimensional point cloud data of a large steel structure, and according to the flight height and field of view range of a given UAV, determining the point cloud set that can fall within the field of view of the UAV to obtain the inspection range of the UAV; the point cloud set outside the inspection range of the UAV is the inspection range of the wall-climbing robot; respectively planning the inspection paths of the UAV and the wall-climbing robot within the inspection ranges of the UAV and the wall-climbing robot; among them, planning the inspection path of the UAV includes: Extracting the outer layer data from the point cloud data within the inspection range of the UAV and expanding it by a set distance, and fitting to obtain the inspection flight plane of the UAV; Clustering the steel structure point cloud data within the inspection range of the UAV to obtain the point cloud data of multiple steel segments, extracting the center line according to the point cloud data of each steel segment to obtain the center line spatial network; performing three-dimensional grid division on the center line spatial network, and for each grid through which the center line passes, extracting a detection view point; For each detection view point, calculating the intersection point when its distance from the inspection flight plane of the UAV is the shortest, clustering all the intersection points according to a set distance threshold to obtain multiple intersection point clusters, and taking the clustering center of each intersection point cluster as an inspection fixed-point position of the UAV; Within the inspection range of the wall-climbing robot, planning the inspection path of the wall-climbing robot includes: Taking a set position as the origin to construct a spatial coordinate system to obtain the coordinate information of the point cloud data; according to the inspection range of the wall-climbing robot, obtaining the boundary coordinate information of the inspection range of the wall-climbing robot; Dividing the steel structure point cloud data within the boundary into steel segments, projecting all the steel segments onto a set two-dimensional plane, and calculating the projection matrix; extracting detection view points for each projected steel segment through grid sampling, and calculating the line-of-sight directions corresponding to all the detection view points based on the plane normal vector feature; According to the projection matrix, restoring all the view points and line-of-sight directions to three-dimensional space, obtaining the intersection points of all the line-of-sight directions and the nearest steel segment, and determining the steel segments where these intersection points are located; these steel segments are recorded as the steel segments to be climbed, and these intersection points are recorded as the inspection fixed-point positions of the wall-climbing robot.

2. The intelligent operation and maintenance system for the whole life cycle of a large steel structure according to claim 1, characterized in that, The method for constructing the digital twin model includes: Constructing an in-service steel structure geometric model based on multi-viewpoint cloud data and design drawings; Obtaining multi-source heterogeneous data reflecting the health status of the steel structure; Performing cross-modal fusion on the multi-source heterogeneous data to obtain cross-modal fusion features, specifically including: Obtaining strain-temperature feature mapping and vibration feature mapping based on the multi-source heterogeneous data of the steel structure; Calculating the characterization value of the information at any position in the strain-temperature feature mapping as a unary function; Calculating the correlation between the information at any position in the strain-temperature feature mapping and the information at any position in the vibration feature mapping in the form of an embedded Gaussian function as a binary function; Performing cross-modal fusion based on the unary function and the binary function to obtain cross-modal fusion features; Combining the steel structure geometric model data and the cross-modal fusion feature data to construct a digital twin model corresponding to the steel structure geometric model; Performing dynamic prediction and correction on the constructed digital twin model to obtain a corrected digital twin model.

3. The intelligent operation and maintenance system for the whole life cycle of a large steel structure according to claim 2, characterized in that, The formula for performing cross-modal fusion based on the unary function and the binary function to obtain cross-modal fusion features is: Among them, the unary function is used to calculate the feature mapping at the position of the information representation value. The binary function calculates the correlation between the information at the position in the feature mapping and the information at the position in the feature mapping in the form of an embedded Gaussian function.

4. A large steel structure full-life-cycle intelligent operation and maintenance system according to claim 2, characterized in that The construction of the digital twin model corresponding to the steel structure geometric model by combining the steel structure geometric model data and the cross-modal fusion feature data includes: Aligning the steel structure geometric model data and the cross-modal fusion feature data in the time dimension and the space dimension; Training the deduction model based on the aligned steel structure geometric model data and the fusion feature data before and after the change of the steel structure parameters to obtain a trained deduction model; Combining the multi-source heterogeneous data reflecting the health status of the steel structure monitored in real time and the finite element simulation analysis results, and using the trained deduction model to perform mapping from the monitoring data to the three-dimensional model, that is, completing the construction of the digital twin model.

5. The intelligent operation and maintenance system for the whole life cycle of a large steel structure according to claim 2, characterized in that, The dynamic prediction and correction of the constructed digital twin model to obtain a corrected digital twin model is based on the influence differences of multi-source data on different nodes, and different weights are selected for multi-source data fusion to achieve the goal of minimizing the sum of the squares of the Euclidean distances between the fusion result and each data.

6. The intelligent operation and maintenance system for the whole life cycle of a large steel structure according to claim 1, characterized in that The operation and maintenance system further includes a damage alarm module for sending an alarm message when damage is identified, and the alarm message includes a damage image and the position information of the steel section where the damage is located.

7. The intelligent operation and maintenance system for the whole life cycle of a large steel structure according to claim 1, characterized in that, The operation and maintenance system further includes an intelligent repair module configured to control the repair robot and the welding robot to perform autonomous repair on the steel structure according to the location of the damage, as well as the damage type and the degree of damage.

Images