Full-life-cycle intelligent operation and maintenance system for large steel structure

Abstract

The invention relates to the technical field of large-scale steel structure operation and maintenance, and discloses a large-scale steel structure full-life-cycle intelligent operation and maintenance system, which comprises an intelligent identification and diagnosis module used for acquiring an inspection image shot by inspection equipment in an inspection process, and carrying out damage identification and positioning based on a pre-trained damage identification model; the digital twinborn model construction module is used for constructing a digital twinborn model and performing online synchronization based on strain, temperature or vibration distribution data acquired by the real-time data acquisition system; and the digital twinborn model visualization module is used for responding to the visualization request, acquiring a current digital twinborn model, synchronizing the damage identified based on the inspection image and the inspection equipment information, the welding equipment and the repairing equipment in the working state to the digital twinborn model, and carrying out visualization. On the basis of realizing the digital twinborn model, damage and related operation and maintenance equipment are superposed, and the real-time state of the large steel structure can be remotely displayed to operation and maintenance personnel more intuitively.

Description

Technical Field

[0001] The present invention belongs to the technical field of large-scale steel structure operation and maintenance, and in particular relates to a full life cycle intelligent operation and maintenance system for large-scale steel structures. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Large steel structures have the characteristics of irregular component shapes, complex interconnections of many rods, and staggered obstructions. It is difficult for a single automated inspection device to achieve full coverage of the field of view. For example, drones and ground inspection robots are difficult to take into account the internal rods with obstructions, especially because many steel structures have external enclosures and internal ceilings, which are not applicable at all. Therefore, large steel structure inspections still have to be mainly manual, and manual methods require high-altitude operations, which are extremely risky; at the same time, manual inspections usually rely on the experience of operation and maintenance personnel. When he subjectively judges that a certain location may be damaged, he uses detection equipment for inspection, which is easy to miss; and people cannot reach places with narrow spaces, so full coverage cannot be achieved. In addition, manual methods require high professional knowledge of operation and maintenance personnel, who need to be familiar with the mechanism of action of various loads on steel structures and the applicability of different detection tools. The above status quo has brought one-sided, static, passive, and blind challenges to the operation and maintenance of complex large steel structures. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an intelligent operation and maintenance system for the entire life cycle of large-scale steel structures. On the basis of realizing the online synchronization of the digital twin model, by superimposing the damage identification and analysis results obtained based on the inspection images, the real-time status of the large-scale steel structure can be more intuitively displayed to the operation and maintenance personnel remotely.

[0005] To achieve the above objectives, one or more embodiments of the present invention provide a large-scale steel structure full life cycle intelligent operation and maintenance system, including a real-time data acquisition system, inspection and repair equipment, and an integrated platform, wherein the integrated platform configuration includes: Collaborative control module, used for collaborative control of inspection equipment information, welding equipment and repair equipment; Equipment status monitoring module, used to obtain the real-time positioning of inspection equipment, welding equipment and repair equipment; The intelligent identification and diagnosis module is used to obtain the inspection images taken by the inspection equipment during the inspection process, and to identify and locate damage in the inspection images based on the pre-trained damage identification model; A digital twin model building module is used to build a digital twin model and perform online synchronization based on the strain, temperature or vibration distribution data obtained by the real-time data acquisition system; The digital twin model visualization module is used to respond to a visualization request, obtain the current digital twin model, and synchronize the damage identified based on the inspection image, as well as the inspection equipment information, welding equipment and repair equipment in working state to the digital twin model for visualization.

[0006] In some embodiments, the inspection equipment includes a drone and a wall-climbing robot; and the inspection path planning method based on the drone and the wall-climbing robot includes: Obtain the 3D point cloud data of large steel structures, and determine the point cloud set that can fall within the drone's field of view according to the given drone's flight altitude and field of view, and obtain the drone's inspection range; the point cloud set outside the drone's inspection range is the wall-climbing robot's inspection range; The inspection paths of the UAV and the wall-climbing robot are planned within the inspection range of the UAV and the wall-climbing robot respectively.

[0007] In some embodiments, planning an inspection path for a drone within the inspection range of the drone includes: According to the point cloud data within the inspection range of the UAV, the outer layer data is extracted and expanded by a set distance to obtain the UAV inspection flight surface by fitting; Clustering the point cloud data of steel structures within the inspection range of the drone to obtain point cloud data of multiple steel sections, extracting the center line based on the point cloud data of each steel section to obtain a center line space network; performing three-dimensional grid division on the center line space network, and extracting a detection viewpoint for each grid through which the center line passes; According to the set distance threshold, all intersection points are clustered to obtain multiple intersection clusters, and the cluster center of each intersection cluster is used as a fixed inspection point for the UAV.

[0008] In some embodiments, within the inspection range of the wall-climbing robot, planning the inspection path of the wall-climbing robot includes: Taking the set position as the origin, a spatial coordinate system is constructed to obtain the coordinate information of the point cloud data; according to the inspection range of the wall-climbing robot, the boundary coordinate information of the inspection range of the wall-climbing robot is obtained; The steel structure point cloud data within the boundary range is divided into steel segments, all steel segments are projected onto the set two-dimensional plane, and the projection matrix is ​​calculated; the detection viewpoints are extracted from each projected steel segment through grid sampling, and the line of sight directions corresponding to all detection viewpoints are calculated based on the plane normal vector features; According to the projection matrix, all viewpoints and sight lines are restored to three-dimensional space, the intersections of all sight lines and the nearest steel segments are obtained, and the steel segments where these intersections are located are determined; these steel segments are recorded as the steel segments to be crawled, and these intersections are recorded as the inspection fixed points of the wall-climbing robot.

[0009] In some embodiments, the digital twin model construction method includes: The geometric model of the in-service steel structure is constructed based on multi-view point cloud data and design drawings; Acquire multi-source heterogeneous data reflecting the health status of steel structures; Cross-modal fusion of multi-source heterogeneous data is performed to obtain cross-modal fusion features, including: Based on the multi-source heterogeneous data of steel structure, strain and temperature characteristic mapping and vibration characteristic mapping are obtained; Calculate the representative value of the information of the strain and temperature characteristic map at any position as a univariate function; The correlation between the information at any position in the strain and temperature feature map and the information at any position in the vibration feature map is calculated as an embedded Gaussian function as a binary function; Cross-modal fusion is performed based on univariate functions and binary functions to obtain cross-modal fusion features; The digital twin model corresponding to the steel structure geometric model is constructed by combining the steel structure geometric model data and the cross-modal fusion feature data; The constructed digital twin model is dynamically predicted and corrected to obtain a corrected digital twin model.

[0010] In some embodiments, the formula for obtaining the cross-modal fusion feature by performing cross-modal fusion based on the unary function and the binary function is:

[0011] Among them, the one-variable function Used to calculate feature maps In Location Information Characteristic value, binary function Compute feature maps as embedded Gaussian functions Middle position Information and feature maps Middle position Information of relevance.

[0012] In some embodiments, the digital twin model corresponding to the steel structure geometric model is constructed by combining the steel structure geometric model data and the cross-modal fusion feature data, including: Align the steel structure geometry model data and the cross-modal fusion feature data in the time dimension and the space dimension; The deduction model is trained based on the aligned steel structure geometric model data and the fused feature data before and after the steel structure parameter changes to obtain a trained deduction model; Combining the real-time monitoring of multi-source heterogeneous data reflecting the health status of the steel structure and the results of finite element simulation analysis, the trained deduction model is used to map the monitoring data to the three-dimensional model, thus completing the construction of the digital twin model.

[0013] In some embodiments, the constructed digital twin model is dynamically predicted and corrected to obtain a corrected digital twin model. The basis for the correction is to select different weights for multi-source data fusion according to the differences in the impact of multi-source data on different nodes, so as to achieve the goal of minimizing the sum of squared Euclidean distances between the fusion results and each data.

[0014] In some embodiments, the operation and maintenance system further includes a damage alarm module for issuing an alarm message when damage is identified, wherein the alarm message includes a damage image and location information of the steel section where the damage is located.

[0015] In some embodiments, the operation and maintenance system also includes an intelligent repair module, which is configured to control the repair robot and the welding robot to autonomously repair the steel structure according to the location of the damage, as well as the type and degree of damage.

[0016] In one or more of the above technical solutions, strain, temperature or vibration distribution data, and inspection image data are respectively obtained. On the basis of realizing online synchronization of the digital twin model, by superimposing the damage identification results obtained based on the inspection images, the real-time status of the large steel structure can be more intuitively displayed to the operation and maintenance personnel remotely. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0018] Figure 1 This is a functional architecture diagram of the intelligent operation and maintenance system for the entire life cycle of a large steel structure in an embodiment of the present invention; Figure 2 This is an overall flow chart of the path planning method in an embodiment of the present invention; Figure 3 This is a schematic diagram of the path planning principle of a drone in an embodiment of the present invention; Figure 4 This is a flow chart of a method for constructing a large-scale steel structure digital twin model by integrating multi-source data in an embodiment of the present invention; Figure 5 is a schematic diagram of a real-time data acquisition system provided by an embodiment of the present invention; Figure 6 It is a schematic diagram of cross-modal fusion features provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0020] In the description of the embodiments of the present application, the term “including” and similar terms should be understood as open inclusion, that is, “including but not limited to.” The term “based on” should be understood as “based at least in part on.”

[0021] In view of the existing problems in the operation and maintenance of large steel structures, one or more embodiments of the present invention provide a large steel structure full life cycle intelligent operation and maintenance system, including a real-time data acquisition system, inspection equipment, repair equipment, an integrated platform and a client. The real-time data acquisition system, inspection equipment, and repair equipment are all connected to the integrated platform, and the client can establish a communication connection with the integrated platform to access the integrated platform.

[0022] The real-time data acquisition system collects real-time monitoring data using a steel structure key parameter monitoring system based on distributed optical fiber sensing. The system includes a pulse light generator, a circulator, a wavelength division multiplexer, two photodetectors, a sensing optical fiber and a computer. The computer sends a control signal to the pulse light generator and the photodetector and starts collecting signals through three channels at the same time. The pulse light generator emits pulse light, which reaches the sensing optical fiber arranged at a set position on the surface of the steel structure through the circulator and the wavelength division multiplexer. The backward Rayleigh reflected light is collected by the first photodetector through the wavelength division multiplexer and the circulator. The backward Brillouin reflected light includes Stokes light and anti-Stokes light. Due to different wavelengths, the Stokes light and the anti-Stokes light are collected by the second photodetector through the wavelength division multiplexer respectively. The second photodetector converts the collected reflected light signal into an electrical signal and transmits it to the main control computer for analysis, so as to obtain the strain, temperature or vibration distribution of the steel structure surface at the current moment, and obtain multi-source heterogeneous data reflecting the health status of the steel structure.

[0023] As an example, Figure 5As shown, when the system works normally, the main control computer sends control signals to the pulse light generator and photodetector (APD1, APD2) and simultaneously collects signals through three channels (channel 1, channel 2 and channel 3) through the data acquisition card. The pulse light generator emits infrared pulse light with a central wavelength of C band, which reaches the sensing optical fiber arranged at a set position on the surface of the steel structure through a circulator and a wavelength division multiplexer. Due to changes in external stress, temperature or vibration, the scattering information of the pulse light in the sensing optical fiber changes. This system mainly collects backward Rayleigh scattered light and two-way backward Brillouin scattered light, among which the backward Rayleigh reflected light is collected by the photodetector APD1 through the wavelength division multiplexer and the circulator, and the Stokes light and anti-Stokes light in the backward Brillouin reflected light are collected by the photodetector APD2 through the wavelength division multiplexer respectively. After the photoelectric detector collects the reflected light signal, it converts it into an electrical signal and transmits it to the main control computer. After the main control computer performs signal preprocessing such as denoising and decoupling on the collected signal, it generates the corresponding strain, temperature or vibration curve graph to obtain the strain, temperature or vibration distribution of the steel structure surface at the current moment.

[0024] When the system detects abnormal strain, temperature or vibration at a certain location, the location of the steel structure damage can be determined through signal decoupling analysis. Secondly, the intensity of the scattered signal can be analyzed to determine whether the optical fiber currently laid on the surface of the steel structure is broken or bent, thereby ensuring the reliability of the acquired data.

[0025] The inspection device is configured as follows: Inspections are performed based on the set inspection paths, and during the inspection process, fixed-point inspection images are transmitted back to the integrated platform.

[0026] In order to reduce the pressure of data transmission, the inspection equipment can be built with a preliminary defect discrimination model to preliminarily screen the captured images during the inspection process, exclude normal images, and only transmit suspected defective images to the integrated platform. As an example, using images of defect-free steel segments and defective steel segments, a binary classification model is trained based on a support vector machine (SVM) to distinguish only whether there are defects or not, and the images of steel segments determined to be defective are used as suspected defective steel segment images and transmitted to the integrated platform.

[0027] like Figure 1 As shown, the integrated platform is configured as: The data storage subsystem is used to obtain and store the inspection images of each fixed point, as well as the strain, temperature or vibration distribution of the steel structure surface at each moment. Based on this, for each inspection point, the inspection images of the entire life cycle can be obtained. Understandably, in order to ensure the transparency and traceability of each link, blockchain, access passwords and digital summaries can be used, combined with the information of the entire life cycle in the cloud, to build a full-chain information traceability mechanism, which is helpful for analyzing the evolution mechanism of different types of damage in large steel structures and providing a data basis for building a large steel structure damage identification inversion model.

[0028] The device control subsystem is configured to include: The equipment information management module is used to manage the inspection equipment information, welding equipment and repair equipment. The inspection equipment includes one or more wall-climbing robots and / or one or more drones. The welding equipment is such as a welding robot, and the repair equipment is such as a wall-climbing rust repair robot, which are not specifically limited here.

[0029] Collaborative control module, used for collaborative control of inspection equipment information, welding equipment and repair equipment; The equipment status monitoring module is used to obtain the real-time positioning and real-time operating parameters of inspection equipment, welding equipment and repair equipment to determine whether they are operating normally and whether they are operating according to the prescribed path.

[0030] The inspection execution subsystem is configured to include: The inspection path planning module is used to plan the inspection path for one or more inspection devices for large steel structures to obtain the inspection path for each inspection device.

[0031] As a specific implementation method, the inspection equipment includes two types: drones and wall-climbing robots, and an air-ground collaborative inspection path planning method is adopted, such as Figure 2 As shown, specifically including: Step 1: Obtain the 3D point cloud data of the large steel structure. According to the given flight altitude and field of view of the drone, determine the point cloud set that can fall within the field of view of the drone, and obtain the inspection range of the drone. The point cloud set outside the inspection range of the drone is the inspection range of the wall-climbing robot. Step 2: Plan the inspection paths of the UAV and the wall-climbing robot within the inspection range of the UAV and the wall-climbing robot respectively.

[0032] In step 1, first, a spatial coordinate system is constructed with the set position as the origin to obtain the coordinate information of the point cloud data; as an example, the ground center point of the steel structure building can be used as the origin. The inspection range of the drone is mainly analyzed based on the performance parameters of the drone, such as the detection accuracy and monitoring range. The flight altitude of the drone is the maximum distance between the drone and the steel structure, which is related to the resolution of the visual system carried thereon. Within the maximum flight altitude of the drone, the altitude at which a defect of a given size can be clearly photographed shall prevail. The field of view is related to the field of view of the visual system and the degree of freedom of the gimbal where it is located. When the parameters of the drone, its gimbal and the visual system are determined, the flight altitude and field of view can be obtained by calculation.

[0033] In step 2, Figure 3 As shown in the figure, the inspection path planning for the drone within the inspection range of the drone specifically includes: (1) Based on the point cloud data within the inspection range of the drone, the outer layer data is extracted and expanded by a set distance to obtain the drone inspection flight surface. The set distance is the distance between the drone and the steel structure during the inspection. It can be understood that when the ground center point of the steel structure building is taken as the origin, the point farthest from the coordinate origin in the same ray direction starting from the coordinate origin is recorded as the outer layer data.

[0034] (2) Clustering the point cloud data of the steel structure within the inspection range of the UAV to obtain point cloud data of multiple steel sections, extracting the center line based on the point cloud data of each steel section to obtain a center line space network; dividing the center line space network into a three-dimensional grid (cube grid), and extracting a detection viewpoint for each grid through which the center line passes; (3) For each detection viewpoint, calculate the intersection point where the distance between the detection viewpoint and the inspection flight surface of the UAV is the shortest, and record the direction from the detection viewpoint to the intersection point as the line of sight direction; (4) According to the set distance threshold, all intersection points are clustered to obtain multiple intersection clusters. The cluster center of each intersection cluster is used as a fixed inspection point for the drone. The drone takes pictures of the corresponding viewpoint at the fixed inspection point.

[0035] (5) Based on all the inspection locations of the drone, the shortest path is planned.

[0036] By extracting the center line within the inspection range and dividing it into three-dimensional grids, a viewpoint is extracted in each three-dimensional grid. This ensures that the field of view of the subsequent drone can cover all steel structure sections within the inspection range during inspection, avoiding missed views or defects.

[0037] After obtaining all the inspection fixed-point positions of the drone, it is necessary to organize these inspection fixed-point positions into a coherent path through path planning. As an example, the ant colony algorithm can be used for path planning.

[0038] Since the wall-climbing robot works by attaching to the surface of the steel structure, its detection field of view is limited. Therefore, in order to take into account both work efficiency and energy saving, it is necessary to make the crawling path as short as possible while ensuring that the field of view of the wall-climbing robot can cover the entire inspection range.

[0039] Based on this, within the inspection range of the wall-climbing robot, the inspection path planning of the wall-climbing robot specifically includes: (1) Clustering the point cloud data of the steel structure within the inspection range of the wall-climbing robot to obtain point cloud data of multiple steel segments, extracting the center line based on the point cloud data of each steel segment to obtain a center line space network; dividing the center line space network into a three-dimensional grid (cube grid), and extracting a detection viewpoint for each grid through which the center line passes; (2) For each detection viewpoint, determine the multiple steel segments that can be connected to the detection viewpoint without obstacles. That is, for each detection viewpoint, a set of steel segments that can be visible to it can be obtained. These steel segment sets are analyzed, and the steel segment with the most visible viewpoints is preferentially selected. The steel segment with the least steel segments is taken as the goal, and the steel segment corresponding to each viewpoint is obtained; (3) These steel segments are recorded as the steel segments to be climbed, and the intersection between the detection viewpoint and the corresponding steel segment is recorded as the inspection fixed point position of the wall-climbing robot, which is used to shoot the viewpoint on the corresponding line of sight.

[0040] (4) Based on the steel sections to be crawled and the fixed inspection positions on each steel section, the shortest path planning is performed. It can be understood that the shortest path is a coherent path covering all the steel sections to be crawled and the fixed inspection positions on each steel section.

[0041] Since it is necessary to ensure that the wall-climbing robot passes through all the sections to be crawled and is allowed to pass through the steel sections repeatedly, the above path planning problem can be equivalent to an arc routing problem, that is, it is required to find a shortest path so that each steel section to be crawled in the steel structure is passed at least once. As an example, the following method can be used: an undirected road network graph is established based on all the steel sections to be crawled, the endpoints of each steel section are nodes of the graph, the steel section is the edge connecting two nodes, the length of each steel section determines the weight of the edge, and the solution for solving problems under existing large-scale road networks is used for path planning.

[0042] The path is solved based on the above method, and all fixed inspection positions in each steel section are smoothly connected to generate the final path.

[0043] Based on this, the inspection range division and inspection route planning of drones and wall-climbing robots are realized. For large steel structures, a single drone or a single wall-climbing robot cannot meet the daily inspection needs due to its limitations such as endurance. It may require multiple devices to cooperate to achieve this.

[0044] Based on this, for large steel structures, it also includes: Step 3: Based on the performance parameters of the drone and wall-climbing robot, determine whether a single device can complete the inspection task. If not, further calculate the required number of drones and wall-climbing robots, and allocate the inspection path.

[0045] By extracting viewpoints based on point cloud data, the comprehensiveness of subsequent inspection viewpoints is ensured, and then the lines of sight are clustered to determine the fixed inspection positions of the inspection equipment during the inspection process. While ensuring comprehensive coverage of the inspection field of view, the inspection equipment can capture a larger range at a vertex inspection position, so that the inspection equipment can stay longer when capturing dense areas of steel sections.

[0046] The intelligent expert subsystem is configured to include: The intelligent identification and diagnosis module is used to perform damage identification based on the inspection images and the pre-trained damage identification model. As an example, the damage identification model can be trained using an existing neural network model to achieve accurate positioning of rust, feature identification and area calculation, bolt loosening identification, and pixel-level extraction of weld cracks.

[0047] The damage alarm module is used to issue an alarm message when damage is identified. The alarm message includes a damage image and location information of the steel section where the damage is located.

[0048] The intelligent repair module is configured to control the repair robot and the welding robot to autonomously repair the steel structure according to the location of the damage, as well as the type and degree of damage.

[0049] The visualization subsystem is configured to include: BIM model visualization module is used to visualize the BIM model of large steel structures.

[0050] Digital twin model building module for online synchronization of digital twin models.

[0051] The digital twin model visualization module obtains the damage identified based on the inspection image, synchronizes the damage to the digital twin model according to the location and size of the damage, and visualizes it.

[0052] Among them, Figure 4 As shown, the method for constructing the digital twin model includes the following steps: Step 1: Construct a geometric model of the in-service steel structure based on multi-view point cloud data and design drawings; Step 2: Obtain multi-source heterogeneous data reflecting the health status of the steel structure; Step 3: Perform cross-modal fusion on multi-source heterogeneous data to obtain cross-modal fusion features; Step 4: Combine the steel structure geometry model data and the cross-modal fusion feature data to construct a digital twin model corresponding to the steel structure geometry model; Step 5: Perform dynamic prediction and correction on the constructed digital twin model to obtain the corrected digital twin model.

[0053] Wherein, the step 1 specifically includes: The three-dimensional model of the in-service steel structure is constructed based on the acquired multi-view point cloud data; Build a BIM design model of the in-service steel structure based on the design drawings; The 3D model of the in-service steel structure is compared with the BIM design model, and the characteristic point matrix of the 3D point cloud model of the steel component is matched to obtain the geometric 3D model of the steel structure.

[0054] During on-site construction, in order to solve the geometric deviation and shape difference between the BIM design model and the on-site steel components, the quality monitoring of the steel components can be realized by comparing the BIM design model with the 3D reconstruction model of the steel structure scanned on-site. Those skilled in the art can understand that the above-mentioned 3D model construction method based on point cloud data and design drawings can be realized by existing methods, which are not specifically limited here.

[0055] After obtaining the multi-source heterogeneous data reflecting the health status of the steel structure, the step 2 performs data preprocessing, including: denoising the multi-source heterogeneous data based on the constructed denoising autoencoder; normalizing the denoised multi-source heterogeneous data. Multi-source heterogeneous data is structured data mainly based on dynamic time series data. By performing data cleaning work on multi-source data to delete duplicate data and complete missing data, dynamic correction of existing erroneous data is completed. In view of the problem that the signal acquired by the system is accompanied by noise interference and the signal confidence is reduced, wavelet denoising, adaptive filtering and other methods are used to design a denoising autoencoder network to perform denoising preprocessing on the collected data.

[0056] The multi-source heterogeneous data collected during the operation and maintenance of large steel structures contains a large amount of information. How to extract key features from heterogeneous multi-source information, filter redundant information, and achieve deep fusion of multi-source data and three-dimensional models is a difficult problem that needs to be solved urgently in the current multi-source heterogeneous data fusion digital twin model. Step 3 adopts a feature extraction method based on a pre-trained model to enable the network to pay more attention to useful feature information for dynamic data and improve its ability to suppress redundant information. Then, the different modal data are comprehensively processed through feature fusion technology, and a cross-modal attention mechanism is introduced to further perform feature interaction between multi-source heterogeneous data, thereby improving the data fusion effect and providing more comprehensive, accurate and reliable information. Step 3 specifically includes: The distributed optical fiber strain, temperature and vibration time series data collected based on the multi-source heterogeneous data monitoring platform are input into the pre-trained model to obtain the strain and temperature feature mapping and vibration feature mapping In this embodiment, the pre-trained model can use a deep neural network (such as a convolutional neural network, CNN, or a recurrent neural network, RNN) as the basic architecture, and effectively extract features from the input data by training the features learned on a large amount of labeled data. The feature extraction method based on the pre-trained model makes the network pay more attention to the useful feature information of the distributed optical fiber dynamic monitoring data, and improves its ability to suppress redundant information.

[0057] Then, the data of different modalities are comprehensively processed through feature fusion technology, and the cross-modal attention mechanism is introduced to further perform feature interaction between multi-source heterogeneous data, thereby improving the data fusion effect and providing more comprehensive, accurate and reliable information.

[0058] Figure 6 The structure diagram of the cross-modal attention module structure. The two inputs of this module are the strain and temperature feature maps extracted by the neural network. and vibration feature mapping The cross-modal attention module can be defined as:

[0059] in, Yes Features and positions at all positions on The result of normalized weighted aggregation of features; yes The index of a position in yes The index of a position in ; unary function Used to calculate feature maps In Location Information The characterization value is usually calculated using a linear weighted approach, as shown below: ,in As the weight parameters are optimized during the model training process, a 1×1 convolution operation can be used to implement it in actual operations.

[0060] In addition, in this embodiment, the binary function Compute feature maps as embedded Gaussian functions Middle position Information and feature maps Middle position Information Correlation, a binary function The expression is:

[0061] in, and As two embedded terms, the calculation process is as follows:

[0062]

[0063] Among them, the weight matrix and A 1×1 convolution operation is used to achieve optimization.

[0064] Finally, the cross-modal fusion features of the output of the cross-modal attention module By feature map and Adding them element by element gives:

[0065] in, As the weight matrix that needs to be trained, it is also implemented using a 1×1 convolution operation.

[0066] The step 4 specifically includes the following steps: First, to ensure accurate data mapping and consistency, we use resampling, time series alignment and other methods to synchronize multi-source heterogeneous data with the 3D model in both time and space dimensions. The deduction model is trained based on multi-view point cloud data and multi-source heterogeneous data fusion features that reflect the changes in the health data of the steel structure before and after, and a trained neural network deduction model is obtained; It is expressed as: , , in Represents the reconstructed geometric model of the steel structure, Indicates the data fusion characteristics before the key parameters of the steel structure change. It represents the data fusion characteristics after the key parameters of the steel structure change at time t. It represents the mapping relationship between the multi-source data monitoring field and the reconstructed model deformation field at time t. represents the neural network deduction model, Represents the monitoring data at time t; The distributed fiber optic monitoring technology is used to collect real-time information on multiple physical quantities such as temperature, stress, vibration, etc., combined with the results of finite element simulation analysis, and the trained deduction model is used to map and deduce the data to achieve accurate mapping of the monitoring data to the three-dimensional model, thus completing the construction of the digital twin model.

[0067] The monitoring results under the current monitoring status are compared with the mapping results, and the accuracy and credibility of the digital twin deduction model are verified by monitoring the differences between the measured data and the mapping results.

[0068]

[0069] in represents the average calibration result, Represents the measured data, Represents the model mapping result.

[0070] In order to further improve the accuracy of the digital twin model, a dynamic prediction and correction method for the digital twin model of multi-source data fusion is proposed. According to the difference in the impact of multi-source data on different nodes, different weights are selected for multi-source data fusion to achieve the goal of minimizing the sum of squares of the Euclidean distances between the fusion results and each data. The calculation formula is:

[0071] in, For the Data sources in The spatial context information at the moment, For the The weight of each data source is set according to its impact on the steel structure. For the The bias item of the data source is used to adjust the fusion result and can be used as a part of the network optimization parameters.

[0072] The twin model is modified by combining the fused data with the simulation data of the finite element model to ensure the high restoration of the digital twin model. The generative decoder is used to predict the distributed optical fiber monitoring data at the current moment and directly output the multi-step prediction results; the specific steps include the following: The collected steel structure monitoring data is used as the "start character", and after all the predicted data are set to 0, it is input into the decoder as the decoder input data. The trained decoder predicts the decoder input data and puts the prediction result into the prediction content for output; The decoding process of the entire decoder abandons the dynamic decoding process and uses a forward process to decode the entire output sequence.

[0073] In addition, MSE is selected as the loss function during training, and the loss is calculated for the entire prediction content to obtain the final prediction result. The dimension of the fully connected layer output depends on the dimension of the variable to be predicted.

[0074] It should be noted that the original prediction method is point-by-point dynamic decoding, that is, the hidden layer state of the previous step and the output of the previous step are input to calculate the hidden layer state of the current step, and then the output data of the next step is predicted. This method has the disadvantages of a sharp drop in prediction speed and a sharp increase in loss as the monitoring time becomes longer. The use of a generative decoder can speed up the prediction speed of the network and reduce the cumulative error propagation during inference.

[0075] Through the dynamic prediction mechanism, the digital twin model is ensured to reflect the state changes of the steel structure entity, and finally the digital twin model construction that is coordinated and linked with the steel structure entity in real time is realized. This enables the operation and maintenance personnel to intuitively understand the global state of the large steel structure. In addition, by superimposing the identification results of the apparent damage obtained by the inspection operation, the availability of the digital twin model is further improved. It helps to transform "passive maintenance" into "active maintenance" and then develop a new generation of intelligent and safe operation and maintenance system for large steel structures.

[0076] Although adopting specific order to describe each operation, this should be understood as requiring such operation to be performed in the specific order shown or in sequential order, or requiring all illustrated operations to be performed to obtain desired results. Under certain environment, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the application. Some features described in the context of separate embodiments can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in the mode of any suitable sub-combination.

[0077] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the scope of protection of the present application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are only exemplary forms.

Claims

1. A large-scale steel structure full life cycle intelligent operation and maintenance system, characterized in that: It includes a real-time data acquisition system, inspection equipment and an integrated platform, wherein the integrated platform is configured to include: Collaborative control module, used for collaborative control of inspection equipment information, welding equipment and repair equipment; Equipment status monitoring module, used to obtain the real-time positioning of inspection equipment, welding equipment and repair equipment; The intelligent identification and diagnosis module is used to obtain the inspection images taken by the inspection equipment during the inspection process, and to identify and locate damage in the inspection images based on the pre-trained damage identification model; A digital twin model building module is used to build a digital twin model and perform online synchronization based on the strain, temperature or vibration distribution data obtained by the real-time data acquisition system; The digital twin model visualization module is used to respond to a visualization request, obtain the current digital twin model, and synchronize the damage identified based on the inspection image, as well as the inspection equipment information, welding equipment and repair equipment in working state to the digital twin model for visualization.

2. A large-scale steel structure full life cycle intelligent operation and maintenance system as claimed in claim 1, characterized in that: The inspection equipment includes drones and wall-climbing robots; The inspection path planning method based on the UAV and the wall-climbing robot includes: Obtain the 3D point cloud data of large steel structures, and determine the point cloud set that can fall within the drone's field of view according to the given drone's flight altitude and field of view, and obtain the drone's inspection range; the point cloud set outside the drone's inspection range is the wall-climbing robot's inspection range; The inspection paths of the UAV and the wall-climbing robot are planned within the inspection range of the UAV and the wall-climbing robot respectively.

3. A large-scale steel structure full life cycle intelligent operation and maintenance system as claimed in claim 2, characterized in that: Planning the inspection path for the drone within the inspection range includes: According to the point cloud data within the inspection range of the UAV, the outer layer data is extracted and expanded by a set distance to obtain the UAV inspection flight surface by fitting; Clustering the point cloud data of steel structures within the inspection range of the drone to obtain point cloud data of multiple steel sections, extracting the center line based on the point cloud data of each steel section to obtain a center line space network; performing three-dimensional grid division on the center line space network, and extracting a detection viewpoint for each grid through which the center line passes; According to the set distance threshold, all intersection points are clustered to obtain multiple intersection clusters, and the cluster center of each intersection cluster is used as a fixed inspection point for the UAV.

4. A large-scale steel structure full life cycle intelligent operation and maintenance system as claimed in claim 2, characterized in that: Within the inspection range of the wall-climbing robot, the inspection path planning of the wall-climbing robot includes: Taking the set position as the origin, a spatial coordinate system is constructed to obtain the coordinate information of the point cloud data; according to the inspection range of the wall-climbing robot, the boundary coordinate information of the inspection range of the wall-climbing robot is obtained; The steel structure point cloud data within the boundary range is divided into steel segments, all steel segments are projected onto the set two-dimensional plane, and the projection matrix is ​​calculated; the detection viewpoints are extracted from each projected steel segment through grid sampling, and the line of sight directions corresponding to all detection viewpoints are calculated based on the plane normal vector features; According to the projection matrix, all viewpoints and sight lines are restored to three-dimensional space, the intersections of all sight lines and the nearest steel segments are obtained, and the steel segments where these intersections are located are determined; these steel segments are recorded as the steel segments to be crawled, and these intersections are recorded as the inspection fixed points of the wall-climbing robot.

5. A large-scale steel structure full life cycle intelligent operation and maintenance system as claimed in claim 1, characterized in that: The digital twin model construction method includes: The geometric model of the in-service steel structure is constructed based on multi-view point cloud data and design drawings; Acquire multi-source heterogeneous data reflecting the health status of steel structures; Cross-modal fusion of multi-source heterogeneous data is performed to obtain cross-modal fusion features, including: Based on the multi-source heterogeneous data of steel structure, strain and temperature characteristic mapping and vibration characteristic mapping are obtained; Calculate the representative value of the information of the strain and temperature characteristic map at any position as a univariate function; The correlation between the information at any position in the strain and temperature feature map and the information at any position in the vibration feature map is calculated as an embedded Gaussian function as a binary function; Cross-modal fusion is performed based on univariate functions and binary functions to obtain cross-modal fusion features; The digital twin model corresponding to the steel structure geometric model is constructed by combining the steel structure geometric model data and the cross-modal fusion feature data; The constructed digital twin model is dynamically predicted and corrected to obtain a corrected digital twin model.

6. A large-scale steel structure full life cycle intelligent operation and maintenance system as claimed in claim 5, characterized in that: The formula for obtaining the cross-modal fusion feature by cross-modal fusion based on unary function and binary function is: Among them, the one-variable function Used to calculate feature maps In Location Information Characteristic value, binary function Compute feature maps as embedded Gaussian functions Middle position Information and feature maps Middle position Information of relevance.

7. A large-scale steel structure full life cycle intelligent operation and maintenance system as claimed in claim 5, characterized in that: The digital twin model corresponding to the steel structure geometric model is constructed by combining the steel structure geometric model data and the cross-modal fusion feature data, including: Align the steel structure geometry model data and the cross-modal fusion feature data in the time dimension and the space dimension; The deduction model is trained based on the aligned steel structure geometric model data and the fused feature data before and after the steel structure parameter changes to obtain a trained deduction model; Combining the real-time monitoring of multi-source heterogeneous data reflecting the health status of the steel structure and the results of finite element simulation analysis, the trained deduction model is used to map the monitoring data to the three-dimensional model, thus completing the construction of the digital twin model.

8. A large-scale steel structure full life cycle intelligent operation and maintenance system as claimed in claim 5, characterized in that: The constructed digital twin model is dynamically predicted and corrected to obtain a corrected digital twin model. The correction is based on the difference in the impact of multi-source data on different nodes, and different weights are selected to fuse multi-source data to achieve the goal of minimizing the sum of squared Euclidean distances between the fusion result and each data.

9. A large-scale steel structure full life cycle intelligent operation and maintenance system as claimed in claim 1, characterized in that: The operation and maintenance system also includes a damage alarm module, which is used to issue an alarm message when damage is identified. The alarm message includes a damage image and location information of the steel section where the damage is located.

10. The large-scale steel structure full life cycle intelligent operation and maintenance system according to claim 1, characterized in that: The operation and maintenance system also includes an intelligent repair module, which is configured to control the repair robot and the welding robot to autonomously repair the steel structure according to the location of the damage, as well as the type and degree of damage.

Images