Multi-modal data display method of BIM model

Abstract

The invention relates to a multi-modal data display method of a BIM model, and relates to the technical field of data interaction display. According to the multi-modal data display method of the BIM model, three-dimensional point cloud and multi-angle image data of each device are obtained, a structure attribute set is extracted, an industrial factory building BIM model is constructed, and a device abnormal state perception index is obtained through analysis; combining the equipment motion behavior data with the operation voiceprint signal data, extracting a behavior abnormity coordination index and a voiceprint abnormity identification index, and forming an operation evaluation set; according to the method, linkage display is carried out on each equipment primitive in the industrial factory building BIM model through the equipment heterogeneous risk indexes, so that coupling modeling, risk quantification and dynamic display among multimode anomalies are realized, and the method has the advantages that the equipment heterogeneous risk indexes of the equipment primitives in the industrial factory building BIM model are comprehensively analyzed; and therefore, the visualization depth, the anomaly recognition precision and the state evolution tracking capability of the BIM in equipment operation and maintenance can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data interactive display, and in particular to a multimodal data display method for a BIM model. Background Art

[0002] BIM, or Building Information Modeling, has been widely used in the planning, construction, and operation and maintenance management of industrial plants. BIM models can intuitively present information such as equipment distribution and space utilization at the spatial structure level, providing important support for visual management and equipment deployment at industrial sites. Traditional BIM systems mainly display the geometric position and annotation attributes of equipment during the operation and maintenance stage, but are unable to reflect behavioral changes, structural offsets, abnormal sound patterns, or vibration anomalies during equipment operation in real time. Even if some systems attempt to access sensor data, most of them only perform numerical superposition or alarm pop-ups, lacking the ability to model and analyze the intrinsic relationships between data. This results in an isolated existence between equipment graphics and multimodal data, making it difficult to form a unified perception-driven data view, affecting the application effect of BIM models in operation and maintenance scenarios such as intelligent diagnosis and fault diagnosis.

[0003] Prior art, such as the patent application with publication number CN118885650A, discloses a BIM-based multimodal data display method and system, which includes separating BIM model data, assembling the separated BIM models to obtain an assembled BIM model for rendering; processing multimodal single-point data to integrate the multimodal single-point data; binding the integrated multimodal single-point data with the rendering BIM model; and displaying the bound multimodal single-point data in a linked manner. The BIM-based multimodal data display method of the present invention includes BIM model digital-analog separation, multimodal single-point data processing, and multimodal single-point data digital-analog binding. Two integration schemes are proposed for integrating two different types of multimodal data, single-point multimodal data and homologous multi-point multimodal data, with the BIM model, respectively. This breaks the BIM data island phenomenon and combines multimodal data with BIM data.

[0004] Based on the above solution, it is found that the limitations of the existing technology include at least the following problems. The existing technology fails to establish the modeling capability of the correlation between multi-modal operating data such as behavioral changes, structural disturbances and acoustic anomalies during the operation of the equipment, and lacks a risk assessment mechanism to convert multi-source dynamic data into quantifiable, comparable and decision-making risk assessment. For example, in the operation and maintenance scenario of industrial plant equipment, a rotating equipment simultaneously experiences the following abnormal phenomena during operation: its internal rotor is slightly eccentric, resulting in unstable operating vibration frequency; the inertial measurement unit feedback shows that its posture has a continuous deviation trend; at the same time, the operating soundprint signal collected by the acoustic acquisition module shows a high amplitude energy leakage near the main frequency. For such operating states where the three modes simultaneously show abnormal characteristics, it is difficult for the existing technology to conduct joint modeling and analysis, thereby ignoring the synergistic effects and risk enhancement trends between multiple anomalies, resulting in coarse granularity and low accuracy of anomaly identification, and it is impossible to form a detailed expression of the anomaly type, severity level and its evolution trend. Summary of the Invention

[0005] In response to the technical problems existing in the prior art, the present invention provides a multimodal data display method for BIM models, which solves the problem that the prior art is difficult to display linkage based on multi-modal anomaly-driven graphic elements, and thus difficult to express the entire process of equipment status evolution.

[0006] The technical solution of the present invention to solve the above technical problems is as follows: a multimodal data display method of a BIM model, comprising the following steps: obtaining three-dimensional point cloud data of each device in a set industrial plant and image data at each angle, and inputting them into a pre-trained attribute recognition model for attribute extraction processing to obtain a structural attribute set of each device in the set industrial plant, and constructing an industrial plant BIM model, including a number of device primitives; inputting the structural attribute set of each device in the set industrial plant into the industrial plant BIM model for comprehensive analysis to obtain a device anomaly perception index for each device primitive in the industrial plant BIM model. ; and obtain the operating data of each equipment in the set industrial plant in real time, including equipment motion behavior data and operating voiceprint signal data, and input it into the industrial plant BIM model for feature analysis to obtain the operating evaluation set of each equipment element in the industrial plant BIM model, including the behavior abnormality coordination index and the voiceprint abnormality recognition index; conduct a comprehensive analysis of the operating evaluation set and equipment anomaly perception index of each equipment element in the industrial plant BIM model to obtain the equipment heterogeneous risk index of each equipment element in the industrial plant BIM model; based on the equipment heterogeneous risk index, each equipment element in the industrial plant BIM model is linked and displayed.

[0007] Furthermore, the specific formula for calculating the equipment heterogeneity risk index of a certain equipment in the industrial plant BIM model is as follows: ;in, The equipment heterogeneity risk index of a certain equipment in the industrial plant BIM model, It is the equipment anomaly perception index of a certain equipment in the BIM model of the industrial plant. is the abnormal perception adjustment coefficient stored in the database, The abnormal coordination index of a device in the BIM model of an industrial plant. is the behavior adjustment coefficient stored in the database, The voiceprint anomaly recognition index of a device in the BIM model of an industrial plant. is the voiceprint adjustment coefficient stored in the database, is the structural adjustment coefficient stored in the database.

[0008] Furthermore, the three-dimensional point cloud data specifically includes the voxel value and three-dimensional coordinates of each voxel point, the image data specifically includes the pixel value, two-dimensional coordinates and corresponding depth information value of each pixel point, the structural attribute set includes a geometric attribute set, a structural construction attribute set, a material attribute set, and a structural risk attribute set, and the attribute recognition model is specifically a multi-modal fusion recognition network, which includes a coding input layer, a feature fusion layer, a semantic construction layer, a risk modeling layer, and an attribute output layer.

[0009] Furthermore, the specific steps for obtaining the structural attribute set of each device in the set industrial plant are as follows: in the encoding input layer of the multi-mode fusion recognition network, the three-dimensional point cloud data of each device in the set industrial plant and the image data of each angle are received, and encoding processing is performed respectively to obtain the point cloud feature vector and image feature vector of each device in the set industrial plant; in the feature fusion layer of the multi-mode fusion recognition network, the point cloud feature vector and image feature vector of each device in the set industrial plant are fused and enhanced to obtain the fused feature vector of each device in the set industrial plant; in the semantic construction layer of the multi-mode fusion recognition network, the fused feature vector of each device in the set industrial plant is processed. Attribute semantic mapping processing is performed to obtain the attribute semantic vector of each device in the set industrial plant; in the risk modeling layer of the multi-mode fusion recognition network, risk expansion processing is performed on the attribute semantic vector of each device in the set industrial plant to obtain the attribute risk semantic vector of each device in the set industrial plant; in the attribute output layer of the multi-mode fusion recognition network, attribute prediction processing is performed on the attribute risk semantic vector of each device in the set industrial plant to obtain the geometric attribute set, structural construction attribute set, material attribute set, and structural risk attribute set of each device in the set industrial plant; the structural risk attribute set includes an obstruction risk index, a stability risk index, a corrosion risk index, and a fatigue risk index.

[0010] Furthermore, the specific steps for obtaining the equipment anomaly perception index of each equipment graphic element in the BIM model of the industrial plant are as follows: reading the obstruction risk index, stability risk index, corrosion risk index, and fatigue risk index of each equipment in the set industrial plant, and performing a comprehensive analysis to obtain the equipment anomaly perception index of each equipment in the set industrial plant; performing correlation processing on the equipment anomaly perception index of each equipment in the set industrial plant based on the BIM model of the industrial plant, and obtaining the equipment anomaly perception index of each equipment graphic element in the BIM model of the industrial plant.

[0011] Furthermore, the specific steps for obtaining the operation evaluation set of each equipment element in the industrial plant BIM model are as follows: based on the industrial plant BIM model, the operation data of each equipment in the set industrial plant are correlated and processed to obtain the operation data of each equipment element in the industrial plant BIM model; the operation data of each equipment element in the industrial plant BIM model are comprehensively analyzed to obtain the behavior abnormality coordination index and voiceprint abnormality recognition index of each equipment element in the industrial plant BIM model.

[0012] Furthermore, the equipment motion behavior data includes vibration frequency value, inertial bias response index, posture offset value, multi-axis phase difference index, and inertial offset loop area value. The specific steps for obtaining the abnormal behavior coordination index of each equipment element in the industrial plant BIM model are as follows: comprehensively analyze the equipment motion behavior data of each equipment element in the industrial plant BIM model to obtain a behavior evaluation set of each equipment element in the industrial plant BIM model, including an inertial configuration index and a disturbance imbalance index; and comprehensively analyze the behavior evaluation set of each equipment element in the industrial plant BIM model to obtain the abnormal behavior coordination index of each equipment element in the industrial plant BIM model.

[0013] Furthermore, the specific formula for calculating the behavioral anomaly coordination index of a certain equipment element in the industrial plant BIM model is as follows: ;in, It is the abnormal coordination index of a certain equipment element in the BIM model of the industrial plant. is the disturbance imbalance index of a certain equipment element in the industrial plant BIM model. is the disturbance adjustment coefficient stored in the database, It is the inertial configuration index of a certain equipment element in the BIM model of the industrial plant. is the inertia adjustment coefficient stored in the database, is the inhibition adjustment coefficient stored in the database.

[0014] Furthermore, the specific steps for obtaining the behavior evaluation set of each equipment element in the industrial plant BIM model are as follows: based on the genetic algorithm, the inertial bias response index and posture offset value of each equipment element in the industrial plant BIM model are comprehensively analyzed to obtain the inertial configuration index of each equipment element in the industrial plant BIM model; based on the genetic algorithm, the vibration frequency value, multi-axis phase difference index, and inertial offset loop area value of each equipment element in the industrial plant BIM model are comprehensively analyzed to obtain the disturbance imbalance index of each equipment element in the industrial plant BIM model.

[0015] Furthermore, the operating voiceprint signal data is specifically each instantaneous sound pressure amplitude, and the specific steps for obtaining the voiceprint anomaly recognition index of each equipment element in the industrial plant BIM model are as follows: based on the fast Fourier transform, the operating voiceprint signal data of each equipment element in the industrial plant BIM model is comprehensively analyzed to obtain a voiceprint evaluation index set of each equipment element in the industrial plant BIM model, including a voiceprint energy offset index and an abnormal bandwidth expansion index; and based on the genetic algorithm, the voiceprint evaluation index set of each equipment element in the industrial plant BIM model is comprehensively analyzed to obtain a voiceprint anomaly recognition index of each equipment element in the industrial plant BIM model.

[0016] The beneficial effects of the present invention are: based on the structural attributes, behavioral data and voiceprint signals of industrial equipment, a multi-source information fusion equipment graphic element operation evaluation system is constructed, that is, by introducing a structural attribute recognition network and a real-time operation data processing mechanism, a behavioral anomaly coordination index, a voiceprint anomaly recognition index and an equipment anomaly perception index are established, and further integrated to construct an equipment heterogeneous risk index, driving the color linkage, indicator display and subordinate information of the equipment graphic element status in the BIM model to be synchronously expanded, thereby realizing continuous perception and visual expression of state changes, and the graphic element linkage mechanism supports risk level color display and interactive response, and can expand detailed operation evaluation information when the user clicks, thereby realizing a closed-loop linkage of equipment state attribute modeling, operation collection, risk assessment, and graphic element display, and then realizing coupled modeling, risk quantification and dynamic display between multi-mode anomalies, thereby effectively improving the visualization depth, anomaly recognition accuracy and state evolution tracking capability of the BIM system in equipment operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a multimodal data display method for a BIM model of the present invention.

[0018] Figure 2 This is a flowchart of the specific steps for obtaining and setting the structural attribute set of each equipment in an industrial plant in a multimodal data display method of a BIM model of the present invention.

[0019] Figure 3This is a flowchart of the specific steps for obtaining the abnormal behavior coordination index of each equipment element in the BIM model of an industrial plant in a multimodal data display method of a BIM model of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0021] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the described features. In the description of this application, "plurality" means two or more, unless otherwise specifically specified.

[0022] In the description of this application, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art will recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0023] See also Figure 1, an embodiment of the present invention provides a technical solution: a multimodal data display method of a BIM model, comprising the following steps: obtaining three-dimensional point cloud data of each device in a set industrial plant and image data of each angle, and inputting them into a pre-trained attribute recognition model for attribute extraction processing to obtain a structural attribute set of each device in the set industrial plant, and constructing an industrial plant BIM model, including a number of device primitives; inputting the structural attribute set of each device in the set industrial plant into the industrial plant BIM model for comprehensive analysis to obtain a device anomaly perception index of each device primitive in the industrial plant BIM model; and obtaining the operating data of each device in the set industrial plant in real time (in It should be noted that real-time is not instantaneous, but takes an extremely short time window, such as 200ms as a time window), including equipment motion behavior data and operation voiceprint signal data, and inputs them into the industrial plant BIM model for feature analysis to obtain an operation evaluation set of each equipment element in the industrial plant BIM model, including a behavior abnormality coordination index and a voiceprint abnormality recognition index; a comprehensive analysis is performed on the operation evaluation set and equipment anomaly perception index of each equipment element in the industrial plant BIM model to obtain the equipment heterogeneous risk index of each equipment element in the industrial plant BIM model; and each equipment element in the industrial plant BIM model is linked and displayed based on the equipment heterogeneous risk index.

[0024] The three-dimensional point cloud data specifically includes the voxel value and three-dimensional coordinates of each voxel point (with the center position of the industrial plant as the origin of the coordinate system, the X-axis is used to indicate the horizontal spatial distribution direction of the industrial plant, pointing to the front of the equipment or the left-right direction of the space; the Y-axis is used to indicate the longitudinal spatial extension direction of the industrial plant, pointing to the right of the equipment or the front-back direction of the space; the Z-axis is used to indicate the height change of the industrial plant in the vertical direction, pointing directly upwards). The image data specifically includes the pixel value and two-dimensional coordinates of each pixel point (the upper left corner of the image is the origin, the X-axis indicates the horizontal direction of the image, increasing to the right, reflecting the pixel column index; the Y-axis indicates the vertical direction of the image, increasing downward, reflecting the pixel row index) and the corresponding depth information value. The structural attribute set includes the geometric attribute set, the structural construction attribute set, the material attribute set, and the structural risk attribute set. The attribute recognition model is specifically a multi-modal fusion recognition network (through the fusion of PointNet++ and ResNet-50). The multi-modal fusion recognition network includes an encoding input layer, a feature fusion layer, a semantic construction layer, a risk modeling layer, and an attribute output layer.

[0025] Among them, the geometric attribute set includes: length, width, height, volume, surface area, contour curvature, and spatial envelope size.

[0026] The structural construction attribute set includes: installation method, support structure type, connection type, number of interfaces and spatial distribution.

[0027] The material property set includes: main material type, surface treatment method, material thickness level, and aging level.

[0028] The encoding input layer is used to extract geometric and visual features from device point cloud and image data.

[0029] The feature fusion layer is used to align, fuse and enhance point cloud features with image features into a unified fused feature vector.

[0030] The semantic construction layer is used to convert the fused features into vector expressions with device attribute semantics.

[0031] The risk modeling layer is used to strengthen the feature dimensions related to equipment structure risks based on attribute semantics.

[0032] The attribute output layer is used to output a complete set of attributes such as geometry, construction, material, and structural risk.

[0033] The specific steps to build an industrial plant BIM model are as follows: Based on the geometric attribute set, the spatial dimensions and outer contour data of each equipment component are used as 3D modeling input to generate the corresponding geometric model. The spatial envelope size is then used to accurately layout its installation position, footprint, and spatial relationship in the BIM model. Then, based on the structural construction attribute set, the connection logic and support method between each equipment element and the plant structure or other equipment are defined. The connection relationship between equipment and components such as pipelines, foundations, and brackets is generated in the BIM platform, and the system-level equipment pipelines are automatically laid out based on the spatial distribution of interfaces. Finally, based on the material attribute set, the material, surface treatment, and degradation state of each equipment are bound to the BIM element as component attributes for subsequent material list generation and visual rendering (such as metal / spray effect simulation). The three attribute sets jointly support the automatic conversion of identified industrial equipment into 3D equipment elements with visible structure, resolvable connections, and controllable materials, and then bind them to the industrial plant BIM model.

[0034] The specific steps for linkage display of each equipment element in the industrial plant BIM model based on the equipment heterogeneity risk index are as follows: the equipment heterogeneity risk index of each equipment element in the industrial plant BIM model is judged and analyzed with the preset equipment heterogeneity risk index threshold range; if the equipment heterogeneity risk index of each equipment element in the industrial plant BIM model is lower than the lower limit of the preset equipment heterogeneity risk index threshold range, the equipment element is displayed in green; if the equipment heterogeneity risk index of each equipment element in the industrial plant BIM model is within the preset equipment heterogeneity risk index threshold range, the equipment element is displayed in green. Displayed in yellow; if the device heterogeneity risk index of each device element in the industrial plant BIM model is higher than the preset upper limit of the device heterogeneity risk index threshold range, the device element will be displayed in red; and when the user selects or clicks any device element in the industrial plant BIM model, the system will simultaneously expand the operation status detailed information window associated with the device element, and display the corresponding operation raw data summary of the device and the various indicator values ​​calculated in the window, including but not limited to the behavior abnormality coordination index, voiceprint abnormality recognition index, device anomaly perception index and the final generated device heterogeneity risk index.

[0035] The specific formula for calculating the equipment heterogeneity risk index of a certain equipment in the BIM model of an industrial plant is as follows: ;in, The equipment heterogeneity risk index of a certain equipment in the industrial plant BIM model, It is the equipment anomaly perception index of a certain equipment in the BIM model of the industrial plant. is the abnormal perception adjustment coefficient stored in the database, The abnormal coordination index of a device in the BIM model of an industrial plant. is the behavior adjustment coefficient stored in the database, The voiceprint anomaly recognition index of a device in the BIM model of an industrial plant. is the voiceprint adjustment coefficient stored in the database, is the structural adjustment coefficient stored in the database.

[0036] What needs to be explained is that the specific form of the tanh function is: ,in, is a natural constant, and in this embodiment, it can be taken as 2.71, and the domain is ( , ), the value range is ( , ).

[0037] 、 、 、 It can be obtained through the following steps: using historical data, combined with the device anomaly perception index, behavior abnormality coordination index and voiceprint abnormality recognition index, to conduct statistical regression analysis, quantify the specific impact of each factor on the device heterogeneous risk index, and thus fit the initial weight value; secondly, using the sensitivity analysis method, adjust the value range of each coefficient, observe its impact on the device heterogeneous risk assessment results, ensure the stability and rationality of the model, and based on the device characteristics and actual conditions, correct and optimize the preliminary fitted coefficients, and finally determine the coefficient value applicable to the specific device.

[0038] Specifically, such as Figure 2 As shown, the specific steps to obtain the structural attribute set of each device in the industrial plant are as follows: In the encoding input layer of the multimodal fusion recognition network, the 3D point cloud data of each device in the set industrial plant and the image data of each angle are received and encoded respectively (the point cloud data is input into the voxel feature extraction sub-network built based on the PointNet++ network structure. First, the original voxel point set is spatially downsampled by the Farthest Point Sampling algorithm, and the representative backbone voxel points are retained to reduce redundancy and maintain the global geometric distribution. Then, a local neighborhood is constructed around each sampling point, and the Ball Query, spherical neighborhood search selects a set of voxel points within a fixed radius to form a local substructure. In each local substructure, relative position encoding is used to align the position of the neighborhood point with the center point, and then the voxel value and coordinate information are input as joint features to the local feature learning module, including three layers of point-by-point multilayer perceptrons, namely MLP, which are used to extract local spatial geometric relationships and inter-point structural distribution features; then the maximum pooling operation is used to aggregate local features to obtain the local representation of each central sampling point, and the multi-level local features are stacked layer by layer to construct a global point cloud feature tensor. In the final output layer, a fixed-length point cloud feature vector is obtained through full connection mapping to characterize the volume morphology, edge structure, support contour and local structural change characteristics of the equipment components; the image data includes an image sequence covering multiple perspectives of the equipment, and each image is input as an independent input frame in turn to the image feature extraction subnetwork built on ResNet-50, which adopts co- The shared weight structure ensures the consistency and comparability of features from multiple perspectives. Each image first passes through the first two layers of convolution modules to extract low-level visual features such as color edges and surface textures, and then enters the residual block structure to extract material texture patterns, boundary continuity and color difference distribution information in multiple residual units. At the same time, the depth value and spatial position in the depth channel are fused into the intermediate feature map through position encoding to enhance the representation of complex surface conditions such as concave and convex areas, corrosion spots, and pollution boundaries. The output features of all residual blocks are processed by global average pooling to generate a fixed-dimensional image feature vector. The image feature vectors from all perspectives are then input into the angle fusion module. Through attention weighted fusion or angle smoothing averaging operations, a unified image feature vector is generated to comprehensively express the surface material type, color consistency and surface degradation signs of equipment components). The point cloud feature vector and image feature vector of each device in the set industrial plant are obtained; In the feature fusion layer of the multimodal fusion recognition network, the point cloud feature vector and image feature vector of each device in the set industrial plant are fused and enhanced (first, the two types of feature vectors are linearly transformed to make their dimensions consistent, and then used as input to construct a fusion representation. During the fusion process, the depth information value of each pixel in the image data is introduced. The fusion weight coefficient matrix is ​​generated based on the matching relationship between the depth value and the spatial coordinates in the point cloud structure. It is used to guide the enhancement of channels in the image feature vector that have a high degree of correspondence with the geometric structure and filter out image interference features that do not have spatial consistency. Subsequently, the point cloud features and the weighted image features are fused with cross-modal attention. The semantic correlation between the image channel and the point cloud channel is modeled using the cross-attention mechanism, and the feature return and stability optimization are performed through the residual enhancement module. The fusion result is then adaptively adjusted for the importance of various semantic features through the channel attention mechanism. Finally, the fused feature vector of each device in the set industrial plant is output as the input basis for component attribute identification and structural risk derivation). In the semantic construction layer of the multimodal fusion recognition network, attribute semantic mapping is performed on the fused feature vectors of each device in the set industrial plant. (The fused feature vectors are input into the attribute semantic mapping module, which consists of a multi-layer perceptron network (MLP). This module gradually enhances the semantic expression capability of the features through continuous nonlinear transformations and constructs a mapping channel between the fused features and the device attributes. Subsequently, spatial position encoding is introduced during the feature propagation process, and the relative coordinate information of the device in the plant is embedded as the position into the feature expression to strengthen the correspondence between geometric attributes and spatial semantics. A local association modeling mechanism is also introduced to construct logical adjacency relationships in the feature vector to simulate the combined semantics between the internal connection mode, installation structure, and morphological constraints of the device. Then, through feature channel grouping and interactive attention mechanism, the potential information under different semantic dimensions is classified, summarized, and selectively retained, constructing a high-order semantic space representation that includes geometric form, structural form, material texture, and layout features.) In the risk modeling layer of the multimodal fusion recognition network, the attribute semantic vector of each device in the set industrial plant is subjected to risk expansion processing (the attribute semantic vector is input into the risk modeling module, and adaptive reweighting processing is performed on each feature channel within it. Based on the changes in the eigenvalue distribution between channels, the semantic component responses that are highly correlated with risk factors are automatically enhanced. Risk factors include but are not limited to unstable equipment connections, surface corrosion, structural fatigue, and spatial occlusion. Subsequently, the risk-enhanced features are compressed and normalized using a nonlinear activation function to retain the differential expression between features. The enhanced features are then fused with the original attribute semantic vector using a residual connection mechanism to ensure that the risk dimension information is embedded while maintaining the integrity of the original semantics such as geometric attributes, structural attributes, and material attributes). The attribute risk semantic vector of each device in the set industrial plant is obtained; In the attribute output layer of the multi-mode fusion recognition network, attribute prediction processing is performed on the attribute risk semantic vector of each device in the set industrial plant (the semantic vector is mapped to four independent output branches through channel division and weight distribution, corresponding to the geometric attribute channel, structural construction attribute channel, material attribute channel and structural risk attribute channel respectively. In the geometric attribute channel, the feature dimensions reflecting the volume shape and boundary structure of the device in the semantic vector are extracted. This type of dimension mainly comes from the local spatial structure features learned by the point cloud channel in the encoding stage, such as neighborhood density, boundary position change rate, etc. A regression sub-network is constructed through a multi-layer perceptron (MLP) to map these spatial structure semantics into clear geometric parameters. Numerical values ​​are obtained to obtain a set of geometric attributes, including: length, width, and height, which correspond to the boundary dimensions of the equipment in the direction of the main axis of three-dimensional space; volume: obtained by fitting the comprehensive boundary scale and morphological complexity characteristics; surface area: estimating the surface expansion area of ​​the equipment based on the density of the point cloud and the irregularity of the boundary; contour curvature: fitting the curvature of the edge morphology using the trend of structural changes between points; spatial envelope size: outputting the structural enclosing range for BIM layout reference by calculating the minimum three-dimensional envelope box of the fitted boundary point set; in the structural construction attribute channel, semantic features related to the equipment support form, connection method and interface configuration are extracted. These semantics are jointly encoded in the image and point cloud fusion features, reflecting the connection area The boundary continuity, support position characteristics and topological morphology of the device are input into the classification sub-network, and the Softmax strategy is used to perform multi-label recognition on various structural features, and the structural attribute set is output, including: installation method: judged as ground, hoisted, embedded or suspended; support structure type: classified as single point, multi-point or suspended according to the proportional relationship between the center of gravity of the equipment and the bottom support structure; connection type: classified as rigid, flexible, plug-in or snap-on connection based on boundary clues; number of interfaces and spatial distribution: identify the number of device ports and their distribution vectors in three-dimensional space through the feature index mechanism; in the material attribute channel, extract the image feature vector generated by the image encoding network related to color change, surface texture and edge The semantic components related to clarity are used to construct a classification and regression joint output structure, branching processing on the shared basic feature extraction layer to output a set of material attributes, including: Main material type: Material classification is based on information such as color distribution and edge clarity, such as carbon steel, stainless steel, and aluminum alloy; Surface treatment method: Based on the overall contrast and texture continuity in the image, it is judged as painting, electroplating, polishing, sandblasting, etc.; Material thickness level: Based on the semantic representation of the change trend of the device boundary, it is estimated as thin, medium, or thickened through a regression model; Aging level: Based on the surface fading, texture disturbance and other features encoded in the image channel, a continuous score in the range of 0-1 is output to reflect the degree of degradation of the material surface state;In the structural risk attribute channel, the feature dimensions enhanced by the risk modeling layer are extracted from the attribute risk semantic vector. This type of semantic feature has automatically focused on key factors related to potential risks of the equipment through the front-layer network, such as weak installation structure, insufficient boundary contact area, surface damage or space obstruction and other risk signs. The risk attribute channel adopts a regression structure to establish prediction branches for different categories of risk factors, and through feature compression and normalization processing, a continuous risk index in the range of 0~1 is generated for each type of risk factor, and a set of structural risk attributes is output, including: Obstruction risk index: Based on the position of the equipment in the spatial layout, the interface orientation and the envelope size, it is evaluated whether the equipment has problems such as blocking the passage path and hindering maintenance operations, reflecting Spatial accessibility risk; Stability risk index: Combined with the equipment's support structure, connection type, and force distribution semantics, it infers its anti-overturning or anti-vibration capabilities, indicating the structural stability risk of the equipment during operation or under external force interference; Corrosion risk index: Derived from visual features such as surface color difference changes and edge texture disturbances encoded in the image channel, it quantifies the risk of corrosion, rust, or oxidation degradation on the surface; Fatigue risk index: Based on the combined feature expression of sharp corners, local stress concentration features, or thin material areas in the geometric shape, it regresses to assess the fatigue damage risk that may exist after long-term operation of the structure), and obtains the geometric attribute set, structural construction attribute set, material attribute set, and structural risk attribute set of each equipment in the set industrial plant; The structural risk attribute set includes obstruction risk index, stability risk index, corrosion risk index, and fatigue risk index.

[0039] Among them, the pre-training process of the multi-modal fusion recognition network is as follows: Acquire annotated multimodal image datasets of industrial equipment, including 3D point cloud data and multi-angle image data of each equipment component, along with precise annotation information (such as equipment bounding boxes, support connection labels, material type annotations, and local structure risk level annotations). Unify the format of the raw data, perform point cloud voxelization, and enhance image standardization. The data are then divided into multimodal training and validation sets to ensure balanced coverage of different component types, installation methods, and surface materials.

[0040] Initialize the multimodal fusion recognition network and randomly initialize all trainable weights using the Kaiming method. At the same time, load some encoder weights pre-trained on general 3D model libraries (such as ShapeNet) and image classification datasets (such as ImageNet) to enhance the expressive power of multimodal input channels and the cross-modal migration performance of the model.

[0041] The network is pre-trained based on a multi-modal training set, and the training cycle is set (e.g., 100 rounds). In each round of training, forward propagation is performed in sequence (point cloud and image data are input into the network, and after encoding, fusion, semantic construction, and attribute prediction modules, geometric, structural, material, and risk attribute results are output), multi-task loss function calculation (including: geometric regression loss, structural construction cross entropy loss, material classification loss, and risk index regression loss, which are integrated into the total loss using a weighted strategy), backpropagation, and parameter update (network parameter iterative update is performed using the AdamW optimizer combined with gradient clipping, learning rate hot start, and Cosine Decay strategies).

[0042] During the training process, the model is evaluated on the multi-modal validation set after each round, and the prediction results of each attribute are output and the accuracy indicators are calculated separately, including the mean square error (MSE) of geometric attributes, the accuracy (Accuracy) and IoU indicators of structure and material classification, the mean deviation (MAE) and F1 score of risk prediction, etc. At the same time, the training loss and indicator change curves are plotted to monitor the model convergence progress; if the verification performance does not improve after several consecutive rounds, the Early Stopping mechanism is activated to automatically terminate the training early to prevent overfitting.

[0043] Finally, the converged multi-mode fusion recognition network model weight file is saved and exported to a deployment format to support subsequent component attribute identification and structural risk analysis tasks in the industrial plant BIM system.

[0044] The specific steps for obtaining the equipment anomaly perception index of each equipment element in the BIM model of the industrial plant are as follows: read the obstruction risk index, stability risk index, corrosion risk index, and fatigue risk index of each equipment in the set industrial plant, and conduct a comprehensive analysis to obtain the equipment anomaly perception index of each equipment in the set industrial plant; based on the BIM model of the industrial plant, associate the equipment anomaly perception index of each equipment in the set industrial plant (that is, obtain the unique identification information of each equipment, the identification information is the preset equipment number, and standardize the identification format to meet the matching requirements with the equipment elements in the BIM model of the industrial plant, and then parse the component elements of the industrial plant BIM model to extract the equipment elements in the model. The basic attribute fields of the element instance of the class are obtained, including element number, system affiliation, component type, installation location and other information, and the element attribute structure index is established. In the target equipment element in each BIM model, an extended field for carrying external status values ​​is added to receive the equipment anomaly perception index corresponding to the element. Then, based on the mapping relationship between equipment identification information and element number, the equipment anomaly perception index of each device is written into the extended field of the corresponding equipment element in the BIM model to form a one-to-one data binding relationship. Finally, after completing the writing of the status values ​​of all equipment elements, the overall status of the BIM model is synchronized to obtain the equipment anomaly perception index of each equipment element in the BIM model of the industrial plant.

[0045] The specific formula for calculating the device abnormality perception index of a certain device in a given industrial plant is as follows: ;in, To set the device anomaly perception index of a certain device in an industrial plant, To set the obstruction risk index for a piece of equipment in an industrial plant, is the barrier adjustment coefficient stored in the database, To set the stability risk index of a piece of equipment in an industrial plant, is the stability adjustment coefficient stored in the database, To set the corrosion risk index for a piece of equipment in an industrial plant, is the corrosion adjustment coefficient stored in the database, To set the fatigue risk index for a piece of equipment in an industrial plant, is the fatigue adjustment coefficient stored in the database, is the interaction adjustment coefficient stored in the database.

[0046] What needs to be explained is that 、 、 、 、 It can be obtained through the following steps: Based on historical data, determine the initial impact weight of each variable (obstruction risk index, stability risk index, corrosion risk index, fatigue risk index) on the equipment anomaly perception index through statistical regression analysis. Then, use the sensitivity analysis method to adjust the value range of the coefficient to evaluate the stability and applicability of these parameters to the formula output. Next, further fit the weights through model optimization (such as machine learning algorithms) to ensure that the formula can accurately reflect the anomaly perception status of the actual equipment. Fine-tune the coefficients based on different equipment characteristics to ensure that they are suitable for specific equipment anomaly perception evaluation needs.

[0047] The following is a specific implementation example of calculating the device abnormality perception index of a certain device in a given industrial plant. The following data is available, including the obstruction risk index, stability risk index, corrosion risk index, and fatigue risk index of three devices (randomly selected) in the given industrial plant, as shown in Table 1: Table 1 Example of risk attribute set data for equipment sequence structure in industrial plants

[0048] Obstacle adjustment coefficient stored in the database Approximately: 0.294; Stability adjustment coefficient stored in the database Approximately: 0.306; Corrosion adjustment factors stored in the database Approximately: 0.216; Fatigue adjustment factors stored in the database Approximately: 0.349; Interaction adjustment coefficients stored in the database Approximately: 1.573; Substituting the data in Table 1 and the above adjustment coefficient into the specific formula for calculating the device abnormality perception index of a certain device in a given industrial plant, we obtain: Set the device anomaly perception index of the first device in the industrial plant to be √(0.231^0.294×0.318^0.306×0.213^0.216×0.438^0.349) / exp(-1.573×0.231×0.318×0.213×0.438)≈0.502; Set the device anomaly perception index of the second device in the industrial plant to be √(0.128^0.294×0.289^0.306×0.267^0.216×0.364^0.349) / exp(-1.573×0.128×0.289×0.267×0.364)≈0.479; The device anomaly perception index of the third device in the industrial plant is set to √(0.157^0.294×0.374^0.306×0.183^0.216×0.516^0.349) / exp(-1.573×0.57×0.374×0.183×0.516)≈0.485.

[0049] In this implementation scheme, through deep analysis and semantic construction of the multimodal raw data of industrial equipment, the multi-dimensional attribute information such as the geometric structure, connection structure, surface material and potential structural risks of the equipment is comprehensively extracted. Secondly, a multimodal recognition network integrating PointNet++ and ResNet-50 is adopted to introduce multi-angle image texture semantics while maintaining the structural details of the point cloud. In this way, the final recognition result not only has the ability to accurately express the spatial structure, but also can effectively reflect the surface treatment process and degradation status of the equipment. Through the step-by-step abstraction of the semantic construction layer and the risk modeling layer, a mapping mechanism from raw perception data to high-order structural attributes is established. Finally, the output attribute set provides semantic closed-loop support for subsequent equipment status assessment, BIM component generation and intelligent linkage display, thereby enhancing the stability, interpretability and scalability of attribute recognition, and thus being suitable for structural modeling and multimodal data visualization application scenarios of heterogeneous equipment in complex industrial environments.

[0050] Specifically, the specific steps for obtaining the operation evaluation set of each equipment element in the BIM model of the industrial plant are as follows: based on the BIM model of the industrial plant, the operation data of each equipment in the set industrial plant are correlated (consistent with the logic of correlating the equipment anomaly perception index of each equipment in the set industrial plant) to obtain the operation data of each equipment element in the BIM model of the industrial plant; the operation data of each equipment element in the BIM model of the industrial plant are comprehensively analyzed to obtain the behavior abnormality coordination index and voiceprint abnormality recognition index of each equipment element in the BIM model of the industrial plant.

[0051] The equipment motion behavior data includes vibration frequency value, inertial bias response index, attitude offset value, multi-axis phase difference index, and inertial offset loop area value. The specific steps for obtaining the abnormal behavior coordination index of each equipment element in the industrial plant BIM model are as follows: comprehensively analyze the equipment motion behavior data of each equipment element in the industrial plant BIM model to obtain a behavior evaluation set of each equipment element in the industrial plant BIM model, including the inertial configuration index and the disturbance imbalance index; and comprehensively analyze the behavior evaluation set of each equipment element in the industrial plant BIM model to obtain the abnormal behavior coordination index of each equipment element in the industrial plant BIM model.

[0052] The vibration frequency value is the mechanical vibration generated by the reciprocating, rotating or colliding components of the equipment during operation, which can be obtained through a vibration sensor.

[0053] The inertial bias response index (IBRI) measures the offset strength of the device's inertial response during operation (e.g., the asymmetry and deviation exhibited by the device's inertial response). The inertial bias response index (IBRI) is calculated by obtaining the angular velocity along three axes using a three-axis gyroscope and the acceleration along three axes using a three-axis accelerometer. These values ​​are then averaged to obtain the mean angular velocity and mean acceleration. These values ​​are then normalized and weighted based on the normalization results. The resulting value is the IBI.

[0054] The attitude offset value is the change in the spatial attitude angle of the device's overall structure during operation. It uses a 9-axis IMU (three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer) to obtain the device's acceleration, angular velocity, and geomagnetic direction signals in the X, Y, and Z axes. The raw data collected by the IMU is input into the embedded attitude calculation module, which uses complementary filtering, Mahony filtering, or the Madgwick algorithm to fuse these three types of signals to calculate the device's current pitch and roll angles. After the device is first installed, the pitch and roll angles at that moment are recorded and marked as the pitch reference angle and roll reference angle, respectively. The current pitch and roll angles are then subtracted from the pitch reference angle and roll reference angle, respectively, to obtain the current pitch angle difference and roll angle difference. These values ​​are then normalized and weighted based on the normalization results. The resulting value is the attitude offset value.

[0055] The multi-axis phase difference index quantifies the degree of phase synchronization deviation between the vibration signals of each axis of the equipment during operation. It can obtain the vibration acceleration signals of the equipment in the X, Y, and Z directions through a three-axis accelerometer and perform high-frequency synchronous sampling (≥1kHz). The three-axis acceleration signal is then bandpass filtered, and the main operating frequency range of the equipment (such as 10-500Hz) is selected to eliminate DC offset and high-frequency interference. Subsequently, the short-time Fourier transform (STFT) or Hilbert transform is used to extract the main frequency phase of the three-axis signal, and the real-time phase difference between any two axes is calculated. The calculated result is weighted and processed to obtain the result of the multi-axis phase difference index.

[0056] The inertial offset loop area value is the relative change trend and energy asymmetry between the angular velocity and linear acceleration responses during device operation. This value is obtained by synchronously acquiring angular velocity and linear acceleration signals through an inertial measurement unit (i.e., a three-axis gyroscope and a three-axis accelerometer). The X-axis is selected as the principal axis signal output. The currently acquired angular velocity is used as the X-axis coordinate, and the acceleration as the Y-axis coordinate. The current inertial state point on the two-dimensional characteristic plane is constructed. This point forms an inertial trajectory arc with the point at the previous moment, which is continuously connected over time to form a dynamic open loop trajectory. The infinitesimal area calculation is then performed on the current data point pair (i.e., the current inertial state point and the inertial state point at the previous moment). The incremental area is solved using the trapezoidal method or the vector cross product method. The result is the inertial offset loop area value (which is 0 at the first moment).

[0057] The specific steps for obtaining the behavior evaluation set of each equipment element in the BIM model of the industrial plant are as follows: Based on the genetic algorithm, the inertia bias response index and posture offset value of each equipment element in the BIM model of the industrial plant are comprehensively analyzed (that is, first standardized, and weighted based on the standardized processing results, and the weight coefficients corresponding to the inertia bias response index and the posture offset value are obtained through the genetic algorithm. Specifically, a weight coefficient search space is constructed, and the weight coefficients corresponding to the inertia bias response index and the posture offset value are recorded as A and B respectively, and they are set to meet the normalization constraint condition, that is, A+B=1, and at the same time, their value range is limited to between 0-1 to ensure the validity and distinguishability of the weight distribution. Then, a fitness evaluation mechanism of the genetic algorithm is constructed based on the pre-acquired historical structure sample set, and the historical inertia bias response index and posture offset value of multiple equipment elements are used as input samples. Combined with their known inertial configuration state labels, the sample prediction error is constructed as the objective function, specifically the difference between the inertial configuration index calculated under each set of weights and the reference annotation value. The mean square error is used as the fitness function, i.e., the evaluation criterion for the quality of weight combinations. Next, the genetic algorithm population is initialized, and several groups of weight coefficient combinations that meet the constraints are used as the initial population individuals to form the first generation of candidate solutions. During the genetic algorithm iteration process, individual selection, crossover recombination, and mutation operations are performed respectively. The selection operation uses a roulette or tournament method to retain excellent solutions based on fitness ranking. The crossover operation generates new individuals by exchanging parent weight fragments at random intersections. The mutation operation enhances population diversity by applying small perturbations to individual weight values, preventing them from falling into local optimality. Subsequently, the weight coefficient combination is iteratively updated based on the fitness value of each individual in each generation of the population. The optimization process is terminated when the preset maximum number of iterations is reached or the fitness change of multiple generations of the population is below the convergence threshold. Finally, the weight combination with the best fitness is extracted as the optimal weight coefficient for the inertial bias response index and attitude offset value, and the inertial configuration index (which measures the degree of mismatch between the inertial direction distribution and structural configuration of the equipment during operation) of each equipment element in the industrial plant BIM model is obtained. Based on the genetic algorithm, a comprehensive analysis is performed on the vibration frequency value, multi-axis phase difference index, and inertia offset loop area value of each equipment element in the industrial plant BIM model (first standardization is performed, and then weighted processing is performed based on the normalization result. The weight coefficients corresponding to the vibration frequency value, multi-axis phase difference index, and inertia offset loop area value are obtained through the genetic algorithm, and the logic is consistent with the weight coefficients corresponding to the inertia bias response index and posture offset value obtained through the genetic algorithm). The disturbance imbalance index of each equipment element in the industrial plant BIM model is obtained.

[0058] The specific formula for calculating the behavioral anomaly coordination index of a certain equipment element in the industrial plant BIM model is as follows: ;in, It is the abnormal coordination index of a certain equipment element in the BIM model of the industrial plant. is the disturbance imbalance index of a certain equipment element in the industrial plant BIM model. is the disturbance adjustment coefficient stored in the database, It is the inertial configuration index of a certain equipment element in the BIM model of the industrial plant. is the inertia adjustment coefficient stored in the database, is the inhibition adjustment coefficient stored in the database.

[0059] What needs to be explained is: This item is used to adjust the inhibitory effect of the inertial configuration index on the disturbance imbalance index to avoid the abnormal behavior coordination index being too high or too low.

[0060] 、 、 It can be obtained through the following steps: Based on historical data, determine the initial influence weight of each variable (disturbance imbalance index, inertial configuration index) on the behavioral abnormal coordination index through statistical regression analysis. Then, use the sensitivity analysis method to adjust the value range of the coefficient to evaluate the stability and applicability of these parameters to the formula output. Next, further fit the weights through model optimization (such as multi-objective optimization) to ensure that the formula can accurately reflect the abnormal state of actual behavioral abnormal coordination.

[0061] In this implementation scheme, five types of physically interpretable behavioral characteristic indicators are extracted from the high-frequency inertial measurement data obtained during the operation of the equipment, and the optimal weighted combination of multiple parameters is achieved by combining genetic algorithms, thereby constructing two intermediate evaluation quantities, the inertial configuration index and the disturbance imbalance index. The behavioral abnormality coordination index is then derived through the nonlinear coupling relationship, which can characterize the comprehensive behavioral performance of the equipment in terms of posture stability, structural dynamic response, vibration coordination, etc., and then quantify the abnormal offset and imbalance trend during the movement of the equipment. This step realizes the fusion modeling of multi-dimensional inertial behavioral parameters, which has higher stability, robustness and dynamic recognition capabilities, thereby providing high-resolution behavioral feature support for the operation status assessment, risk modeling and multi-modal display of equipment graphics in the BIM model of industrial plants, and effectively enhancing the model's recognition granularity and judgment reliability of dynamic anomalies of complex equipment.

[0062] Specifically, the operating voiceprint signal data is each instantaneous sound pressure amplitude (within the time window), and the specific steps for obtaining the voiceprint anomaly recognition index of each equipment element in the industrial plant BIM model are as follows: based on the fast Fourier transform, the operating voiceprint signal data of each equipment element in the industrial plant BIM model are comprehensively analyzed to obtain a voiceprint evaluation index set of each equipment element in the industrial plant BIM model, including a voiceprint energy offset index and an abnormal bandwidth expansion index; and based on the genetic algorithm, the voiceprint evaluation index set of each equipment element in the industrial plant BIM model is comprehensively analyzed (that is, first normalization is performed, and weighted processing is performed based on the normalization result, and the weight coefficients corresponding to the voiceprint energy offset index and the abnormal bandwidth expansion index are obtained through the genetic algorithm, and are logically consistent with the weight coefficients corresponding to the inertial bias response index and the posture offset value obtained through the genetic algorithm), to obtain the voiceprint anomaly recognition index of each equipment element in the industrial plant BIM model.

[0063] Among them, the specific steps of obtaining the voiceprint evaluation index set of each equipment element in the industrial plant BIM model are as follows: perform window function weighted processing on the operating voiceprint signal data of each equipment element in the industrial plant BIM model, use Hamming window to suppress spectrum leakage, and perform fast Fourier transform (FFT) on the windowed operating voiceprint signal data of each equipment element in the industrial plant BIM model to obtain the frequency domain energy spectrum of each equipment element in the industrial plant BIM model, including the sound energy of each frequency point, which is used to characterize the energy distribution of the voiceprint signal at each frequency; perform weighted averaging processing on the sound energy of each frequency point of each equipment element in the industrial plant BIM model to obtain the spectrum centroid value of each equipment element in the industrial plant BIM model, and obtain the reference spectrum centroid value of each equipment element in the industrial plant BIM model (by obtaining the historical spectrum centroid values ​​of several historical time periods and performing average processing to obtain the reference spectrum centroid value), and perform ratio processing (that is, the absolute value of the spectrum centroid value and the reference spectrum centroid value). The sound energy offset index of each equipment element in the industrial plant BIM model is obtained by calculating the sound energy difference / reference spectrum centroid value. The sound energy of each frequency point of each equipment element in the industrial plant BIM model is arranged in ascending order, and the cumulative energy proportion of each frequency point is calculated in turn, that is, the proportion of the sum of its energy from the lowest frequency point to the current frequency point to the total energy of the entire frequency domain. By calculating the cumulative energy proportion of each frequency point in turn, a complete frequency cumulative energy distribution curve can be constructed. Two specific frequency points are counted, namely the lower boundary frequency point where the cumulative energy reaches 5% and the upper boundary frequency point where the cumulative energy reaches 95%, and the frequency difference between the two is calculated, which is called the main energy distribution bandwidth of each equipment element in the industrial plant BIM model, representing the main coverage range of the equipment sound energy in the spectrum. The frequency point with the largest energy value is counted to determine the main frequency position. Finally, the ratio of the main energy distribution bandwidth of each equipment element in the industrial plant BIM model to the corresponding main frequency value is calculated to obtain the abnormal bandwidth expansion index.

[0064] In this implementation, by converting the original sound pressure signal during the operation of the equipment into a frequency domain energy spectrum, efficient modeling and anomaly identification of acoustic behavior characteristics are achieved, thereby significantly improving the fine-grained perception capability of the industrial plant BIM system of equipment status changes. That is, through window function weighting and fast Fourier transform, the energy distribution of the voiceprint signal at each frequency point can be accurately extracted, and highly expressive voiceprint evaluation indexes such as the spectrum centroid and main frequency bandwidth can be constructed, thereby effectively characterizing the energy offset characteristics and frequency diffusion trends during the operation of the equipment. A genetic algorithm is further introduced to perform weighted integration of multiple voiceprint evaluation indexes to construct a unified voiceprint anomaly identification index, thereby realizing adaptive fusion analysis among multiple indicators and enhancing the robust recognition capability of complex voiceprint abnormal states, thereby providing a high-precision acoustic decision-making basis for the linkage of graphic element display.

[0065] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0066] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0070] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0071] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A multimodal data display method for a BIM model, characterized in that: The following steps are involved: Obtain the 3D point cloud data and image data from each angle of each device in the set industrial plant, and input them into the pre-trained attribute recognition model for attribute extraction processing to obtain the structural attribute set of each device in the set industrial plant, and construct the industrial plant BIM model, including several equipment primitives; Input the structural attribute set of each equipment in the industrial plant into the industrial plant BIM model for comprehensive analysis to obtain the equipment anomaly perception index of each equipment element in the industrial plant BIM model; The system also acquires the operating data of each device in the designated industrial plant in real time, including the device motion behavior data and operating voiceprint signal data, and inputs them into the industrial plant BIM model for feature analysis. This system then obtains the operating evaluation set of each device element in the industrial plant BIM model, including the abnormal behavior coordination index and the abnormal voiceprint recognition index. Comprehensively analyze the operation evaluation set and equipment anomaly perception index of each equipment element in the industrial plant BIM model to obtain the equipment heterogeneity risk index of each equipment element in the industrial plant BIM model; Based on the equipment heterogeneous risk index, each equipment element in the industrial plant BIM model is displayed in a linked manner.

2. The multimodal data display method of the BIM model according to claim 1, characterized in that: The specific formula for calculating the equipment heterogeneity risk index of a certain equipment in the BIM model of an industrial plant is as follows: ； in, 、 、 、 They are the equipment heterogeneity risk index, equipment anomaly perception index, behavior anomaly coordination index, and voiceprint anomaly recognition index of a certain equipment in the BIM model of the industrial plant. 、 、 、 They are the abnormal perception adjustment coefficient, behavior adjustment coefficient, voiceprint adjustment coefficient, and structure adjustment coefficient stored in the database respectively.

3. The multimodal data display method of the BIM model according to claim 1, characterized in that: The three-dimensional point cloud data specifically includes the voxel value and three-dimensional coordinates of each voxel point; the image data specifically includes the pixel value, two-dimensional coordinates and corresponding depth information value of each pixel point; the structural attribute set includes a geometric attribute set, a structural construction attribute set, a material attribute set, and a structural risk attribute set; the attribute recognition model is specifically a multi-modal fusion recognition network, which includes a coding input layer, a feature fusion layer, a semantic construction layer, a risk modeling layer, and an attribute output layer.

4. The multimodal data display method of the BIM model according to claim 3, characterized in that: The specific steps to obtain the structural attribute set of each device in the industrial plant are as follows: In the encoding input layer of the multi-modal fusion recognition network, the three-dimensional point cloud data of each device in the set industrial plant and the image data of each angle are received and encoded respectively to obtain the point cloud feature vector and image feature vector of each device in the set industrial plant; In the feature fusion layer of the multi-mode fusion recognition network, the point cloud feature vector and image feature vector of each device in the set industrial plant are fused and enhanced to obtain the fused feature vector of each device in the set industrial plant; In the semantic construction layer of the multi-modal fusion recognition network, attribute semantic mapping is performed on the fusion feature vector of each device in the set industrial plant to obtain the attribute semantic vector of each device in the set industrial plant; In the risk modeling layer of the multi-mode fusion recognition network, the attribute semantic vector of each device in the set industrial plant is subjected to risk expansion processing to obtain the attribute risk semantic vector of each device in the set industrial plant; In the attribute output layer of the multi-mode fusion recognition network, attribute prediction processing is performed on the attribute risk semantic vector of each device in the set industrial plant, and the geometric attribute set, structural attribute set, material attribute set, and structural risk attribute set of each device in the set industrial plant are obtained; The structural risk attribute set includes an obstruction risk index, a stability risk index, a corrosion risk index, and a fatigue risk index.

5. The multimodal data display method of the BIM model according to claim 4, characterized in that: The specific steps to obtain the equipment anomaly perception index of each equipment element in the industrial plant BIM model are as follows: Read the obstruction risk index, stability risk index, corrosion risk index, and fatigue risk index of each device in the set industrial plant, and conduct a comprehensive analysis to obtain the device abnormality perception index of each device in the set industrial plant; Based on the BIM model of the industrial plant, the equipment anomaly perception index of each equipment in the set industrial plant is associated and processed to obtain the equipment anomaly perception index of each equipment element in the BIM model of the industrial plant.

6. The multimodal data display method of the BIM model according to claim 1, characterized in that: The specific steps to obtain the operation evaluation set of each equipment element in the industrial plant BIM model are as follows: Based on the industrial plant BIM model, the operating data of each device in the set industrial plant is correlated and processed to obtain the operating data of each device element in the industrial plant BIM model; The operation data of each equipment element in the BIM model of the industrial plant is comprehensively analyzed to obtain the behavior abnormality coordination index and voiceprint abnormality recognition index of each equipment element in the BIM model of the industrial plant.

7. The multimodal data display method of the BIM model according to claim 6, characterized in that: The equipment motion behavior data includes vibration frequency value, inertia bias response index, attitude offset value, multi-axis phase difference index, and inertia offset loop area value. The specific steps for obtaining the abnormal behavior coordination index of each equipment element in the industrial plant BIM model are as follows: Comprehensively analyze the equipment motion behavior data of each equipment element in the industrial plant BIM model to obtain a behavior evaluation set for each equipment element in the industrial plant BIM model, including the inertial configuration index and the disturbance imbalance index; A comprehensive analysis is also conducted on the behavior evaluation set of each equipment element in the industrial plant BIM model to obtain the behavioral abnormal coordination index of each equipment element in the industrial plant BIM model.

8. The multimodal data display method of the BIM model according to claim 7, characterized in that: The specific formula for calculating the behavioral anomaly coordination index of a certain equipment element in the industrial plant BIM model is as follows: ； in, 、 、 They are the abnormal coordination index, disturbance imbalance index, and inertial configuration index of a certain equipment element in the BIM model of the industrial plant. 、 、 They are the disturbance adjustment coefficient, inertia adjustment coefficient, and suppression adjustment coefficient stored in the database respectively.

9. The multimodal data display method of the BIM model according to claim 7, characterized in that: The specific steps to obtain the behavior evaluation set of each equipment element in the industrial plant BIM model are as follows: Based on the genetic algorithm, the inertial bias response index and attitude offset value of each equipment element in the industrial plant BIM model are comprehensively analyzed to obtain the inertial configuration index of each equipment element in the industrial plant BIM model; Based on the genetic algorithm, the vibration frequency value, multi-axis phase difference index and inertia offset loop area value of each equipment element in the industrial plant BIM model are comprehensively analyzed to obtain the disturbance imbalance index of each equipment element in the industrial plant BIM model.

10. The multimodal data display method of the BIM model according to claim 6, characterized in that: The operation voiceprint signal data is specifically each instantaneous sound pressure amplitude. The specific steps for obtaining the voiceprint anomaly recognition index of each equipment element in the industrial plant BIM model are as follows: Based on the fast Fourier transform, the operating voiceprint signal data of each equipment element in the industrial plant BIM model is comprehensively analyzed to obtain a set of voiceprint evaluation indices for each equipment element in the industrial plant BIM model, including the voiceprint energy deviation index and the abnormal bandwidth expansion index. Based on the genetic algorithm, a comprehensive analysis of the voiceprint evaluation index set of each equipment element in the industrial plant BIM model is performed to obtain the voiceprint anomaly recognition index of each equipment element in the industrial plant BIM model.

Images