PC component BIM twinborn collaborative exchange method based on deep learning

Abstract

The invention discloses a PC component BIM twinborn collaborative exchange method based on deep learning, and aims to solve the problems that unstructured multi-modal data of a site or a factory is difficult to automatically identify the identity, the state and the quality of a PC component and accurately map the unstructured multi-modal data with BIM twinborn objects one by one, and increment updating capable of realizing multi-party collaborative exchange is difficult to form. According to the method, time synchronization and coordinate calibration preprocessing is carried out on image data, video data and point cloud data, component instance recognition and feature extraction are carried out by using a deep learning model, and component state information and component quality information are generated; analyzing a BIM twinborn model to obtain priori features such as a unique identifier of a component, a component type, a size parameter, a spatial position and component geometry, performing cross-modal feature fusion under priori guidance, generating a candidate matching relationship, and constructing an optimal transmission cost matrix containing feature difference and constraint penalty to obtain a soft matching matrix; one-to-one mapping is obtained by adopting matching confidence rejection and assignment solution, an identification result and an evidence data index are written into a twinborn object, component-level difference is carried out on the twinborn object and a previous version to generate evidence increment exchange data, and the technical effects of high-reliability automatic updating and traceable cooperative exchange of the component-level twinborn model are achieved.

Description

Technical Field

[0001] This invention relates to the field of building information technology (BIM), and more particularly to a deep learning-based method for collaborative exchange of PC component BIM twins. Background Technology

[0002] In prefabricated construction, the production, transportation, storage management, and on-site installation of precast concrete components typically rely on Building Information Modeling (BIM) for planning, location, and quality traceability. With the development of BIM and digital twin technologies, the industry has gradually moved from manual ledgers and two-dimensional drawing management to component-level information management centered on BIM, and has begun to introduce perception methods such as image acquisition, video capture, and laser point cloud scanning to obtain actual status data from the site or factory. Meanwhile, deep learning has matured in object detection, instance segmentation, and 3D point cloud understanding, making it possible to automatically identify components from unstructured data and write back to the twin model; in terms of multi-party collaboration, there is also a growing demand for cross-participant synchronization based on model files or exchanged data.

[0003] Existing technologies still have shortcomings in automatically transforming unstructured sensed data into collaboratively exchangeable component-level twin incremental updates, mainly including:

[0004] First, there are problems such as time asynchrony, coordinate inconsistency, noise and occlusion among multimodal data, which leads to unstable component instance recognition and feature representation, making it difficult to simultaneously take into account component identity recognition, state recognition and quality judgment.

[0005] Second, the one-to-one mapping between the identified components and the building information model components usually relies on manual confirmation or greedy matching based on feature similarity. It lacks a global optimal matching mechanism that utilizes prior constraints such as component type, size parameters, and spatial location, which is prone to mismatch, rematch, and omission. Furthermore, it lacks the process of rejecting and verifying low-confidence results.

[0006] Third, twin information updates and collaborative exchanges often adopt whole-model overlay updates or coarse-grained synchronization methods, which makes it difficult to achieve semantic differential incremental exchanges at the unique identifier level of components. They also lack evidence data and verification mechanisms corresponding to the changed content, resulting in low collaborative efficiency and insufficient traceability.

[0007] Therefore, a method for collaborative exchange of PC component BIM twins that can overcome the shortcomings of the existing technology is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0008] One objective of this invention is to propose a deep learning-based collaborative exchange method for PC component BIM twins. Addressing the challenges of existing technologies where on-site or factory image, video, and point cloud data are difficult to automatically identify component identities, states, and qualities, and where accurate one-to-one mapping with BIM twin model components is difficult, and update results are hard to form collaboratively exchangeable component-level incremental data, this invention proposes a technical solution that involves time synchronization and coordinate calibration preprocessing of multimodal data, using a deep learning model for component instance identification and feature extraction to obtain evidence data, parsing the BIM twin model to generate prior features of components, performing cross-modal fusion under prior guidance, generating candidate matching relationships based on constraints, constructing an optimal transmission cost matrix including feature difference terms and constraint penalty terms to obtain a soft matching matrix, performing rejection and one-to-one assignment solutions based on matching confidence, writing the state, quality, and confidence back to the twin object, and generating evidence-based incremental exchange data based on semantic differential at the component unique identifier granularity using the previous version. This invention achieves the technical effects of improving component matching accuracy and robustness, reducing the risk of mismatch and omission, realizing traceable component-level incremental updates, and supporting multi-party collaborative exchange.

[0009] This invention provides a deep learning-based BIM twin collaborative exchange method for PC components, comprising:

[0010] S1. Acquire unstructured data of the target scene and preprocess it to obtain multimodal input data; S2. Use the multimodal input data to perform instance recognition and feature extraction on PC components through a component recognition deep learning model, obtaining the first component feature, component state information, and component quality information of each identified PC component, and extract evidence data corresponding to each identified PC component from the multimodal input data; S3. Acquire a BIM twin model corresponding to the target scene, extract BIM components, extract the unique identifier and BIM prior information of each component, and generate BIM prior features based on the BIM prior information; S4. Under the guidance of the BIM prior features, perform cross-modal fusion processing on the first component features to obtain fused component features, and generate candidate matching relationships between the identified PC components and BIM components based on BIM prior information and preset constraints; S5. Construct the optimal transmission cost matrix based on the fused component features, BIM prior features, and candidate matching relationships, the elements of which are the feature difference terms between the fused component features and the BIM prior features and preset constraints. The constraint penalty terms corresponding to the constraints are jointly determined, and the optimal transmission solution is performed to obtain the soft matching matrix; S6, the matching confidence of each identified PC component is calculated based on the soft matching matrix, and a rejection identification information is generated with the preset confidence threshold. For the identified PC components that are not rejected, the assignment solution is performed based on the soft matching matrix to obtain a one-to-one mapping result; S7, based on the one-to-one mapping result, the component status information, component quality information, and matching confidence are written into the twin object associated with the corresponding component unique identifier in the BIM twin model, and the twin object update value is generated. Based on the rejection identification information, the rejected identified PC components are associated with the BIM spatial partition object in the BIM twin model, and a review mark and the evidence data index corresponding to the rejected identified PC components are written; S8, using the previous version of the twin data of the BIM twin model, the twin object update value and the review mark are differentially calculated with the previous version of the twin data at the component unique identifier granularity to obtain component-level incremental update information and generate incremental exchange data for multi-party collaborative exchange.

[0011] Optionally, S1 includes:

[0012] The unstructured data includes image data, video data, and point cloud data;

[0013] The image data, video data, and point cloud data are collected in the target scene, and a collection timestamp is recorded for each frame of the image data, each frame of the video data, and each frame of the point cloud data.

[0014] A unified time base is established based on the acquisition timestamp, and time synchronization is performed on the image data, the video data, and the point cloud data, so that the frames of the image data, the frames of the video data, and the frames of the point cloud data form a corresponding relationship under the unified time base;

[0015] Based on a preset engineering coordinate system, coordinate calibration is performed on the image data, the video data, and the point cloud data to determine the coordinate transformation relationship between the point cloud coordinate system and the preset engineering coordinate system, and the point cloud data is transformed to the preset engineering coordinate system.

[0016] The image data, video data, and point cloud data that have undergone time synchronization and coordinate calibration are subjected to noise filtering processing to obtain the multimodal input data, which consists of the image data, video data, and point cloud data after time synchronization, coordinate calibration, and noise filtering processing.

[0017] Optionally, S2 includes:

[0018] Using the multimodal input data and the component recognition deep learning model, PC component instance segmentation is performed frame-by-frame on the image data and the video data to obtain component segmentation masks for each frame. PC component instance segmentation is then performed on the point cloud data to obtain point cloud component segmentation masks. A correspondence is established between the component segmentation masks and the point cloud component segmentation masks based on coordinate calibration results to obtain multimodal component segmentation results. Based on the multimodal component segmentation results, component region feature aggregation is performed on the features generated by the component recognition deep learning model to obtain the first component feature for each identified PC component. Based on the first component feature, component state information for each identified PC component is generated through a component state prediction network, and component quality information for each identified PC component is generated through a component quality prediction network. Based on the multimodal component segmentation results, evidence data corresponding to each identified PC component is extracted from the multimodal input data. The evidence data includes at least one of component local images, component local video clips, and component local point cloud clips.

[0019] The component local image is an image segment extracted from the image data according to the region corresponding to the component segmentation mask; the component local video segment is a video segment extracted from the video data according to the region corresponding to the component segmentation mask and containing multiple consecutive frames; and the component local point cloud segment is a point cloud segment extracted from the point cloud data according to the point cloud component segmentation mask point set.

[0020] Optionally, S3 includes:

[0021] Obtain a BIM twin model corresponding to the target scene, and parse the BIM twin model to extract multiple BIM components;

[0022] For each BIM component, read the component's unique identifier and extract the component type information, size parameter information, spatial location information in the preset engineering coordinate system, and component geometric information, which are the BIM prior information.

[0023] Based on the BIM prior information, the component type information is categorized and coded; the size parameter information and spatial location information are numerically normalized; geometric description parameters are generated for the component geometric information; and the BIM prior features used to characterize the BIM component are generated.

[0024] Optionally, S4 includes:

[0025] Based on the first component features and calling the BIM prior features, guidance information for the attention mechanism is constructed based on the BIM prior features. The attention mechanism is then used to perform cross-modal fusion of the first component features to obtain the fused component features of each identified PC component. Based on the BIM prior information and combined with preset constraints, candidate matching relationships are generated between each identified PC component and each BIM component. The candidate matching relationships are obtained by calculating the feature similarity between the fused component features and the BIM prior features, and selecting the top K BIM components from the BIM components that meet the preset constraints in descending order of feature similarity, where K is a preset positive integer. The candidate matching relationships satisfy the preset constraints, which include at least one of component type constraints, size parameter constraints, and spatial location constraints.

[0026] Optionally, S5 includes:

[0027] For each identified PC component and each BIM component in its corresponding candidate matching relationship, a feature difference term is calculated based on the fused component features and the BIM prior features.

[0028] Based on the preset constraints, at least one of the component type constraint penalty value, size parameter constraint penalty value, and spatial position constraint penalty value is calculated to obtain the constraint penalty item;

[0029] The optimal transmission cost is obtained by weighted summation of the feature difference term and the constraint penalty term, and the optimal transmission cost matrix is ​​constructed accordingly.

[0030] Matrix elements that do not belong to the candidate matching relationship are set to a preset large generation value or masking value;

[0031] The optimal transmission cost matrix is ​​solved by performing an optimal transmission solution. The optimal transmission solution is obtained by Sinkhorn iteration with entropy regularization term to obtain a soft matching matrix representing the matching probability between each identified PC component and each BIM component.

[0032] Optionally, S6 includes:

[0033] Based on the soft matching matrix, the matching probability corresponding to each BIM component is determined for each identified PC component, and the maximum value of the matching probability is determined as the matching confidence of the identified PC component.

[0034] Based on the comparison between the matching confidence level and the preset confidence threshold, a rejection label is generated for the identified PC component whose matching confidence level is lower than the preset confidence threshold;

[0035] Based on the rejection identification information, the rows corresponding to the rejected PC components are removed from the soft matching matrix to obtain the matching probability submatrix used for assignment and solution.

[0036] The assignment solution is performed based on the matching probability submatrix. The assignment solution uses the Hungarian algorithm or linear programming to generate matching relationships that satisfy the one-to-one correspondence constraint, and obtains a one-to-one mapping result. The one-to-one mapping result is the correspondence between the unique identifier of each unrejected PC component and the corresponding BIM component.

[0037] Optionally, the S7 includes:

[0038] Based on the one-to-one mapping result, locate the twin object in the BIM twin model corresponding to the unique identifier of the component. For each identified PC component that is not rejected, write the component status information into the component status field of the twin object, write the component quality information into the component quality field of the twin object, and write the matching confidence into the matching confidence field of the twin object to generate an updated value for the twin object. Based on the rejection identifier information and the spatial position of the rejected identified PC component in the preset engineering coordinate system, associate the rejected identified PC component with a pre-established BIM spatial partition object in the BIM twin model. Write an updated value containing a verification mark and an evidence data index corresponding to the rejected identified PC component into the BIM spatial partition object. The BIM spatial partition object includes at least one of a floor object, a region partition object, or a component container object.

[0039] Optionally, S8 includes:

[0040] Read the previous version of the BIM twin model's twin data, and align the updated values ​​of the twin objects and the updated values ​​of the markers to be reviewed with the previous version of the twin data according to the unique identifier of the components;

[0041] Compare the field values ​​of each component's unique identifier before and after the change to determine the changed field and generate the change value corresponding to the changed field to obtain component-level incremental update information.

[0042] Based on the component-level incremental update information, select the evidence data corresponding to the change field from the evidence data, and calculate the hash check value for the evidence data;

[0043] Incremental exchange data for multi-party collaborative exchange is generated according to a preset exchange data format. The incremental exchange data includes a unique component identifier, a change field, a change value, a timestamp, a matching confidence level, evidence data, and the hash verification value.

[0044] Optionally, the optimal transmission solution employs unbalanced optimal transmission or partially optimal transmission, allowing the soft matching matrix to contain identified PC or BIM components that have not been assigned quality, thus characterizing at least one of the following: newly added components, missed components, or duplicated components.

[0045] The beneficial effects of this invention are:

[0046] 1. By leveraging BIM prior information to guide cross-modal feature fusion and combining constraints such as component type, size parameters, and spatial location to generate candidate matching relationships, the stability and distinguishability of component feature representation under image data, video data, and point cloud data are improved, thereby enhancing the accuracy and robustness of PC component identification and subsequent mapping.

[0047] 2. By constructing an optimal transmission cost matrix that includes feature difference terms and constraint penalty terms, a soft matching matrix is ​​obtained. A two-stage strategy of matching confidence threshold rejection and assignment solution is adopted to achieve a globally consistent one-to-one mapping, reduce the risk of mismatch, rematch and omission, and automatically include low confidence results in the pending review process.

[0048] 3. Write the component status information, component quality information and matching confidence into the twin object associated with the component's unique identifier, and perform semantic differential at the granularity of the component's unique identifier to generate incremental exchange data. The incremental packet carries the corresponding evidence data and its hash verification value, which improves the efficiency of collaborative exchange and the traceability of updates, and facilitates multiple parties to verify and hold accountable the changed content. Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0050] Figure 1This is a flowchart of a deep learning-based BIM twin collaborative exchange method for PC components proposed in this invention. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0052] refer to Figure 1 A deep learning-based BIM twin collaborative exchange method for PC components includes:

[0053] S1. Acquire unstructured data of the target scene and preprocess it to obtain multimodal input data; S2. Use the multimodal input data to perform instance recognition and feature extraction on PC components through a component recognition deep learning model, obtaining the first component feature, component state information, and component quality information of each identified PC component, and extract evidence data corresponding to each identified PC component from the multimodal input data; S3. Acquire a BIM twin model corresponding to the target scene, extract BIM components, extract the unique identifier and BIM prior information of each component, and generate BIM prior features based on the BIM prior information; S4. Under the guidance of the BIM prior features, perform cross-modal fusion processing on the first component features to obtain fused component features, and generate candidate matching relationships between the identified PC components and BIM components based on BIM prior information and preset constraints; S5. Construct the optimal transmission cost matrix based on the fused component features, BIM prior features, and candidate matching relationships, the elements of which are the feature difference terms between the fused component features and the BIM prior features and preset constraints. The constraint penalty terms corresponding to the constraints are jointly determined, and the optimal transmission solution is performed to obtain the soft matching matrix; S6, the matching confidence of each identified PC component is calculated based on the soft matching matrix, and a rejection identification information is generated with the preset confidence threshold. For the identified PC components that are not rejected, the assignment solution is performed based on the soft matching matrix to obtain a one-to-one mapping result; S7, based on the one-to-one mapping result, the component status information, component quality information, and matching confidence are written into the twin object associated with the corresponding component unique identifier in the BIM twin model, and the twin object update value is generated. Based on the rejection identification information, the rejected identified PC components are associated with the BIM spatial partition object in the BIM twin model, and a review mark and the evidence data index corresponding to the rejected identified PC components are written; S8, using the previous version of the twin data of the BIM twin model, the twin object update value and the review mark are differentially calculated with the previous version of the twin data at the component unique identifier granularity to obtain component-level incremental update information and generate incremental exchange data for multi-party collaborative exchange.

[0054] In this specific embodiment, S1 includes:

[0055] Industrial cameras, video cameras, and 3D laser scanners are deployed in the target scene to collect image data, video data, and point cloud data respectively. The image data is stored as an image frame sequence, the video data is decoded as a video frame sequence, and the point cloud data is stored as a point cloud frame sequence according to the scanning cycle.

[0056] Three types of acquisition devices are connected to the same time source and synchronously acquire data using hardware trigger pulses. At each rising edge of the trigger, a timestamp is recorded for the image frame, video frame, and point cloud frame, respectively. The collection timestamp The timing origin and timing unit are consistent across all three types of equipment;

[0057] A unified time base is established based on all timestamps. The unified time base adopts the start time. To align with the origin and with a fixed sampling interval Given an increasing sequence of discrete time points, for each discrete time point, search for the image frame sequence, video frame sequence, and point cloud frame sequence for the time point with the minimum absolute difference in timestamps that does not exceed [a certain value]. The corresponding frames are then grouped into a multimodal frame group at the same time, and the three types of corresponding frames are combined. If any modality does not satisfy the condition at that discrete time... The corresponding frame is discarded at that discrete moment, and the search continues for the remaining discrete moments to ensure that each multimodal frame group has a one-to-one corresponding image frame, video frame and point cloud frame.

[0058] Establish a preset engineering coordinate system during the coordinate calibration stage. The preset engineering coordinate system It is a right-handed system with the project's measurement control points as the origin and the building's longitudinal direction as the axis. Axial direction, with the building's horizontal direction as Positive axis, with vertical upward as positive axis, simultaneously defining the point cloud coordinate system The coordinate system of the laser scanner itself;

[0059] In the target scene, at least six spatially discrete and non-coplanar spherical targets are fixedly installed, and the center of each target is obtained in the preset engineering coordinate system. The coordinates of the target points are obtained by performing RANSAC spherical fitting on the target point set in the point cloud frame, and then using the coordinates of the target points in the point cloud coordinate system. The coordinates below are used to form the center of the target ball of the same name. Based on the corresponding point set, the point cloud coordinate system is solved using SVD rigid body registration. To the preset engineering coordinate system The coordinate transformation relationship is obtained and the homogeneous transformation matrix is ​​obtained. ,in for Matrix and by Rotation matrix and Composed of translation vectors;

[0060] Perform coordinate transformation on the point cloud frames in each multimodal frame group, transforming any point within the point cloud frame from the point cloud coordinate system. Transform to the preset engineering coordinate system satisfy:

[0061] ;

[0062] in The point indicates the location in the preset engineering coordinate system. The homogeneous coordinate column vector is given by the following four components: , Indicates the point in the point cloud coordinate system The homogeneous coordinate column vector is given by the following four components: Indicates from the point cloud coordinate system To the preset engineering coordinate system The homogeneous coordinate transformation matrix, Represents the point cloud coordinate system. Indicates the preset engineering coordinate system;

[0063] In the noise filtering stage, bilateral filtering is performed frame-by-frame on both image and video frames for noise reduction. The side length of the bilateral filtering window is set to 7, and the spatial domain standard deviation is set to [value missing]. Pixels and the standard deviation of pixel value range are set to To suppress imaging noise and preserve component edges, statistical outlier removal and voxel downsampling are performed on the point cloud frames after coordinate transformation. Statistical outlier removal calculates the difference between each point and its corresponding voxel. The average distance of the nearest neighbors is used to delete points whose average distance is greater than the global average distance plus twice the global standard deviation to suppress flying points and multipath noise. The voxel downsampling voxel side length is set to... The average coordinates of points within a voxel are used to replace all points within the voxel to reduce redundancy density.

[0064] After completing time synchronization, coordinate calibration, and noise filtering, all remaining multimodal frame groups are organized into multimodal input data in discrete time sequence. The multimodal input data includes image frames and video frames strictly aligned under a unified time reference, as well as data transformed to a preset engineering coordinate system. And point cloud frames that have completed noise filtering.

[0065] In this specific embodiment, S2 includes:

[0066] Read multimodal input data and process it frame by frame according to a unified time base. The multimodal frame group at each moment contains the image frame, video frame and the frame transformed to the preset engineering coordinate system at the same moment. The point cloud frame below;

[0067] Image frames and video frames are respectively input into the two-dimensional instance segmentation branch of the component recognition deep learning model, and component segmentation masks are obtained frame by frame. The two-dimensional instance segmentation branch adopts a Mask R-CNN structure with a backbone network of ResNet-50 and FPN feature pyramid. The RPN anchor box scale is set to... pixels and aspect ratio RoIAlign output size is The mask output size of the mask head is The category set consists of PC component category and background category, and the instance confidence threshold is set to 0.70. NMS with an IOU threshold of 0.50 is used to remove duplicates to obtain a two-dimensional component instance set for each frame.

[0068] The point cloud frame is input into the 3D instance segmentation branch of the deep learning model for component recognition, and a point cloud component segmentation mask is obtained. The 3D instance segmentation branch adopts a voxelized sparse convolutional U-Net structure with a voxel side length set to 0.02 m. The network includes 4 downsampling stages and 4 upsampling stages, with the number of channels in each stage being as follows: The convolutional kernel size is 3 and the downsampling stride is 2. The network outputs point-level semantic categories and point-level instance embedding vectors at the same time. Mean-shift clustering is performed based on the point-level instance embedding vectors and the bandwidth is set to 0.50 m to form a set of point cloud component instances and generate a point cloud component segmentation mask.

[0069] Based on the coordinate calibration results, establish the correspondence between the component segmentation mask and the point cloud component segmentation mask. Specifically, this involves translating each point in the point cloud frame from the preset engineering coordinate system. The projection is projected onto the camera pixel coordinates, and the projection landing point is used to determine which 2D component segmentation mask region it belongs to. The overlap ratio between 3D instances and 2D instances is then calculated, and an overlap ratio of not less than 0.60 is used as the criterion for determining if they belong to the same component, thus generating multimodal component segmentation results. The projection satisfies the following:

[0070] ;

[0071] in Represents the x-coordinate of pixels, with the unit being pixels. Represents the ordinate of pixels, with the unit being pixels. This represents the projection scale factor, and its value is equal to the point's position in the camera coordinate system. The depth below, This represents the camera intrinsic parameter matrix, which is composed of focal length and principal point parameters and is obtained from camera calibration. Indicates from the preset engineering coordinate system To the camera coordinate system of The homogeneous transformation matrix is ​​obtained by extrinsic parameter calibration. The point indicates the location in the preset engineering coordinate system. The three-dimensional coordinate components are shown below, with the unit being meters. Indicates the preset engineering coordinate system. Indicates the camera coordinate system;

[0072] Based on the multimodal component segmentation results, the features generated by the component recognition deep learning model are aggregated to obtain the first component feature of each recognized PC component. Specifically, in the two-dimensional instance segmentation branch, global average pooling is performed on the RoIAlign features of each component instance and then... The fully connected layer obtains the 2D instance feature vector. In the 3D instance segmentation branch, max pooling is performed on the point-level features belonging to the same 3D component instance, and then... The fully connected layer obtains the three-dimensional instance feature vector, and then the two-dimensional instance feature vector and the three-dimensional instance feature vector are concatenated in sequence to form the first component feature with a dimension of 1024;

[0073] Based on the first component features, component state information for each identified PC component is generated through a component state prediction network. The component state prediction network is a three-layer perceptron with layer widths of [missing information]. Each hidden layer has a ReLU activation function and a Softmax output layer that outputs a 4-dimensional state probability vector. The state label corresponding to the highest probability is taken as the component state information.

[0074] Based on the first component features, component quality information for each identified PC component is generated through a component quality prediction network. The component quality prediction network is a three-layer perceptron with layer widths of [missing information]. Each hidden layer has a ReLU activation function and an output layer has a Softmax function, which outputs a 4-dimensional quality probability vector. The four quality labels are qualified, cracked, chipped edges and corners, and exposed reinforcement. The quality label with the highest probability is taken as the component quality information.

[0075] Based on the multimodal component segmentation results, evidence data corresponding to each identified PC component is extracted from the multimodal input data. The component local image is an image segment obtained by extending the outer rectangular region of the component segmentation mask in the image frame by 10 pixels in all directions, while simultaneously saving the pixel coordinate range of the image segment in the original image frame. The component local video segment is a sequence of 16 consecutive frames formed by taking 8 frames forward and 7 frames backward from the current frame as the center in the video frame sequence, and then cropping the segment sequence after extending the outer rectangular region of the component segmentation mask in all directions for each frame, while simultaneously saving the original video frame number corresponding to each frame. The component local point cloud segment is a sequence of all points belonging to the component instance selected from the point cloud frame according to the point cloud component segmentation mask and stored together with the intensity value of the points, while simultaneously saving the point index set of the point cloud segment in the original point cloud frame. The first component feature, component state information, component quality information and corresponding evidence data of each identified PC component are output.

[0076] In this specific embodiment, S3 includes:

[0077] The BIM twin model corresponding to the target scene is read and the model file is parsed using a BIM parser. The BIM twin model is stored in the form of a collection of component objects, and each component object contains a unique component identifier field, a component type field, a geometric representation field, and a coordinate placement field.

[0078] During parsing, all component objects are traversed and only the BIM components corresponding to precast concrete (PC) components are retained. The BIM components corresponding to the PC components are determined by the joint determination of the component type field and the material field, with the material field value being concrete and the component type field value belonging to the preset set of PC component types.

[0079] For each BIM component, the component's unique identifier is read and used as the primary key for subsequent write-back and differential operations. The component's unique identifier is a globally unique string and remains unchanged throughout the entire lifecycle of the BIM twin model.

[0080] For each BIM component, BIM prior information is extracted, including component type information, dimensional parameter information, and information in a preset engineering coordinate system. The spatial location information and component geometric information are as follows, where the component type information is taken from the component type field and mapped to a fixed-length category coding vector. The category coding vector uses a length of... The one-hot encoding and one-hot index are determined by the order of the types in the preset PC component type set and are fixed during system deployment;

[0081] Dimensional parameter information is derived from the component geometry information in a preset engineering coordinate system. The calculations are as follows: specifically, the geometric representation of the component is discretized into a triangular mesh, and the coordinates of the mesh vertices are uniformly transformed to a preset engineering coordinate system. Then, the component is calculated in the preset engineering coordinate system. The lower axis is aligned with the bounding box, and the three-axis side lengths of the bounding box are used as dimensional parameters, which are recorded as length, width and height in sequence.

[0082] Spatial location information is obtained by calculating the center point of the axis-aligned bounding box and placing the center point in a preset engineering coordinate system. The three-dimensional coordinates below are used as spatial location information;

[0083] In addition to being used for size and position calculations, the component's geometric information is also used to generate geometric description parameters. The geometric description parameters include the volume and surface area of ​​the component's triangular mesh. The volume is calculated by summing the areas of all triangular meshes after checking the consistency of the closed triangular meshes using the tetrahedral summation method.

[0084] When generating BIM prior features, the component type information is directly represented using the category encoding vector. Numerical normalization is performed on the dimensional parameters, spatial location information, and geometric description parameters, and these are then concatenated to form the BIM prior feature vector for each BIM component. The numerical normalization uses min-max normalization and satisfies the following for any numerical term:

[0085] ;

[0086] in This represents a numerical term to be normalized, corresponding to one of the following: length, width, height, center point coordinate components, volume, or surface area. This indicates the minimum value of this numerical item across all BIM components. This indicates the maximum value of this numerical item across all BIM components. Represents the normalized numerical term and its range is 1. ;

[0087] The and By performing a full scan of the corresponding numerical items of all BIM components and binding them with the BIM twin model version number to ensure normalization consistency under the same version, we obtain the BIM prior features used to characterize each BIM component and establish a correlation with the unique identifier of the corresponding component.

[0088] In this specific embodiment, S4 includes:

[0089] For each identified PC component, its first component feature is read and split into a two-dimensional instance feature vector and a three-dimensional instance feature vector according to its source. The two-dimensional instance feature vector is the result of the convergence of component region features from image data and video data and has a dimension of 512. The three-dimensional instance feature vector is the result of the convergence of component region features from point cloud data and has a dimension of 512.

[0090] Simultaneously, based on the point cloud component segmentation mask corresponding to the identified PC component in the preset engineering coordinate system The bounding box of the identified PC component is calculated to obtain its dimensional and spatial position estimates. The dimensional estimates are composed of the three side lengths of the bounding box, namely length, width, and height, in that order. The spatial position estimates are the center point of the bounding box in the preset engineering coordinate system. The three-dimensional coordinates below;

[0091] All BIM prior features are invoked, and the size estimate and spatial location estimate are used as retrieval conditions to construct the guidance information for the attention mechanism. Specifically, BIM components whose spatial location is no more than 5.0 m from the spatial location estimate are selected from all BIM components, and the mean value of their BIM prior features is calculated dimension by dimension to obtain the guidance vector corresponding to the identified PC component. The guidance vector is used as the prior input of the attention mechanism to characterize the BIM statistical prior of the local area where the identified PC component is located.

[0092] A cross-modal fusion network is constructed based on the guiding vector, and the fusion of two-dimensional instance feature vectors and three-dimensional instance feature vectors is performed. The cross-modal fusion network is a gated attention structure and includes a set of weight generation subnetworks and a set of feature fusion subnetworks. The weight generation subnetworks are two fully connected layers with layer widths of [missing information]. The input is the concatenation result of the guiding vector, the two-dimensional instance feature vector, and the three-dimensional instance feature vector. The output is the two-dimensional modal weight and the three-dimensional modal weight, which are normalized by Softmax to make their sum equal to 1. The feature fusion subnetwork weights the two-dimensional instance feature vector according to the two-dimensional modal weight and the three-dimensional instance feature vector according to the three-dimensional modal weight, and then sums them to obtain the fusion intermediate vector. Then, the fusion intermediate vector is mapped from 512 dimensions to 256 dimensions through a fully connected layer and ReLU activation is used to obtain the fusion component features of the identified PC component.

[0093] After obtaining the features of the fused components, candidate matching relationships between the identified PC components and each BIM component are generated based on the prior information of BIM and the preset constraints. The preset constraints include component type constraints, size parameter constraints and spatial position constraints, all of which are hard constraints. The component type constraint is that the component type label of the identified PC component is consistent with the component type information of the BIM component. The component type label of the identified PC component is determined by the instance category output of the two-dimensional instance segmentation branch.

[0094] The dimensional parameter constraint is that the relative error between the estimated dimensions of the identified PC component and the dimensional parameter information of the BIM component is no greater than [value missing]. ;

[0095] The spatial location constraint is that the Euclidean distance between the estimated spatial location of the identified PC component and the spatial location information of the BIM component is no greater than 1.5 m;

[0096] For BIM components that meet the preset constraints, their BIM prior features are first mapped from 40 dimensions to 256 dimensions through a fully connected layer and then... Normalization is performed to ensure that the features of the fused component are in the same feature space. Then, the feature similarity between the fused component features and the corresponding features of the BIM component is calculated, and the features are sorted from high to low similarity and selected. A candidate matching relationship is formed by BIM components, and the feature similarity is based on cosine similarity and satisfies:

[0097] ;

[0098] in Indicates the first The identified PC component and the first Feature similarity of BIM components Indicates the first The fused component features of the identified PC components and the dimension is Indicates the first The 256-dimensional vector obtained by fully connected mapping of the BIM prior features of a BIM component. This indicates the transpose operation. express The norm outputs the fused component features of each identified PC component and candidate matching relationships that meet preset constraints and are of controlled size.

[0099] In this specific embodiment, S5 includes:

[0100] Let the number of PC components identified be... And the number of BIM components is For each identified PC component Read its fusion component features And for each BIM component Read the BIM prior features that are in the same feature space as the fused component features. Simultaneously read the candidate matching relationships and then... The set of candidate BIM component indices corresponding to each identified PC component is denoted as . and ;

[0101] Construct the optimal transmission cost matrix And for any matrix element Assign values ​​according to the candidate relationship if and only if Calculate the value of time, otherwise it will Set as mask value Solving for optimal transmission by preventing non-candidate matches from entering the optimal transmission;

[0102] when At that time, based on cosine similarity Obtain the feature difference term and define the feature difference term as follows: Next, calculate the component type constraint penalty value, dimensional parameter constraint penalty value, and spatial position constraint penalty value in the constraint penalty item, among which the component type constraint penalty value... It is a binary quantity and when the PC component is identified Component type labels and BIM components The value is 0 if the component type information is consistent, otherwise it is 1. (Dimensional parameter constraint penalty value) From the identified PC component Size estimate With BIM components Dimensional parameter information The spatial position constraint penalty value is obtained by calculating the relative error dimension by dimension and summing the absolute values ​​of the three-dimensional relative errors. From the identified PC component Spatial location estimate With BIM components Spatial location information The Euclidean distance is calculated and normalized to 1.5 m, and the normalized result is truncated to the interval [0,1].

[0103] The optimal transfer cost is obtained by weighted summation of the feature difference term and each constraint penalty term:

[0104] ;

[0105] in Indicates the first The identified PC component and the first Optimal transfer value between BIM components Represents the weight of the feature difference term and takes Indicates the penalty weight for dimensional parameter constraints and takes Represents the spatial location constraint penalty weight and takes Indicates the component type constraint penalty weight and takes Indicates the characteristics of fused components With BIM prior features The cosine similarity between them and the range of values ​​is 1. Indicates by and The calculated relative error penalty value for dimensions is within the range of [value missing]. Indicates by and The calculated location distance penalty value has a range of values. This indicates a penalty value for inconsistent component types, and the value can be either 0 or 1.

[0106] Finish After construction, Sinkhorn iteration with entropy regularization is performed to obtain the soft-matching matrix. The entropy regularization coefficient for the Sinkhorn iteration is set to . And the maximum number of iterations is set to The edge distributions of the identified PC components and BIM components are set as uniform distribution vectors respectively. and And each component is respectively and The iteration termination condition is set as follows: the maximum deviation between the row sum and column sum relative to the target edge distribution does not exceed [a certain value]. ;

[0107] The soft-matching matrix output by the Sinkhorn iteration elements Indicates the first The identified PC component and the first The matching probability between BIM components and satisfying .

[0108] In this specific embodiment, S6 includes:

[0109] Reading the soft matching matrix And interpret it row by row as a matching probability distribution from the identified PC component to the BIM component, where This indicates the number of PC components identified, and there is a one-to-one correspondence between the indices of the PC components identified in steps S2 and S4. This indicates the number of BIM components and corresponds one-to-one with the index of the BIM components in step S3. This represents the soft-matching matrix obtained by solving for optimal transmission. Indicates the first The identified PC component and the first The matching probability between BIM components and satisfying ;

[0110] For each identified PC component Calculate its matching confidence score and define the maximum matching probability of that row as the matching confidence score. ,in Indicates the first The matching confidence of each identified PC component. Represents the soft matching matrix In the Line number The probability of matching a column. This represents the BIM component index, with values ​​ranging from 1 to... Indicates the number of BIM components;

[0111] Set the preset reliability threshold to And The identified PC component is marked as rejected to generate rejection identification information, which includes the index of the identified PC component. Matching confidence And the rejection reason code, which is fixed to "low confidence";

[0112] The indexes of identified PC components that were not rejected are aggregated into a reserved index set and extracted from the soft matching matrix. The matching probability submatrix is ​​obtained by removing all rejected rows. ,in Indicates the number of identified PC components that were not rejected and When it appears Time based on matching confidence From low to high, perform secondary rejection on identified PC components that were not rejected until the condition is met. The secondary rejection result is incorporated into the rejection identification information to ensure that there is a feasible solution for subsequent assignments.

[0113] right The assignment solution is performed to obtain matching relationships that satisfy the one-to-one correspondence constraint. The assignment solution uses the Hungarian algorithm and is implemented in a cost-minimizing form, with each probability element... Convert to cost element To ensure that a higher matching probability corresponds to a lower cost, the cost matrix is ​​padded to a square matrix according to the number of rows to meet the input requirements of the Hungarian algorithm. The padding method is as follows: Add below The minimum cost solution is to create virtual rows and set the cost of each virtual row to zero, allowing unused BIM component columns without affecting the real rows.

[0114] The assignment result output by the Hungarian algorithm is a unique column index corresponding to each unrejected identified PC component in all BIM components. Based on this, a one-to-one mapping result is generated and represented as an index of the unrejected recognized PC components. Unique identifier of the corresponding BIM component The correspondence between them, where For step S3, read from the BIM twin model and match with the first... A unique identifier for a BIM component.

[0115] In this specific embodiment, S7 includes:

[0116] Read the one-to-one mapping result and expand it into an index of the recognized PC components that were not rejected. Unique identifier for corresponding BIM component The paired list reads component status information, component quality information, and matching confidence level simultaneously. ,in Indicates the index of the identified PC component. Indicates the relationship with the first A unique identifier for each BIM component. Indicates the first The matching confidence of each identified PC component;

[0117] For each identified PC component that was not rejected, In the data storage of the BIM twin model, the twin object is located. The twin object is a data node strongly bound to the unique identifier of the component and includes a component status field, a component quality field, a matching confidence field, and an update timestamp field. The component status field stores discrete status labels, and its value is limited to one of the following sets: "Production Completed," "Out of Factory," "In Transit," "Arrived on Site," "In Place," and "Installation Complete." The component quality field stores discrete quality labels, and its value is limited to one of the following sets: "Qualified," "Cracked," "Cut Edges / Corners," and "Exposed Reinforcement." The matching confidence field stores a floating-point number with a value range of [missing information]. The update timestamp field stores the unified time base timestamp of the write-back event.

[0118] Perform an atomic update operation on the located twin object and then update the first twin object. The component status information of the first identified PC component is written into the component status field, and the first... The component quality information of each identified PC component is written into the component quality field. Write the matching confidence field, and simultaneously write the update timestamp field to generate a match with the specified data. The associated twin object update value is recorded in a key-value structure of "field name - field value - timestamp" and includes the update value of the component status field, the update value of the component quality field and the update value of the matching confidence field.

[0119] For the PC component whose rejection information is generated in step S6, its coordinates in the preset engineering coordinate system are read. Spatial location estimate The BIM spatial partitioning objects are located in the BIM twin model. These objects are pre-established during the modeling phase and include at least floor objects and area partitioning objects. A spatial boundary volume is configured for each spatial partitioning object. The spatial boundary volume of a floor object is composed of the floor's elevation range and the floor's outer plane contour. The spatial boundary volume of an area partitioning object is composed of the area's three-dimensional bounding volume, and its coordinate system is aligned with the preset engineering coordinate system. Consistent;

[0120] in accordance with Perform spatial attribution determination and associate the rejected identified PC component with a unique BIM spatial partition object. The spatial attribution determination is first performed according to... The vertical coordinate components fall within the elevation range of a certain floor object to determine the floor object, and then the floor object is determined by... The region partitioning object is determined by the spatial boundary volume of the region partitioning object. Spatial boundary bodies that do not fall into any area partitioning object are associated with component container objects under that floor object, and the component container object is the default storage node of that floor object;

[0121] The rejection flag information is written into the pending review list field of the associated BIM space partition object, along with a pending review marker. The pending review marker is a Boolean field with a value of true to indicate that manual review is required. Simultaneously, the evidence data index corresponding to the rejected PC component is written into the field. ,in The evidence data index is an immutable reference identifier pointing to the evidence data stored in step S2, and is stored as a structured string containing "bucket path, filename, frame number range, and point index set summary," enabling the reviewer to access the evidence data. The system directly backtracks the component's local images, local video clips, and local point cloud clips and verifies the reasons for rejection. After completing the write-back of all non-rejected identifications and the association of all rejected identifications, it outputs the updated value of the twin object and the updated value of the spatial partition object containing the pending review markers and evidence data index.

[0122] In this specific embodiment, S8 includes:

[0123] Read the previous version of the BIM twin model's twin data and load it into a previous version object table. The previous version object table uses the unique identifier of the component as the key and the set of object fields as the value. The set of object fields includes at least the component status field, component quality field, matching confidence field, update timestamp field, pending review mark field, and evidence data index field. At the same time, read the update values ​​of the twin objects and the update values ​​of the spatial partition objects and merge them into the current update table. The current update table also uses the unique identifier of the component as the key, and its key covers the corresponding twin objects and BIM spatial partition objects of the BIM components for unified alignment.

[0124] Align the previous version's object table with the current update table based on the unique identifier of each component, and assign a unique identifier to each component. A field-by-field comparison of the values ​​before and after the change, and the generation of difference markers, satisfies the following:

[0125] ;

[0126] in Indicates the unique identifier of the component. This indicates a field index that corresponds to one of the following fields: component status field, component quality field, match confidence field, update timestamp field, pending review marker field, or evidence data index field. The component is uniquely identified as And the field index is The changed field values ​​are taken from the currently updated table. The component is uniquely identified as And the field index is The field values ​​before the change are taken from the object table of the previous version. This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and otherwise takes the value _____. This indicates a field-level difference marker and is used to indicate whether a field has changed.

[0127] If and only if When that happens, write the field name into the change field and... Write the changed values ​​to form component-level incremental update information. At the same time, write the update timestamp field value from the current update table to the timestamp field of the component-level incremental update information and write the matching confidence field value from the current update table to the matching confidence field of the component-level incremental update information.

[0128] Based on the change field, evidence data corresponding to the changed content is selected from the evidence data to form an evidence list. The selection rules for the evidence list are fixed as follows: when the change field contains the component quality field, the component local image and component local point cloud fragment corresponding to the component's unique identifier are selected; when the change field contains the component status field, the component local video fragment corresponding to the component's unique identifier is selected; when the change field contains the pending review mark field or the evidence data index field, the component local image, component local video fragment, and component local point cloud fragment corresponding to the component's unique identifier are selected. All evidence data are located and read as binary byte streams through the evidence data index EID recorded in the evidence data index field.

[0129] For each piece of evidence in the evidence list, calculate the hash check value and output a 32-byte hash digest using the SHA-256 algorithm, storing it as a hexadecimal string as the hash check value field;

[0130] Incremental exchange data for multi-party collaborative exchange is generated according to a preset exchange data format. The preset exchange data format is a JSON object with a version number field, a generation timestamp field, and an incremental record array field at the top level. Each incremental record in the incremental record array contains a component unique identifier field, a change field, a change value field, a timestamp field, a matching confidence field, an evidence data field, and a hash verification value field. The evidence data field stores the binary byte stream corresponding to the evidence list and uses Base64 encoding to form transmittable text. At the same time, a hash verification value field corresponding one-to-one with the evidence data field is written in the same incremental record to support the receiver to perform integrity verification after decoding. The output is incremental exchange data that carries component-level semantic differential results and has the ability to trace and verify evidence.

[0131] In this specific embodiment, the optimal transmission solution in step S5 adopts unbalanced optimal transmission to allow for the existence of identified PC or BIM components with unassigned quality in the soft matching matrix. Specifically, this is achieved by using the constructed optimal transmission cost matrix... Based on this, a marginal distribution relaxation term is introduced and the soft-matching matrix is ​​obtained by solving the problem. ,in Indicates the number of identified PC components. Indicates the number of BIM components. This represents the optimal transmission cost matrix. This represents the soft-matching matrix output from the solution of the imbalanced optimal transport problem. This represents the nominal edge distribution vector on the side of the identified PC component, with each component having a uniform mass. This represents the nominal edge distribution vector on the BIM component side, with each component having a uniform mass.

[0132] The objective function for unbalanced optimal transmission is set as follows:

[0133] ;

[0134] in Represents the optimal soft-matching matrix. This represents the independent variable that minimizes the objective function. express All elements are non-negative. express and The Frobenius dot product is equal to the sum of the products of all elements. Denotes the entropy regularity coefficient and takes Represents the entropy regularization term and according to The element-level entropy is calculated to encourage soft allocation and improve numerical stability. This represents the relaxation penalty coefficient distributed on the side edges of the identified PC component, and takes... This represents the relaxation penalty coefficient for the distribution of BIM component side edges, and takes... Kullback-Leibler divergence is used to measure the degree of deviation between the actual and nominal marginal distributions. The dimension is A column vector of all 1s is used to calculate rows and vectors, The dimension is A column vector of all 1s is used to calculate Columns and vectors, Indicates the transpose operation;

[0135] The objective function is through imbalance Iterative solution, using iteration and Construct the exponential kernel and scale the vectors by rows With column scaling vector Perform alternating updates, during which... The element-wise ratio of the sum of the current rows to the power of the sum of the current rows is taken as the exponent. And on The element-wise ratio to the sum of the current column is given by the power of the power. ,in Represents the row scaling vector. Represents a column scaling vector. and They represent respectively by and The relaxation update index is determined jointly and a fixed value is taken in this embodiment to fix the edge relaxation strength;

[0136] During iterative initialization, and All components are set to 1.0, and the maximum number of iterations is set to [value]. The termination condition is set to the maximum relative change between the row sum vector and the column sum vector obtained in two consecutive iterations not exceeding [a certain value]. ;

[0137] Output soft-matching matrix The system no longer mandates that the row sum of each identified PC component be strictly equal to its nominal quality, nor does it mandate that the column sum of each BIM component be strictly equal to its nominal quality. This results in the row quality of the corresponding identified PC component being significantly lower than its nominal quality when there are newly added or duplicate identified components in the target scene. In step S6, this leads to a rejection process due to the reduced matching confidence. Simultaneously, when there are missed components in the target scene, the column quality of the corresponding BIM component is significantly lower than its nominal quality. This is reflected in the subsequent incremental data exchange as the unique identifier of the component not being updated in the status or quality fields, thus forming a clue to be verified.

[0138] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0139] This invention addresses the technical problem of "difficulty in automatically identifying PC components and mapping them one-to-one with BIM twin components from unstructured site or factory data, and the difficulty in forming collaboratively exchangeable incremental updates." It employs a closed-loop processing chain consisting of algorithms combining "multimodal component identification and feature extraction, optimal transmission soft matching and one-to-one assignment, twin write-back and component-level differential incremental exchange." By synchronizing and calibrating image, video, and point cloud data in time, multimodal data can be aligned under a unified time reference and engineering coordinate system, thereby improving the reliability of component instance segmentation and feature aggregation. Global matching based on the soft matching matrix, combined with one-to-one assignment, avoids mismatches and rematches caused by greedy matching relying solely on local similarity. After writing component status information, component quality information, and matching confidence into a twin object associated with the component's unique identifier, semantic differential analysis at the component's unique identifier granularity is performed on the previous version to generate incremental exchange data carrying evidence data and verification information. This ensures reliable updates while improving the efficiency and traceability of multi-party collaborative exchange.

[0140] In terms of algorithm structure, this invention addresses the aforementioned technical problems by improving upon engineering priors and uncertainties: First, it introduces BIM prior information to guide cross-modal fusion, making the fusion features more discriminative under component type, size parameters, spatial location, and geometric prior constraints. It also generates candidate matching relationships based on prior constraints to reduce invalid matches and improve robustness. Second, it adds a constraint penalty term to the optimal transmission cost and masks non-candidate matches, ensuring that soft matching results simultaneously satisfy both appearance feature consistency and engineering constraint consistency, thus reducing the probability of mismatches globally. Third, it employs a two-stage strategy of rejection and assignment based on matching confidence, automatically isolating low-confidence results and placing them in the pending review process to avoid erroneous write-back to the twin. Fourth, incremental exchange uses an evidentiary semantic differential packaging mechanism, establishing a connection between changed content and corresponding evidentiary data, making it verifiable and further enhancing the credibility and verifiability of collaborative updates.

Claims

1. A deep learning-based BIM twin collaborative exchange method for PC components, comprising: S1. Acquire unstructured data of the target scene and preprocess it to obtain multimodal input data; S2. Using multimodal input data, perform instance recognition and feature extraction on PC components through a component recognition deep learning model to obtain the first component feature, component state information, and component quality information of each recognized PC component, and extract evidence data corresponding to each recognized PC component from the multimodal input data; S3. Obtain a BIM twin model corresponding to the target scene, extract BIM components, extract the unique identifier and BIM prior information for each component, and generate BIM prior features based on the BIM prior information; S4. Under the guidance of BIM prior features, perform cross-modal fusion processing on the features of the first component to obtain fused component features. Based on BIM prior information and preset constraints, generate candidate matching relationships between the identified PC component and the BIM component. S5. Construct the optimal transmission cost matrix based on the fused component features, BIM prior features, and candidate matching relationships. Its elements are jointly determined by the feature difference terms between the fused component features and BIM prior features and the constraint penalty terms corresponding to the preset constraint conditions. Perform the optimal transmission solution to obtain the soft matching matrix. S6. Calculate the matching confidence of each identified PC component based on the soft matching matrix, generate rejection identification information with the preset confidence threshold, and perform assignment solving on the identified PC components that are not rejected based on the soft matching matrix to obtain one-to-one mapping results. S7. Based on the one-to-one mapping results, write the component status information, component quality information and matching confidence into the twin object associated with the corresponding component unique identifier in the BIM twin model, generate the twin object update value, and based on the rejection identification information, associate the rejected identified PC component with the BIM space partition object in the BIM twin model, and write the review mark and the evidence data index corresponding to the rejected identified PC component. S8. Using the previous version of the BIM twin model's twin data, perform differential calculations at the component unique identifier granularity between the twin object update value and the verification mark and the previous version of the twin data to obtain component-level incremental update information and generate incremental exchange data for multi-party collaborative exchange.

2. The deep learning-based PC component BIM twin collaborative exchange method according to claim 1, S1 includes: The unstructured data includes image data, video data, and point cloud data; The image data, video data, and point cloud data are collected in the target scene, and a collection timestamp is recorded for each frame of the image data, each frame of the video data, and each frame of the point cloud data. A unified time base is established based on the acquisition timestamp, and time synchronization is performed on the image data, the video data, and the point cloud data, so that the frames of the image data, the frames of the video data, and the frames of the point cloud data form a corresponding relationship under the unified time base; Based on a preset engineering coordinate system, coordinate calibration is performed on the image data, the video data, and the point cloud data to determine the coordinate transformation relationship between the point cloud coordinate system and the preset engineering coordinate system, and the point cloud data is transformed to the preset engineering coordinate system. The image data, video data, and point cloud data that have undergone time synchronization and coordinate calibration are subjected to noise filtering processing to obtain the multimodal input data, which consists of the image data, video data, and point cloud data after time synchronization, coordinate calibration, and noise filtering processing.

3. The deep learning-based PC component BIM twin collaborative exchange method according to claim 1, S2 includes: Using the multimodal input data and the component recognition deep learning model, PC component instance segmentation is performed frame by frame on the image data and the video data to obtain the component segmentation mask for each frame. PC component instance segmentation is also performed on the point cloud data to obtain the point cloud component segmentation mask. Based on the coordinate calibration results, establish the correspondence between the component segmentation mask and the point cloud component segmentation mask to obtain the multimodal component segmentation results; Based on the multimodal component segmentation results, the features generated by the component recognition deep learning model are aggregated to obtain the first component feature of each recognized PC component; Based on the first component characteristics, component state information of each identified PC component is generated through a component state prediction network, and component quality information of each identified PC component is generated through a component quality prediction network based on the first component characteristics. Based on the multimodal component segmentation results, evidence data corresponding to each identified PC component is extracted from the multimodal input data. The evidence data includes at least one of component local image, component local video clip, and component local point cloud clip. The component local image is an image segment extracted from the image data according to the region corresponding to the component segmentation mask; the component local video segment is a video segment extracted from the video data according to the region corresponding to the component segmentation mask and containing multiple consecutive frames; and the component local point cloud segment is a point cloud segment extracted from the point cloud data according to the point cloud component segmentation mask point set.

4. The deep learning-based PC component BIM twin collaborative exchange method according to claim 1, S3 includes: Obtain a BIM twin model corresponding to the target scene, and parse the BIM twin model to extract multiple BIM components; For each BIM component, read the component's unique identifier and extract the component type information, size parameter information, spatial location information in the preset engineering coordinate system, and component geometric information, which are the BIM prior information. Based on the BIM prior information, the component type information is categorized and coded; the size parameter information and spatial location information are numerically normalized; geometric description parameters are generated for the component geometric information; and the BIM prior features used to characterize the BIM component are generated.

5. The deep learning-based PC component BIM twin collaborative exchange method according to claim 1, S4 includes: Based on the first component features and calling the BIM prior features, guidance information for the attention mechanism is constructed based on the BIM prior features, and the first component features are fused across modes through the attention mechanism to obtain the fused component features of each identified PC component. Based on BIM prior information and combined with preset constraints, candidate matching relationships are generated between each identified PC component and each BIM component. The candidate matching relationships are obtained by calculating the feature similarity between the fused component features and the BIM prior features, and selecting the top K BIM components from the BIM components that meet the preset constraints in descending order of feature similarity, where K is a preset positive integer. The candidate matching relationships satisfy the preset constraints, which include at least one of component type constraints, size parameter constraints, and spatial location constraints.

6. The PC component BIM twin collaborative exchange method based on deep learning according to claim 1, S5 includes: For each identified PC component and each BIM component in its corresponding candidate matching relationship, a feature difference term is calculated based on the fused component features and the BIM prior features. Based on the preset constraints, at least one of the component type constraint penalty value, size parameter constraint penalty value, and spatial position constraint penalty value is calculated to obtain the constraint penalty item; The optimal transmission cost is obtained by weighted summation of the feature difference term and the constraint penalty term, and the optimal transmission cost matrix is ​​constructed accordingly. Matrix elements that do not belong to the candidate matching relationship are set to a preset large generation value or masking value; The optimal transmission cost matrix is ​​solved by performing an optimal transmission solution. The optimal transmission solution is obtained by Sinkhorn iteration with entropy regularization term to obtain a soft matching matrix representing the matching probability between each identified PC component and each BIM component.

7. The deep learning-based PC component BIM twin collaborative exchange method according to claim 1, S6 includes: Based on the soft matching matrix, the matching probability corresponding to each BIM component is determined for each identified PC component, and the maximum value of the matching probability is determined as the matching confidence of the identified PC component. Based on the comparison between the matching confidence level and the preset confidence threshold, a rejection label is generated for the identified PC component whose matching confidence level is lower than the preset confidence threshold; Based on the rejection identification information, the rows corresponding to the rejected PC components are removed from the soft matching matrix to obtain the matching probability submatrix used for assignment and solution. The assignment solution is performed based on the matching probability submatrix. The assignment solution uses the Hungarian algorithm or linear programming to generate matching relationships that satisfy the one-to-one correspondence constraint, and obtains a one-to-one mapping result. The one-to-one mapping result is the correspondence between the unique identifier of each unrejected PC component and the corresponding BIM component.

8. The deep learning-based PC component BIM twin collaborative exchange method according to claim 1, S7 includes: Based on the one-to-one mapping results, locate the twin object in the BIM twin model corresponding to the unique identifier of the component, and for each identified PC component that is not rejected, write the component status information into the component status field of the twin object, write the component quality information into the component quality field of the twin object, and write the matching confidence into the matching confidence field of the twin object to generate the twin object update value. Based on the rejection identification information, and based on the spatial position of the rejected PC component in the preset engineering coordinate system, the rejected PC component is associated with a pre-established BIM spatial partition object in the BIM twin model. An update value containing a verification mark and an evidence data index corresponding to the rejected PC component is written into the BIM spatial partition object. The BIM spatial partition object includes at least one of a floor object, a region partition object, or a component container object.

9. The deep learning-based PC component BIM twin collaborative exchange method according to claim 1, S8 includes: Read the previous version of the BIM twin model's twin data, and align the updated values ​​of the twin objects and the updated values ​​of the markers to be reviewed with the previous version of the twin data according to the unique identifier of the components; Compare the field values ​​of each component's unique identifier before and after the change to determine the changed field and generate the change value corresponding to the changed field to obtain component-level incremental update information. Based on the component-level incremental update information, select the evidence data corresponding to the change field from the evidence data, and calculate the hash check value for the evidence data; Incremental exchange data for multi-party collaborative exchange is generated according to a preset exchange data format. The incremental exchange data includes a unique component identifier, a change field, a change value, a timestamp, a matching confidence level, evidence data, and the hash verification value.

10. A deep learning-based BIM twin collaborative exchange method for PC components according to claim 6, characterized in that, The optimal transmission solution employs unbalanced optimal transmission or partially optimal transmission, allowing the soft matching matrix to contain identified PC or BIM components that have not been assigned quality, thus characterizing at least one of the following situations: newly added components, missed components, or duplicated components.

Images