A method for constructing digital exhibition hall scenes based on digital twins
By using surface identity association and an improved PointConv model to eliminate transient occlusion data and locate scene difference intervals, and by adjusting parameters using the Northern Eagle optimization algorithm, the problems of data confusion and repeated reconstruction in the construction of digital exhibition hall scenes were solved, achieving accurate and efficient scene updates.
Patent Information
- Application Number
- CN202611368731.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-04
- Publication Date
- 2026-11-17
AI Technical Summary
Existing methods for constructing digital exhibition hall scenarios struggle to accurately distinguish between instantaneous changes and actual changes in exhibits, leading to data confusion and inaccurate reconstruction. Furthermore, extensive repetitive reconstruction increases the amount of data processing required.
By combining surface identity association, ordered surface intersection sequence and improved PointConv model, transient occlusion data is removed, the sequence difference interval in adjacent scene versions is located, and the connection and interpolation control parameters are adjusted by the Northern Eagle optimization algorithm to achieve local reconstruction of the changed surface segment.
It improves the accuracy and continuity of digital exhibition hall scene updates, reduces the amount of data processing, and is suitable for scene construction and maintenance where exhibits are frequently adjusted.
Smart Images

Figure CN123118913A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer graphics technology, and in particular to a method for constructing digital exhibition hall scenes based on digital twins. Background Technology
[0002] Existing methods for constructing digital exhibition hall scenes typically collect images or point cloud data from the physical exhibition hall, generate a digital twin scene through coordinate registration, point cloud segmentation, and surface reconstruction, and update the position, orientation, and surface condition of exhibits based on newly acquired data. With frequent adjustments to exhibits in the exhibition hall, the digital twin scene needs to continuously reflect these changes.
[0003] Existing methods typically rely on differences in point clouds at adjacent time points to determine scene changes. This makes it difficult to distinguish between instantaneous changes such as people passing by or temporary occlusion and actual changes in exhibits, and it is easy to write occlusion data into the digital twin scene. At the same time, there is a lack of stable surface identity associations between adjacent scene versions, making it difficult to accurately determine the changed surface range after exhibits move or occlusion relationships change.
[0004] In addition, existing point cloud reconstruction methods mostly use fixed feature transfer and interpolation methods. Historical surface features are easily involved in the reconstruction of the current changed area. Data confusion can easily occur between newly added surfaces, continued surfaces and temporarily invisible surfaces, resulting in inaccurate local scene updates. Furthermore, large-scale repeated reconstruction increases the amount of data processing required for scene construction.
[0005] Therefore, how to provide a method for constructing digital exhibition hall scenarios based on digital twins is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] One objective of this invention is to propose a method for constructing digital exhibition hall scenes based on digital twins. This invention employs a combination of surface identity association, ordered surface intersection sequences, and an improved PointConv model. By eliminating transient occlusion data and locating sequence difference intervals in adjacent scene versions, it reconstructs cross-version jump connections and restricts PointDeconv interpolation propagation to achieve local reconstruction of changed surface segments. Based on this, it combines the Northern Eagle optimization algorithm to adjust connection and interpolation control parameters and generates surface construction instructions based on the optimized surface observation state to complete the digital twin scene version update. This method has the advantages of accurate location of change range, less interference from historical features, continuous scene updates, and low data processing volume.
[0007] A method for constructing a digital exhibition hall scene based on digital twins according to an embodiment of the present invention includes the following steps: S1. Collect observation data of the exhibition hall and register it to the exhibition hall coordinate system, remove instantaneous occlusion data, and arrange it into a point cloud sequence of the exhibition hall according to the collection time; S2. Divide the surface segments of exhibits according to the point cloud sequence of the exhibition hall, associate the same surface segments at adjacent times, and form adjacent scene versions corresponding to the surface identity; S3. Project the exhibits whose spatial range changes in adjacent scene versions to the acquisition field of view, generate an ordered surface intersection sequence, align them according to surface identity and extract the first difference to the sequence difference interval of the re-consistent sequence. S4. Improve the PointConv model to encode surface point clouds within the difference interval of the sequence, reconstruct cross-version jump connections according to the difference interval and restrict PointDeconv interpolation propagation to form the initial surface observation state. S5. Project the surface points corresponding to the initial surface observation state onto the acquisition field of view and compare them with the observation data. Calculate the scene recovery deviation. Use the Northern Eagle optimization algorithm to search for structural control parameters and feed back the improved PointConv model to reprocess the difference interval to form an optimized surface observation state. S6. Determine the surface segment construction method based on the optimized surface observation status, and generate surface construction instructions; S7. Update the surface segments according to the surface construction instructions to build a digital twin scene version; S8. Optimize the surface observation status and write it into the digital twin scene version. Associate the entity exhibits with the surface identity to form a digital exhibition hall scene and update the scene change comparison benchmark.
[0008] Optionally, S1 specifically includes: S11. Acquire exhibition hall observation data according to the acquisition time, record the correspondence between the position and posture of the acquisition device and the acquisition time, and form exhibition hall observation data with position and posture information; S12. Based on the corresponding pose, transform the exhibition hall observation data to the exhibition hall coordinate system, and perform point cloud registration and fusion on the overlapping observation areas at the same acquisition time to form exhibition hall point cloud frames; S13. Compare continuous exhibition hall point cloud frames, identify point sets that appear only at a single acquisition moment and whose corresponding spatial locations recover to the same surface at adjacent moments as instantaneous occlusion data and remove them. S14. Arrange the exhibition hall point cloud frames after removing data with momentary occlusion according to the collection time to form the exhibition hall point cloud sequence.
[0009] Optionally, S2 specifically includes: S21. Perform exhibit instance segmentation on the exhibition hall point cloud sequence frame by frame, divide the point cloud of each exhibit according to the continuity of normal, and form the surface segment of the exhibit. S22. Estimate the pose transformation of the same exhibit at adjacent time points, transform the surface segment of the exhibit at the previous time point to the current time point, and match the same surface segment according to spatial overlap. S23. Assign new surface identities to surface segments that are not matched at the current time, retain the original surface identities for surface segments that were not matched at the previous time, and organize the surface segments of the exhibits according to adjacent acquisition times to form adjacent scene versions.
[0010] Optionally, S3 specifically includes: S31. Project the exhibits whose spatial range has changed into the acquisition field of view in adjacent scene versions, and merge the projection ranges to form candidate acquisition areas; S32. Generate candidate acquisition rays within the candidate acquisition area, calculate the intersection points of the candidate acquisition rays with the surface segments of the exhibits in the adjacent scene versions, and arrange them along the rays from near to far to form an ordered surface intersection sequence. S33. Align the ordered surfaces of adjacent scene versions according to their surface identities to find the intersection sequence, truncate the first difference position to the re-consistent position, and truncate to the end of the sequence if no re-consistency is found, forming the sequence difference interval.
[0011] Optionally, the improved PointConv model includes a PointConv encoding layer, a cross-version skip connection layer, a restricted PointDeconv layer, and a state output layer connected in sequence. The PointConv coding layer performs hierarchical sampling, neighborhood grouping, and PointConv convolution on the surface point clouds within the sequence difference intervals of adjacent scene versions, forming coding nodes with different resolutions; The cross-version skip connection layer aligns the encoding nodes of adjacent scene versions according to surface identity and intersection order, retains the skip connection between the encoding nodes outside the sequence difference interval and the PointDeconv node of the same resolution, and cuts off the feature propagation of the encoding nodes of the previous scene version within the sequence difference interval. In the current scene version, newly added surface segments are encoded by PointConv encoding layer, which generates encoding nodes and establishes skip connections. Encoding nodes from the previous scene version that do not match the current scene version stop propagating to PointDeconv nodes. The restricted PointDeconv layer selects neighboring points within the surface range corresponding to the preserved or newly created skip connections to perform interpolation, concatenates the interpolated features with the encoded features of the same resolution and performs PointConv convolution to recover the surface point features step by step. The state output layer merges and recovers the surface point features according to the surface identity to form the initial surface observation state.
[0012] Optionally, the cross-version hop connection layer reconstructing the cross-version hop connection specifically includes: According to the resolution level of the PointConv coding layer, the sampling points corresponding to each coding node are assigned to the presentation surface segment, and the intersection position of the presentation surface segment in the ordered surface intersection sequence is recorded. For the exhibit surface segment before the first difference position, maintain the skip connection from the previous scene version encoding node to the same resolution PointDeconv node. Remove the input of the previous scene version encoding node from the first difference position and connect the encoding node of the exhibit surface segment to the same resolution PointDeconv node to the corresponding intersection position in the current scene version. Reconstruct the skip connections corresponding to each intersection position along the sequence difference interval. When the current scene version lacks the same resolution encoding node, retain the features obtained by the PointDeconv layer interpolation of the previous resolution level and do not perform the same resolution encoding feature concatenation. When the surface identity and intersection order are re-consistent, the jump connection from the encoding node corresponding to the previous scene version to the PointDeconv node of the same resolution is restored. If no re-consistent position is found, the connection of the encoding node of the current scene version is maintained to the end of the ordered surface intersection sequence, forming a reconstructed cross-version jump connection.
[0013] Optionally, S5 specifically includes: S51. Project the surface points corresponding to the initial surface observation state onto the acquisition field of view, calculate the positional deviation between the projected points and the corresponding observation points, and summarize them to form the scene restoration deviation. S52, the Northern Eagle optimization algorithm evaluates the connection and interpolation control parameters based on the scene restoration deviation, iteratively updates the connection and interpolation control parameters, compares the scene restoration deviation before and after the update, and retains the parameter results corresponding to the reduction of deviation; S53. Write the optimized connection and interpolation control parameters into the improved PointConv model, reprocess the sequence difference interval, and form an optimized surface observation state.
[0014] Optionally, S6 specifically includes: S61. Based on the surface identity, the optimized surface observation status is mapped to the surface segment of the adjacent scene version, and the surface points are matched with the exhibition hall observation data to determine the current observation coverage area; S62. The current observation coverage area is determined by updating the exhibition hall observation data, and the uncovered surface area with continued surface identity is determined by using the adjacent scene version surface data. S63. Surface segments for which observations are lacking and whose surface identity continuity cannot be confirmed will not be constructed for the time being, and a surface construction instruction will be formed by summarizing these.
[0015] Optionally, S7 specifically includes: S71. Locate the surface segments and corresponding construction methods according to the surface construction instructions; S72. Integrate the exhibition hall observation data into the surface segments constructed using the updated method to form updated surface data; S73. Retain the surface data of adjacent scene versions that have been built and continue to be used, and maintain the unbuilt state of the surface segments that are not yet built; S74. Combine the updated surface data and the retained surface data according to the surface identity to form a digital twin scene version.
[0016] Optionally, S8 specifically includes: S81. Write the optimized surface observation status into the corresponding surface segment in the digital twin scenario version according to the surface identity; S82. Based on the surface identity correspondence, associate the surface sections in the digital twin scene version with the physical exhibits to form a digital exhibition hall scene; S83. The digital twin scenario version is determined as the benchmark for comparing scenario changes.
[0017] The beneficial effects of this invention are: This invention constructs a digital twin scene based on continuously collected exhibition hall observation data. By using coordinate registration and instantaneous occlusion data removal, it reduces the impact of factors such as people passing by and short-term occlusion on the judgment of scene changes. Furthermore, it utilizes surface identity to maintain the correspondence between the same exhibit surface in adjacent scene versions, thereby improving the continuity and accuracy of digital exhibition hall scene updates.
[0018] By generating ordered surface intersection sequences for adjacent scene versions and extracting the sequence difference intervals, this invention can focus the scene processing scope on the surface regions that have actually changed, reducing the repeated calculation of unchanged regions; the improved PointConv model reconstructs cross-version jump connections based on the difference intervals and restricts PointDeconv interpolation propagation, avoiding the participation of failed historical features in the current surface reconstruction and reducing feature interference between different surface states.
[0019] By combining the Northern Eagle optimization algorithm to adjust the connection and interpolation control parameters, and determining the update, reuse, or temporary non-construction mode of surface segments based on the optimized surface observation status, this invention can improve the reliability of changing surface reconstruction and scene version updates, while reducing the data processing burden generated by overall reconstruction. It is suitable for the continuous construction and maintenance of digital exhibition hall scenes under conditions of frequent exhibit adjustments. Attached Figure Description
[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a digital exhibition hall scene construction method based on digital twins proposed in this invention; Figure 2This is a schematic diagram of the improved PointConv model of the digital exhibition hall scene construction method based on digital twin proposed in this invention; Figure 3 This diagram illustrates the cross-version jump connection reconstruction and constrained interpolation propagation of a digital exhibition hall scene construction method based on digital twins proposed in this invention. Detailed Implementation
[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0022] refer to Figures 1-3 A method for constructing a digital exhibition hall scene based on digital twins includes the following steps: S1. Collect observation data of the exhibition hall and register it to the exhibition hall coordinate system, remove instantaneous occlusion data, and arrange it into a point cloud sequence of the exhibition hall according to the collection time; S2. Divide the surface segments of exhibits according to the point cloud sequence of the exhibition hall, associate the same surface segments at adjacent times, and form adjacent scene versions corresponding to the surface identity; S3. Project the exhibits whose spatial range changes in adjacent scene versions to the acquisition field of view, generate an ordered surface intersection sequence, align them according to surface identity and extract the first difference to the sequence difference interval of the re-consistent sequence. S4. Improve the PointConv model to encode surface point clouds within the difference interval of the sequence, reconstruct cross-version jump connections according to the difference interval and restrict PointDeconv interpolation propagation to form the initial surface observation state. S5. Project the surface points corresponding to the initial surface observation state onto the acquisition field of view and compare them with the observation data. Calculate the scene recovery deviation. Use the Northern Eagle optimization algorithm to search for structural control parameters and feed back the improved PointConv model to reprocess the difference interval to form an optimized surface observation state. S6. Determine the surface segment construction method based on the optimized surface observation status, and generate surface construction instructions; S7. Update the surface segments according to the surface construction instructions to build a digital twin scene version; S8. Optimize the surface observation status and write it into the digital twin scene version. Associate the entity exhibits with the surface identity to form a digital exhibition hall scene and update the scene change comparison benchmark.
[0023] In this embodiment, S1 specifically includes: The exhibition hall observation data includes depth images, color images, and acquisition times obtained by the acquisition equipment, as well as camera intrinsic parameters and camera extrinsic parameters obtained through camera calibration and visual positioning, and the pose of the acquisition equipment. Fixed acquisition equipment determines the rotation matrix and translation vector from the camera coordinate system to the exhibition hall coordinate system through a calibration plate, while movable acquisition equipment records the corresponding pose at each acquisition time. When the time difference between the image and the recorded pose does not exceed 20ms, an association is established to form exhibition hall observation data with pose information. When the time difference exceeds 20ms or the corresponding pose is missing, the image is registered as a pose abnormal frame and does not participate in the generation of the current exhibition hall point cloud frame. After the pose data is completed, the association is re-executed. If the data is still not completed after the acquisition ends, the original image and acquisition time are retained but not written into the exhibition hall point cloud sequence.
[0024] The exhibition hall coordinate transformation first reads the x-coordinate, y-coordinate, and depth value of the pixels in the depth image. Then, it uses the lateral focal length, longitudinal focal length, and principal point coordinates from the camera's intrinsic parameters to restore the pixels to spatial points. The x-coordinate in the camera coordinate system is obtained by multiplying the difference between the pixel's x-coordinate and the principal point's x-coordinate by the depth value and then dividing by the lateral focal length. The y-coordinate is calculated in the same way, and the axial coordinate is taken from the depth value. The restored spatial point is first multiplied by the rotation matrix at the corresponding acquisition time, and then the translation vector is superimposed to obtain the point coordinates in the exhibition hall coordinate system. Pixels with depth values less than the minimum measurement distance of the acquisition device, exceeding the maximum measurement distance, or lacking effective depth values are not converted into spatial points.
[0025] After point clouds from different acquisition devices at the same acquisition time complete coordinate transformation, the overlapping observation area is determined by the intersection of each acquisition field of view in the exhibition hall coordinate system. Within the overlapping observation area, iterative nearest-point registration from point to plane is performed. In each round, the vertical distance from the source point to the tangent plane of the corresponding target point is calculated and the rotation and translation are updated. The iteration stops when it reaches 20 times or the average point displacement of two adjacent rounds is less than 0.2mm. After stopping, if the average point-to-plane distance is not greater than 30mm, the registration result is accepted. If it exceeds 30mm, the point cloud of the current acquisition device is not fused and the point cloud is registered as registration abnormal data. The point cloud that accepts registration is divided into voxels with a side length of 10mm. The coordinates of each point in the same voxel are accumulated and divided by the number of points to obtain the coordinates of the fused point. The color channels are averaged in the same way. All the points after fusion form the exhibition hall point cloud frame at the current acquisition time.
[0026] Transient occlusion data is determined by examining point cloud frames from three consecutive acquisition times. First, the nearest point is established between the intermediate point cloud frame and the previous and next frames. If there is no corresponding point within 15mm in either the previous or next frame for the current point, the current point is added to the newly added point set at the intermediate time. Then, the spatial position of the newly added point set is checked. If there are surface points within 15mm in the corresponding positions in both the previous and next frames and the angle between their normals is no greater than 10°, the adjacent time points are considered to have recovered to the same surface. If the newly added point set only appears at the intermediate acquisition time and meets the conditions for surface recovery, it is identified as transient occlusion data and removed from the intermediate point cloud frame. Point sets that appear continuously at two or more consecutive acquisition times are not processed as transient occlusion data.
[0027] When consecutive point cloud frames are missing or a valid correspondence cannot be established between adjacent surfaces, instantaneous occlusion removal of the current point set is not performed, and the original point set is retained. When the missing frame can be reacquired, the retained point cloud frame is compared with the previous and next adjacent frames after the missing frame is filled. If the comparison conditions cannot be met after continuous acquisition, the original point cloud frame is maintained to avoid deleting the real exhibit surface data when there is insufficient evidence.
[0028] The exhibition hall point cloud frames that have completed the instantaneous occlusion processing are arranged from earliest to latest according to the acquisition time. Only the fused single exhibition hall point cloud frame with the same acquisition time is retained. Point cloud frames with duplicate or reversed acquisition times are reordered according to the time record. Point cloud frames that lack a valid acquisition time are not included in the arrangement result. The arranged exhibition hall point cloud frames together form the exhibition hall point cloud sequence, which is used as the input for exhibit instance segmentation and exhibit surface segment division.
[0029] In this embodiment, S2 specifically includes: The exhibit instance segmentation uses a single frame of point cloud in the exhibition hall point cloud sequence as the processing object. Each point input consists of three-dimensional coordinates in the exhibition hall coordinate system, normalized color value, and unit normal vector. The three-dimensional coordinates are scaled to the range of zero to one according to the exhibition hall space boundary, the color channel is divided by 255, and the normal vector is calculated from the coordinate covariance of the 20 neighboring points around the current point. The trained point cloud instance segmentation network outputs exhibit category and instance number point by point. Points with the same instance number form an exhibit point cloud. Isolated point sets with fewer than 30 points and a distance of more than 50mm from the boundary of the nearest exhibit point cloud do not form exhibit instances and are retained in the unassigned point set of the current point cloud frame.
[0030] The surface segments of each exhibit are divided within the point cloud of each exhibit based on the continuity of the normal vector. The coordinates of the 20 neighboring points around each point are subtracted from the coordinates of the neighborhood center, and the covariance matrix is calculated. The unit eigenvector corresponding to the smallest eigenvalue of the covariance matrix is used as the normal vector of the current point. The curvature of the current point is obtained by dividing the smallest eigenvalue by the sum of the three eigenvalues. The surface segment is established starting from the unassigned point with the smallest curvature. Then, neighboring points with a spatial distance of no more than 30mm are searched. The unit normal vectors of the current point and the neighboring points are multiplied by a dot product, and the inverse cosine of the dot product result is calculated to obtain the normal angle. The plane passing through the current point and whose normal vector is the normal vector of the current point is used as the local tangent plane. When the normal angle is no greater than 12° and the vertical distance from the neighboring point to the local tangent plane is no greater than 8mm, the neighboring point is merged into the same surface segment. The newly added point continues to search for neighboring points as the expansion center until no point that meets the requirements is added. The remaining unassigned points are used to establish new surface segments in the same way.
[0031] After the surface segments are divided, surface segments with fewer than 20 points are checked against their nearest adjacent surface segments. If the average normal angle between two surface segments is no greater than 12° and the average distance between their boundary points is no greater than 15mm, they are merged. If the merging requirements are not met, they are retained as independent surface segments. Each surface segment records its associated exhibit, the index of the points it contains, the surface center point, and the average normal vector. The surface center point is obtained by summing the coordinates of all points in the segment and dividing by the number of points. The average normal vector is obtained by summing all unit normal vectors and then normalizing them.
[0032] For pose transformations of the same exhibit at adjacent time points, candidate correspondences are first established based on exhibit category, bounding area size, and center point position. For exhibits of the same category, bounding side length ratios in the three coordinate directions are between 0.8 and 1.25, and the center point distance is no greater than 500mm, rigid registration is performed. Fast point feature histograms are calculated for the sampling points in the point clouds of the two candidate exhibits. These histograms are statistically formed by the relationship between the sampling point and its neighboring points in terms of distance, normal angle, and normal direction. A correspondence is established between the sampling points in the two exhibit point clouds with the smallest feature distance. The corresponding point coordinates are subtracted from their respective centroids, and the outer product of the decentered coordinates from the previous time point and the current time point is accumulated. Singular value decomposition is then performed on the accumulated matrix to obtain the rotation matrix. The translation vector is obtained by subtracting the rotated centroid from the centroid of the previous time point at the current time point. A pose transformation is confirmed when the average distance between the transformed corresponding points is no greater than 20mm and the proportion of effective corresponding points to all corresponding points is no less than 60%. If these requirements are not met, no correspondence is established for the same exhibit, and the exhibit at the current time point is treated as a new exhibit.
[0033] Spatial overlap matching involves multiplying the points in each surface segment at the previous moment by the rotation matrix and superimposing the translation vector, then comparing them with the surface segments in the same exhibit at the current moment. For each transformed point, the nearest current point is found. Points with a distance of no more than 15mm between them and a normal angle of no more than 15° are recorded as overlapping points. The number of overlapping points is divided by the smaller value between the number of points in the previous and next surface segments to obtain the surface overlap rate. When the surface overlap rate is no less than 60%, a candidate match is established. When there are multiple candidate matches for the same surface segment, the one-to-one correspondence is determined in descending order of overlap rate. Surface segments that have already been matched do not participate in the remaining candidate matches.
[0034] Surface identity is composed of a stable exhibit identifier and an incremental number within the exhibit. The current surface segment that has completed a match retains the surface identity from the previous moment. The surface segment that has not matched in the current moment is assigned a new surface identity. The surface segment that has not matched in the previous moment retains its original surface identity and the observation point set is set to empty in the current scene version. When a surface segment from the previous moment matches multiple current surface segments, only the current surface segment with the highest overlap rate retains its original surface identity, and the other current surface segments are assigned new surface identities. When a current surface segment matches multiple surface segments from the previous moment, the current surface segment retains the surface identity from the previous moment with the highest overlap rate, and the other previous surface identities are retained but not written into the current observation point set.
[0035] The exhibit instances, exhibit surface segments, surface identities, and corresponding results at adjacent acquisition times are organized in pairs according to the acquisition time. The previous scene version retains the surface point data and surface identity of the previous moment, while the current scene version writes the surface point data, inherited identities, newly added identities, and unobserved identities of the current moment. This forms the adjacent scene versions corresponding to the surface identities, which serve as inputs for judging changes in the spatial range of exhibits and generating ordered surface intersection sequences.
[0036] In this embodiment, S3 specifically includes: Changes in the spatial range of exhibits are compared using surface segments with the same exhibit identifier in adjacent scene versions. The minimum and maximum values of surface points in each version are calculated in the three coordinate directions of the exhibition hall coordinate system. The minimum and maximum values in each direction are used to determine the enclosing range and center of the enclosing range of the exhibit. When the distance between the center point of the current version and the previous version is not less than 20mm, or the boundary change in any coordinate direction is not less than 20mm, the corresponding exhibit is identified as an exhibit whose spatial range has changed. Exhibits newly added in the current version or exhibits that existed in the previous version but are not matched in the current version are directly included in the changed exhibits. The comparison results form the corresponding data of the spatial range of the changed exhibits and adjacent scene versions.
[0037] The field-of-view projection is acquired using the calibrated camera intrinsics, extrinsics, and effective image size from the exhibition hall observation data. First, the vertices of the bounding range of the changed exhibits in the previous and current scene versions are transformed from the exhibition hall coordinate system to the camera coordinate system. Then, the horizontal and vertical coordinates are divided by the axial depth and multiplied by the corresponding focal length. Finally, the principal point coordinates are superimposed to obtain the pixel position. Vertices with an axial depth less than or equal to 0 do not participate in the projection. The effective projection points of each scene version form rectangular projection ranges with the minimum and maximum values of the horizontal and vertical axes, respectively. Then, the intersection of the rectangular projection range and the effective image range is calculated. When the intersection is empty, the corresponding scene version does not generate an effective projection range.
[0038] When both the previous scene version and the current scene version have valid projection ranges, the union of the two valid projection ranges is taken to form a candidate acquisition area. When only one scene version has a valid projection range, the corresponding projection range is directly used to form a candidate acquisition area. When neither scene version has a valid projection range, the ray generation corresponding to the current acquisition device is terminated.
[0039] The candidate acquisition ray originates from the optical center of the acquisition device and passes through the sampling pixels within the candidate acquisition area. The sampling pixels are arranged with the same interval in both the horizontal and vertical directions. In one embodiment, the pixel interval is 4 pixels. When the width of the candidate acquisition area in any direction is less than 4 pixels, one sampling pixel is generated at the center of the area. The principal point coordinates are subtracted from each sampling pixel, and then divided by the horizontal focal length and the vertical focal length respectively to obtain the ray direction in the camera coordinate system. After the ray direction is normalized, it is rotated to the exhibition hall coordinate system through the camera extrinsic parameters. The optical center of the acquisition device is transformed to the exhibition hall coordinate system through the camera extrinsic parameters. The two together determine the candidate acquisition ray.
[0040] The intersection point of the candidate acquisition ray and the surface section of the exhibit is obtained by finding the intersection point of the local plane. First, the vertical distance from each surface point in the surface section to the ray is calculated. Surface points with a vertical distance of no more than 15 mm and located in the positive direction of the ray are selected as candidate neighbor points. Then, the 20 candidate neighbor points closest along the ray depth are fitted to the local plane. The normal vector of the local plane is the eigenvector corresponding to the smallest eigenvalue of the covariance matrix of the candidate neighbor points. When the absolute value of the dot product of the ray direction and the local plane normal vector is less than 0.05, the ray is considered to be approximately parallel to the local plane and the current intersection is abandoned. In other cases, the intersection point depth is obtained by dividing the dot product of the vector from the ray starting point to the plane reference point and the plane normal vector by the dot product of the ray direction and the plane normal vector. When the intersection point depth is greater than 0 and the distance from the intersection point to the nearest surface point is no more than 15 mm, the intersection point is retained.
[0041] When the same candidate acquisition ray and the same surface segment yield multiple valid intersection points, they are clustered according to their intersection depth. Intersection points with a depth difference of no more than 20 mm are grouped into the same intersection point group, and the average depth of the intersection points within the group is used as the intersection depth of the surface segment. The intersection points of each surface segment are arranged in ascending order of intersection depth, and the surface identity and intersection depth are recorded at each arrangement position. This generates ordered surface intersection sequences for the previous scene version and the current scene version, respectively. When the difference in intersection depth between two surface segments is less than 5 mm, they are compared again according to the average intersection depth. If the depths are still the same, the arrangement position is determined according to the surface identity coding order to ensure that the same input yields consistent arrangement results.
[0042] The ordered surface intersection sequence of adjacent scene versions is compared item by item starting from the end closest to the acquisition device. When the surface identities and arrangement order are the same at the current position, they move to the next position simultaneously. When the surface identities are different for the first time, one side of the sequence lacks a corresponding position, or the arrangement order of the same surface identity changes, the current position is determined as the first difference position. After the first difference position, the search continues to find positions in the two sequences where the surface identities are the same. After finding the same surface identity, the two consecutive arrangement positions starting from that position are compared. When the surface identities and arrangement order of the two consecutive positions are the same, the first consistent position is determined as the new consistent position. If there are less than two remaining positions, all remaining positions are required to be consistent. If no new consistent position is found, the end of the longer sequence is determined as the end of the difference interval.
[0043] When the previous scene version sequence is empty and the current scene version sequence is not empty, the entire sequence of the current scene version is determined as the sequence difference interval. When the current scene version sequence is empty and the previous scene version sequence is not empty, the entire sequence of the previous scene version is determined as the sequence difference interval. When both sequences are empty, the current candidate acquisition ray is discarded. For each valid ray, the surface identity, intersection depth range and corresponding surface points involved are extracted from the first difference position to the re-consistency position to form a single ray difference result.
[0044] The single-ray difference results of multiple candidate acquisition rays are first grouped according to surface identity, and then divided into continuous ray groups according to the spatial adjacency of the sampled pixels. When the index difference between two sampled pixels in the horizontal or vertical direction does not exceed one sampling interval, the corresponding rays are considered to be adjacent. When the same surface identity appears repeatedly in the same continuous ray group, the surface point indices involved in each ray are merged and the minimum starting depth and maximum ending depth are taken as the merging range. When the ray groups corresponding to the same surface identity are not adjacent to each other, they are retained as independent difference ranges. When the nearest distance between the surface points corresponding to different surface identities is not greater than 30mm and the corresponding rays are adjacent, the corresponding results are merged into the same continuous difference range. The merged surface identity, surface point index, ray range and intersection position range together constitute the sequence difference interval, which is used as the surface point cloud range processed by the improved PointConv model.
[0045] In this embodiment, the improved PointConv model includes a PointConv encoding layer, a cross-version skip connection layer, a restricted PointDeconv layer, and a state output layer connected in sequence: In practical implementation, the ordinary PointConv model includes a PointConv encoder, a PointDeconv decoder, and fixed skip connections connecting encoding and decoding nodes of the same resolution. The PointConv encoder reduces the number of points and increases the feature channels layer by layer through hierarchical sampling, neighborhood grouping, and point convolution. The PointDeconv decoder restores the point resolution layer by layer through neighboring point interpolation and concatenates the interpolated features with the same resolution encoded features passed in by the fixed skip connections. The ordinary PointConv model processes the point cloud at a single time step. The fixed skip connections do not adjust with the addition, disappearance, or intersection order of surfaces in adjacent scene versions. Encoded features that have expired in the previous scene version will still enter the current decoding path. The ordinary PointDeconv will also select interpolated neighboring points only based on spatial distance, which can easily propagate historical features across changing surface boundaries.
[0046] The improved PointConv model consists of a PointConv encoding layer, a cross-version skip connection layer, a restricted PointDeconv layer, and a state output layer connected sequentially. The PointConv encoding layer includes an input ordering unit, a hierarchical sampling unit, a neighborhood grouping unit, a point convolution calculation unit, and an encoding node registration unit. The cross-version skip connection layer includes a node correspondence unit, a difference interval localization unit, a connection state update unit, and a connection output unit. The restricted PointDeconv layer includes an effective surface range resolution unit, an interpolation neighbor selection unit, a distance interpolation unit, a same-resolution fusion unit, and a stepwise recovery unit. The state output layer includes a feature readout unit, a position correction unit, a normal correction unit, a confidence value generation unit, and a surface state merging unit.
[0047] The input sorting unit receives the surface point cloud corresponding to the sequence difference interval in adjacent scene versions, and reads the encoded nodes, encoded features, and skip connection indices that have been saved outside the sequence difference interval in the previous scene version. The surface points of the previous scene version and the current scene version within the sequence difference interval enter the PointConv encoding layer for re-encoding. The saved encoded nodes outside the sequence difference interval do not repeat the PointConv encoding, but directly enter the cross-version skip connection layer and maintain the original skip connection. The point features participating in point convolution consist of three-dimensional coordinates, color values, and unit normal vectors, with a total of 9 feature components. The three-dimensional coordinates are divided by the maximum side length of the bounding range after subtracting the center of the change expansion term, the color values are divided by 255, and the unit normal vectors retain three directional components. The scene version label, surface identity, and intersection position are transmitted synchronously with the point features as index information but do not participate in numerical convolution.
[0048] The hierarchical sampling unit processes the point cloud of the previous scene version and the point cloud of the current scene version respectively. In one embodiment, the number of valid surface points input for each scene version does not exceed 4096. If the number exceeds 4096, 4096 points are retained by sampling from the farthest point. Sampling from the farthest point starts from any valid point. In each round, the minimum Euclidean distance from the unselected point to the set of selected points is calculated and the point with the largest minimum Euclidean distance is selected until the target number of points is reached. The encoding process sets four resolution levels. The number of sampling points in each level is 1024, 256, 64 and 16 respectively. The output feature dimensions of each level are 64, 128, 256 and 512 respectively. When the number of valid points is less than the target number of sampling points, all valid points are retained and invalid points are not copied to generate encoding nodes.
[0049] The neighborhood grouping unit takes each sampling center as a reference and selects the 32 nearest neighboring points from the set of points corresponding to the same surface identity. Points corresponding to different surface identities do not enter the current neighborhood. If there are fewer than 32 effective neighboring points in the same surface segment, the farthest effective neighboring point is used to fill the gap. When only the sampling center exists, the sampling center point features are copied. The coordinates of the neighboring points are subtracted from the coordinates of the sampling center to form 3D relative coordinates. The neighboring point features, relative coordinates, and distance between points are all entered into the point convolution calculation unit.
[0050] The point convolution computation unit follows the point convolution processing of PointConv, inputting the relative coordinates of neighboring points into the multilayer perceptron to generate convolution weights corresponding to the input and output channels, and multiplying the features of neighboring points with the convolution weights; the local density of neighboring points is estimated by kernel density based on the distance between neighboring points and the sampling center, and when the local density is less than 0.001, it is used as 0.001 in the calculation. The reciprocal of the local density is compressed to 0 to 1 by fully connected transformation and Sigmoid to form a density compensation value; the product of the feature and the convolution weight is multiplied by the density compensation value, accumulated channel by channel along the current neighborhood and superimposed with the output bias, and the accumulated result is batch normalized and ReLU to form the encoded feature corresponding to the sampling center.
[0051] After completing point convolution at each resolution level, the encoding node registration unit establishes an encoding node index table. Each row of the encoding node index table corresponds to an encoding node, and each column records the scene version tag, resolution level, encoding node number, sampling center point number, surface identity, intersection position, and encoding feature storage address in sequence. The sampling center directly inherits the surface identity and intersection position of the original surface point. When the same sampling neighborhood covers multiple surface segments, the surface segment to which the sampling center belongs is still used as the encoding node affiliation. Neighboring points corresponding to different surface identities have been excluded in the neighborhood grouping unit.
[0052] The corresponding node unit reads the encoding node index table. Within the same resolution level, it first searches for the previous scene version encoding node and the current scene version encoding node with the same surface identity and the same intersection position, and registers the search results as cross-version corresponding nodes. When multiple encoding nodes correspond to the same surface identity and the same intersection position, it calculates the Euclidean distance between the coordinates of the previous scene version sampling center after the item pose transformation and the coordinates of the current scene version sampling center, and establishes a one-to-one correspondence according to the distance from small to large. Encoding nodes that have completed the corresponding matching no longer participate in the remaining matching, and encoding nodes that have not found a corresponding node are registered as single-version nodes.
[0053] The difference interval location unit reads the first difference position and the re-consistency position, and compares the intersection positions in the coding node index table. If the intersection position is less than the first difference position number, the coding node is registered as a node outside the difference interval. If the intersection position starts from the first difference position and is less than the re-consistency position, it is registered as a node inside the difference interval. If no re-consistency position appears, all nodes from the first difference position to the end of the intersection sequence on the ordered surface are registered as nodes inside the difference interval. Coding nodes that lack valid intersection positions do not establish skip connections and are retained inside the coding layer.
[0054] The connection state update unit establishes a connection state table. Each row of the connection state table corresponds to a jump connection from an encoding node to a PointDeconv node of the same resolution. Each column records the resolution level, source scene version, source encoding node number, surface identity, intersection position, target PointDeconv node number, and connection validity value in sequence. When the connection validity value is 1, the encoded features are allowed to enter the target PointDeconv node. When the connection validity value is 0, the target PointDeconv node does not read the corresponding encoded features.
[0055] The connection state update unit keeps the connection validity value of the previous scene version encoding node outside the difference interval at 1, updates the connection validity value of the previous scene version encoding node inside the difference interval to 0, and updates the connection validity value from the current scene version encoding node to the target PointDeconv node to 1 when there is an encoding node of the same resolution in the current scene version. After the newly added surface segment in the current scene version generates an encoding node through the PointConv encoding layer, a new valid connection is established. Encoding nodes in the previous scene version that do not match the current scene version continue to be retained in the encoding node index table, but the connection validity value remains at 0.
[0056] After the connection state is updated, the failed encoding nodes located in the difference interval in the previous scene version can no longer enter the current PointDeconv node. The encoding nodes of the current scene version replace the historical encoding source through valid connections, thereby changing the forward propagation path of the fixed skip connection of the ordinary PointConv, so that the same resolution features read by the decoding path are consistent with the surface changes in the current scene version.
[0057] When there are multiple candidate connections for the same target PointDeconv node, the connection state update unit first reads whether the intersection position is within the difference interval. Within the difference interval, only the connection of the current scene version is retained, and outside the difference interval, only the connection of the previous scene version is retained. When there are still multiple candidate encoding nodes for the same scene version, the encoding node connection with the smallest spatial distance between the sampling center and the corresponding point of the target PointDeconv node is retained, and the connection validity value of other candidate connections is updated to 0, thereby ensuring that there is only one valid encoding source for the same resolution, the same surface identity, and the same intersection position.
[0058] The connection output unit encapsulates records with a connection validity value of 1 into a valid connection index. The valid connection index records the encoded feature storage address, the target PointDeconv node number, the surface identity, and the range of valid surface points in sequence. Records with a connection validity value of 0 are not included in the valid connection index. The restricted PointDeconv layer only reads the valid connection index during the model's forward processing, thereby applying the update results in the connection state table to the encoded feature reading path.
[0059] The effective surface range parsing unit reads the surface identity from the effective connection index and extracts the point index contained in the corresponding surface identity in the current resolution point set to form an effective surface point range that allows interpolation. When the same surface identity corresponds to multiple spatially discontinuous difference ranges, the point index of each difference range is retained, and other surface points between different difference ranges are not merged into the current effective surface point range.
[0060] The interpolation neighbor filtering unit takes each point to be restored as the center and searches for the three nearest neighbor points among low-resolution points with the same surface identity and located within the same effective surface point range. If there are fewer than three effective neighbor points, all effective neighbor points are used. If there are neighbor points that coincide with the coordinates of the point to be restored, the features of the coincident points are read directly. If there are multiple coincident points, the features of the coincident points are averaged channel by channel. If there are no neighbor points within the range of the current effective surface point, the features passed from the PointDeconv layer of the previous resolution level are retained, and neighbor points are not selected from other surface segments.
[0061] Interpolation neighbor selection is limited to points with the same surface identity and the same valid surface points. The propagation path of ordinary PointDeconv, which selects neighboring points across the changing surface boundary based on spatial distance, is cut off. Low-resolution features that are unrelated to the current surface in the previous scene version cannot enter the current surface restoration process through interpolation.
[0062] The distance interpolation unit calculates the Euclidean distance between the point to be recovered and the effective neighboring points. When the Euclidean distance is not less than 0.1 mm, the reciprocal of the distance is taken as the initial interpolation weight. Then, each initial interpolation weight is divided by the sum of the initial interpolation weights to make the sum of all interpolation weights equal to 1. Each neighboring point feature is multiplied by the corresponding interpolation weight channel by channel and then accumulated to form the interpolation feature of the point to be recovered. When the Euclidean distance is less than 0.1 mm, the corresponding coincident point feature is directly used, and the reciprocal of the distance is no longer calculated.
[0063] The same-resolution fusion unit reads the same-resolution encoded features corresponding to the effective connections through the target PointDeconv node number, concatenates the interpolated features and the same-resolution encoded features along the channel direction, multiplies the concatenated feature vector with the shared weight matrix and adds the bias, and then performs batch normalization and ReLU sequentially. When the current scene version lacks same-resolution encoded nodes, it does not read the same-resolution encoded features, but only retains the features obtained by interpolation from the previous resolution level PointDeconv layer, and inputs the interpolated features into the dimension matching fully connected layer. The dimension matching result continues to enter the current PointDeconv processing path.
[0064] The progressive restoration unit starts from the lowest resolution point set, restoring 16 points to 64 points, 64 points to 256 points, 256 points to 1024 points, and then back to the input surface point resolution. At each resolution level, effective surface range resolution, interpolation neighbor filtering, distance interpolation, and same-resolution fusion are performed sequentially. The fusion result is then aggregated with spatial features within the same surface identity range through PointConv convolution. The output of the current layer is used as the low-resolution input of the next resolution level until the restored features corresponding one-to-one with the input surface points are obtained.
[0065] The feature readout unit restores the features to the input point resolution and inputs them into a two-layer fully connected network. The first layer maps the current feature dimension to a 128-dimensional readout feature and superimposes the bias. Then, ReLU sets the components less than 0 to 0. The 128-dimensional readout features are then fed into the position correction unit, normal correction unit, and confidence value generation unit, respectively.
[0066] The position correction unit multiplies the 128-dimensional readout features with a 128x3 position weight matrix and adds a 3-dimensional position bias to form a 3-dimensional position correction. The position correction is compressed to -1 to 1 using a hyperbolic tangent function, multiplied by 20mm, and then added component by component with the input point coordinates to form the corrected coordinates. The normal correction unit multiplies the 128-dimensional readout features with a 128x3 normal weight matrix and adds a 3-dimensional normal bias to form a 3-dimensional normal correction. The normal correction is added component by component with the input unit normal vector and divided by the length of the sum to form the corrected unit normal vector. If the vector length is less than 0.000001, the input unit normal vector is retained.
[0067] The confidence value generation unit multiplies the 128-dimensional readout features with a 128-row, 1-column confidence value weight matrix and adds a 1-dimensional bias, then compresses it to 0 to 1 using a Sigmoid function to form the observation confidence value. When the observation confidence value is not lower than 0.5, the corrected coordinates and corrected unit normal vector are registered as valid observation states. When the observation confidence value is lower than 0.5, the input point coordinates and input unit normal vector are retained and registered as low confidence states.
[0068] The surface state aggregation unit uses the surface identity as an index to write the point index, corrected coordinates, corrected unit normal vector, observation confidence value and confidence state of each surface point in the same surface segment into the surface point state table. Each row of the surface point state table corresponds to a surface point. Different surface identities form independent surface states. After all surface states are aggregated, the initial surface observation state is formed. The initial surface observation state maintains a one-to-one correspondence between the surface identity and the specific surface point.
[0069] The improved PointConv model uses adjacent scene version surface point clouds and the current real observed surface as training samples. The training input is extracted from the sequence difference interval, and the training target is the current surface point obtained by manual correction or high-precision point cloud registration. When the distance between the restored point and the nearest real point is no more than 20 mm and the angle between the unit normal vectors of the two points is no more than 15°, the confidence value supervision flag of the corresponding restored point is set to 1, and the confidence value supervision flag of other restored points is set to 0. The average square distance from each restored point to the nearest real point and the average square distance from each real point to the nearest restored point are calculated separately. The normal error is calculated as the average angle between the unit normal vectors of the effective corresponding points. The confidence value error is calculated as the binary cross-entropy between the observed confidence value and the confidence value supervision flag. In one embodiment, the weights of the three parts of the loss are 0.6, 0.3 and 0.1 respectively, and the weighted results of the three parts are added together to form the total loss.
[0070] The fully connected weights, convolutional weights, and biases in the PointConv encoding layer, restricted PointDeconv layer, and state output layer are updated by gradients using the Adam optimizer. The initial learning rate is 0.001. Training stops when the total loss has not decreased for 10 consecutive epochs after 200 training epochs. The cross-version skip connection layer does not determine the connection endpoints through gradients, but directly changes the forward propagation path by encoding the node index table, the difference interval localization results, and the connection state table.
[0071] After training, the surface point clouds of adjacent scene versions within the sequence difference interval undergo input reordering, hierarchical sampling, neighborhood grouping, point convolution calculation, encoding node registration, connection state update, effective range interpolation, same-resolution fusion, step-by-step recovery, and state readout in sequence. The encoded features of failed historical surfaces within the difference interval stop propagating to the current decoding path because the effective connection value is updated to 0. The restricted PointDeconv layer only completes feature recovery within the current effective surface range, and finally forms the initial surface observation state corresponding to the surface points of the current scene version.
[0072] In this embodiment, the reconstruction of the cross-version jump connection layer specifically includes: When reconstructing cross-version jump connections, the PointConv encoding layer reads the encoding node index table, the ordered surface intersection sequence of adjacent scene versions, and the sequence difference interval. It then determines the surface segment, intersection position, and PointDeconv node of the same resolution corresponding to the encoding node in each resolution level. The connection reconstruction result is written to the connection state table. The restricted PointDeconv layer reads the connection state table to maintain the valid connection and completes the encoding feature selection and surface point feature recovery.
[0073] The sampling points corresponding to the encoding node include the sampling center and the neighboring points that actually participate in the feature aggregation of the current encoding node. The neighborhood grouping unit has excluded neighboring points with different surface identities from the sampling center. Therefore, the encoding node directly inherits the surface identity corresponding to the sampling center. When the sampling center lacks a valid surface identity, the encoding node is registered as an unowned node. Unowned nodes retain the encoding features but do not establish skip connections to the PointDeconv node.
[0074] When a surface segment appears on multiple candidate acquisition rays, the permutation number and intersection depth of the surface identity in each ordered surface intersection sequence are recorded. Then, the occurrence frequency of each permutation number is counted, and the permutation number with the highest occurrence frequency is determined as the intersection position of the surface segment. When multiple permutation numbers have the same occurrence frequency, the average value of the corresponding intersection depths is calculated, and the permutation number with the smaller average intersection depth is determined as the intersection position. In this way, the surface order in multiple rays is merged into a unique intersection position, while the ray index participating in the merging is retained for locating the corresponding spatial range.
[0075] For each resolution level, a correspondence is established between encoding nodes, surface identities, intersection positions, and PointDeconv nodes. Each row in the encoding node index table records the resolution level, scene version marker, encoding node number, surface identity, and intersection position. PointDeconv nodes of the same resolution are searched by the surface identity and intersection position of the point to be restored. When multiple PointDeconv nodes correspond to the same surface identity and the same intersection position, the Euclidean distance between the sampling center of the encoding node and each point to be restored is calculated. The PointDeconv node with the smallest distance is determined as the connection target. Encoding nodes that have been determined as connection targets are not reassigned to other target nodes.
[0076] The surface segments of the exhibits before the first difference position maintain the same surface identity and intersection order in adjacent scene versions. The source node number, target node number, and connection validity value of the encoded node to the same resolution PointDeconv node in the previous scene version remain unchanged. The encoded features of the previous scene version continue to be used as the same resolution input of the current decoding path, without regenerating connections or replacing the encoded source.
[0077] The connection rewriting process begins at the first difference position. The connection state update unit reads the surface segments within the difference interval in ascending order of intersection positions. First, it updates the valid connection values from the previous scene version's encoding node to the same resolution PointDeconv node to 0. Then, it searches for the encoding node in the current scene version that has the same resolution level and belongs to the surface segment of the exhibit at the current intersection position. The found current encoding node is written to the source node number, and the valid connection value is updated to 1. After each intersection position's connection rewriting is completed, the updated connection state is immediately written back to the connection state table before processing the next intersection position in the difference interval. No unordered overall replacement is performed on all difference positions.
[0078] When a new surface segment in the current scene version does not have an encoding node from the previous scene version, the connection state update unit directly reads the corresponding encoding node of the current scene version and establishes a new connection from the current encoding node to the PointDeconv node of the same resolution. Surface segments that exist in the previous scene version but are not matched in the current scene version stop propagating to the decoding path. The corresponding encoding node is still retained in the encoding node index table, but the connection validity value is updated to 0. The restricted PointDeconv layer does not read the corresponding encoding features in the current forward processing.
[0079] The previous resolution layer, PointDeconv, is the layer before the current PointDeconv layer that directly provides low-resolution features in the decoding direction. The decoding process starts with the features corresponding to 16 points and is restored sequentially to 64 points, 256 points, 1024 points, and the input surface point resolution. When the current scene version lacks an encoding node at the current resolution layer, the current PointDeconv node retains the features obtained by interpolation from the previous resolution layer, does not perform concatenation of encoded features at the same resolution, and then multiplies the interpolated features with the dimension matching weight matrix and adds the bias to make the feature dimension meet the input requirements of the current PointDeconv convolution. After completing the current layer's PointConv convolution, it continues to enter the next resolution layer.
[0080] When the surface identity and intersection order reach the re-consistent position, the connection state update unit searches for nodes with the same resolution level, surface identity, and intersection position in the previous scene version encoding nodes. When there are multiple previous scene version encoding nodes at the same position, the encoding node with the smallest distance between the sampling center after the presentation pose transformation and the corresponding point of the target PointDeconv node is determined as the recovery source. The effective value of the current scene version connection is updated to 0, and the effective value of the selected previous scene version connection is restored to 1, so that the re-consistent position and the stable surface thereafter continue to inherit the encoding features of the previous scene version.
[0081] When a previous scene version encoding node is missing at a re-consistent position, historical encoding features without a corresponding relationship are not connected to the current decoding path. The current scene version connection remains valid until the next position with consistent surface identity and intersection position and a valid previous scene version encoding node is found. In the case of missing valid correspondence, a connection anomaly record is written. The connection anomaly record only saves the resolution level, surface identity and intersection position, and does not participate in feature convolution.
[0082] When no new consistent position appears in the intersection sequence of ordered surfaces, the current scene version encoding node is kept connected from the first difference position to the end of the sequence. The connection validity value of the previous scene version encoding node is always 0. The encoding features of the previous scene version must not re-enter the decoding path before the end of the difference interval. When the current scene version lacks encoding nodes of the same resolution at some intersection positions, the interpolation features passed from the PointDeconv layer of the previous resolution level are used to maintain decoding continuity, and the historical encoding nodes corresponding to other surface identities are not called to supplement the features.
[0083] The connection state is implemented in the computer through connection index and connection validity value. The connection index records the storage address of the source encoded features and the target PointDeconv node number. The connection validity value uses 0 or 1 to record whether the connection participates in the current forward processing. The restricted PointDeconv layer reads the connection record corresponding to the target node number and loads the encoded features from the source storage address only when the connection validity value is 1. When the connection validity value is 0, it skips feature reading and concatenation at the same resolution. This allows the connection state update to directly change the feature propagation path inside the model, rather than just filtering the feature results at the model output.
[0084] When the same PointDeconv node is affected by multiple sequence difference intervals, the surface identities corresponding to each difference interval are compared first. Connections with different surface identities are applied to their respective surface point ranges, and cross-surface merging is not performed. When the surface identities are the same and the intersection position ranges overlap, the overlapping intervals are merged into a continuous range from the smallest first difference position to the largest ending position, and the overlapping positions are covered by the current scene version connection. When the surface identities are the same but the intersection position ranges do not overlap, each connection range is retained, and the PointDeconv node reads the connection state within the corresponding range when processing different surface points.
[0085] In one implementation, the ordered surface intersection sequence of the previous scene version sequentially includes surface identity A, surface identity B, surface identity C, and surface identity D, and the ordered surface intersection sequence of the current scene version sequentially includes surface identity A, surface identity E, surface identity C, and surface identity D. Surface identity B is replaced by surface identity E. The first difference position is the second intersection position, and the re-consistency position is the third intersection position. Within each resolution level, the encoding node of the previous scene version corresponding to surface identity A remains connected, the connection corresponding to surface identity B is updated to 0, and the connection of the encoding node of the current scene version corresponding to surface identity E is updated to 1. The restricted PointDeconv layer reads the current encoding feature of surface identity E at the second intersection position, and restores the connection of the encoding node of the previous scene version corresponding to surface identity C when the third intersection position corresponding to surface identity C is reached. Surface identity D continues to use the original connection.
[0086] When a surface identity E lacks a coding node at a certain resolution level, the corresponding PointDeconv node does not read the historical coding features of surface identity B, nor does it read the coding features of other surface identities. Instead, it retains the surface identity E features that have been interpolated at the previous resolution level and completes the convolution of the current layer. The recovered features at each resolution obtained in this way continue to be fed into the state output layer and merged into the initial surface observation state according to the surface identity, so that the connection rewriting results in the sequence difference interval are consistent with the state of the specific surface point.
[0087] Compared to the ordinary PointConv model, the improvement lies in changing the nature of skip connections in the model. Ordinary skip connections are fixed hierarchical channels between the encoding and decoding layers. This invention transforms fixed hierarchical channels into a dynamic feature inheritance structure jointly controlled by scene version, surface identity, and intersection order. Whether historical encoded features can enter the current decoding path is no longer determined by the single condition of the same resolution, but by whether the entity surface maintains true continuity in adjacent scene versions. Changed surfaces lose the qualification for historical feature inheritance, while newly added surfaces gain the qualification for writing current encoded features. Surfaces that temporarily lack encoded nodes maintain continuity through the decoder's own interpolation without borrowing irrelevant historical features. This improvement transforms cross-version point cloud recovery from static feature reuse to selective feature inheritance consistent with the continuity relationship of physical surfaces. It not only preserves the stable geometric information of unchanged surfaces, but also blocks the historical feature pollution of disappearing surfaces and out-of-order surfaces from the source of model forward propagation. This can reduce surface ghosting, erroneous surface patching, and cross-boundary interpolation, and maintain version continuity and surface identity consistency in digital exhibition hall scenes when there are local changes.
[0088] In this embodiment, S5 specifically includes: The effective observation state in the initial surface observation state includes surface identity, surface point index, corrected coordinates, corrected unit normal vector, and observation confidence value. Surface points with an observation confidence value of not less than 0.5 are included in the scene recovery deviation calculation, while surface points with an observation confidence value less than 0.5 are retained in the initial surface observation state but no corresponding observation point is established. The effective surface points are first transformed from the exhibition hall coordinate system to the acquisition device coordinate system. During the transformation, the translation vector in the camera extrinsic parameters is subtracted from the surface point coordinates, and then multiplied with the transpose of the extrinsic parameter rotation matrix to obtain the lateral coordinates, longitudinal coordinates, and axial depth in the camera coordinate system.
[0089] When surface points are projected onto the acquisition field of view, points with an axial depth greater than 0 have their lateral coordinates divided by the axial depth, multiplied by the lateral focal length, and then superimposed with the principal point's lateral coordinates. Similarly, points with their longitudinal coordinates divided by the axial depth, multiplied by the longitudinal focal length, and then superimposed with the principal point's longitudinal coordinates are used to obtain the projected pixel position. Surface points with an axial depth less than or equal to 0, projected pixels exceeding the effective image size, or whose projection calculation results are not finite values are not included in the observation point matching process, and the corresponding points are registered as having no valid projection points.
[0090] The correspondence between the projection point and the observation data in the exhibition hall is established within a 5×5 pixel area around the projection pixel. First, the observation pixels with effective depth values in the area are restored to three-dimensional observation points in the coordinate system of the acquisition device. Then, the Euclidean distance between the projection surface point and each three-dimensional observation point is calculated, and the observation point with the smallest distance is determined as the candidate corresponding point. Point correspondence is established when the minimum distance is not greater than 30mm, and point correspondence is not established when the minimum distance is greater than 30mm or when there is no effective depth value in the area.
[0091] Candidate corresponding points also need to undergo occlusion checks. When the axial depth of a candidate observation point is less than the axial depth of the projection surface point and the difference between the two depths exceeds 20mm, the projection surface point is considered to be located after the currently visible observation surface, and the corresponding point is excluded from the deviation calculation. When the same observation point corresponds to multiple projection surface points at the same time, the point with the smallest three-dimensional Euclidean distance is retained, and other projection surface points are searched again for unused observation points in the area. If there are no available observation points, it is registered as no corresponding point.
[0092] The positional deviation is calculated from the three-dimensional coordinates corresponding to each group of valid points. The coordinate differences between the projected surface point and the corresponding observation point in the three coordinate directions are calculated separately. The three coordinate differences are squared and then added together to obtain the squared distance corresponding to the current point. The squared distances corresponding to all valid points are accumulated and divided by the number of valid points. The square root of the division result is then taken to obtain the scene restoration deviation. When the number of valid points is 0, the scene restoration deviation is recorded as 100mm to avoid the parameter results without valid observations being mistakenly identified as low deviation results.
[0093] The connection and interpolation control parameters apply to the cross-version skip connection layer and the restricted PointDeconv layer, respectively. The connection control parameter is the maximum spatial corresponding distance allowed when an encoding node at each resolution level establishes an effective connection with the target PointDeconv node. The interpolation control parameters include the interpolation search radius, the number of interpolation neighboring points, and the distance weight index for each resolution level. The maximum spatial corresponding distance controls whether the encoding node of the current scene version can access the PointDeconv node of the same resolution. The interpolation search radius controls the range of low-resolution points that the point to be recovered can read. The number of interpolation neighboring points controls the number of points participating in the weighted accumulation. The distance weight index controls the decay rate of the interpolation weight due to the distance between neighboring points.
[0094] For each resolution level, the baseline distance between the sampling centers of the current scene version is first calculated. The baseline distance is the median of the distances from each sampling center to the nearest sampling center of the same surface identity. If there is no adjacent sampling center of the same surface identity, it is twice the baseline distance of the previous resolution level. If the lowest resolution level still cannot calculate, it is 20mm. The initial value of the maximum spatial distance is twice the baseline distance. The lower limit of the search is the baseline distance, and the upper limit of the search is four times the baseline distance. The initial value of the interpolation search radius is three times the baseline distance. The lower limit of the search is 1.5 times the baseline distance, and the upper limit of the search is five times the baseline distance. The initial value of the number of interpolation neighboring points is 3, and the search range is 2 to 8. The initial value of the distance weight index is 1, and the search range is 1 to 3.
[0095] In the Northern Eagle optimization algorithm, a candidate individual encodes the connection and interpolation control parameters for all resolution levels. The four resolution levels are respectively written with a maximum spatial corresponding distance, an interpolation search radius, an interpolation neighbor number, and a distance weight index, forming a 16-dimensional candidate parameter vector. The continuous parameters are generated uniformly between the corresponding search lower limit and search upper limit. The interpolation neighbor number is rounded to an integer and limited to between 2 and 8 after generation. In one embodiment, the candidate population contains 20 candidate individuals.
[0096] When evaluating the fitness of each candidate individual, the four maximum spatial corresponding distances are written into the node corresponding unit of the cross-version jump connection layer, and the four interpolation search radii, four interpolation neighbor number values, and four distance weight exponents are written into the restricted PointDeconv layer. The node corresponding unit establishes an effective connection only when the spatial distance between the encoding node and the target PointDeconv node does not exceed the maximum spatial corresponding distance of the current layer. If the spatial corresponding distance exceeds the maximum spatial corresponding distance, the same resolution encoding node is not connected. The restricted PointDeconv layer selects the corresponding number of neighbor points within the interpolation search radius of the current layer, adds 0.1 mm to the distance, takes the negative power corresponding to the distance weight exponent, and then divides each negative power value by the sum to form the interpolation weight.
[0097] After the parameters are written, the improved PointConv model reprocesses the difference interval of the same sequence, and performs surface point encoding, encoding node registration, connection state update, effective range interpolation, same resolution fusion and state output in sequence. Then, the newly generated surface observation state is projected onto the acquisition field of view and the scene restoration deviation is calculated. The scene restoration deviation is directly used as the fitness of the current candidate individual. The smaller the scene restoration deviation, the higher the degree of restoration consistency between the candidate parameters and the current observation data.
[0098] After the candidate population completes its initial evaluation, the candidate individual with the smallest scene restoration deviation is registered as the current best individual. In each round, prey identification and updating are performed first. Each candidate individual randomly selects another candidate individual as the search target and generates a random number between 0 and 1 and an integer factor with a value of 1 or 2. When the search target deviation is small, the current parameter vector is updated along the direction of the product of the current parameter vector and the search target parameter vector minus the integer factor. When the search target deviation is not small, the current parameter vector is updated along the direction away from the search target parameter vector. When the update result exceeds the upper and lower limits of the parameters, it is truncated to the corresponding boundary. The number of interpolated neighboring points is rounded and limited to between 2 and 8. The updated parameters are rewritten into the improved PointConv model. The update is accepted when the scene restoration deviation decreases, and the original parameters are retained when the deviation does not decrease.
[0099] After the prey identification update is completed, a local pursuit update is performed. The local search radius coefficient is obtained by subtracting the ratio of the current round to the maximum iteration round from 1 and then multiplying by 0.02. Each parameter component is superimposed with the product of the local search radius coefficient, a random number between -1 and 1, and the current parameter component. After the updated parameters are truncated to the upper and lower limits and the integer parameters are rounded, they are rewritten into the improved PointConv model. When the scene recovery deviation decreases, the local update is accepted. When the deviation does not decrease, the parameters updated after the prey identification are retained. After each round, the current best individual with the smallest scene recovery deviation is re-determined.
[0100] In one implementation, the maximum number of iterations for parameter search is 40. The search is stopped early when the reduction in the current optimal scene restoration deviation is less than 0.01mm for 5 consecutive iterations, and stops directly when 40 iterations are reached. If there are no effective projection points, the output coordinates are not finite values, or the number of effective connections is 0 during model reprocessing, the scene restoration deviation of the current candidate individual is recorded as 100mm and the corresponding parameter results are discarded. The failure of a single candidate individual does not terminate the search for other candidate individuals.
[0101] After the search is completed, the maximum spatial distance of the current best individual is written into the cross-version jump connection layer, the interpolation search radius, the number of interpolation neighbor points and the distance weight index are written into the restricted PointDeconv layer, and the sequence difference interval is reprocessed with the optimal parameters. The reprocessed surface point coordinates, unit normal vectors, observation confidence values and confidence states are written into the surface point state table according to the surface identity. After all surface point states are merged, the optimized surface observation state is formed and used as the input for the surface segment observation coverage judgment and the determination of the surface construction method.
[0102] In this embodiment, S6 specifically includes: The surface observation state is optimized by using the surface identity as the corresponding index between the surface segments of adjacent scene versions. The surface point index, corrected coordinates, corrected unit normal vector, observation confidence value and confidence state corresponding to each surface identity are read. Then, the surface segments with the same surface identity are searched in adjacent scene versions. When the same surface identity is found, the optimized surface observation state is written into the temporary matching set of the corresponding surface segment. If the newly added surface identity in the current scene version has no corresponding segment in the adjacent scene version, a new temporary matching set is established with the newly added surface identity. If the surface identity is empty or a duplicate conflict occurs, the construction mode is not entered and the original surface data is retained.
[0103] The current observation coverage is determined by establishing a spatial correspondence between surface points in the optimized surface observation status and exhibition hall observation data. First, starting with surface points with an observation confidence value of not less than 0.5, the nearest observation point in the exhibition hall coordinate system is searched. When the distance between points is not greater than 20mm and the angle between the unit normal vectors of the two points is not greater than 15°, a valid match is established. When the same observation point corresponds to multiple surface points, only the match with the smallest distance between points is retained, and other surface points continue to search for unused observation points. If no observation point meets the distance and normal requirements, the corresponding surface point is registered as an uncovered point.
[0104] The observation coverage ratio of each exhibit surface segment is obtained by dividing the number of valid matching surface points by the total number of valid surface points in the current exhibit surface segment. The total number of valid surface points does not include low-confidence state points and invalid coordinate points. In one embodiment, the current observation is considered to have formed a valid coverage when the observation coverage ratio is not less than 30% and there are not less than 20 valid matching surface points. The current observation is not considered to have formed a valid coverage when the total number of valid surface points is 0 or there are fewer than 20 valid matching surface points. The valid matching surface points are spatially connected and grouped according to the distance between the points not greater than 20 mm. The valid matching surface points contained in each connected group and the neighboring points within the same exhibit surface segment that are no more than 20 mm away from the corresponding matching point together form a current observation coverage area. The unconnected groups form independent coverage areas respectively.
[0105] The surface segments currently covered by observations are updated using exhibition hall observation data. The update targets are limited to surface points within the current observation coverage area. The temporary matching set records the corresponding observation point index and surface point index. The same surface segment that is not included in the current observation coverage area is not updated along with the covered portion. When a new surface identity meets the observation coverage requirements, it is also updated using exhibition hall observation data, and the range of valid matching points corresponding to the new surface identity is used as the scope of construction.
[0106] Surface identity continuation is confirmed by the surface identity correspondence results already established in adjacent scene versions. If the current surface segment does not meet the observation coverage requirements, but there are the same surface identities in adjacent scene versions and the corresponding exhibit instances are consistent, the surface identity is considered to be continuation, and the surface data of the adjacent scene version is determined to be used. The scope of use is all surface points included in the corresponding surface identity in the adjacent scene version. Low confidence state, short-term occlusion, or non-coverage of the current acquisition field of view do not change the surface identity continuation result.
[0107] If there is a lack of current observations and no identical surface identity exists in adjacent scene versions, or if the surface identity is the same but the corresponding exhibit instance conflicts, it is determined that the continuation of the surface identity cannot be confirmed, and the corresponding surface segment is determined to be temporarily not constructed. When not constructed, the current observation point is not written, historical surface data in adjacent scene versions is not deleted, and data of other surface identities are not used to replace the current surface segment. Surface identity conflict records are retained until the next valid observation is obtained and the surface identity correspondence is re-executed.
[0108] When the same surface segment meets multiple judgment results at the same time, first check whether the current observation coverage requirement is met. If the requirement is met, the exhibition hall observation data is used for updating. If the requirement is not met, then check whether the surface identity is continued. If the continuation can be confirmed, the surface data of the adjacent scene version is used. If neither the observation coverage requirement is met nor the continuation of the surface identity can be confirmed, it is determined that it will not be constructed for the time being. This ensures that each surface identity corresponds to only one surface segment construction method in the current processing.
[0109] Surface construction instructions are generated item by item using surface identity as the index. Each instruction records the surface identity, construction method, and scope. The construction method is recorded as update, reuse, or no construction for now. The scope of the update instruction is the current observation coverage area, the scope of the reuse instruction is the surface segment range corresponding to the adjacent scene version, and the scope of the no construction instruction is all point indices contained in the current exhibit surface segment. When a new surface identity has not yet formed valid surface points, the point index range of the corresponding surface identity within the sequence difference interval is taken. The construction instructions corresponding to all surface identities are merged to form the surface construction instruction, which serves as the direct input for surface segment updates, surface data retention, and maintenance of the unconstructed state.
[0110] In this embodiment, S7 specifically includes: Surface construction instructions serve as direct input for surface segment construction. Each instruction includes surface identity, construction method, and scope. First, the corresponding surface segment is searched for in the current exhibit instance and adjacent scene versions based on the surface identity. Then, the surface point to be processed is determined based on the point index within the scope. If the surface identity cannot find the corresponding surface segment and the construction method is not updated, the current instruction is registered as a location failure and the adjacent scene version data remains unchanged. When a surface identity corresponds to multiple surface segments, only surface segments with the same exhibit instance and point index falling within the scope are processed.
[0111] When updating, the construction method reads observation points that have been effectively matched with surface points within the scope of the action from the exhibition hall observation data and transforms the observation points to the exhibition hall coordinate system. For each observation point, the nearest surface point with a distance of no more than 20mm and a normal angle of no more than 15° is found in the corresponding surface segment. If the matching requirements are met, a fusion point pair is established. If the matching requirements are not met but the surface identity belongs to the newly added identity of the current scene version, the observation point is directly registered as a newly added surface point. If the matching requirements are not met and the surface identity already exists, the current observation point is not written to the corresponding surface segment.
[0112] The coordinates of the observation point and the original surface point in the fused point pair are weighted and accumulated to generate the updated coordinates. The weight of the observation point is taken as the observation confidence value in the optimized surface observation state and limited to between 0.5 and 1. The weight of the original surface point is 1 minus the weight of the observation point. The two sets of coordinates are multiplied by their corresponding weights and then added in each coordinate direction. The normal vector of the observation point and the normal vector of the original surface point are accumulated in each component with the same weight and then divided by the vector length of the accumulated result to form the updated unit normal vector. When the vector length is less than 0.000001, the normal vector of the observation point is retained. The updated coordinates, the updated unit normal vector, and the surface identity are combined to form the updated surface point.
[0113] When the same original surface point corresponds to multiple observation points, the Euclidean distance between each observation point and the original surface point is compared first. The observation point with the smallest distance is retained to participate in the fusion, while other observation points continue to search for surface points within their range that have not yet been fused. When the same observation point corresponds to multiple original surface points, it is only fused with the original surface point with the smallest distance. The original surface points that have not been fused continue to retain their original coordinates and original normal vectors to avoid a single observation point being repeatedly written to multiple surface positions.
[0114] If there are original surface points within the update's effective range but no corresponding valid observation points, the original surface points are not directly deleted. If the minimum distance between the original surface point and all valid observation points is greater than 30mm, and the corresponding spatial position falls within the effective field of view of the current acquisition device, the original surface point is marked as a conflicting old point and removed from the updated surface data. If the corresponding spatial position is outside the acquisition field of view or is obscured by the currently visible surface, the original surface point is retained to prevent the accidental deletion of the real surface due to missing current observations.
[0115] When constructing a digital twin scene version, the point coordinates, unit normal vectors, and segment boundaries of the corresponding surface segment are read from the adjacent scene version by the surface identity, and the read results are copied to the current digital twin scene version. The reuse process does not re-perform point cloud interpolation, surface reconstruction, or coordinate weighting. The same surface segment is multiplied by the confirmed rotation matrix and superimposed with the translation vector only when the overall transformation of the exhibition hall coordinate system occurs and the pose transformation of the exhibit has been confirmed. When the pose transformation has not been confirmed, the surface coordinates in the adjacent scene version are directly retained.
[0116] When the construction method is not to be constructed, no data on exhibition hall observation points, optimized surface observation status, or other surface identities will be written to the corresponding scope of action, and the surface identity and unconstructed state will be retained in the current digital twin scenario version. When there is historical surface data of the same surface identity in adjacent scenario versions, the historical surface data will continue to be saved in the adjacent scenario versions but will not be copied as the current valid surface. When there is no available surface data for the newly added identity in the current scenario version, only the unconstructed segment record will be established.
[0117] Before merging the updated surface data and the existing surface data, the coordinate system markers are checked uniformly. If the coordinate system markers are not the exhibition hall coordinate system, the coordinate transformation is completed using the camera extrinsic parameters at the corresponding acquisition time or the already confirmed exhibit pose transformation. Data lacking valid transformation parameters are not included in the current scene version and the original construction instructions are retained. After the coordinate transformation is completed, the exhibit surface segments are grouped by surface identity. The exhibit surface segments constructed using the update are written with updated data within the current observation coverage area. Outside the coverage area, the original surface data of the same exhibit surface segment that did not participate in the fusion are retained. The exhibit surface segments constructed using the existing data copy the corresponding surface data from adjacent scene versions.
[0118] When updated and retained data overlap at the boundary of the exhibit surface section, first calculate the angle between the unit normal vectors of the two boundary points. If the angle is greater than 30°, retain them within their respective surface ranges. If the angle is not greater than 30°, calculate the nearest distance from the two boundary points to the current valid observation point. If the difference between the two nearest distances is greater than 2mm, retain the data point with the smaller nearest distance. If the difference between the two nearest distances is not greater than 2mm, multiply the coordinates of the two points by 0.5 and add them together to form the boundary fusion point. Then, add the two unit normal vectors component by component and normalize them to form the boundary fusion point normal vector.
[0119] Surface segment merging is performed from smallest to largest according to surface identity. Each surface identity is written with updated surface data, retained surface data, or unbuilt state. Point coordinate mixing is not performed between different surface identities. After all surface identities are processed, the surface point data, unit normal vector, segment boundary, and unbuilt state are uniformly encapsulated into the current digital twin scene version. The current processing only forms scene version data, does not perform entity item association, and does not replace the scene change comparison benchmark.
[0120] In this embodiment, S8 specifically includes: The optimization process involves writing the surface observation status into the digital twin scene version using the surface identity as the positioning index. The surface identity, point index, corrected coordinates, corrected unit normal vector, observation confidence value, and confidence status are read item by item from the surface point status table. The same surface identity is then searched for in the digital twin scene version for exhibit surface segments. First, a direct correspondence is established between the status point and the constructed surface point using the point index. If a matching point index is missing, the nearest surface point within the same exhibit surface segment with a distance not exceeding 20mm is searched. When one status point corresponds to multiple surface points, the smallest distance correspondence is retained. When one surface point corresponds to multiple status points, the status point with the larger observation confidence value is retained. When the observation confidence values are the same, the status point with the smaller spatial distance is retained.
[0121] After the state points are mapped to the constructed surface points, the corrected coordinates, corrected unit normal vectors, observation confidence values, and confidence states are added to the corresponding surface points to form a surface observation state record. The surface point coordinates and unit normal vectors generated during the scene version construction phase are not recalculated or overwritten due to the state writing. The surface segments of exhibits constructed using the updated method are written to the corresponding surface observation state record. The surface segments of exhibits constructed using the reuse method are only added to the observation state record when there are valid corresponding state points. The surface segments of exhibits that are not yet constructed only retain the surface identity and the unconstructed state. If a corresponding state point cannot be established, it is not written to the digital twin scene version but is retained in the state anomaly record.
[0122] The association between surface segments and physical exhibits reads the exhibit instance attribution and surface identity correspondence results formed during the exhibit instance segmentation stage. Each surface identity stores the identifier of its respective exhibit instance. In the digital twin scenario version, the surface segment finds the corresponding exhibit instance identifier through the surface identity and writes the surface segment into the surface set of the corresponding physical exhibit. The same surface identity can only be associated with one physical exhibit. When the same surface identity has multiple exhibit instance identifiers, the spatial overlap rate between the current surface segment and the point cloud of each exhibit instance is compared. The exhibit instance with the largest overlap rate is determined as the associated object. If the maximum overlap rate is less than 60%, no physical exhibit association is established.
[0123] The newly added exhibit surface segments in the current scenario version are associated with the current entity exhibit by the exhibit instance identifier saved when the new surface identity is generated. Exhibit surface segments that existed in the previous scenario version but are not matched in the current observation retain the original exhibit instance identifier and the original entity exhibit association, and are registered as unobserved surfaces in the current digital twin scenario version. The surface identity and entity exhibit association of unobserved surfaces are not transferred to other entity exhibits.
[0124] After all surface segments have completed state writing and entity exhibit association, the corresponding valid surface segments are merged with the entity exhibit as the organization object. The surface segments retain the point coordinates, unit normal vector, observation confidence value, confidence state and surface identity in the exhibition hall coordinate system. Unbuilt surface segments only save the unbuilt state and do not generate visible surfaces. Each entity exhibit and its corresponding surface segment together form a digital exhibition hall scene. The surface identity in the digital exhibition hall scene continues to serve as a unified association index between entity exhibits, surface segments and surface point states.
[0125] The scene change comparison benchmark consists of valid surface data, surface identity, exhibit instance identifier, entity exhibit association, unobserved state, and unbuilt state in the current digital twin scene version, and records the collection time corresponding to the current scene version. When the next set of exhibition hall observation data arrives, adjacent scene versions are first established based on the entity exhibits and surface identities in the comparison benchmark, and then the new exhibition hall point cloud is compared with the surface spatial range in the comparison benchmark. The scene version formed by the current processing thus becomes the previous scene version for the next spatial change judgment, surface identity matching, and sequence difference interval determination.
[0126] If a digital twin scene version contains coordinate values that are not finite, multiple entity items associated with the same surface identity, or missing valid surface points or having zero valid surface segments, the current scene version will be registered as a failed construction version. The failed construction version will not replace the original scene change comparison benchmark, nor will it be used as the previous scene version in the next adjacent scene version. The original comparison benchmark will remain valid, and the currently collected data and the reason for failure will be retained until the scene construction is performed again after obtaining complete observation data, thereby preventing the continued propagation of incorrect surface identities and abnormal geometric states.
[0127] Example 1: To verify the feasibility of this invention in practice, it was applied to the continuous construction of a digital twin scene in a comprehensive digital exhibition hall. This exhibition hall features movable display cases, 3D exhibits, and interactive display devices. During operation, there are real changes such as adjustments to exhibit positions, alterations in posture, and partial surface exposure. Simultaneously, visitors frequently cause short-term occlusion of the exhibits. Existing scene construction methods primarily update the digital model based on the overall differences between adjacent point clouds. This easily leads to the identification of visitor occlusion as exhibit changes, requiring the reprocessing of large-scale point clouds after exhibit movement. This results in temporary human silhouettes, historical surface remnants, and localized surface misalignments in the digital scene.
[0128] Four depth cameras were deployed around the exhibition hall, with depth image resolution of 1280×720 pixels and a acquisition frequency of 10 frames per second. After calibrating the camera intrinsic and extrinsic parameters, the exhibition hall observation data acquired by each camera was transformed to a unified exhibition hall coordinate system, and point cloud registration and fusion were performed on overlapping areas. Point sets that appeared only at a single acquisition moment in consecutive point cloud frames and whose corresponding positions recovered to the same surface in adjacent moments were identified as transient occlusion data and removed. Subsequently, instance segmentation and region growing based on normal continuity were performed on the exhibit point cloud to divide the exhibit surface segments and maintain surface identity between adjacent scene versions. After detecting changes in the exhibit spatial range, the changed range was projected onto each acquisition field of view. An ordered surface intersection sequence was generated through candidate acquisition rays. The first difference to the re-consistent sequence difference interval was extracted, and the cross-version jump connection was reconstructed using an improved PointConv model while limiting PointDeconv interpolation propagation. The Northern Eagle optimization algorithm adjusted the connection and interpolation control parameters according to the scene recovery deviation. Finally, the changed surfaces were updated according to the surface construction instructions, the unchanged surfaces were used, and the construction of unconfirmed surface segments was temporarily suspended.
[0129] To verify the application effect of this invention, 150 scene update processes were selected as verification samples, including 60 changes in exhibit position, pose, or surface exposure, and 90 instances of instantaneous occlusion caused by visitors passing by. The reference scene was determined by manual verification combined with high-precision point cloud registration results. The method of this invention was compared with traditional full-scene point cloud registration and reconstruction methods and the PointConv model method using fixed skip connections. The instantaneous occlusion error update rate, change area localization results, surface reconstruction error, surface identity continuity, single scene update time, and actual processed point cloud ratio were statistically analyzed. The statistical results are shown in Table 1 below.
[0130] Table 1 Comparison of the effects of continuous construction and updating of digital exhibition hall scenarios
[0131] As shown in Table 1, the method of this invention outperforms the traditional full-scene point cloud registration and reconstruction method and the fixed-skip connection PointConv model method in terms of instantaneous occlusion error update rate, change interval localization intersection-union ratio, surface reconstruction root mean square error, surface identity continuity rate, single scene update time, and processing point cloud ratio. The traditional full-scene point cloud registration and reconstruction method has an instantaneous occlusion error update rate of 16.7%, a surface reconstruction root mean square error of 13.4 mm, and a single scene update time of 21.8 s. Furthermore, it requires processing the entire point cloud, indicating that temporary occlusion is easily identified as real changes, resulting in large reconstruction errors and high computational costs. While the fixed-skip connection PointConv model method reduces the error update rate to 10.0%, its change interval localization intersection-union ratio is only 0.79, and its surface identity continuity rate is 91.7%, indicating problems with historical feature interference and inaccurate update boundaries. The method of the present invention reduces the false update rate to 3.3%, increases the intersection-union ratio of the change interval to 0.90, reduces the root mean square error of surface reconstruction to 6.1 mm, achieves a surface identity continuity rate of 97.8%, shortens the single update time to 6.2 s, and processes only 29.4% of the point cloud. This indicates that the present invention improves the accuracy of change recognition and local reconstruction, and reduces the repeated calculation of unchanged areas.
[0132] This embodiment unifies the registration of exhibition hall observation data and compares continuous point cloud frames to identify and remove transient occlusion data, avoiding the inclusion of short-term point sets formed by visitor passage into the digital twin scene. It divides exhibit surface segments based on normal continuity and maintains surface identity between adjacent scene versions, ensuring that the correspondence remains even after exhibit movement, pose changes, or partial surface exposure. Upon detecting spatial range changes, an ordered surface intersection sequence is generated using candidate acquisition rays. The first difference position to the re-consistent position is truncated as the sequence difference interval, limiting the processing scope to the actual change area. The improved PointConv model reconstructs cross-version jump connections according to the difference interval, truncates the propagation of mismatched historical encoding nodes to PointDeconv nodes, and restores effective surface point features. The Northern Eagle optimization algorithm adjusts connection and interpolation control parameters based on scene recovery deviations, and then determines whether to update, reuse, or temporarily not construct surface segments based on the optimized surface observation status. The resulting digital twin scene version accurately reflects changes in the physical exhibition hall and retains continuous data of unchanged surfaces, making it suitable for continuous scene construction under conditions of frequent exhibit adjustments.
[0133] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for constructing a digital exhibition hall scene based on digital twins, characterized in that, Includes the following steps: S1. Collect observation data of the exhibition hall and register it to the exhibition hall coordinate system, remove instantaneous occlusion data, and arrange it into a point cloud sequence of the exhibition hall according to the collection time; S2. Divide the surface segments of exhibits according to the point cloud sequence of the exhibition hall, associate the same surface segments at adjacent times, and form adjacent scene versions corresponding to the surface identity; S3. Project the exhibits whose spatial range changes in adjacent scene versions to the acquisition field of view, generate an ordered surface intersection sequence, align them according to surface identity and extract the first difference to the sequence difference interval of the re-consistent sequence. S4. Improve the PointConv model to encode surface point clouds within the difference interval of the sequence, reconstruct cross-version jump connections according to the difference interval and restrict PointDeconv interpolation propagation to form the initial surface observation state. S5. Project the surface points corresponding to the initial surface observation state onto the acquisition field of view and compare them with the observation data. Calculate the scene recovery deviation. Use the Northern Eagle optimization algorithm to search for structural control parameters and feed back the improved PointConv model to reprocess the difference interval to form an optimized surface observation state. S6. Determine the surface segment construction method based on the optimized surface observation status, and generate surface construction instructions; S7. Update the surface segments according to the surface construction instructions to build a digital twin scene version; S8. Optimize the surface observation status and write it into the digital twin scene version. Associate the entity exhibits with the surface identity to form a digital exhibition hall scene and update the scene change comparison benchmark.
2. The method for constructing a digital exhibition hall scene based on digital twins according to claim 1, characterized in that, S1 specifically includes: S11. Acquire exhibition hall observation data according to the acquisition time, record the correspondence between the position and posture of the acquisition device and the acquisition time, and form exhibition hall observation data with position and posture information; S12. Based on the corresponding pose, transform the exhibition hall observation data to the exhibition hall coordinate system, and perform point cloud registration and fusion on the overlapping observation areas at the same acquisition time to form exhibition hall point cloud frames; S13. Compare continuous exhibition hall point cloud frames, identify point sets that appear only at a single acquisition moment and whose corresponding spatial locations recover to the same surface at adjacent moments as instantaneous occlusion data and remove them. S14. Arrange the exhibition hall point cloud frames after removing data with momentary occlusion according to the collection time to form the exhibition hall point cloud sequence.
3. The method for constructing a digital exhibition hall scene based on digital twins according to claim 1, characterized in that, S2 specifically includes: S21. Perform exhibit instance segmentation on the exhibition hall point cloud sequence frame by frame, divide the point cloud of each exhibit according to the continuity of normal, and form the surface segment of the exhibit. S22. Estimate the pose transformation of the same exhibit at adjacent time points, transform the surface segment of the exhibit at the previous time point to the current time point, and match the same surface segment according to spatial overlap. S23. Assign new surface identities to surface segments that are not matched at the current time, retain the original surface identities for surface segments that were not matched at the previous time, and organize the surface segments of the exhibits according to adjacent acquisition times to form adjacent scene versions.
4. The method for constructing a digital exhibition hall scene based on digital twins according to claim 1, characterized in that, S3 specifically includes: S31. Project the exhibits whose spatial range has changed into the acquisition field of view in adjacent scene versions, and merge the projection ranges to form candidate acquisition areas; S32. Generate candidate acquisition rays within the candidate acquisition area, calculate the intersection points of the candidate acquisition rays with the surface segments of the exhibits in the adjacent scene versions, and arrange them along the rays from near to far to form an ordered surface intersection sequence. S33. Align the ordered surfaces of adjacent scene versions according to their surface identities to find the intersection sequence, truncate the first difference position to the re-consistent position, and truncate to the end of the sequence if no re-consistency is found, forming the sequence difference interval.
5. The method for constructing a digital exhibition hall scene based on digital twins according to claim 1, characterized in that, The improved PointConv model includes a PointConv encoding layer, a cross-version jump connection layer, a restricted PointDeconv layer, and a state output layer connected in sequence. The PointConv coding layer performs hierarchical sampling, neighborhood grouping, and PointConv convolution on the surface point clouds within the sequence difference intervals of adjacent scene versions, forming coding nodes with different resolutions; The cross-version skip connection layer aligns the encoding nodes of adjacent scene versions according to surface identity and intersection order, retains the skip connection between the encoding nodes outside the sequence difference interval and the PointDeconv node of the same resolution, and cuts off the feature propagation of the encoding nodes of the previous scene version within the sequence difference interval. In the current scene version, newly added surface segments are encoded by PointConv encoding layer, which generates encoding nodes and establishes skip connections. Encoding nodes from the previous scene version that do not match the current scene version stop propagating to PointDeconv nodes. The restricted PointDeconv layer selects neighboring points within the surface range corresponding to the preserved or newly created skip connections to perform interpolation, concatenates the interpolated features with the encoded features of the same resolution and performs PointConv convolution to recover the surface point features step by step. The state output layer merges and recovers the surface point features according to the surface identity to form the initial surface observation state.
6. The method for constructing a digital exhibition hall scene based on digital twins according to claim 5, characterized in that, The reconstruction of cross-version jump connections by the cross-version jump connection layer specifically includes: According to the resolution level of the PointConv coding layer, the sampling points corresponding to each coding node are assigned to the presentation surface segment, and the intersection position of the presentation surface segment in the ordered surface intersection sequence is recorded. For the exhibit surface segment before the first difference position, maintain the skip connection from the previous scene version encoding node to the same resolution PointDeconv node. Remove the input of the previous scene version encoding node from the first difference position and connect the encoding node of the exhibit surface segment to the same resolution PointDeconv node to the corresponding intersection position in the current scene version. Reconstruct the skip connections corresponding to each intersection position along the sequence difference interval. When the current scene version lacks the same resolution encoding node, retain the features obtained by the PointDeconv layer interpolation of the previous resolution level and do not perform the same resolution encoding feature concatenation. When the surface identity and intersection order are re-consistent, the jump connection from the encoding node corresponding to the previous scene version to the PointDeconv node of the same resolution is restored. If no re-consistent position is found, the connection of the encoding node of the current scene version is maintained to the end of the ordered surface intersection sequence, forming a reconstructed cross-version jump connection.
7. The method for constructing a digital exhibition hall scene based on digital twins according to claim 1, characterized in that, S5 specifically includes: S51. Project the surface points corresponding to the initial surface observation state onto the acquisition field of view, calculate the positional deviation between the projected points and the corresponding observation points, and summarize them to form the scene restoration deviation. S52, the Northern Eagle optimization algorithm evaluates the connection and interpolation control parameters based on the scene restoration deviation, iteratively updates the connection and interpolation control parameters, compares the scene restoration deviation before and after the update, and retains the parameter results corresponding to the reduction of deviation; S53. Write the optimized connection and interpolation control parameters into the improved PointConv model, reprocess the sequence difference interval, and form an optimized surface observation state.
8. The method for constructing a digital exhibition hall scene based on digital twins according to claim 1, characterized in that, S6 specifically includes: S61. Based on the surface identity, the optimized surface observation status is mapped to the surface segment of the adjacent scene version, and the surface points are matched with the exhibition hall observation data to determine the current observation coverage area; S62. The current observation coverage area is determined by updating the exhibition hall observation data, and the uncovered surface area with continued surface identity is determined by using the adjacent scene version surface data. S63. Surface segments for which observations are lacking and whose surface identity continuity cannot be confirmed will not be constructed for the time being, and a surface construction instruction will be formed by summarizing these.
9. A method for constructing a digital exhibition hall scene based on digital twins according to claim 1, characterized in that, Specifically, S7 includes: S71. Locate the surface segments and corresponding construction methods according to the surface construction instructions; S72. Integrate the exhibition hall observation data into the surface segments constructed using the updated method to form updated surface data; S73. Retain the surface data of adjacent scene versions that have been built and continue to be used, and maintain the unbuilt state of the surface segments that are not yet built; S74. Combine the updated surface data and the retained surface data according to the surface identity to form a digital twin scene version.
10. A method for constructing a digital exhibition hall scene based on digital twins according to claim 1, characterized in that, S8 specifically includes: S81. Write the optimized surface observation status into the corresponding surface segment in the digital twin scenario version according to the surface identity; S82. Based on the surface identity correspondence, associate the surface sections in the digital twin scene version with the physical exhibits to form a digital exhibition hall scene; S83. The digital twin scenario version is determined as the benchmark for comparing scenario changes.
