A building digital twin modeling method based on intelligent construction

Abstract

The present invention relates to the field of image data processing technology, and specifically to a building digital twin modeling method based on intelligent construction, comprising acquiring video streams and building three-dimensional point cloud data and performing data fusion to obtain a multi-source data set; dividing the data according to point cloud density to obtain multiple interpolation areas; calculating the point cloud texture coefficient of each interpolation area according to the multi-source data set corresponding to each interpolation area; confirming multiple interpolation points in each interpolation area; calculating the interpolation weight of each neighboring point according to the four neighboring data sets of each interpolation point; and calculating the point cloud coordinates of the interpolation point according to the point cloud texture coefficient of the interpolation area to which the interpolation point belongs, the point cloud coordinates of each neighboring point, and the interpolation weight. The present invention integrates the data of the video image sensor and the lidar sensor, utilizes the complementarity between them to fit and supplement, and performs interpolation processing on the point cloud data, thereby improving the accuracy of the edge details of the digital twin model.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a method for modeling a building digital twin based on intelligent construction. Background Art

[0002] In today's rapidly developing information age, the construction industry is undergoing a profound transformation. With the widespread adoption of cutting-edge technologies such as big data, cloud computing, the Internet of Things, and artificial intelligence, intelligent construction is becoming a key driver of the industry's transformation and upgrading. Building digital twin technology, a core element of intelligent construction, offers unprecedented opportunities for building lifecycle management through its unique real-time simulation, decision-making optimization, and efficient management capabilities. This significantly improves building quality, reduces operating costs, and plays a crucial role in achieving sustainable development.

[0003] However, given the high complexity of building structures and systems, existing digital models often struggle to achieve the desired accuracy. This leads to discrepancies between simulation results and actual building performance, thus undermining the effectiveness of decision support. Furthermore, as intelligent construction continues to evolve, digital twin technology lacks the ability to update and respond to building status in real time, failing to accurately reflect immediate changes in physical buildings. This limits the model's effectiveness in responding to emergencies or performing real-time optimization. Therefore, there is an urgent need to develop a building digital twin modeling method that can accurately reflect actual building status in real time. Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a building digital twin modeling method based on intelligent construction to solve the above problems. In order to achieve the above purpose, the technical solutions adopted by the present invention are as follows:

[0005] The present application provides a building digital twin modeling method based on intelligent construction, the method comprising: acquiring a video stream and building three-dimensional point cloud data of a target building in real time; fusing the video stream of the target building and the building three-dimensional point cloud data to obtain a multi-source data set, the multi-source data set comprising a feature video frame, a portion of the pixel points of the feature video frame comprising point cloud coordinates; dividing the multi-source data set corresponding to the target building according to point cloud density to obtain a plurality of multi-source data sets corresponding to interpolation areas; calculating the point cloud texture coefficient of each interpolation area according to the multi-source data set corresponding to each interpolation area; confirming a plurality of interpolation points in each interpolation area; calculating the interpolation weight of each neighboring point according to four neighboring data sets of each interpolation point, each neighboring data set comprising the point cloud coordinates and grayscale value corresponding to a neighboring point; performing interpolation calculation according to the point cloud texture coefficient of the interpolation area to which the interpolation point belongs, the point cloud coordinates of each neighboring point and the interpolation weight to obtain the point cloud coordinates of each interpolation point; constructing a building digital twin model according to the point cloud coordinates of all the interpolation points and the building three-dimensional point cloud data.

[0006] In one possible implementation, data fusion is performed on the video stream of the target building and the three-dimensional point cloud data of the building to obtain a multi-source data set, including: performing image screening on the video stream to obtain a target video frame, wherein the target video frame has the smallest image data change amplitude within the video stream of a preset time length; performing entity object segmentation on the target video frame to obtain at least one entity object and two-dimensional image data corresponding to each entity object; extracting all point cloud coordinates corresponding to each entity object in the building point cloud data according to the time point corresponding to the target video frame; and performing two-dimensional image data mapping on each point cloud coordinate based on the two-dimensional image data corresponding to each entity object to obtain a multi-source data set.

[0007] In one possible implementation, the video stream is image-filtered to obtain a target video frame, including: intercepting the target video stream in the video stream according to a preset number of frames, the cutoff time of the target video stream being the current time; converting the grayscale of each frame of the target video stream to obtain a target grayscale video stream; calculating the grayscale change value of each grayscale image frame in the target grayscale video stream frame by frame, the grayscale change value being the sum of the absolute values of the grayscale changes of all pixels in the grayscale image in consecutive frames; and filtering the target video frame according to the grayscale change value corresponding to each frame of the grayscale image, the grayscale change value corresponding to the target video frame being the minimum value.

[0008] In a possible implementation, the point cloud texture coefficient of each interpolation area is calculated based on the multi-source data set corresponding to each interpolation area, including: calculating the point cloud density based on the multi-source data set corresponding to the interpolation area; calculating the grayscale image texture information complexity based on the grayscale values corresponding to all pixels in the interpolation area; and calculating the point cloud texture coefficient based on the grayscale image texture information complexity and the point cloud density.

[0009] In one possible implementation, the point cloud density is calculated based on the multi-source data set corresponding to the interpolation area, including: counting the number of point cloud coordinates and the number of pixel points in the interpolation area; and calculating the point cloud data density based on the number of point cloud coordinates and the number of pixel points.

[0010] In one possible implementation, the grayscale image texture information complexity is calculated based on the grayscale values corresponding to all pixels in the interpolation area, including: calculating the grayscale variance based on the grayscale values corresponding to all the pixels in the interpolation area; calculating the grayscale co-occurrence matrix based on the grayscale values corresponding to all the pixels in the interpolation area; calculating the entropy of the grayscale co-occurrence matrix to obtain an entropy value; and calculating the grayscale image texture information complexity based on the entropy value and the grayscale co-occurrence matrix.

[0011] In one possible implementation, the interpolation weight of each neighboring point is calculated based on the four neighboring data sets of each interpolation point, including: calculating the grayscale feature deviation corresponding to each neighboring point based on the image data corresponding to each neighboring point, and the grayscale feature deviation is used to characterize the degree of change of the grayscale value of the neighboring point; calculating the curvature of each neighboring point based on all point cloud coordinates in the interpolation area to which the interpolation point belongs; calculating the curvature deviation degree of each neighboring point based on the curvature of each neighboring point; and calculating the interpolation weight of each neighboring point based on the curvature deviation degree and the grayscale value change degree corresponding to each neighboring point.

[0012] In one possible implementation, the grayscale feature deviation corresponding to each of the neighboring points is calculated based on the image data corresponding to each of the neighboring points, including: calculating the grayscale mean based on the grayscale values corresponding to all of the neighboring points; and calculating the grayscale feature deviation corresponding to each pixel point based on the grayscale value corresponding to each pixel point and the grayscale mean.

[0013] In one possible implementation, the curvature deviation degree of each of the neighboring points is calculated based on the curvature of each of the neighboring points, including: performing mean calculation based on the curvature of each of the neighboring points to obtain the curvature mean; and calculating the curvature deviation degree corresponding to each of the neighboring points based on the ratio of the difference between the curvature of each of the neighboring points and the curvature mean.

[0014] In a possible implementation, interpolation calculation is performed based on the point cloud texture coefficient of the interpolation area to which the interpolation point belongs, the point cloud coordinates of each of the neighboring points, and the interpolation weight to obtain the point cloud coordinates of each interpolation point, including: calculating the first target coordinates based on the point cloud texture coefficient and the point cloud coordinates of all the neighboring points; calculating the second target coordinates based on the point cloud coordinates of each of the neighboring points and the interpolation weight; and performing weighted calculation based on the first target coordinates and the second target coordinates to obtain the point cloud coordinates of the interpolation point.

[0015] The present invention has the following beneficial effects:

[0016] The present invention first combines the video stream collected by the video image sensor with the three-dimensional point cloud data of the building scanned by the lidar sensor. Through separate analysis, the point cloud texture coefficient of each interpolation area of the physical object in the target building is confirmed, and the interpolation weight of each neighboring point is calculated by combining the point cloud coordinates and grayscale values of the four neighboring points around the interpolation point. Finally, the interpolation weight and point cloud texture coefficient are combined to assign a value to each interpolation point to enrich the edge details of the building digital twin model.

[0017] The present invention first combines the video stream data collected by the video image sensor with the three-dimensional point cloud data of the building obtained by scanning with the lidar sensor. The point cloud texture coefficients of each interpolation area of each physical object within the target building are independently analyzed. Next, the interpolation weight of each neighboring point is calculated by combining the point cloud coordinates and grayscale values of the four neighboring points around the interpolation point. Finally, by combining these interpolation weights and point cloud texture coefficients, each interpolation point is accurately assigned a value, effectively enriching the edge details of the building digital twin model, making it more realistic and detailed. This not only improves the accuracy of the digital twin model, but also enhances its practicality and reliability in various application scenarios.

[0018] Combining video stream data with 3D LiDAR point cloud data, the point cloud texture coefficients of physical objects within the target building are analyzed. Interpolation weights for neighboring points are calculated and, combined with the texture coefficients, the interpolated points are accurately assigned values, enhancing the edge details of the building digital twin model for greater realism and detail. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1A schematic diagram of a flow chart of a building digital twin modeling method based on intelligent construction provided in Example 1 of the present invention;

[0021] Figure 2 This is a flow chart of step S2 provided in Example 1 of the present invention;

[0022] Figure 3 This is a flow chart of step S4 provided in Example 1 of the present invention;

[0023] Figure 4 This is a flow chart of step S6 provided in Example 1 of the present invention;

[0024] Figure 5 This is a flow chart of step S7 provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0025] To further illustrate the technical means and effectiveness of the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a building digital twin modeling method based on intelligent construction proposed by the present invention. In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0026] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0027] Example 1:

[0028] The following describes in detail a specific solution of a building digital twin modeling method based on intelligent construction provided by the present invention in conjunction with the accompanying drawings.

[0029] See also Figure 1 , which shows a flow chart of a building digital twin modeling method based on intelligent construction provided by an embodiment of the present invention, see steps S1-S8.

[0030] S1. Obtain the video stream and 3D point cloud data of the target building in real time.

[0031] In this embodiment, high-definition video sensors and lidar sensors are used to comprehensively extract the features of buildings under construction. While video sensors (such as the Hikvision DS-2CD7A26G0 / P) can capture details of the built environment but cannot directly generate 3D models, lidar sensors (such as the Velodyne Puck (VLP-16)) can provide highly accurate 3D data. However, in the later stages of large-scale construction, as the building's complexity increases, the point cloud density may decrease, leading to a loss of detail. Furthermore, highly reflective materials can cause lidar detection failures, resulting in missing data. To address these issues, this embodiment combines video sensor and lidar data to achieve high-precision building recognition.

[0032] It should also be noted that: before installing the sensors, the building area needs to be surveyed to determine the scale and layout of the current buildings, and then determine the coverage and installation height of the video image sensor and the lidar sensor to ensure that the data coverage is appropriate. The specific coverage range can be determined by those skilled in the art according to actual conditions, and this embodiment does not impose specific restrictions. At the same time, when installing the sensors at the predetermined location, the lidar sensor and the video image sensor need to be installed at the same height to achieve the purpose of consistent collection areas. At the same time, it should also be noted that the target buildings mentioned in this embodiment can be artificial structures such as shopping malls, high-rise buildings, residential areas, office buildings, and industrial facilities.

[0033] In this embodiment, by combining data from video image sensors and lidar sensors, the details and 3D form of buildings under construction can be captured in all directions. The video image sensors provide detailed visual information, while the lidar sensors provide high-precision 3D data. This multi-source data fusion lays a solid foundation for subsequent data processing and the construction of building digital twins, improving the accuracy and reliability of building recognition.

[0034] S2. Perform data fusion on the video stream of the target building and the three-dimensional point cloud data of the building to obtain a multi-source dataset, wherein the multi-source dataset includes a feature video frame, and a portion of pixel points of the feature video frame include point cloud coordinates.

[0035] In this step, a multi-source data joint analysis method is used to improve the accuracy of the building digital twin. Data fusion refers to the process of mapping the building's 3D point cloud data into a 2D image based on positional relationships.

[0036] The specific steps are as follows: First, using a calibration plate or other calibration tool, the video image sensor captures an image of the plate and detects corner points. This allows the video image sensor's internal parameters, including focal length and distortion coefficients, to be calculated. Subsequently, by synchronously acquiring images of the calibration plate from the lidar sensor and the video image sensor, the position of the plate within both images is used to calculate the lidar's external parameters, namely the rotation matrix and translation vector, using optimization algorithms such as the least squares method. Finally, a transformation matrix is constructed to convert the pixel data corresponding to each pixel in the two-dimensional image into the lidar's three-dimensional building point cloud data. It should be noted that the rotation matrix represents the rotation relationship between the lidar coordinate system and the camera coordinate system. It is typically a 3x3 matrix, describing the angular change between the coordinate systems through three-dimensional rotation. The translation vector represents the translation relationship between the lidar coordinate system and the camera coordinate system. It is typically a 3x1 vector, describing the displacement between the two coordinate systems. Together, the rotation matrix and translation vector form the transformation matrix, which describes the transformation relationship from the video image sensor coordinate system to the lidar sensor coordinate system.

[0037] This phase of the process achieved data fusion between the LiDAR sensor and the video image sensor, generating a multi-source dataset containing both video image and point cloud data. This dataset more accurately reflects the details and geometric features of the target building, thereby improving the accuracy and real-time update capabilities of the building's digital twin model, ensuring that the digital model reflects the building's dynamic changes in real time. This provides a reliable foundation for subsequent data processing and model building, making the digital twin model of the target object more accurate and comprehensive.

[0038] At the same time, in this embodiment, considering that data mapping and subsequent building digital twin modeling are performed for each frame of video, it will consume a lot of computing resources. At the same time, in actual applications, it is not necessary to update the building digital twin modeling so frequently. Therefore, this embodiment also includes a method and steps for reducing computing resources, which can be found in detail in Figure 2 , Figure 2 It is shown that step S2 includes steps S21 to S24.

[0039] S21 . Screen the video stream to obtain a characteristic video frame, wherein the characteristic video frame has the smallest image data variation within the video stream of a preset duration.

[0040] Specifically, in this embodiment, step S21 also includes steps S211 to S214.

[0041] S211. Intercept a target video stream from the video stream according to a preset number of frames, where the end time of the target video stream is the current time.

[0042] Specifically, a target video segment with a preset duration of 30 minutes is selected in the video stream, and the target video segment is intercepted in a manner of tracing back the time range from the current moment, and the preset number of frames is the total number of frames within 30 minutes.

[0043] S212: Convert each frame of the target video stream into grayscale to obtain a target grayscale video stream.

[0044] In this step, it's important to understand that grayscale conversion involves linearly combining the red, green, and blue (RGB) channels of a color image according to specific weights (e.g., 0.2989, 0.5870, and 0.1140) to create a single-channel grayscale image. This grayscale image retains the original image's brightness information while simplifying computational complexity, facilitating subsequent image analysis and processing.

[0045] S213 , calculating the grayscale change value of each grayscale image frame in the target grayscale video stream frame by frame, where the grayscale change value is the sum of the absolute values of grayscale changes of all pixels in the grayscale image in consecutive frames.

[0046] Specifically, this step is based on the absolute value of the grayscale value of each pixel in each frame of the grayscale image between two consecutive frames, and accumulates these absolute values to obtain the grayscale change value of the entire frame image.

[0047] S214 , obtaining the target video frame by screening according to the grayscale change value corresponding to each grayscale image frame, wherein the grayscale change value corresponding to the target video frame is the minimum value.

[0048] The above steps, i.e., calculating the grayscale change values in the image frame by frame, can effectively reduce the interference caused by the movement of people within the building area, which could negatively impact the quality of information collection from a single physical object. Therefore, through the above steps, this embodiment can quantitatively analyze each frame in the video stream to determine the degree of its dynamic change. This process helps to select relatively stable video frames as target video frames. This screening process provides solid basic data support for subsequent analysis of the target video frames, ensuring the accuracy and reliability of the analysis.

[0049] S22: Segment the target video frame into entity objects to obtain at least one entity object and two-dimensional image data corresponding to each entity object.

[0050] In this step, we process the selected video frames using image segmentation to identify and extract physical objects within the building. Specifically, we utilize a convolutional neural network (Mask R-CNN model) for image segmentation. The Mask R-CNN model classifies each pixel in the video frame, effectively separating physical objects from their background and generating two-dimensional image data for each physical object. This 2D image data captures visual information such as the object's shape, size, and position, providing a precise basis for subsequent analysis and processing.

[0051] S23. Extract all point cloud coordinates corresponding to each of the physical objects from the building point cloud data according to the time point corresponding to the target video frame.

[0052] In this step, the point cloud coordinates of each physical object are extracted from the corresponding building point cloud data based on the time point of the feature video frame. The video frame and point cloud data are then linked using timestamps to ensure that the extracted point cloud coordinates correspond to the location of the physical object.

[0053] S24. Based on the two-dimensional image data corresponding to each of the physical objects, perform two-dimensional image data mapping on each point cloud coordinate to obtain a multi-source data set.

[0054] In this step, a multi-source dataset is generated by mapping the 2D image data of each physical object to its corresponding point cloud coordinates. This is done by fusing the 2D image data with the point cloud coordinates using a transformation matrix, ensuring that each point cloud coordinate contains the corresponding 2D image information.

[0055] S3. Divide the multi-source dataset corresponding to the target building according to point cloud density to obtain multi-source datasets corresponding to multiple interpolation regions, wherein the point cloud density in all the interpolation regions of the same entity object in the target building is the same.

[0056] In this embodiment, in order to ensure the consistency and accuracy of subsequent data processing, a method of dividing the multi-source data set of the target building based on point cloud density is adopted. Specifically, a spatial grid method can be used, that is, the point cloud data is divided into spatial grids, and each grid represents an interpolation area. First, the grid size is defined, and then the point cloud data is distributed according to the grid. This method can be adjusted according to the size and complexity of the building to ensure that the point cloud density within each grid is uniform. In addition, the density-based Mean Shift algorithm can also be used for division. The Mean Shift algorithm is a density-based clustering method that iteratively moves each point in the direction of maximizing density, and finally divides the point cloud set into multiple interpolation areas according to the density. The point cloud density of each interpolation area is the same.

[0057] At the same time, it should be pointed out that when dividing the interpolation region, the various physical objects within the target building must also be considered. These physical objects include but are not limited to columns, stairs, walls, ceilings, floors, doors and windows, beams, supporting structures, and decorative elements. These physical objects in the building have different geometric characteristics and materials, requiring accurate calibration and division within a multi-source dataset. For example, columns and beams may have a high structural point cloud density, while stairs and decorative elements may have complex geometric forms and high texture features. Therefore, in this embodiment, image segmentation is first performed based on the feature video frames to identify and extract the physical objects in the building. Then, each physical object is divided into multiple interpolation regions.

[0058] S4. Calculating a point cloud texture coefficient of each interpolation region according to the multi-source data set corresponding to each interpolation region.

[0059] In this embodiment, we consider that different physical objects have different distances from the acquisition location. Therefore, the point cloud density corresponding to each physical object varies. At the same time, for physical objects with richer surface texture features and unique material characteristics, if the number of surface acquisition points is small, it is difficult to fully express them in the building digital twin function construction level. Therefore, in this embodiment, it is necessary to obtain the point cloud texture coefficient of the physical object based on the surface point cloud data corresponding to a single entity and the grayscale features presented in the image.

[0060] In this embodiment, the point cloud density corresponding to different physical objects varies due to their varying distances from the acquisition location. Furthermore, for physical objects with rich surface texture features and unique material characteristics, insufficient surface acquisition points can hinder their full representation within the performance building of the building digital twin. Therefore, in this embodiment, the point cloud texture coefficient for a single physical object is derived based on the surface point cloud data corresponding to that object and the grayscale features presented in the image.

[0061] See Figure 3 , the figure shows the An interpolation area is used as an example to illustrate how steps S41 to S43 calculate the point cloud texture coefficients of an interpolation area.

[0062] S41. Calculate the point cloud density based on the multi-source data set corresponding to the interpolation area.

[0063] In this step, we first count the The number of point cloud coordinates in the interpolation area and the number of pixels ; Then, the density of the point cloud data is calculated based on the number of the point cloud coordinates and the number of the pixel points.

[0064] In this embodiment, after data fusion, the point cloud coordinates in the 3D building point cloud data are mapped to the 2D feature video frames based on their positional relationships. Therefore, the point cloud density within an interpolation region can be expressed not by conventional area but by the number of point cloud coordinates scattered around a pixel to represent the density of the point cloud data.

[0065] Specifically, the calculation formula for the density of point cloud data in this step is as follows:

[0066]

[0067] in, Indicates the The density of the point cloud data in the interpolation area; Indicates the The number of point cloud coordinates within the interpolation area; Indicates the The number of pixels in the interpolation area. It should be noted that: since each interpolation area is a part of the physical object in the target video frame, Not zero.

[0068] S42. Calculate the grayscale image texture information complexity according to the grayscale values corresponding to all pixels in the interpolation area.

[0069] In this embodiment, it is considered that when the grayscale value of an image changes significantly, it will be reflected in the increase in the grayscale value variance of the image. At the same time, the entropy value of an image is a measure of the amount of image information. If an image has no texture, the grayscale co-occurrence matrix is almost a zero matrix and the entropy value is close to zero. Therefore, in this embodiment, the entropy value and variance are combined as the basis for measuring the complexity of the texture information of the grayscale image in the interpolation area. For details, please refer to the following content:

[0070] In this step, first The grayscale variance is calculated based on the grayscale values corresponding to all the pixels in the interpolation area; The grayscale values corresponding to all the pixel points in the interpolation area are calculated to obtain a grayscale co-occurrence matrix; the entropy of the grayscale co-occurrence matrix is then calculated to obtain an entropy value; and finally, the grayscale image texture information complexity is calculated based on the entropy value and the grayscale co-occurrence matrix.

[0071] Specifically, in this step, the calculation formula for the complexity of grayscale image texture information is as follows:

[0072]

[0073] in, Indicates the The complexity of the grayscale image texture information of the interpolation area; represents the maximum and minimum normalization function; Indicates the Grayscale variance of the interpolation area; Indicates the The entropy value of the interpolation area.

[0074] S43. Calculate a point cloud texture coefficient according to the complexity of the grayscale image texture information and the density of the point cloud.

[0075] In this step, it is considered that: when the entity object When the point cloud data density in the interpolation area is relatively low, that is, the intervals between point cloud data points are large and the distribution is sparse, in order to ensure the continuity of texture and the richness of details, the interpolation area with more lost feature details should be compensated.

[0076] At the same time, this embodiment also considers the complexity of the grayscale image texture information. If the grayscale image texture information of an interpolation area is relatively complex, that is, the texture is rich in details and has many variations, then retaining these complex texture features during the interpolation process also requires a corresponding higher texture coefficient.

[0077] In this regard, the entity object in this embodiment The calculation formula of the texture coefficient of the interpolation area point cloud is:

[0078]

[0079] in, Indicates the Point cloud texture coefficients of the interpolation area; represents the maximum and minimum normalization function; Indicates the The complexity of the grayscale image texture information of the interpolation area; Indicates the The density of point cloud data in the interpolation area, Not zero.

[0080] In the above formula, the larger the point cloud texture coefficient of an interpolation region, the more significant the grayscale image changes in that interpolation region, which in turn manifests as an increase in the complexity of the grayscale image texture information. However, this also means that relatively few point cloud coordinates are collected, that is, the density of the point cloud data is reduced. Therefore, directly using the collected 3D building point cloud data to model the building digital twin may result in a significant loss of feature information during the modeling process. In view of this, when interpolating interpolation regions with high point cloud texture coefficients, the relevant parameter weights should be adjusted to accommodate the changes in the texture coefficient. The specific operation method can be referred to the interpolation process below.

[0081] S5. Confirm and obtain multiple interpolation points in each interpolation area.

[0082] In this embodiment, the area where each interpolation point is located is determined based on the bilinear interpolation method, that is, each interpolation point is located inside a quadrilateral defined by four known point cloud coordinates, where these four known point cloud coordinates can also be called the corresponding point cloud coordinates of the four neighboring points of the interpolation point.

[0083] S6. Calculate the interpolation weight of each neighboring point according to four neighboring data sets of each interpolation point, where each neighboring data set includes point cloud coordinates and grayscale values corresponding to a neighboring point.

[0084] In this embodiment, it is considered that the point cloud coordinates of each interpolation point should be affected by the point cloud coordinates of the four neighboring points around it and the corresponding image data. To explain this more clearly, see Figure 4 , Figure 4 It is shown that step S6 in this embodiment includes steps S61 to S64, which are In the interpolation area The interpolation point The interpolation weight calculation process of neighboring points is explained as an example.

[0085] S61. Calculate the grayscale feature deviation corresponding to each of the neighboring points based on the image data corresponding to each of the neighboring points, where the grayscale feature deviation is used to characterize the degree of change in the grayscale value of the neighboring point.

[0086] In this embodiment, the grayscale feature deviation is used to measure and characterize the degree of change in the grayscale value between neighboring points. Through this measurement, the brightness change of the four neighboring points around the interpolation point in the image can be effectively reflected. The degree of change in this grayscale value is of great significance for judging the significance of neighboring points in the image, because it can reveal which neighboring points are more prominent and important in the local area of the image. Furthermore, the calculation result of the grayscale feature deviation will provide the necessary basic information for the subsequent interpolation weight calculation. In the interpolation process, using the grayscale feature deviation as a weight can make the interpolation result closer to the actual situation, thereby improving the quality and effect of the subsequent building digital twin model.

[0087] In this step, the grayscale mean is first calculated based on the grayscale values corresponding to all the neighboring points; and then the grayscale feature deviation corresponding to each pixel is calculated based on the grayscale value corresponding to each pixel and the grayscale mean.

[0088] Specifically, in this step, the grayscale feature deviation is calculated as follows:

[0089]

[0090] in, Indicates the In the interpolation area The interpolation point The grayscale feature deviation of neighboring points; represents the maximum and minimum normalization function; Indicates the In the interpolation area The interpolation point Grayscale values corresponding to neighboring points; Indicates the In the interpolation area The grayscale mean corresponding to the interpolation point is given by The grayscale value of the four neighboring points corresponding to the interpolation point is averaged. Since the grayscale value of a pixel in the image is 0, it means that the pixel is pure black, which means that the pixel in the image represents a material that completely absorbs light (which will not appear in the actual building environment) or there is no object here. At the same time, each neighboring point corresponds to a point cloud data, that is, the location of the pixel is an objective object, that is, the grayscale value of each neighboring point will not be 0, that is, It is also not zero.

[0091] In the above formula, if the grayscale deviation of one of the interpolation point's neighbors is significant, this means that the grayscale value of that neighboring point in the feature video frame differs significantly from that of the other neighboring points. In this case, the interpolation process requires special attention to the characteristics of the neighboring point with the large grayscale deviation. In other words, when the grayscale deviation of a neighboring point is large, its weight in the interpolation process should be increased accordingly. This ensures that the interpolation result more accurately reflects the grayscale distribution in the feature video frame, thereby improving the accuracy and effectiveness of the interpolation.

[0092] At the same time, this embodiment also considers that curvature can be used to measure the geometric complexity of the area where adjacent points are located, reflecting the local geometric characteristics of the point cloud data. Therefore, this embodiment also takes into account the curvature changes of the point cloud data. For details, see steps S62 and S63.

[0093] S62: Calculate the curvature of each of the neighboring points based on all point cloud coordinates within the interpolation area to which the interpolation point belongs.

[0094] It should be noted that the curvature calculation mentioned in this step can first fit a plane or curved surface based on the point cloud data corresponding to the interpolation area, and then calculate the curvature of each adjacent point based on the fitting result. This is a prior art and will not be repeated in this embodiment.

[0095] In this step, the curvature calculation method involved can first fit a plane or surface based on the point cloud data corresponding to the interpolation area. Specifically, the discrete point cloud data can be fitted into a continuous plane or surface using least squares methods, Bezier surface fitting, or other high-order surface fitting methods. This ensures that the fitting result is as close to the actual geometric shape as possible.

[0096] After the fitting is completed, the curvature of each adjacent point is calculated based on the fitting result. The specific curvature calculation details are all existing technologies and are therefore not described in detail in this embodiment.

[0097] S63. Calculate the curvature deviation of each of the adjacent points based on the curvature of each of the adjacent points.

[0098] In this step, first, the mean curvature is calculated based on the curvature of each of the neighboring points; then, the curvature deviation degree corresponding to each of the neighboring points is calculated based on the difference ratio between the curvature of each of the neighboring points and the curvature mean.

[0099] The calculation formula for the curvature deviation degree in this step is as follows:

[0100]

[0101] in, Indicates the In the interpolation area The interpolation point The degree of curvature deviation of adjacent points; Indicates the In the interpolation area The interpolation point The curvature corresponding to the adjacent points; Indicates that based on In the interpolation area The interpolation point corresponds to the mean curvature, which is given by The curvature of the four neighboring points corresponding to the interpolation point is calculated by averaging. A non-zero constant to prevent the denominator from being zero.

[0102] In the above calculation formula, if the curvature deviation of one of the neighboring points of a certain interpolation point is large, this means that the local curvature of the neighboring point in three-dimensional space is significantly different from that of the other neighboring points. In this case, when performing the interpolation calculation, it is necessary to consider the three-dimensional characteristics of the neighboring point more to ensure the accuracy of the interpolation result. In other words, if the grayscale value of a neighboring point deviates greatly from that of other neighboring points, then the influence of the neighboring point will be greater when assigning a specific value to the interpolation point. This is because a significant deviation in the grayscale value usually indicates that the feature of the point in the image is more prominent, and therefore more weight needs to be given in the interpolation process. Therefore, in this embodiment, the above-mentioned curvature deviation degree can ensure that the interpolation result can better reflect the local characteristics of the point.

[0103] S64. Calculate the interpolation weight of each neighboring point according to the curvature deviation degree and grayscale value change degree corresponding to each neighboring point.

[0104] In this step, the curvature deviation degree and grayscale value change degree corresponding to each neighboring point are first multiplied to obtain the influence degree corresponding to each neighboring point, and then the normalized weight calculation is performed to obtain the interpolation weight. Specifically, the interpolation weight calculation formula is as follows:

[0105]

[0106] in, Indicates the Interpolation weights of neighboring points; Indicates the In the interpolation area The interpolation point The degree of influence of neighboring points; Indicates the In the interpolation area The interpolation point The degree of curvature deviation of adjacent points; Indicates the In the interpolation area The interpolation point The grayscale feature deviation of neighboring points.

[0107] S7. Perform interpolation calculation according to the point cloud texture coefficient of the interpolation area to which the interpolation point belongs, the point cloud coordinates of each of the neighboring points, and the interpolation weight to obtain the point cloud coordinates of each interpolation point.

[0108] Through the above process, this embodiment divides a single physical object into multiple interpolation regions of equal density by dividing it based on point cloud data density. The point cloud texture coefficients for each interpolation region are calculated based on the division results. Simultaneously, the interpolation weights for the adjacent points around a single interpolation point are obtained by analyzing the grayscale values and curvature characteristics of the adjacent points.

[0109] Therefore, this embodiment adopts a comprehensive approach to assigning interpolation points. It not only relies on the image features represented by the point cloud texture coefficients, but also incorporates the feature differences of neighboring points represented by the interpolation weights. This comprehensive approach enhances the detail of the building digital twin while avoiding oversharpening.

[0110] To better understand this process, please refer to the Figure 5 The figure shows that step S7 also includes steps S71-S73. These steps describe how to combine point cloud texture coefficients and interpolation weights to assign interpolation points. This approach effectively enhances the visual quality of the building digital twin while maintaining its authenticity and naturalness.

[0111] S71. Calculate first target coordinates based on the point cloud texture coefficients and the point cloud coordinates of all the neighboring points.

[0112] In this embodiment, it is considered that when the point cloud texture coefficient is larger, the actual texture features of the interpolation area are richer. Therefore, it is necessary to consider more the features of the surrounding neighboring points when assigning values to the interpolation area. When the point cloud texture coefficient is smaller, the overall interpolation area appears smoother and the overall texture is richer. Therefore, it is necessary to reduce sharpening and enhance smoothness when assigning values.

[0113] In this embodiment, we fully consider the impact of the point cloud texture coefficient on the interpolation region. Specifically, when the point cloud texture coefficient is large, it means that the actual texture features of the interpolation region are relatively rich and complex. In this case, to more accurately reflect these texture features, we need to more carefully consider the texture features of the surrounding points when assigning values to the interpolation region. This ensures that the interpolation result better captures texture details and variations, thereby improving the overall texture representation.

[0114] Conversely, when the point cloud texture coefficient is small, this generally indicates that the interpolated area exhibits a smoother surface overall, with a relatively uniform and rich texture. In this case, to maintain this smooth visual effect, we need to reduce sharpening when assigning values to the interpolated area to enhance smoothness. This avoids over-emphasizing texture details and thus unnecessary visual noise, ensuring that the interpolated result presents a more natural and soft texture effect.

[0115] Therefore, in this step, the first target coordinate calculation formula is as follows:

[0116]

[0117] in, Indicates the In the interpolation area The first target coordinate of the interpolation point axis coordinates; Indicates the In the interpolation area The interpolation point Neighboring points axis coordinates; Indicates the Point cloud texture coefficients of the interpolation area; Indicates the In the interpolation area The first target coordinate of the interpolation point axis coordinates; Indicates the In the interpolation area The interpolation point Neighboring points axis coordinates; Indicates the In the interpolation area The first target coordinate of the interpolation point axis coordinates; Indicates the In the interpolation area The interpolation point Neighboring points Axis coordinates.

[0118] S72. Obtain second target coordinates based on the point cloud coordinates and interpolation weights of each of the neighboring points.

[0119] In this embodiment, inspired by the bilinear interpolation method, the coordinate result of an interpolation point should be affected by the different interpolation weights of the four neighboring points. Therefore, in this step, the second target coordinate calculation formula is as follows:

[0120]

[0121] in, Indicates the In the interpolation area The second target coordinates of the interpolation points axis coordinates; Indicates the Point cloud texture coefficients of the interpolation area; Indicates the In the interpolation area The interpolation point Neighboring points axis coordinates; Indicates the Interpolation weights of neighboring points; Indicates the In the interpolation area The second target coordinates of the interpolation points axis coordinates; Indicates the In the interpolation area The interpolation point Neighboring points axis coordinates; Indicates the In the interpolation area The second target coordinates of the interpolation points Axis coordinates.

[0122] S73. Perform weighted calculation according to the first target coordinates and the second target coordinates to obtain the point cloud coordinates of the interpolation point.

[0123] In this step, the point cloud coordinates of the interpolation points are calculated as follows:

[0124]

[0125] in, Indicates the In the interpolation area The interpolation points correspond to axis coordinates; Indicates the In the interpolation area The interpolation points correspond to axis coordinates; Indicates the In the interpolation area The interpolation points correspond to axis coordinates; Indicates the In the interpolation area The first target coordinate of the interpolation point axis coordinates; Indicates the In the interpolation area The second target coordinates of the interpolation points axis coordinates; Indicates the In the interpolation area The first target coordinate of the interpolation point axis coordinates; Indicates the In the interpolation area The second target coordinates of the interpolation points axis coordinates; Indicates the In the interpolation area The first target coordinate of the interpolation point axis coordinates; Indicates the In the interpolation area The second target coordinates of the interpolation points Axis coordinates.

[0126] S8. Construct a building digital twin model based on the point cloud coordinates of all the interpolation points and the building three-dimensional point cloud data.

[0127] In this step, 3D modeling software (such as CloudCompare, MeshLab, and AutoCAD) can be used to build and visualize the model.

[0128] In this embodiment, the above-described method utilizes the video stream and three-dimensional point cloud data obtained from the target building during full-time monitoring, constructing a digital twin model of the building every 30 minutes. This allows for comprehensive, real-time monitoring of the building twin model, thereby achieving dynamic monitoring. Furthermore, the building details provided by the video image sensor partially compensate for the reduced point density and severe loss of detail caused by lidar sensors in large environments, as well as the potential for highly reflective materials to render lidar detection ineffective and result in data loss. This allows for the high-precision construction of a digital twin model of the target building.

[0129] In this embodiment, 3D modeling software, such as CloudCompare, MeshLab, and AutoCAD, can be used to build and visualize the model. These software tools can accurately and intuitively display the building digital twin model.

[0130] This example utilizes video streams and 3D point cloud data from target buildings monitored throughout the entire building lifecycle. A digital twin model of the building is constructed every 30 minutes, enabling comprehensive, real-time monitoring of the building twin. This approach not only captures building changes in real time but also dynamically displays its status, achieving dynamic monitoring.

[0131] Furthermore, this embodiment uses video image sensors to provide detailed information about buildings. This approach effectively compensates for the reduced point density and severe loss of detail experienced by LiDAR sensors in large environments. This is particularly true in environments with highly reflective materials, where LiDAR detection may fail, resulting in missing data. By supplementing this with video image sensors, we can better capture these details, enabling the high-precision construction of a digital twin model of the target building.

[0132] That is, by combining the advantages of three-dimensional modeling software and video image sensors in this embodiment, a digital twin model of a building can be constructed more comprehensively and accurately, thereby providing strong technical support for the monitoring and management of the building.

[0133] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0134] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A building digital twin modeling method based on intelligent construction, characterized in that: The method comprises: Acquire the target building’s video stream and 3D point cloud data in real time; Performing data fusion on the video stream of the target building and the three-dimensional point cloud data of the building to obtain a multi-source dataset, wherein the multi-source dataset includes a feature video frame, and a portion of pixels of the feature video frame include point cloud coordinates; Dividing the multi-source dataset corresponding to the target building according to point cloud density to obtain multi-source datasets corresponding to multiple interpolation areas; Calculating a point cloud texture coefficient of each interpolation area according to the multi-source data set corresponding to each interpolation area; Confirming and obtaining a plurality of interpolation points in each of the interpolation areas; Calculating an interpolation weight for each neighboring point based on four neighboring data sets of each interpolation point, each neighboring data set including point cloud coordinates and grayscale values corresponding to a neighboring point; Performing interpolation calculation based on the point cloud texture coefficient of the interpolation area to which the interpolation point belongs, the point cloud coordinates of each of the neighboring points, and the interpolation weight to obtain the point cloud coordinates of each interpolation point; Constructing a building digital twin model according to the point cloud coordinates of all the interpolation points and the building three-dimensional point cloud data; Calculating a point cloud texture coefficient of each interpolation region according to the multi-source data set corresponding to each interpolation region includes: Calculating the point cloud density based on the multi-source data set corresponding to the interpolation area; The grayscale image texture information complexity is calculated based on the grayscale values corresponding to all pixels in the interpolation area; The point cloud texture coefficient is calculated according to the complexity of the grayscale image texture information and the density of the point cloud.

2. The building digital twin modeling method based on intelligent construction according to claim 1 is characterized in that: The video stream of the target building and the three-dimensional point cloud data of the building are fused to obtain a multi-source data set, including: Performing image screening on the video stream to obtain a target video frame, wherein the target video frame has the smallest image data change within the video stream of a preset duration; Performing entity object segmentation on the target video frame to obtain at least one entity object and two-dimensional image data corresponding to each entity object; Extracting all point cloud coordinates corresponding to each of the physical objects in the building point cloud data according to the time point corresponding to the target video frame; According to the two-dimensional image data corresponding to each of the physical objects, two-dimensional image data mapping is performed on each point cloud coordinate to obtain a multi-source data set.

3. The building digital twin modeling method based on intelligent construction according to claim 2 is characterized in that: Perform image filtering on the video stream to obtain the target video frame, including: intercepting a target video stream within the video stream according to a preset number of frames, wherein the cutoff time of the target video stream is the current time; Convert each frame of the target video stream into grayscale to obtain a target grayscale video stream; Calculating the grayscale change value of each grayscale image in the target grayscale video stream frame by frame, where the grayscale change value is the sum of the absolute values of the grayscale changes of all pixels in the grayscale image in consecutive frames; The target video frame is obtained by screening according to the grayscale change value corresponding to each frame of the grayscale image, and the grayscale change value corresponding to the target video frame is the minimum value.

4. The building digital twin modeling method based on intelligent construction according to claim 1 is characterized in that: The point cloud density is calculated based on the multi-source data set corresponding to the interpolation area, including: Counting the number of point cloud coordinates and the number of pixels in the interpolation area; The density of the point cloud data is calculated based on the number of the point cloud coordinates and the number of the pixel points.

5. The building digital twin modeling method based on intelligent construction according to claim 1 is characterized in that: The grayscale image texture information complexity is calculated based on the grayscale values corresponding to all pixels in the interpolation area, including: Calculating the grayscale variance according to the grayscale values corresponding to all the pixels in the interpolation area; A gray level co-occurrence matrix is calculated based on the gray values corresponding to all the pixels in the interpolation area; Calculating the entropy of the gray-level co-occurrence matrix to obtain an entropy value; The complexity of the grayscale image texture information is calculated based on the entropy value and the gray-level co-occurrence matrix.

6. The building digital twin modeling method based on intelligent construction according to claim 1 is characterized in that: The interpolation weight of each neighboring point is calculated based on the four neighboring data sets of each interpolation point, including: Calculating the grayscale feature deviation corresponding to each of the neighboring points based on the image data corresponding to each of the neighboring points, wherein the grayscale feature deviation is used to characterize the degree of change in the grayscale value of the neighboring point; Calculating the curvature of each of the neighboring points based on all point cloud coordinates within the interpolation area to which the interpolation point belongs; Calculating the curvature deviation of each of the adjacent points based on the curvature of each of the adjacent points; The interpolation weight of each neighboring point is calculated according to the curvature deviation degree and gray value change degree corresponding to each neighboring point.

7. The building digital twin modeling method based on intelligent construction according to claim 6 is characterized in that: Calculating the grayscale feature deviation corresponding to each of the neighboring points based on the image data corresponding to each of the neighboring points includes: Calculate the grayscale mean according to the grayscale values corresponding to all the adjacent points; The grayscale feature deviation corresponding to each pixel is calculated based on the grayscale value corresponding to each pixel and the grayscale mean.

8. The building digital twin modeling method based on intelligent construction according to claim 6 is characterized in that: The curvature deviation degree of each of the adjacent points is calculated based on the curvature of each of the adjacent points, including: Calculating the mean value of the curvature of each of the adjacent points to obtain a curvature mean; The curvature deviation degree corresponding to each of the adjacent points is calculated according to the difference ratio between the curvature of each of the adjacent points and the curvature mean.

9. The building digital twin modeling method based on intelligent construction according to claim 1 is characterized in that: The point cloud coordinates of each interpolation point are obtained by performing interpolation calculation according to the point cloud texture coefficient of the interpolation area to which the interpolation point belongs, the point cloud coordinates of each of the neighboring points, and the interpolation weight, including: Calculate the first target coordinates according to the point cloud texture coefficient and the point cloud coordinates of all the neighboring points; Calculating the second target coordinates according to the point cloud coordinates of each of the neighboring points and the interpolation weight; The point cloud coordinates of the interpolation point are obtained by performing weighted calculation according to the first target coordinates and the second target coordinates.

Images