Automatic data conversion and fusion method for three-dimensional model component

Abstract

The invention discloses an automatic data conversion and fusion method for a three-dimensional model component, and relates to the technical field of data processing. According to the method, the two-dimensional image of the three-dimensional model is extracted through the SAM model based on the image data of the three-dimensional model, the two-dimensional image is segmented to obtain the first component segmentation result, and then the spatial geometric features of the component in the three-dimensional model are determined through the NeRF model based on the image data of the three-dimensional model and the first component segmentation result; the method comprises the steps of obtaining a second component segmentation result, then mapping the second component segmentation result to a three-dimensional model, segmenting the three-dimensional model to obtain the component, determining attribute information of the component through a semantic recognition model, and finally generating standard data of the component based on the component and the attribute information, thereby improving the accuracy of three-dimensional model component data conversion.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for automatic data conversion and fusion of three-dimensional model components. Background Art

[0002] In the construction industry, Building Information Modeling (BIM) can store and manage the geometry, attributes, and relationship data of building components. When converting ordinary 3D models into BIM data, a key step is to identify each component in the 3D model and assign corresponding attribute information to each component.

[0003] At present, the method of converting 3D models into BIM data is mainly for professionals to disassemble and mark the 3D models according to the rule base and common sense, and then identify and classify the properties and parameters of different components according to the rule base, and manually add attribute information for each component. However, the above rule base is difficult to adapt to all types of 3D models. When faced with diversified and complex 3D models, errors are prone to occur in component identification and attribute association, resulting in low accuracy in the conversion of 3D models to BIM data.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present application is to provide a method for automatic conversion and fusion of data of three-dimensional model components, aiming to solve the technical problem of low accuracy of standard data conversion of three-dimensional model components.

[0006] To achieve the above purpose, the present application proposes a method for automatic data conversion and fusion of three-dimensional model components, the method comprising: Based on the image data of the three-dimensional model, extracting a two-dimensional image of the three-dimensional model through the SAM model, and segmenting the two-dimensional image to obtain a first component segmentation result; Based on the image data of the three-dimensional model and the first component segmentation result, determine the spatial geometric features of the component in the three-dimensional model through the NeRF model to obtain a second component segmentation result; Mapping the second component segmentation result to the three-dimensional model, segmenting the three-dimensional model to obtain the component, and determining the attribute information of the component through a semantic recognition model; Based on the component and the attribute information, standard data of the component is generated.

[0007] In one embodiment, the step of extracting a two-dimensional image of the three-dimensional model based on the image data of the three-dimensional model through the SAM model and segmenting the two-dimensional image to obtain a first component segmentation result includes: Extracting the image data of the three-dimensional model frame by frame to obtain the two-dimensional images at different viewing angles; Based on the geometric features of the two-dimensional image, segment the two-dimensional image into at least one segmented area, where the segmented area corresponds to a component in the three-dimensional model; Determining a spatial position of the segmented region based on a position of the segmented region in the two-dimensional image at different viewing angles; The first component segmentation result is determined based on the segmented area and the spatial position of the segmented area.

[0008] In one embodiment, the step of determining the spatial geometric features of the component in the three-dimensional model by using a NeRF model based on the image data of the three-dimensional model and the first component segmentation result to obtain the second component segmentation result comprises: Determining spatial geometric features of the component in the three-dimensional model based on the image data of the three-dimensional model; The second component segmentation result is determined based on the first component segmentation result and the spatial geometric features of the component in the three-dimensional model.

[0009] In one embodiment, before the step of determining the second component segmentation result based on the first component segmentation result and the spatial geometric features of the component in the three-dimensional model, the step further includes: Determining the area of ​​the segmented region in the first component segmentation result; If there is an area of ​​the segmented region that is smaller than a preset area threshold, determining the similarity between the segmented region and the adjacent region based on the color and / or texture between the segmented region and the adjacent region; If the similarity between the segmented region and the adjacent region is greater than a preset similarity threshold, the segmented region and the adjacent region are merged to obtain a new first component segmentation result.

[0010] In one embodiment, the attribute information includes first attribute information, and the step of determining the attribute information of the component by using a semantic recognition model includes: Determining initial attribute features of the component based on the segmentation results of the component and the second component; Inputting the initial attribute features into the semantic recognition model; The first attribute information output by the semantic recognition model based on the initial attribute feature is obtained, where the first attribute information includes type information and function information of the component.

[0011] In one embodiment, the attribute information includes second attribute information, and after the step of obtaining the first attribute information output by the semantic recognition model based on the initial attribute feature, the step further includes: Inputting the first attribute information and the image data of the three-dimensional model into the semantic recognition model; The second attribute information of the component is determined by the semantic recognition model, where the second attribute information includes type information, function information, material information and manufacturing information of the component.

[0012] In one embodiment, after the step of determining the attribute information of the component by using the semantic recognition model, the method further includes: Get the attribute modification information entered by the user; inputting the attribute correction information and the attribute information into the semantic recognition model again; The new attribute information output by the semantic recognition model is obtained.

[0013] In one embodiment, the step of generating standard data of the component based on the component and the attribute information includes: Get the standard component data attribute table; Based on the standard component data attribute table, the attribute information is associated with the component and saved to generate standard data of the component.

[0014] The present application provides a method for automatic data conversion and fusion of three-dimensional model components. Based on the image data of the three-dimensional model, a two-dimensional image of the three-dimensional model is extracted through a SAM model, and the two-dimensional image is segmented to obtain a first component segmentation result. Then, based on the image data of the three-dimensional model and the first component segmentation result, the spatial geometric features of the component in the three-dimensional model are determined through a NeRF model to obtain a second component segmentation result. Thereafter, the second component segmentation result is mapped to the three-dimensional model, the three-dimensional model is segmented to obtain components, and the attribute information of the components is determined through a semantic recognition model. Finally, based on the components and the attribute information, standard data of the components is generated.

[0015] This method achieves high-precision segmentation of three-dimensional model components by combining the collaborative work of the SAM model and the NeRF model, effectively overcoming the problems of inaccurate component identification and insufficient conversion accuracy faced by traditional rule-based methods when dealing with complex and diverse three-dimensional models. In addition, this method also introduces a semantic recognition model to conduct in-depth analysis of each component to obtain its detailed attribute information. Finally, based on the segmented components and their attribute information, standard three-dimensional model component data is generated. This process reduces the errors that may be introduced by manual annotation, thereby significantly improving the accuracy of three-dimensional model component data conversion when converting three-dimensional models into BIM data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic diagram of a flow chart provided for a method for automatic data conversion and fusion of a three-dimensional model component according to a first embodiment of the present application; Figure 2 A flow chart of a method for automatic data conversion and fusion of a three-dimensional model component provided in Embodiment 2 of the present application; Figure 3 A flow chart of a method for automatic data conversion and fusion of a three-dimensional model component provided in Embodiment 3 of the present application; Figure 4 A schematic diagram of a flow chart of a method for automatic data conversion and fusion of a three-dimensional model component in an embodiment of the present application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in a method for automatic data conversion and fusion of three-dimensional model components in an embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] In the construction industry, building information models can store and manage the geometry, attributes, and relationship data of building components. When converting ordinary 3D models to BIM data, a key step is to identify each component in the 3D model and assign corresponding attribute information to each component. At present, the method of converting 3D models to BIM data is mainly for professionals to disassemble and mark the 3D model according to the rule base and common sense, and then identify and classify the attributes and parameters of different components according to the rule base, and manually add attribute information for each component. However, the above rule base is difficult to adapt to all types of 3D models. When faced with diversified and complex 3D models, errors are prone to occur in component identification and attribute association, resulting in low accuracy in the conversion of 3D models to BIM data.

[0023] In view of the above problems, the present application proposes a method for automatic data conversion and fusion of three-dimensional model components. Based on the image data of the three-dimensional model, the two-dimensional image of the three-dimensional model is extracted through the SAM model, and the two-dimensional image is segmented to obtain a first component segmentation result. Then, based on the image data of the three-dimensional model and the first component segmentation result, the spatial geometric features of the component in the three-dimensional model are determined through the NeRF model to obtain a second component segmentation result. After that, the second component segmentation result is mapped to the three-dimensional model, the three-dimensional model is segmented to obtain the component, and the attribute information of the component is determined through the semantic recognition model. Finally, based on the component and the attribute information, standard data of the component is generated.

[0024] This method achieves high-precision segmentation of 3D model components through the collaborative work of the SAM model and the NeRF model, effectively overcoming the problems of inaccurate component identification and insufficient conversion accuracy faced by traditional rule-based methods when dealing with complex and diverse 3D models. In addition, this method also introduces a semantic recognition model to conduct in-depth analysis of each component to obtain its detailed attribute information. Finally, based on the segmented components and their attribute information, standard 3D model component data is generated. This process reduces the errors that may be introduced by manual annotation, thereby significantly improving the accuracy of generating standard data for 3D model components when converting 3D models into BIM data.

[0025] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes a three-dimensional model component standard data generation device as an example to illustrate this embodiment and the following embodiments.

[0026] Based on this, the first embodiment proposed in this application provides a method for automatically converting and fusing data of a three-dimensional model component, referring to Figure 1 In this embodiment, the method for automatic conversion and fusion of data of the three-dimensional model component includes steps S10 to S40: Step S10: Based on the image data of the three-dimensional model, extract the two-dimensional image of the three-dimensional model through the SAM model, and segment the two-dimensional image to obtain a first component segmentation result.

[0027] It should be noted that SAM, which stands for Segment Anything Model, is an image segmentation model that can automatically identify and segment objects in an image. In this embodiment, the SAM model is used to extract two-dimensional images from a three-dimensional model and accurately segment these two-dimensional images, so as to preliminarily identify different components in the three-dimensional model. The image data of a three-dimensional model refers to digital model data containing spatial information such as buildings and objects generated by three-dimensional modeling software or obtained by scanning, including information such as the appearance, shape, and texture of the three-dimensional model. A two-dimensional image refers to a two-dimensional projection image generated from different perspectives of a three-dimensional model. The first component segmentation result includes geometric information such as the boundaries, shapes, and textures of each component in the two-dimensional image. For example, it can be a three-dimensional line mask layer, in which the lines represent the boundaries of the components, and the transparency, color, and other properties of the mask layer represent the texture color information of the components.

[0028] Exemplarily, the input three-dimensional model is first preprocessed, including format conversion and coordinate system one. For example, the three-dimensional model is converted into a format acceptable to the SAM model, ensuring that all model data is in a unified coordinate system, etc. Then, the SAM model is used to extract a two-dimensional image from the preprocessed three-dimensional model. Specifically, the three-dimensional model is projected onto different planes to obtain two-dimensional images at different angles, for example, the three-dimensional model is projected from the X, Y, and Z axis directions, and each projection can be regarded as a two-dimensional image observed from a specific perspective. Then, the SAM model is used to segment the input three-dimensional model from at least three mutually orthogonal perspectives to identify different components in the two-dimensional image.

[0029] Optionally, step S10 includes steps S11 to S14: Step S11, extracting the image data of the three-dimensional model frame by frame to obtain the two-dimensional images of different viewing angles.

[0030] Exemplarily, the image data of the 3D model is input into the SAM model. When the SAM model processes the continuous or dynamic 3D model image data, it is split into independent, static 2D image frames to generate a series of 2D images of different perspectives, which retain key information in the 3D model, such as the shape, size and position of the object.

[0031] Optionally, two-dimensional views of the three-dimensional model at different viewing angles are first generated by three-dimensional modeling software, and then the two-dimensional views of the three-dimensional model at different viewing angles are input into the SAM model.

[0032] For example, in 3D modeling software, the position and direction of the camera are set to simulate the viewing angles in different directions. According to the viewing angle requirements, the camera's focal length, field of view, position and direction and other parameters are adjusted, and then the 3D model is rendered into a 2D view according to the set viewing angle and camera parameters. During the rendering process, different rendering quality, lighting effects and texture mapping parameters can be selected to generate high-quality 2D views and improve the accuracy of SAM model segmentation.

[0033] Step S12: based on the geometric features of the two-dimensional image, segment the two-dimensional image into at least one segmented area, where the segmented area corresponds to a component in the three-dimensional model.

[0034] It should be noted that the geometric features of a two-dimensional image refer to the attributes such as the shape, size, position and direction of a component in the two-dimensional image. The relative positions or spatial relationships between different components can also be reflected in two-dimensional images from different perspectives. After the image data of the three-dimensional model is input into the SAM model, the SAM model will segment the components according to the geometric features of the components in the image data and the two-dimensional images from different perspectives to obtain a preliminary first component segmentation result, which includes the boundaries and positions of each component.

[0035] Exemplarily, the SAM model first identifies geometric features in a two-dimensional image, including edges, textures, corners, and contours. For example, the Canny edge detection algorithm is used to identify edge contours in an image, and the Harris corner detection algorithm is used to identify corners in an image. Then, based on the above-mentioned geometric features, the two-dimensional image is segmented into multiple segmented areas. For example, starting from the corners, the image is segmented into multiple segmented areas along the contour lines, and each area corresponds to an object or subject in the image, that is, a component in a three-dimensional model.

[0036] Step S13: determining the spatial position of the segmented area based on the position of the segmented area in the two-dimensional image at different viewing angles.

[0037] Step S14: determining the first component segmentation result based on the segmented area and the spatial position of the segmented area.

[0038] It should be noted that when objects in the three-dimensional model are projected onto a two-dimensional image through a camera, the relative positional relationship between the objects will be maintained. Therefore, the spatial positional relationship of the components in the three-dimensional model corresponding to the segmented area can be inferred based on the positional relationship of the segmented area in the two-dimensional image at different perspectives.

[0039] Optionally, the geometric features in each two-dimensional image are determined, and the similarity of the geometric features of each segmented area in different two-dimensional images is determined by feature descriptors and / or matching algorithms, and the feature points of the same three-dimensional model component corresponding to the segmented areas in the two-dimensional images of different viewing angles are determined based on the similarity. The relative position relationship of the feature points corresponding to each three-dimensional model component in the images of different viewing angles is analyzed, and the segmented areas corresponding to the same three-dimensional model component in the two-dimensional images of different viewing angles are merged according to the relative position relationship.

[0040] For example, in two-dimensional images A and two-dimensional image B at different viewing angles, rectangular window components are included. Geometric features such as edges, textures, and corners of the window components in two-dimensional images A and two-dimensional image B are extracted through the SAM model, and the geometric features of the window components in the two two-dimensional images are converted into vector forms through feature descriptors. The similarity of the geometric features of the window components in two-dimensional images A and two-dimensional image B is determined by calculating the Euclidean distance of the two vectors. If the similarity is greater than the similarity threshold, it is determined that the window components in two-dimensional images A and two-dimensional image B correspond to the same window component in the three-dimensional model, and the position of the window component in the three-dimensional model is determined according to the relative position of each component in the two-dimensional image, and the segmentation result of the window component is obtained.

[0041] Optionally, after receiving the two-dimensional image as input, the SAM model can also receive user-entered prompt information such as points, boxes, texts, etc. to specify the target to be segmented. Specifically, the SAM model first uses an image encoder to convert the original two-dimensional image into a series of image embeddings for generating and evaluating segmentation masks. At the same time, the prompt encoder encodes the prompt information provided by the user, such as points, boxes, texts, etc., into prompt vectors. The mask decoder combines the image embedding and the prompt vector, analyzes the geometric features of the components in the image through a deep learning algorithm, and combines the positional relationship of the components in different two-dimensional images. The self-attention mechanism and cross-attention mechanism of the Transformer (transformer model architecture) are used to fuse the image content and user intentions to determine the exact boundary of each component. Finally, the SAM model generates a segmentation mask to represent the probability that each pixel in the image belongs to the foreground or background, where the foreground pixels usually correspond to the components in the image, and the background pixels correspond to the parts outside the components. Therefore, by binarizing the segmentation mask, such as setting the foreground pixels to 1 and the background pixels to 0, a binary image containing only the components can be obtained. Furthermore, the binary image is processed by contour extraction and the like to identify and extract various components in the image.

[0042] Optionally, a two-dimensional image dataset containing the target component is collected in advance, and the components on the two-dimensional image are annotated. The SAM model is trained using the annotated two-dimensional image dataset so that the SAM model learns the geometric features and segmentation rules of the target component. The trained SAM model is applied to the image to be segmented, and a segmentation mask is generated according to the prompts provided by the user. The segmentation results are post-processed such as morphological processing, boundary smoothing, etc. to improve the segmentation accuracy and visual effect.

[0043] Optionally, a corresponding grayscale value threshold or color value threshold is determined according to a grayscale histogram or color histogram of the two-dimensional image, and the two-dimensional image is segmented based on the grayscale value threshold or color value threshold. Exemplarily, a suitable threshold is selected according to the grayscale histogram or color histogram of the image, and the grayscale value or color value of each pixel in the image is compared with the threshold, and pixels greater than the threshold are classified as foreground, and pixels less than the threshold are classified as background.

[0044] In practical applications, appropriate segmentation methods can be selected according to specific needs and image characteristics. For component segmentation tasks in complex scenes, the deep learning segmentation method combined with the SAM model can more effectively utilize the geometric features and positional relationships of two-dimensional images and more accurately identify and segment components in images.

[0045] Step S20: Based on the image data of the three-dimensional model and the first component segmentation result, the spatial geometric features of the component in the three-dimensional model are determined by the NeRF model to obtain a second component segmentation result.

[0046] It is understandable that the first component segmentation result is obtained through two-dimensional image segmentation, focusing on the surface features in the image, such as the boundaries, textures, and positions of the components. These surface features are crucial for understanding and identifying components in a three-dimensional model, and can provide rich information about the shape and appearance of the components. However, two-dimensional image segmentation has limitations in mining the three-dimensional information of components. Since a two-dimensional image is only a projection of a three-dimensional space, it cannot directly provide accurate information about the three-dimensional properties of the component, such as size and depth. This means that although the first component segmentation result can well reflect the characteristics of the component on the two-dimensional plane, the accuracy of this information may not be enough in three-dimensional space. Therefore, the NeRF model is used to further analyze the spatial geometric characteristics of the component in the three-dimensional model based on the image data of the three-dimensional model and the first component segmentation result to improve the accuracy of component segmentation.

[0047] It should be noted that NeRF is the full name of Neural Radiance Fields, which is used for three-dimensional scene reconstruction and rendering. In this embodiment, the NeRF model can use the two-dimensional image information of multiple perspectives and the first component segmentation result to reconstruct the components in the three-dimensional scene and determine the geometric features of each component in the three-dimensional space. Spatial geometric features refer to the position, shape, size, direction and other features of the component in the three-dimensional space. For example, when two-dimensional images of multiple perspectives are input into the NeRF model, the NeRF model will reconstruct the walls, doors, windows and other components in the three-dimensional scene, and then combine the first component segmentation result to determine the position, size and shape of these components in the three-dimensional model. The second component segmentation result is a three-dimensional data set or model including points, lines, surfaces or volume pixels, such as a three-dimensional line graph that can display the internal component structure, which is used to define the spatial position and boundaries of the component in the three-dimensional model.

[0048] Exemplarily, the NeRF model is trained using an annotated data set. Different types of three-dimensional model image data and corresponding first component segmentation results are obtained in advance. The image data is preprocessed, such as denoising, contrast enhancement, and size adjustment, to improve the training effect. At the same time, the first component segmentation result is adjusted and optimized to ensure that it matches the image data. The processed image data is used as the input of the NeRF model, and the NeRF model will use these image data to learn the geometric features of the component in three-dimensional space. Through iterative training, the NeRF model will gradually learn how to extract three-dimensional spatial geometric features from image data. During the training process, the parameters of the model including learning rate, batch size, regularization parameter, etc. are continuously adjusted, and the gradient descent algorithm, momentum, Adam (Adaptive Moment Estimation) optimizer, etc. are used to accelerate the training process and improve the accuracy and efficiency of the segmentation results. For example, by calculating indicators such as intersection-over-union ratio, precision, and recall rate, the error between the second component segmentation result and the actual three-dimensional model component is determined to evaluate the accuracy of the NeRF model. If the error is greater than the preset error threshold, the network structure of the NeRF model is adjusted, such as increasing or decreasing the number of layers, changing the activation function, etc., to improve the model performance.

[0049] Optionally, step S20 includes steps S21-S22: Step S21, determining the spatial geometric features of the component in the three-dimensional model based on the image data of the three-dimensional model.

[0050] Exemplarily, a trained NeRF model is loaded, image data of the three-dimensional model acquired from multiple perspectives is converted into a format that can be correctly read and processed by the NeRF model, and then the prepared image data of the three-dimensional model is input into the NeRF model as input data of the model.

[0051] Optionally, the NeRF model selects a suitable viewing angle for rendering based on the input image data of the 3D model, according to the complexity of the 3D model and the distribution of components, sets rendering parameters such as resolution, lighting conditions, camera parameters, etc., generates rendered images at different viewing angles using the volume rendering capability of the NeRF model, and determines the spatial geometric features of the components by comparing the rendered images at different viewing angles.

[0052] Optionally, feature points such as corners, edges, etc. are extracted from the image data through the NeRF model, and the spatial position and direction of the component are determined by matching the feature points at different viewing angles. Specifically, feature points such as corners, edges, etc. are detected in the rendered image of each viewing angle using a feature point detection algorithm or a deep learning algorithm, and the feature points at different viewing angles are matched using a feature point matching algorithm such as FLANN (Fast Library for Approximate Nearest Neighbors) to establish a correspondence between the feature points, and the spatial position and direction of the component are calculated based on the matched feature points. For example, after obtaining the matched feature point pairs, the coordinates of the matching points in the three-dimensional space are calculated based on the projection relationship of the matched feature points at multiple viewing angles, the three-dimensional structure of the component is restored, and then the spatial position of the component is determined by calculating the centroid or other geometric center of the component, or the feature vectors such as the normal vector and the main direction in the three-dimensional structure of the component are analyzed to determine the spatial geometric features such as the direction of the component.

[0053] Step S22: determining the second component segmentation result based on the first component segmentation result and the spatial geometric features of the component in the three-dimensional model.

[0054] The first component segmentation result and the spatial geometric features of the component in the three-dimensional model are fused, and the first component segmentation result is further optimized and adjusted using the spatial geometric features output by the NeRF model to obtain a more accurate second component segmentation result. For example, a deep learning model is trained to optimize the segmentation result. The model can receive the first component segmentation result and the spatial geometric features of the component as input, and output a more accurate second component segmentation result. Alternatively, the deep learning model is incorporated into the NeRF model, and the NeRF model finally outputs the second component segmentation result.

[0055] Exemplarily, the positional relationship of each component in the first component segmentation result is compared with the spatial geometric features output by the NeRF model to determine the segmentation area and parameters that need to be optimized, and optimization algorithms such as support vector machines, random forests, and convolutional neural networks are selected to adjust the first component segmentation result according to the spatial geometric features of the components. The optimization process is iterated so that the second component segmentation result conforms to the spatial geometric features of each component in the actual three-dimensional model.

[0056] Optionally, after obtaining the second component segmentation result, the second component segmentation result is post-processed, including removing noise, filling holes, merging small areas, etc., to obtain a more accurate segmentation result.

[0057] For example, in the second component segmentation result, there may be some small, irregular areas caused by errors or interference in the image acquisition and processing process, which are regarded as noise. The step of removing noise is to identify and delete these small, irregular areas to reduce misjudgment and interference of the segmentation result. For the holes or missing areas that may exist in the second component segmentation result, these missing areas are filled by interpolation method, etc., so that the segmentation result is more complete and continuous.

[0058] Through the above steps, the segmentation result of the first component can be fused with the spatial geometric features of the component in the three-dimensional model, and the fused information can be used to perform segmentation optimization to improve the accuracy and precision of the segmentation result.

[0059] Step S30, mapping the second component segmentation result to the three-dimensional model, segmenting the three-dimensional model to obtain the component, and determining the attribute information of the component through a semantic recognition model.

[0060] The three-dimensional coordinates of the second component segmentation result and the original three-dimensional model are aligned, the second component segmentation result is accurately located at the corresponding position in the three-dimensional model, the three-dimensional model is segmented, and each component in the three-dimensional model is obtained. According to the size, shape, texture and position characteristics of the component, other attribute information of the component is further analyzed through the semantic recognition model.

[0061] Optionally, the attribute information includes first attribute information. Step S30 includes steps S31 to S33: Step S31: determining initial attribute features of the component based on the segmentation results of the component and the second component.

[0062] It should be noted that the initial features of a component refer to the attribute features of the component that can be directly obtained from the image data of the three-dimensional model and / or the second component segmentation result, such as the shape and size features, texture features, color features and position features of the component.

[0063] Step S32: inputting the initial attribute features into the semantic recognition model.

[0064] The initial attribute features of the component that can be directly obtained from the image data of the three-dimensional model and / or the second component segmentation result are passed as input to the semantic recognition model. The semantic recognition model is a trained machine learning model or deep learning network that can analyze and reason about the input initial attribute features to identify the semantic information of the component.

[0065] Step S33: obtaining first attribute information output by the semantic recognition model based on the initial attribute feature, wherein the first attribute information includes type information and function information of the component.

[0066] The semantic recognition model analyzes and infers the input initial attribute features, and then outputs the first attribute information. The first attribute information includes the type information and function information of the component. The type information refers to the category or type of the component, such as doors, windows, columns, beams, etc. This information helps to classify the component into a specific building category. Functional information refers to the function or purpose of the component, such as load-bearing, sound insulation, ventilation, decoration, etc.

[0067] Optionally, the attribute information further includes second attribute information. After step S33, the method for automatic conversion and fusion of data of the three-dimensional model component further includes steps S34 to S35: Step S34: input the first attribute information and the image data of the three-dimensional model into the semantic recognition model.

[0068] Step S35: determining second attribute information of the component through the semantic recognition model, where the second attribute information includes type information, function information, material information and manufacturing information of the component.

[0069] In order to more comprehensively understand the attribute information of the component, the first attribute information previously obtained from the initial attribute features and the image data of the three-dimensional model are input into the semantic recognition model. Combining the first attribute information and the overall three-dimensional model, the semantic recognition model analyzes and infers based on the input first attribute information and the image data of the three-dimensional model, which can more accurately identify the type and function of the component, and further infer the material and manufacturing information of the component. Among them, the second attribute information is expanded on the basis of the first attribute information, and not only includes the type information and function information of the component, but also includes material information and manufacturing information. Among them, the material information indicates the type of material used by the component, such as wood, metal, plastic, glass, etc., and the manufacturing information indicates the manufacturing process, connection method, processing accuracy and other aspects of the component.

[0070] Optionally, collect BIM data sets of standard components, which include three-dimensional models, image data and corresponding detailed attribute information of components, among which other attribute information to be identified can be adjusted according to the needs of actual applications, such as physical attribute information, use and maintenance information of components, etc. The collected data are preprocessed, including cleaning data, removing noise, normalization, etc., to ensure the quality and consistency of the data. At the same time, the data is labeled to assign the correct attribute information label to each component. Then, according to the task requirements and data characteristics, select a suitable machine learning model or deep learning network as a semantic recognition model. For example, a convolutional neural network can be used to process image data, and a recurrent neural network or a graph neural network can be used to process the geometric and topological information of components. Next, design the specific structure of the semantic recognition model, including an input layer, a hidden layer and an output layer. The input layer is used to receive the initial attribute features, the first attribute information and the image data of the three-dimensional model; the neurons and the number of layers of the hidden layer are set to capture the complex characteristics of the data; the output layer is used to output the detailed attribute information of the component. The preprocessed standard component data set is input into the semantic recognition model for training. During the training process, the parameters of the semantic recognition model are continuously adjusted to minimize the error between the predicted result and the true label, and a trained semantic recognition model is obtained. In practical applications, the initial attribute features and image data of the 3D model component to be identified are input into the trained semantic recognition model, which will perform inference based on the input initial attribute features and image data and output detailed attribute information of the component.

[0071] Step S40: generating standard data of the component based on the component and the attribute information.

[0072] After obtaining the segmented components and their attribute information, this information is integrated into a unified data structure according to the preset data standards and formats. The data structure can be a database table, XML (eXtensibleMarkup Language) file or a data container in other formats. It is then saved in a data format that complies with specific specifications or standards to facilitate data exchange, sharing and application.

[0073] Optionally, step S40 includes steps S41-S42: Step S41, obtaining a standard component data attribute table.

[0074] Step S42: Based on the standard component data attribute table, the attribute information is associated with the component and saved to generate standard data of the component.

[0075] It should be noted that the standard component data attribute table is a table or data structure that defines all standard component attribute information and its format, which serves as the basis for subsequent data integration and standardization. The standard component data attribute table includes information such as attribute name, data type, value range, etc., which can be used to verify and format the input attribute information.

[0076] Exemplarily, according to the open standards of industrial standards or other BIM-related standards, a set of standard attributes of components is defined, including: basic attributes such as component identifier, name, type, etc., used to identify and distinguish components; geometric attributes such as size (length, width, height, etc.), coordinates, shape, etc., describing the geometric characteristics of components; physical attributes such as material, weight, strength, etc., describing the physical characteristics of components; functional attributes such as use (such as load-bearing walls, soundproof walls, etc.), performance (such as thermal insulation performance, fire resistance, etc.), etc., describing the functions and performance of components. Create a three-dimensional model database, establish indexes and links, associate related components and attribute information, and then export the information in the database into a data file that meets the standard according to the BIM standard (such as the IFC standard). Through the above steps, it is possible to accurately associate the identified attribute information with the corresponding components and generate a data file that meets the BIM standard, providing strong support for the information management and intelligent application of construction projects.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 Before step S22, the method for automatic conversion and fusion of data of the three-dimensional model component further includes steps S23 to S25: Step S23: determining the area of ​​the segmented region in the first component segmentation result.

[0078] It should be noted that there may be some areas in the first segmentation result that are too small or inaccurately segmented, which are caused by noise, edge effects or limitations of the segmentation algorithm itself. For segmented areas that are too small, if they are highly similar to neighboring areas in color and / or texture, they belong to the same component. Merging these similar areas into one segmented area can improve the continuity and integrity of the segmentation results and reduce segmentation fragments.

[0079] For example, in a three-dimensional model, a segmented region may represent a wall or a window. If the area of ​​a segmented region is very small, it may indicate that the segmentation is inaccurate or there is noise. All segmented regions in the segmentation result of the first component are traversed, and the area of ​​each region is calculated.

[0080] Step S24: if there is a segmented region whose area is smaller than a preset area threshold, determining the similarity between the segmented region and the adjacent region based on the color and / or texture between the segmented region and the adjacent region.

[0081] The area of ​​each segmented region is compared with the preset area threshold to obtain a list of all segmented regions whose area is smaller than the preset area threshold. For each small segmented region, its neighboring regions are determined, and the neighboring regions are other segmented regions that are spatially adjacent to the current segmented region. For each small segmented region and its neighboring regions, color and / or texture features are extracted.

[0082] Among them, color features can represent the color distribution of the region through statistical methods such as color histogram and color moment. Texture features can extract the texture information of the region through algorithms such as local binary pattern and gray level co-occurrence matrix. The color similarity or texture similarity between the segmented region and the adjacent region is calculated through Euclidean distance, cosine similarity, etc., or the color similarity and texture similarity are combined into a comprehensive similarity value using weighted average or other combination methods.

[0083] Step S25: if the similarity between the segmented region and the adjacent region is greater than a preset similarity threshold, the segmented region and the adjacent region are merged to obtain a new first component segmentation result.

[0084] If the color similarity and / or texture similarity between the segmented region and the adjacent region is greater than or equal to the similarity threshold, they are considered to belong to the same component, and the segmented region is merged with the adjacent region.

[0085] In this embodiment, by calculating the area of ​​each segmented region in the first component segmentation result, the region with too small area or inaccurate segmentation is identified. And by evaluating the similarity of these small segmented regions with the adjacent regions in color and / or texture, it is determined whether they belong to the same component. If they are determined to be the same component, they are merged to enhance the continuity and integrity of the segmentation result.

[0086] Based on the above embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 3 After step S30, the method for automatic conversion and fusion of data of the three-dimensional model component further includes steps S50 to S70: Step S50: Acquire the attribute modification information input by the user.

[0087] It should be noted that attribute correction information is the correction or supplementary information provided by the user in response to errors or inaccurate information in the attribute information output by the model. Attribute correction information is the user's feedback on the attribute information output by the model, and is intended to improve the accuracy and completeness of the attribute information.

[0088] Exemplarily, a user interaction interface is created to allow the user to input attribute modification information through the interface, keyboard, voice, etc., and the information includes multiple forms such as text, picture, voice, etc.

[0089] Step S60: input the attribute correction information and the attribute information into the semantic recognition model again.

[0090] Step S70, obtaining the new attribute information output by the semantic recognition model.

[0091] The attribute correction information input by the user and the original attribute information are input into the semantic recognition model together for a new round of recognition and processing. The semantic recognition model can re-parse and recognize the attributes based on the new input information and output more accurate and complete attribute information.

[0092] Optionally, the semantic recognition model can directly output new attribute information for the user to view or use. Or it can output the original attribute information and the new attribute information generated after modification for the user to compare and view. For the attribute information in the image or 3D model, the new recognition result can be displayed through visualization means such as annotation and highlighting.

[0093] In this embodiment, the user can provide corrections to the attribute information output by the model through various methods such as interface input, keyboard input or voice input. The attribute correction information input by these users is input into the semantic recognition model together with the original attribute information for a new round of recognition and processing. The semantic recognition model can re-parse and identify the attributes of the component based on this new input information, and output more accurate and complete attribute information. The above steps enable the semantic recognition model to better adapt to different usage scenarios and user needs, and improve the accuracy of the output of the semantic recognition model.

[0094] For example, in order to help understand the implementation process of the method for automatically converting and fusing data of a three-dimensional model component obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 4 , Figure 4 A brief flow chart of a method for automatic data conversion and fusion of three-dimensional model components is provided, specifically: A 3D model is input, and the quality check and preprocessing of the input 3D model are performed to ensure the accuracy and availability of the model data. The preprocessed model is converted into image data to prepare for the subsequent steps. A specific view angle is selected and the image data is converted to generate a 2D view of the 3D model with multiple views. The generated 2D view will be used for subsequent segmentation and recognition. The 2D view is segmented using the SAM model to obtain the first component segmentation result. Then, the NeRF model is used to determine the spatial geometric features of the component. The first component segmentation result and the spatial geometric features of the component are combined to obtain a more accurate second component segmentation result. The second component segmentation result is mapped back to the 3D model to obtain a recognizable standard 3D model component. The initial attribute information is extracted from the segmented component, and the initial attribute information is analyzed using the semantic recognition model to obtain the first attribute information. The first attribute information is then combined with the image data of the 3D model for further analysis to obtain the second attribute information. The second attribute information is corrected and mapped to conform to the attributes of the BIM standard data structure. Combined with the standard component data attribute table, the attribute information is associated with the component to generate the standard data of the 3D model component. Among them, the segmented standard 3D model component can be reorganized as a new 3D model for re-segmentation. The attribute information output by the semantic recognition model can also be repeatedly input into the semantic recognition model for repeated analysis to improve the accuracy of attribute information recognition.

[0095] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for automatic data conversion and fusion of a three-dimensional model component of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0096] The present application also provides a device for automatic data conversion and fusion of three-dimensional model components, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for automatic data conversion and fusion of three-dimensional model components in the above-mentioned embodiment one.

[0097] Reference below Figure 5 , which shows a schematic diagram of a structure of a data automatic conversion and fusion device for a three-dimensional model component suitable for implementing the embodiment of the present application. The data automatic conversion and fusion device for the three-dimensional model component in the embodiment of the present application may include but is not limited to mobile terminals such as laptop computers, tablet computers (PAD, Portable Application Description), etc., and fixed terminals such as desktop computers. Figure 5The data automatic conversion and fusion device for a three-dimensional model component shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0098] like Figure 5 As shown, the data automatic conversion and fusion device of the three-dimensional model component may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM) 1004. In the random access memory 1004, various programs and data required for the operation of the data automatic conversion and fusion device of the three-dimensional model component are also stored. The processing device 1001, the read-only memory 1002 and the random access memory 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the data automatic conversion and fusion device of the three-dimensional model component to communicate wirelessly or wired with other devices to exchange data. Although the figure shows the data automatic conversion and fusion device of the three-dimensional model component with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0099] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0100] The data automatic conversion and fusion device of the three-dimensional model component provided by the present application adopts the data automatic conversion and fusion method of the three-dimensional model component in the above embodiment, which can solve the technical problem of low accuracy of standard data conversion of the three-dimensional model component. Compared with the prior art, the beneficial effects of the data automatic conversion and fusion device of the three-dimensional model component provided by the present application are the same as the beneficial effects of the data automatic conversion and fusion method of the three-dimensional model component provided by the above embodiment, and the other technical features of the data automatic conversion and fusion device of the three-dimensional model component are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0101] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0102] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0103] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the method for automatic data conversion and fusion of three-dimensional model components in the above-mentioned embodiment.

[0104] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequencies (RF, Radio Frequency), etc., or any suitable combination of the above.

[0105] The computer-readable storage medium may be included in the data automatic conversion and fusion device of the three-dimensional model component; or it may exist independently without being assembled into the data automatic conversion and fusion device of the three-dimensional model component.

[0106] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the data automatic conversion and fusion device of the three-dimensional model component, the data automatic conversion and fusion device of the three-dimensional model component can be written in one or more programming languages ​​or a combination thereof to write computer program codes for performing the operations of the present application. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, or as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0109] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned method for automatic data conversion and fusion of three-dimensional model components, and can solve the technical problem of low accuracy of standard data conversion of three-dimensional model components. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the method for automatic data conversion and fusion of three-dimensional model components provided by the above-mentioned embodiment, and will not be repeated here.

[0110] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for automatic conversion and fusion of data of a three-dimensional model component, characterized in that: The method for automatic conversion and fusion of data of the three-dimensional model component comprises: Based on the image data of the three-dimensional model, extracting a two-dimensional image of the three-dimensional model through the SAM model, and segmenting the two-dimensional image to obtain a first component segmentation result; Based on the image data of the three-dimensional model and the first component segmentation result, determine the spatial geometric features of the component in the three-dimensional model through the NeRF model to obtain a second component segmentation result; Mapping the second component segmentation result to the three-dimensional model, segmenting the three-dimensional model to obtain the component, and determining the attribute information of the component through a semantic recognition model; Based on the component and the attribute information, standard data of the component is generated.

2. The method for automatic data conversion and fusion of a three-dimensional model component according to claim 1, characterized in that: The step of extracting a two-dimensional image of the three-dimensional model based on the image data of the three-dimensional model through the SAM model and segmenting the two-dimensional image to obtain a first component segmentation result includes: Extracting the image data of the three-dimensional model frame by frame to obtain the two-dimensional images at different viewing angles; Based on the geometric features of the two-dimensional image, segment the two-dimensional image into at least one segmented area, where the segmented area corresponds to a component in the three-dimensional model; Determining a spatial position of the segmented region based on a position of the segmented region in the two-dimensional image at different viewing angles; The first component segmentation result is determined based on the segmented area and the spatial position of the segmented area.

3. The method for automatic data conversion and fusion of a three-dimensional model component according to claim 1, characterized in that: The step of obtaining a second component segmentation result by determining the spatial geometric features of the component in the three-dimensional model through a NeRF model based on the image data of the three-dimensional model and the first component segmentation result comprises: Determining spatial geometric features of the component in the three-dimensional model based on the image data of the three-dimensional model; The second component segmentation result is determined based on the first component segmentation result and the spatial geometric features of the component in the three-dimensional model.

4. The method for automatic conversion and fusion of data of a three-dimensional model component according to claim 3, characterized in that: Before the step of determining the second component segmentation result based on the first component segmentation result and the spatial geometric features of the component in the three-dimensional model, the step further includes: Determining the area of ​​the segmented region in the first component segmentation result; If there is an area of ​​the segmented region that is smaller than a preset area threshold, determining the similarity between the segmented region and the adjacent region based on the color and / or texture between the segmented region and the adjacent region; If the similarity between the segmented region and the adjacent region is greater than a preset similarity threshold, the segmented region and the adjacent region are merged to obtain a new first component segmentation result.

5. The method for automatic conversion and fusion of data of a three-dimensional model component according to claim 1, characterized in that: The attribute information includes first attribute information, and the step of determining the attribute information of the component by using a semantic recognition model includes: Determining initial attribute features of the component based on the segmentation results of the component and the second component; Inputting the initial attribute features into the semantic recognition model; The first attribute information output by the semantic recognition model based on the initial attribute feature is obtained, where the first attribute information includes type information and function information of the component.

6. The method for automatic conversion and fusion of data of a three-dimensional model component according to claim 5, characterized in that: The attribute information includes second attribute information, and after the step of obtaining the first attribute information output by the semantic recognition model based on the initial attribute feature, the step further includes: Inputting the first attribute information and the image data of the three-dimensional model into the semantic recognition model; The second attribute information of the component is determined by the semantic recognition model, where the second attribute information includes type information, function information, material information and manufacturing information of the component.

7. The method for automatic data conversion and fusion of a three-dimensional model component according to claim 1, characterized in that: After the step of determining the attribute information of the component by the semantic recognition model, the method further includes: Get the attribute modification information entered by the user; inputting the attribute correction information and the attribute information into the semantic recognition model again; The new attribute information output by the semantic recognition model is obtained.

8. The method for automatic conversion and fusion of data of a three-dimensional model component according to claim 1, characterized in that: The step of generating standard data of the component based on the component and the attribute information comprises: Get the standard component data attribute table; Based on the standard component data attribute table, the attribute information is associated with the component and saved to generate standard data of the component.

Images