A BIM object query system and method for image segmentation and semantic recognition

Abstract

The application discloses a BIM object query system for image segmentation and semantic recognition, which comprises an image segmentation module, a semantic recognition module, a coordinate mapping module and an interactive query output module; the image segmentation module is used for obtaining a building real scene image, pre-processing and segmenting the building real scene image, and extracting a building component region and coordinates; the semantic recognition module is used for performing semantic recognition on a segmentation result, and obtaining a component category and attributes; the coordinate mapping module is used for establishing a mapping relationship between image coordinates and BIM model space coordinates; and the interactive query output module is used for querying BIM object attributes according to user interaction and outputting a visual result. The application solves the problems of low BIM object query efficiency and difficulty in associating real scene images with models, and significantly improves the query efficiency and visual interaction.

Description

Technical Field

[0001] This invention relates to the field of building information modeling technology, specifically to a BIM object query system and method for image segmentation and semantic recognition. Background Technology

[0002] With the widespread application of Building Information Modeling (BIM) technology in the design, construction, and operation and maintenance phases of buildings, the level of informatization in the construction industry has been significantly improved. BIM models can not only provide three-dimensional geometric information of buildings, but also integrate multi-dimensional information such as material, component properties, and construction progress, and are gradually becoming an important tool for the whole life cycle management of buildings.

[0003] However, in practical applications, existing BIM object query and interaction methods still have many shortcomings. On the one hand, if users need to query the attributes of a certain object in BIM, they usually need to use professional software to locate it layer by layer in the model or rely on keyword searches. This is not only complex to operate but also inefficient, with limited value for on-site applications. On the other hand, with the development of technologies such as drone aerial photography, laser scanning, and mobile terminal photography, a large number of real-world images have been generated at construction sites. However, there is no direct correspondence between these images and the BIM model, making it difficult for on-site personnel to quickly locate model objects through images. In addition, although some studies have attempted to use image processing methods to identify components in real-world images, traditional methods rely on geometric features or manual annotation, which is difficult to cope with complex and diverse construction environments. Furthermore, the progress of deep learning in image segmentation and classification has not yet been fully integrated with the semantic attributes of BIM objects, making it difficult to transform image recognition results into effective model query information. Summary of the Invention

[0004] The purpose of this invention is to provide a BIM object query system and method based on image segmentation and semantic recognition. This invention combines real-scene image segmentation and semantic recognition technologies to achieve an efficient query method that automatically associates images with BIM model objects, thereby improving the convenience and practicality of building information management.

[0005] To achieve this objective, the present invention provides a BIM object query system based on image segmentation and semantic recognition, comprising: The image segmentation module is used to acquire real-world building images, preprocess the real-world building images to obtain standardized images, perform target segmentation on the standardized images, extract each building component, the area where each building component is located, and the two-dimensional coordinate information of the area where each building component is located from the real-world building images, and generate segmentation masks for each building component. The semantic recognition module is used to perform semantic recognition on the segmentation mask of each building component through the semantic recognition model, and to obtain the category, attributes and bounding box coordinate information of the area where each building component is located. The coordinate mapping module is used to establish the mapping relationship between the two-dimensional coordinate information of the area where each building component is located and the spatial coordinates of the BIM model, and to generate mapping relationship data; The interactive query output module is used to query the corresponding BIM object attribute information in the BIM model space based on the target area selected by the user, mapping relationship data, and the category, attributes and boundary coordinate information of the area where each building component is located; based on the BIM object attribute information, the module queries the BIM database to obtain a visualized real-world image of the building with semantic tags and boundary annotations.

[0006] Preferably, the building scene image is preprocessed. The preprocessing steps include reading the building scene image to obtain the coordinate points of the building scene image; determining whether the size of the building scene image is consistent with the size of the mask; if they are inconsistent, linear transformation is performed on the coordinate points of all building scene images according to the size ratio of the building scene image and the mask, and the linearly transformed coordinate points are mapped to the spatial coordinate system of the mask so that the size of the building scene image is aligned with the size of the mask, thus obtaining a standardized image.

[0007] Preferably, the image segmentation module supports at least one of the following: panoramic segmentation mode, point prompt segmentation mode, box prompt segmentation mode, or text prompt segmentation mode.

[0008] Among them, the panoramic segmentation mode automatically detects and segments all identifiable objects in the image using a deep learning image segmentation model; The dot-hint segmentation mode generates a target segmentation mask by receiving user click coordinates, foreground labels, or background labels, based on a mask area sorting and positive / negative dot hint fusion strategy. The box suggestion segmentation mode selects the optimal matching mask by calculating the IoU value between the user input bounding box and the candidate mask; The text prompt segmentation mode calculates the similarity between the user-input text description and the cropped image region generated by the preliminary segmentation model by using a visual-language model pre-trained based on contrastive learning, and determines the final segmentation result by combining a mask overlap ratio expansion strategy.

[0009] The preferred method for the dot-hint segmentation pattern is as follows: Based on the user-selected target area and the size ratio of the building image to the BIM model mask, a linear transformation is performed to the spatial coordinate system of the mask. All candidate segmentation masks generated by the initial segmentation model are obtained and sorted in descending order by area. All candidate masks are traversed. If a mask covers a point marked as foreground by the user, the entire building image pixel area of ​​that mask is marked as selected. If a mask covers a point marked as background by the user, the entire building image pixel area of ​​that mask is marked as excluded. All selected and not excluded mask areas are merged to form a precise single object mask as the final result.

[0010] The preferred method for establishing the mapping relationship between the two-dimensional coordinate information of the area where each building component is located and the spatial coordinates of the BIM model is as follows: After reconstructing the two-dimensional coordinate information of the area where each building component is located in the real building image, a three-dimensional scene is obtained. The reconstructed three-dimensional scene is aligned with the BIM model in space through spatial registration and geometric alignment. Then, the correspondence between specific objects of building components in the reconstructed three-dimensional scene is found from the BIM model through a dual constraint matching strategy of semantics and space.

[0011] Preferably, the specific method for querying the corresponding BIM object attribute information in the BIM model space based on the user-selected target area, mapping relationship data, and the category, attributes, and boundary coordinate information of each building component's area is as follows: On the interactive interface provided by the system, users can select one or more target building components from a real-world image with semantic labels and bounding boxes using a set selection method. The system captures the user's interaction and immediately matches it against the category, attribute, and bounding box coordinate information of the area where each building component is located. It then searches for the globally unique identifier of the corresponding BIM model object through mapping relationship data and obtains the BIM object attribute information using the globally unique identifier of the obtained BIM model object.

[0012] Preferably, the specific method for obtaining a visualized building reality image with semantic tags and boundary annotations by searching and matching in the BIM database based on the BIM object attribute information is as follows: Based on the BIM object attribute information, the corresponding BIM object is found in the BIM database, and the globally unique identifier of the corresponding BIM object is obtained. Based on the globally unique identifier, a forward query is performed through a bidirectional index table generated by the mapping relationship data to find one or more building components associated with the attribute information of the BIM object. The corresponding building scene image is retrieved according to the found one or more building components. Based on the category, attribute and bounding box coordinate information of the area where the building component is located, a highlight box, mask and semantic label are drawn on the area where the corresponding building component is located on the building scene image to obtain a visualized building scene image with semantic labels and boundary annotations.

[0013] Preferably, it also includes a visualization module, which is used to receive the segmentation mask of each building component from the image segmentation module and the category, attribute and bounding box coordinate information of the area where each building component is located from the semantic recognition module, standardize the format of the segmentation mask, and obtain the height and width of the building real scene image; Depending on the hardware system load and configuration, select to use CPU or GPU for rendering calculations, randomly generate different RGB colors for the segmentation mask of each building component, or specify fixed colors for specific categories of components, set the transparency of the segmentation mask, and sort all segmentation masks by area. Apply the set colors and transparency to the corresponding segmentation mask to generate a mask layer; draw white outlines at the edges of each segmentation mask; based on the height and width of the original image, overlay the generated mask layer and white outlines with the architectural real-world image to obtain a visualization result.

[0014] The beneficial effects of this invention: This invention proposes a BIM object query system and method for image segmentation and semantic recognition, which can effectively solve the problems of lack of direct correspondence between real-world images and BIM models, insufficient accuracy of object recognition, and low query interaction efficiency in the prior art.

[0015] By employing a deep learning image segmentation model to perform multi-modal target segmentation on real-world building images, the system accurately extracts building component regions and their two-dimensional coordinate information. Combined with a semantic recognition model, it automatically identifies the category, attributes, and bounding box coordinates of the components, achieving semantic-level association between building components in real-world images and BIM model objects. This effectively solves the problems of poor robustness and low efficiency caused by traditional methods relying on manual annotation or geometric features, and significantly improves the automation level and accuracy of object query.

[0016] By establishing a mapping relationship between image coordinates and BIM model spatial coordinates, and adopting a dual-constraint matching strategy, accurate matching and bidirectional indexing of real-world components and BIM objects were achieved, avoiding the tedious operation of manually selecting control points. At the same time, the spatial deviation was controlled within 5cm through RMS error assessment, ensuring mapping accuracy and reliability, and providing a solid spatial data foundation for subsequent queries.

[0017] By designing an interactive query interface that supports multiple interaction methods such as point selection, box selection, and natural language input, and combining it with mask visualization technology, users can intuitively select target areas in real-world images and obtain the attribute information of corresponding BIM objects in real time. This significantly reduces the operating threshold of professional software and improves the convenience and visual interactivity of queries, making it especially suitable for non-professional user scenarios such as construction sites and operation and maintenance management.

[0018] Through a modular system architecture and asynchronous task processing mechanism, high-concurrency image processing and resource optimization are achieved. At the same time, combined with security mechanisms such as file upload verification and temporary storage management, the system's stability and efficiency in processing large-scale real-scene images are guaranteed, providing technical support for BIM full lifecycle management.

[0019] This invention effectively solves problems such as the separation between real-scene images and BIM models, low query efficiency, and poor interactivity by integrating multiple technologies and modular design. It significantly improves the level of intelligence in building information management and has broad application value and promotion prospects in the fields of building construction, operation and maintenance management, and intelligent construction. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 A BIM object query system based on image segmentation and semantic recognition, such as Figure 1As shown, it includes: The image segmentation module is used to acquire real-world building images, preprocess the real-world building images to obtain standardized images, perform target segmentation on the standardized images, extract each building component, the area where each building component is located, and the two-dimensional coordinate information of the area where each building component is located from the real-world building images, and generate segmentation masks for each building component. The semantic recognition module is used to perform semantic recognition on the segmentation mask of each building component through the semantic recognition model, and to obtain the category, attributes and bounding box coordinate information of the area where each building component is located. The coordinate mapping module is used to establish the mapping relationship between the two-dimensional coordinate information of the area where each building component is located and the spatial coordinates of the BIM model, and to generate mapping relationship data; The interactive query output module is used to query the corresponding BIM object attribute information in the BIM model space based on the target area selected by the user, mapping relationship data, and the category, attributes and boundary coordinate information of the area where each building component is located; based on the BIM object attribute information, the module queries the BIM database to obtain a visualized real-world image of the building with semantic tags and boundary annotations.

[0023] By defining the overall architecture of the system, including an image segmentation module, a semantic recognition module, a coordinate mapping module, and an interactive query output module, the complex BIM query process is decomposed into collaborative subsystems. This achieves separation of responsibilities and efficient processing, enabling the system to process different tasks in parallel, reducing coupling, improving overall query speed and reliability, while facilitating maintenance and expansion, lowering the professional threshold, and enhancing interactivity.

[0024] In some preferred embodiments, the building scene image is preprocessed. The preprocessing steps include reading the building scene image to obtain the coordinate points of the building scene image; determining whether the size of the building scene image is consistent with the size of the mask; if they are inconsistent, performing a linear transformation on the coordinate points of all building scene images according to the size ratio of the building scene image to the mask, and mapping the linearly transformed coordinate points to the spatial coordinate system of the mask, so that the size of the building scene image is aligned with the size of the mask, thus obtaining a standardized image.

[0025] Regarding the mask size, in some preferred embodiments, the mask size refers to the standard resolution of the segmentation result (i.e., the mask) generated by the deep learning model's internal processing during image segmentation. The mask is a two-dimensional array, where the value of each pixel indicates which target the pixel belongs to or whether it is selected. The mask size is obtained through image normalization. The specific steps are as follows: Input a real-world architectural image with its original size, such as 4000x3000 pixels. During image normalization, in order to make the image conform to the input requirements of the deep learning model, the image is scaled proportionally to ensure that all input images have the same size. The scaled image is fed into the image segmentation model, which performs inference on the scaled image, calculates and outputs the segmentation result. The mask size output by the model is determined by the model's input size.

[0026] In some preferred embodiments, a preprocessing step is used to linearly transform the coordinate points to ensure that the image size is aligned with the mask size. This helps to standardize the data, avoid positioning errors caused by resolution differences, enhance the robustness and accuracy of the system, ensure the consistency of segmentation and recognition blocks when processing real-world images from different sources, reduce calculation deviations in subsequent steps, thereby improving the accuracy of BIM object queries and reliably handling diverse image inputs for complex construction environments.

[0027] In some preferred embodiments, the image segmentation module supports at least one of panoramic segmentation mode, point hint segmentation mode, box hint segmentation mode, or text hint segmentation mode.

[0028] Among them, the panoramic segmentation mode automatically detects and segments all identifiable objects in the image using a deep learning image segmentation model; The dot-hint segmentation mode generates a target segmentation mask by receiving user click coordinates, foreground labels, or background labels, based on a mask area sorting and positive / negative dot hint fusion strategy. The box suggestion segmentation mode selects the optimal matching mask by calculating the IoU value between the user input bounding box and the candidate mask; The text prompt segmentation mode calculates the similarity between the user-input text description and the cropped image region generated by the preliminary segmentation model by using a visual-language model pre-trained based on contrastive learning, and determines the final segmentation result by combining a mask overlap ratio expansion strategy.

[0029] In some preferred embodiments of deep learning image segmentation models, pixel-level classification is performed, that is, for each pixel in the image, it is determined which specific category or object the pixel belongs to.

[0030] For contrastive learning pre-trained visual-language models, in some preferred embodiments, the contrastive learning pre-trained visual-language model is a multimodal large model. Through contrastive learning pre-training, it understands the relationship between images and text. This model is used to learn to bring the feature vectors of matching images and text closer in the semantic space and push away mismatched ones, establishing a common understanding space. This enables the model to understand image content and describe it with text, and it can also understand text instructions and find the corresponding content in the image.

[0031] In some preferred embodiments, by specifying that the image segmentation module supports multiple segmentation modes, a more flexible interaction method is provided to adapt to different user scenarios and needs, which significantly improves the adaptability and ease of use of the system, simplifies the operation process, reduces manual intervention, and improves query efficiency.

[0032] In some preferred embodiments, the specific method for the dot-hint segmentation pattern is as follows: Based on the user-selected target area and the size ratio of the building image to the BIM model mask, a linear transformation is performed to the spatial coordinate system of the mask. All candidate segmentation masks generated by the initial segmentation model are obtained and sorted in descending order by area. All candidate masks are traversed. If a mask covers a point marked as foreground by the user, the entire building image pixel area of ​​that mask is marked as selected. If a mask covers a point marked as background by the user, the entire building image pixel area of ​​that mask is marked as excluded. All selected and not excluded mask areas are merged to form a precise single object mask as the final result.

[0033] The specific method for using the pop-up tooltip split mode is as follows: Based on the user-drawn bounding box coordinates, and according to the size ratio between the actual building image and the internal mask, the image is linearly transformed to the spatial coordinate system of the mask; all candidate segmentation masks generated by the preliminary segmentation model are obtained; the intersection-over-union (IoU) value between each candidate mask and the user-input bounding box is calculated; and the candidate mask with the largest IoU value is selected as the final segmentation result.

[0034] The specific method for text prompt segmentation mode is as follows: All candidate segmentation masks generated by the initial segmentation model are obtained, and corresponding image slices are cropped from the architectural scene image based on the minimum bounding rectangle of each mask. The user-input natural language description and all image slices are encoded into feature vectors using a pre-trained multimodal model, and the similarity score between them is calculated. The image slice with the highest similarity score to the text description is retrieved, and its index in the original candidate mask list is traced. The mask corresponding to the index is determined as the initial matching result. In an optional embodiment, based on the initial matching mask, it is searched whether there is a mask with a larger area and an overlap exceeding a set threshold. If so, the mask with the larger area is updated as the final result to obtain a semantically more complete object.

[0035] The specific method for text prompt segmentation mode is as follows: For the standardized image, a deep learning segmentation model is used to perform automated object detection and segmentation, which identifies and segments all identifiable objects in the image in one go, and generates a corresponding segmentation mask for each object.

[0036] In some preferred embodiments, by refining the specific methods of the point hint segmentation mode, including mask area sorting and positive and negative point hint fusion, the ambiguity problem in target selection is solved, the accuracy of segmentation and the matching degree of user intent are improved, misidentification of local fragments is avoided, and accurate masks can still be generated in overlapping component scenarios, thereby enhancing the reliability of BIM object query and reducing the risk of misoperation.

[0037] In some preferred embodiments, the specific method for establishing the mapping relationship between the two-dimensional coordinate information of the area where each building component is located and the spatial coordinates of the BIM model is as follows: After reconstructing the two-dimensional coordinate information of the area where each building component is located in the real building image, a three-dimensional scene is obtained. The reconstructed three-dimensional scene is aligned with the BIM model in space through spatial registration and geometric alignment. Then, the correspondence between specific objects of building components in the reconstructed three-dimensional scene is found from the BIM model through a dual constraint matching strategy of semantics and space.

[0038] In some preferred embodiments, the specific method for finding the correspondence between specific objects of building components in the reconstructed 3D scene from the BIM model through a dual constraint matching strategy of semantics and space is as follows: A spatial index structure is built for all objects to be queried in the BIM model. The geometric center point of the target building component identified from the real 3D scene or the center of its 3D bounding box is used as the reference point. With the reference point as the center and a preset threshold distance as the radius, all BIM objects falling within the spherical space range are quickly retrieved from the previously built spatial index. The retrieved BIM objects constitute a spatial candidate object set. Semantic features are extracted from the semantic recognition results of each building component. Cosine similarity and other measurement methods are used to calculate the semantic similarity score between the feature vector of the real-world component and the feature vector of each spatial candidate BIM object. A semantic similarity threshold is set. When the similarity score between a BIM candidate object and the real-world component is greater than the similarity threshold, it is considered to have passed the semantic constraint. Traverse the set of candidate objects in space, find BIM objects that simultaneously satisfy the conditions that the spatial distance between the real-world component and the real-world component is less than a preset threshold distance, and that the similarity score between the BIM candidate object and the real-world component is greater than the similarity threshold, and select the BIM object with the highest semantic similarity as the final matching target.

[0039] In some preferred embodiments, spatial registration and geometric alignment are used to associate the two-dimensional coordinates with the spatial coordinates of the BIM model, thereby achieving seamless integration of the real scene and BIM, ensuring automatic alignment of components in the image with BIM objects, and making the query results have practical application value.

[0040] In some preferred embodiments, the specific method for querying the corresponding BIM object attribute information in the BIM model space based on the user-selected target area, mapping relationship data, and the category, attributes, and boundary coordinate information of the area where each building component is located is as follows: On the interactive interface provided by the system, users can select one or more target building components from a real-world image with semantic labels and bounding boxes using a set selection method. The system captures the user's interaction and immediately matches it against the category, attribute, and bounding box coordinate information of the area where each building component is located. It then searches for the globally unique identifier of the corresponding BIM model object through mapping relationship data and obtains the BIM object attribute information using the globally unique identifier of the obtained BIM model object.

[0041] In some preferred embodiments, by defining specific methods for interactive queries, including allowing users to select target areas through the interface and match BIM object attributes, the user operation process is simplified, enabling instant feedback and efficient retrieval. This reduces the time consumption of traditional layer-by-layer retrieval, improves user experience, and is suitable for on-site personnel to quickly obtain component attributes, thereby lowering the professional threshold.

[0042] In some preferred embodiments, the specific method for obtaining visualized building reality images with semantic tags and boundary annotations by searching and matching in the BIM database based on the BIM object attribute information is as follows: Based on the BIM object attribute information, the corresponding BIM object is found in the BIM database, and the globally unique identifier of the corresponding BIM object is obtained. Based on the globally unique identifier, a forward query is performed through a bidirectional index table generated by the mapping relationship data to find one or more building components associated with the attribute information of the BIM object. The corresponding building scene image is retrieved according to the found one or more building components. Based on the category, attribute and bounding box coordinate information of the area where the building component is located, a highlight box, mask and semantic label are drawn on the area where the corresponding building component is located on the building scene image to obtain a visualized building scene image with semantic labels and boundary annotations.

[0043] In some preferred embodiments, the visualization of the results is enhanced by generating visual images, including drawing highlight boxes, masks, and semantic labels. This provides intuitive visual feedback, makes the query results easy to understand, supports users in quickly verifying and making decisions, improves the system's usability and interactivity, and ensures the close integration of BIM information with real-world images.

[0044] In some preferred embodiments, it also includes a visualization module for receiving the segmentation mask of each building component from the image segmentation module and the category, attribute and bounding box coordinate information of the area where each building component is located from the semantic recognition module, standardizing the format of the segmentation mask, and obtaining the height and width of the building real scene image. Depending on the hardware system load and configuration, select to use CPU or GPU for rendering calculations, randomly generate different RGB colors for the segmentation mask of each building component, or specify a fixed color for a specific category of components, set the transparency of the segmentation mask, and sort all segmentation masks from largest to smallest area. Apply the set colors and transparency to the corresponding segmentation mask to generate a mask layer; draw white outlines at the edges of each segmentation mask; based on the height and width of the original image, overlay the generated mask layer and white outlines with the architectural real-world image to obtain a visualization result.

[0045] In some preferred embodiments, by introducing a visualization module, rendering performance and visual effects are optimized, the system's ability to process large-scale images is improved, and different components are distinguished by random or fixed colors, enhancing the clarity of multi-object display, thereby supporting concurrent processing and resource optimization, and is suitable for high-resolution image scenarios.

[0046] In some preferred embodiments, the interactive query output module outputs real-world images of buildings with semantic tags and boundary labels, and at the same time, outputs corresponding structured query results. The structured query results include the category, attributes, and bounding box coordinate information of the building component area. The structured query results refer to machine-readable data for use by computers or other systems. They are well-defined, standardized structures that facilitate computer retrieval and querying.

[0047] In an optional implementation, the system framework consists of two core modules: an image segmentation module and a semantic recognition module. Working together, they achieve intelligent segmentation of real-scene images, semantic recognition, and automatic association and querying of BIM objects. Specifically, they include: 1 Image segmentation methods 1.1 Implementation of Segmentation Based on Point Cueing This method solves the ambiguity problem of target selection by combining mask area sorting with positive and negative point hints. Even if the point selection is not precise enough, the algorithm logic can infer the complete object that best matches its intention.

[0048] 1.1.1 Coordinate Space Normalization The system first receives a set of original image coordinate points (points) and their corresponding labels (point_label), where 1 represents foreground and 0 represents background. To align with the mask size processed by the internal model, the system checks if the original image size matches the mask size. If they do not match, the system performs a linear transformation on all input point coordinates according to the size ratio, mapping them to the spatial coordinate system of the mask. Example logic is shown in the following code: if h != target_height or w != target_width: points = [ [int(point[0] * w / target_width), int(point[1] * h / target_height)] for point in points ] 1.1.2 Mask Priority Sorting To handle overlapping objects, the system first obtains a set of candidate segmentation masks generated by preliminary segmentation models (YOLO, FastSAM). A crucial step is that the system sorts these masks in descending order based on their area. This area-based priority sorting ensures that when a point falls within an overlapping region, the system prioritizes larger objects with more complete boundaries that contain that point, thus avoiding misidentifying the main body as local fragments. Example logic is shown in the following code: masks = sorted(masks, key=lambda x: x['area'], reverse=True) 1.1.3 Point-based mask filtering and fusion The system creates a blank canvas of the same size as the mask, and then iterates through all the sorted candidate masks. For each mask, it checks all user input points.

[0049] Positive selection: If a mask covers a foreground point (`point_label[i] == 1`), then all the pixels occupied by the mask are marked as 1 (i.e. "selected") on the segmentation mask.

[0050] Negative exclusion: If a mask covers a background point (`point_label[i] == 0`), then the entire pixel area occupied by that mask is forcibly marked as 0 on the segmentation mask (i.e., "excluded"). Since the masks are already sorted by area, this exclusion operation can effectively "cut out" unwanted small parts from a large selected area. Example logic is shown in the following code: #Blank Canvas onemask = np.zeros((h, w)) # Traverse all sorted candidate masks for i, annotation in enumerate(masks): if type(annotation) == dict: mask = annotation['segmentation'] else: mask = annotation # Check if each user input point falls within the current mask for i, point in enumerate(points): if mask[point[1], point[0]] == 1 and point_label[i] == 1: onemask[mask] = 1 if mask[point[1], point[0]] == 1 and point_label[i] == 0: onemask[mask] = 0 1.1.4 Generating the final mask After the traversal is complete, all regions with values ​​greater than or equal to 1 on the segmentation mask are considered the final segmentation result. This forms a precise single-object mask that incorporates all positive selections and excludes all negative selections.

[0051] 1.2 Segmentation based on bounding box cues This embodiment describes a method that allows users to quickly select a target by drawing a rough rectangle.

[0052] 1.2.1 Coordinate Space Normalization Similar to dot hints, the system receives a bounding box (bbox [x1, y1, x2, y2]) drawn by the user. The system also checks and transforms the bounding box's coordinates from the original image space to the mask space according to the size ratio. Simultaneously, the system performs boundary checks on the transformed coordinates to ensure they do not exceed the size range of the mask.

[0053] 1.2.2 IoU (Intersection over Union) Calculation and Best Match The core of this method is to calculate the IoU between the user-input bounding box and all candidate masks.

[0054] The system first calculates the area bbox_area of ​​the region selected by the user.

[0055] Then, for each candidate mask, the system efficiently calculates the area of ​​the region where it intersects with the bbox (masks_area, i.e. the part of the mask inside the bbox) and the total area of ​​the mask itself (orig_masks_area).

[0056] According to the formula IoU = Intersection / Union, where `Union = bbox_area + orig_masks_area - masks_area`, the system calculates an IoU score for each mask.

[0057] 1.2.3 Determine the final mask: The system finds the maximum value among all IoU scores and uses the candidate mask corresponding to the maximum value as the final segmentation result.

[0058] 1.3 Text-based segmentation This embodiment describes a method that allows users to segment targets by inputting natural language descriptions. It combines the capabilities of image segmentation with multimodal models such as CLIP, and the core logic is implemented by the text_prompt function.

[0059] 1.3.1 Candidate pruning The system first calls the `crop_image` function. This function iterates through all the initially segmented masks, calculates the minimum bounding rectangle of each mask, and then crops out individual image slices `cropped_boxes` containing only the object from the original image based on this rectangle. Example code logic is as follows: cropped_boxes, cropped_images, not_crop, origin_id, annotations_ =crop_image( annotations, img_path ) 1.3.2 Multimodal Feature Matching The system loads a pre-trained CLIP model and then calls the retriev function, which is responsible for encoding the user-input text and all cropped image slices into high-dimensional text feature vectors and image feature vectors, respectively.

[0060] By calculating the cosine similarity between the text feature vector and each image feature vector, the system obtains a score list, `scores`, which represents the degree of matching between each segmented object and the text description. Then, the image slice with the highest score is found, and its index `max_idx` in the original mask list is traced back. The mask corresponding to this index is the segmentation result that best matches the text. Example logic is shown in the code below: import clip clip_model, preprocess = clip.load("ViT-B / 32", device=device) # Call the retriev function to calculate the similarity score between each image slice and the input text.

[0061] scores = retriev( clip_model, preprocess, cropped_boxes, text, device=device ) # Sort the scores and find the index of the highest score. argsort() returns the indices in ascending order.

[0062] max_idx_local = scores.argsort()[-1] # Map the local index (max_idx_local) in the clipping list back to the global index (max_idx) in the original mask list.

[0063] max_idx = origin_id[int(max_idx_local)] 1.3.3 Extension of Related Areas To obtain a semantically more complete object (e.g., after identifying "wheels," the user might want the entire "car"), this method provides an extension option. It iterates through all masks with a larger area, starting with the best-matching mask `mask0`. If the overlap between a larger mask and `mask0` exceeds a threshold (e.g., 0.9), the system considers this larger mask to be the parent or main body of `mask0` and updates the final result with this larger mask. Example logic is shown in the code below: # Get the current best matching mask (mask0) and its area.

[0064] mask0 = annotations_[max_idx]["segmentation"] area0 = np.sum(mask0) Sort all candidate masks by area from largest to smallest.

[0065] areas = [(i, np.sum(mask["segmentation"])) for i, mask in enumerate(annotations_) if i in origin_id] areas = sorted(areas, key=lambda area: area[1], reverse=True) indices = [area[0] for area in areas] # Iterate through all masks that are greater than (or equal to) mask0.

[0066] for index in indices: # Calculate the intersection area of ​​the current traversed mask and mask0, and divide it by the area of ​​mask0 to obtain the overlap ratio.

[0067] overlap_ratio = np.sum(annotations_[index]["segmentation"] & mask0) / area0 # If the overlap ratio exceeds the threshold (e.g., 0.9), it is considered that a more complete subject has been found.

[0068] if index == max_idx or overlap_ratio > threshold: # Update the best index to this larger, more complete index of the subject mask.

[0069] max_idx = index break # Exit the loop immediately after finding the target.

[0070] # Returns the final optimal mask and its index.

[0071] return annotations_[max_idx]["segmentation"], max_idx 2. Semantic Recognition Methods 2.1 Object Recognition and Coordinate Extraction The main task of this part is to call the LMM (Low-Modity Matrix) to perform semantic recognition and spatial localization of objects in the image. The implementation process is as follows: 2.1.1 Image Encoding and Model Input Preparation The input image data is converted from OpenCV's BGR format to RGB format, then converted to a standard image object using PIL, and finally encoded into a Base64 string. This is a common image input format that the multimodal model can accept. Example logic is shown in the code below: resized_img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) pil_resized = PILImage.fromarray(resized_img_rgb) base64_image = image_array_to_base64(pil_resized, format='JPEG', isfr=True) 2.1.2 Constructing the message body The two components are: first, the encoded image data; and second, the specifically designed prompt word (prompt_detect_box). This prompt word guides the model to output object information in a structured manner, including the object name (label) and the coordinates of the 2D bounding box (bbox_2d). Example code is shown below: messages = [ { "role": "user", "content": [ {"image": base64_image}, {"text": prompt_detect_box} ] } ] 2.1.3 The message body is sent to the specified model (such as the Qwen-VL series) through the MultiModalConversation.call interface. The model analyzes the image content and returns the recognition result. The example logic is shown in the following code: response = MultiModalConversation.call( api_key=settings.qwen_config["api_key"], model=settings.qwen_config["model"], messages=messages, temperature=0, top_k=1, seed=3407, vl_high_resolution_images=True ) 2.2 Model Result Analysis and Structured Output This part uses the ObjectDetectionService.detect_objects() function to perform a semantic inventory of the entire image.

[0072] The system encodes the images and inputs them into the LMM. Using the statistical prompt word prompt_statistic, it guides the model to output a structured list of objects and quantity statistics, thereby generating a "semantic summary" that provides basic data for subsequent BIM object mapping.

[0073] Model outputs are typically returned as text and may contain non-standard formats such as Markdown wrappers or comments. The `_parse_objects_and_coordinates` function is responsible for parsing these outputs into directly usable structured data. This includes: cleaning up redundant characters, such as removing JSON block markers, comments, and invalid blank lines; attempting to parse it into JSON using `json.loads`; if that fails, using regular expressions to extract [...] or {...} fragments from the text for secondary parsing; iterating through the results and validating fields to ensure that each object contains a `name` (semantic label) and `coordinates` (coordinates) fields; and finally outputting a list of objects in a uniform format. Example logic is shown in the following code: [ {"name": "car", "coordinates": [120, 230, 450, 600]}, {"name": "person", "coordinates": [500, 300, 700, 850]} ] This process ensures that the unstructured text output by the model can be stably parsed into a programmable data structure by the system, which is a key aspect of module robustness.

[0074] 2.3 Coordinate Space Restoration During the recognition phase, the input image is typically scaled to fit the model input size.

[0075] To ensure that the final coordinates correspond to the original image, the module performs coordinate inversion during the output stage based on the scaling ratio, that is, the coordinate values ​​returned by the model are divided by the original scaling ratio.

[0076] Example logic is shown in the following code: if scale < 1.0: for obj in object_list: obj["coordinates"] = [coord / scale for coord in obj["coordinates"]] This restoration step ensures the consistency of semantic recognition results in the spatial dimension, making the labeled boxes accurately aligned with the original image.

[0077] 2.4 Semantic Result Annotation and Output After completing the recognition and parsing, the module calls the mark_image method to draw the object name and bounding box on the original image, thus realizing the visualization output of the result.

[0078] The main process includes: calling the mark_image_box utility function to draw bounding boxes and label text on the image; converting the generated labeled image into Base64 encoding so that it can be directly returned to the front-end system for display; and saving the results to a specified local directory with the filename containing a timestamp for easy traceability.

[0079] The output consists of three parts: marked_objects: a list of identified semantic objects; marked_image_base64: a Base64 string representing the semantically annotated image; and marked_image_path: the storage path of the annotated image. This function achieves a closed loop from semantic understanding to visualization.

[0080] 3. File upload and temporary file management mechanism It provides a complete process for security verification, storage, and temporary resource cleanup of uploaded files, ensuring that the system is stable, secure, and resource-controllable when processing user-uploaded files.

[0081] 3.1 File format and size verification When receiving uploaded files, the system will automatically detect the file format and size to prevent illegal or excessively large files from affecting the system.

[0082] 3.1.1 Extension Verification The `allowed_file` method extracts the filename extension and compares it with the system's default `ALLOWED_IMAGE_FORMATS` whitelist to ensure that only files of specific formats are allowed to be uploaded. Example code logic is as follows: def allowed_file(filename: str) -> bool: if not filename or '.' not in filename: return False extension = filename.rsplit('.', 1)[1].lower() return extension in settings.ALLOWED_IMAGE_FORMATS 3.1.2 Size Limitations After reading the file content, determine its byte length. If it exceeds the MAX_FILE_SIZE setting, immediately reject the upload request and return an HTTP 413 error code. Example logic is shown in the code below: content = await file.read() if len(content) > settings.MAX_FILE_SIZE: raise HTTPException(status_code=413, detail="File size exceeds limit") 3.2. Persistent File Storage 3.2.1 Asynchronous File Persistent Storage To improve concurrency performance, the system uses aiofiles for asynchronous writing and writes to disk block by block to avoid memory spikes caused by reading all at once. Example code is shown below: async with aiofiles.open(destination, 'wb') as f: while chunk := await file.read(8192): await f.write(chunk) Before writing, the system will automatically create the target directory: destination.parent.mkdir(parents=True, exist_ok=True) This logic implements idempotent directory creation, avoiding exceptions caused by non-existent directories during concurrent writes.

[0083] 3.2.2 Asynchronous Write to Storage The system supports asynchronous writing and storage of compliant files to improve concurrent processing efficiency and avoid blocking the main thread.

[0084] The storage process includes the following key steps: resetting the file pointer to ensure that the content can be read completely; using aiofiles asynchronous writing method to write to disk in blocks, reducing memory usage and improving performance; and returning the actual storage path of the file after a successful write.

[0085] 3.3 Temporary File Lifecycle Management To prevent the upload directory from growing indefinitely, a cleanup mechanism based on a time threshold is used. Example logic is shown in the code below: `cutoff_time = time.time() - (max_age_hours * 3600)` for file_path in directory.iterdir(): if file_path.is_file() and file_path.stat().st_mtime < cutoff_time: file_path.unlink() This mechanism determines expiration by comparing the file modification time (st_mtime) with a timestamp, implementing an automatic file recycling strategy without external task dependencies. Abnormal deletions are logged but do not interrupt the overall process, enhancing fault tolerance.

[0086] 4. Quick Visualization of Masks in Image Processing This invention proposes a fast mask visualization method, which mainly improves the processing efficiency of large-scale or high-resolution masks by supporting CPU and GPU rendering; enhances the visualization effect of the mask and the original image through color randomization, transparency control, and outline drawing; and supports flexible adjustment of parameters such as mask quality, outline thickness, and bounding box display to achieve customized visualization. Furthermore, it ensures the reasonable display of overlapping areas through area sorting and first non-zero index selection, thereby significantly improving the rendering efficiency of instance segmentation masks while ensuring visualization quality.

[0087] 4.1. Mask format standardization and image preprocessing 4.1.1 Standardized Mask Format The system receives raw data including real-scene images (denoted as "image") and a set of masks (denoted as "annotations"). The elements of the annotations exist in two formats: one is a two-dimensional array directly representing the mask region, and the other is a dictionary containing segmentation keys (keys and values ​​are mask data). To avoid format differences causing subsequent processing errors, format detection and unification must be performed first: first, determine the type of the first element of the annotations; if it is a dictionary, extract the segmentation keys of all elements and convert them to the standard two-dimensional array format; if it is a two-dimensional array, proceed directly to the next step, ensuring consistent data format throughout the entire process. Example logic is shown in the following code: if isinstance(annotations[0], dict): annotations = [annotation['segmentation'] for annotation inannotations] 4.1.2 Image Size Matching and Mask Quality Optimization: The system first obtains the original image dimensions `original_h` and `original_w` to ensure that the subsequent mask perfectly matches the pixel dimensions of the image, avoiding misalignment during overlay. It also provides two mask quality optimization modes—"high precision" and "high speed"—to balance accuracy and efficiency requirements in different scenarios. The high precision mode uses a combination of "closing operations + opening operations": first, a closing operation is performed using a 3×3 rectangular structural element to fill small holes with a diameter less than 3 pixels inside the mask, ensuring the integrity of the instance area; then, an opening operation is performed using an 8×8 rectangular structural element to remove isolated small noise points (such as single-pixel interference) at the edges, improving edge smoothness, suitable for scenarios involving the recognition of small BIM components. The high speed mode directly uses the original mask, skipping morphological operations to reduce computation time, suitable for rapid previewing of large-scale images. Example logic is shown in the following code: mask = cv2.morphologyEx(mask.astype(np.uint8), cv2.MORPH_CLOSE,np.ones((3,3),np.uint8)) mask = cv2.morphologyEx(mask.astype(np.uint8), cv2.MORPH_OPEN,np.ones((8,8),np.uint8)) True: Use the closing operation to remove small holes, and then use the opening operation to remove noise.

[0088] False: Use the original mask directly to improve processing speed.

[0089] 4.2 Mask Rendering Parameter Configuration This step achieves efficient rendering and differentiated display of masks by adapting to hardware devices, customizing display parameters, and handling overlapping areas, thus resolving issues such as stuttering and multi-instance confusion in large-scale masks. 4.2.1. Adaptive Rendering Device Selection The rendering mode switches based on the user-input device parameter ("cpu" or "gpu") to adapt to different data scales. CPU rendering calls the `fast_show_mask` function, requiring no GPU resources and suitable for small-scale mask processing such as single BIM components; GPU rendering calls the `fast_show_mask_gpu` function, utilizing parallel computing capabilities to improve the processing efficiency of large-scale masks such as entire building images and 4K high-resolution images, avoiding lag. Example code logic is shown below: The system selects the rendering method based on the device parameters: CPU rendering calls fast_show_mask; GPU rendering calls fast_show_mask_gpu. if device == 'cpu': inner_mask = fast_show_mask(...) else: inner_mask = fast_show_mask_gpu(...) 4.2.2. Customizing Mask Color and Transparency To differentiate between different instance masks, two modes are supported: "random color" and "fixed color," with a configurable transparency parameter. Random color generates independent RGB values ​​to avoid color repetition across multiple instances; fixed color is suitable for identifying specific semantic categories (such as uniformly labeling wall components). With transparency set to 0.6, color transparency assignment for the mask area is achieved through dimensional expansion and merging, ensuring the mask is clearly visible without obscuring details such as texture and dimension annotations of the original image. Example logic is shown in the following code: color = np.random.random((mask_sum,1,1,3)) # Random color (assigns independent RGB values ​​to each mask) color = np.ones((mask_sum,1,1,3)) * [0,0,1] # Example of blue # Transparency control: Fixed transparency at 0.6, merge color and transparency into four-channel data. transparency = np.ones((mask_sum,1,1,1)) * 0.6 # Dimensional Expansion: Matches the mask to four-channel data to achieve color-to-transparency assignment. mask_image = np.expand_dims(annotation, -1) * np.concatenate([color,transparency], axis=-1) 4.2.3. Overlapping Mask Sorting and Display Control For overlapping areas of multiple masks (such as "window-wall" overlap), a "area sorting + first non-zero index selection" approach is used to ensure reasonable display. First, the pixel area of ​​each mask is calculated and sorted in descending order of area (larger areas have higher priority to avoid being covered by smaller elements). Then, the index of the first non-zero mask at each pixel position (i.e., the mask with the highest priority) is obtained and assigned based on the index to avoid visual clutter in overlapping areas. Example code logic is shown below: areas = np.sum(annotation, axis=(1,2)) sorted_indices = np.argsort(areas)[::1] annotation = annotation[sorted_indices] index = (annotation != 0).argmax(axis=0) mask[h_indices, w_indices, :] = mask_image[indices] 4.3 Enhanced Mask Visualization and Image Overlay Output The system highlights mask boundaries through contour drawing and generates the final visualization result through layer overlay, providing clear visual support for BIM object query.

[0090] 4.3.1. Mask Outline Drawing Optional contour enhancement operations can be performed. First, extract the mask contour: use RETR_TREE mode to preserve the contour hierarchy (e.g., "walls contain windows"), and CHAIN_APPROX_SIMPLE mode to compress contour points (reducing computation). Then, draw a white contour on a transparent temporary layer, setting the line width to 2 / / scale (scale is the image scaling factor) to ensure consistent contour thickness across images of different resolutions, enhancing the distinction between the mask and the original image. Example logic is shown in the following code: contours, _ = cv2.findContours(annotation, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) cv2.drawContours(temp, contour_all, -1, (255,255,255), 2 / / scale) 4.3.2. Image Overlay and Output The "mask + outline" layer is overlaid with the original image to generate an RGBA format image. First, the mask layer is converted to an 8-bit unsigned integer array to generate an RGBA format overlay (preserving the alpha channel to ensure only the mask and outline areas are displayed). Then, with the top left corner of the image as the origin, the overlay is superimposed on the original image, using the overlay as the mask and retaining only the non-transparent areas. The final output RGBA image can be directly used in the query interface, allowing users to intuitively observe the correspondence between the mask and the image and quickly locate the target BIM component.

[0091] overlay_inner = Image.fromarray((inner_mask*255).astype(np.uint8), 'RGBA') image.paste(overlay_inner, (0,0), overlay_inner) 5. Service-oriented architecture and concurrent processing system 5.1 High-concurrency processing and resource management 5.1.1 Asynchronous Task Processing By employing a reasonable task splitting and asynchronous execution strategy, CPU-intensive image preprocessing, I / O-intensive AI inference, and memory-intensive post-processing are separated. Combined with intelligent task identification and optimized scheduling, it effectively avoids resource waste caused by I / O waiting while supporting large-scale requests. It also provides complete task status tracking and management, ensuring optimal system resource utilization and execution efficiency. Example code is shown below: async def get_marked_image(self, image_path: Path) -> Dict[str, Any]: image_path = Path(image_path).resolve() # Synchronous preprocessing (CPU intensive) img, scale, original_width, original_height = self.resize_image(image_path) # Asynchronous AI Inference (IO-intensive) object_list = await self._detect_objects_with_coordinates(img, scale) This code demonstrates an asynchronous image processing procedure.

[0092] 5.1.2. Equipment Resource Management This section implements intelligent selection and optimized configuration of computing devices. It automatically selects the best-performing computing device through a multi-level device detection mechanism (following the priority of CUDA GPU → MPS → CPU). This module not only provides dynamic resource allocation and memory management but also integrates device-specific performance optimization configurations (such as CUDA convolution acceleration), thus ensuring cross-platform compatibility while guaranteeing optimal system performance in different hardware environments. Example code is shown below: device = torch.device( "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu") 5.1.3. Memory Optimization Strategies The memory optimization strategy module optimizes the entire data processing lifecycle: during the data loading phase, lazy loading and block processing are used to reduce instantaneous memory pressure; during processing, efficient encoding formats are employed to achieve a balance between quality and memory usage, and intelligent caching and eviction strategies are used to improve data reuse; after the task is completed, large memory objects are ensured to be released in a timely manner. This combined strategy significantly reduces the overall system memory footprint and ensures memory stability for large-scale image processing tasks.

[0093] 6. Image format recognition and conversion 6.1 Intelligent Image Format Recognition To ensure the system's stability and compatibility under multi-source image input and cross-platform processing, this invention further designs a format conversion mechanism based on the image format recognition module. This mechanism, through a unified encoding and decoding process, achieves automatic conversion and standardized processing between different image formats, ensuring consistency and usability of image data during file reading and writing, network transmission, and model input. This section mainly introduces bidirectional conversion methods from image files to Base64 format and from in-memory images to Base64 format, providing fundamental support for the system's efficient transmission and visualization.

[0094] 6.1.1. Format Detection To ensure the reliability of image processing, this module first automatically detects the input image format (supporting common types such as JPEG, PNG, and BMP). When an unsupported format is encountered, the system will activate a default format fallback mechanism to ensure the process continues. Furthermore, the module provides comprehensive error handling and detailed logging for all abnormal situations, effectively improving the system's fault tolerance and maintainability. Example logic is shown in the following code: def get_image_format(image_path): try: with open(image_path, 'rb') as f: image_type = imghdr.what(f) if image_type: return image_type return 'jpeg' # Default format except Exception as e: print(f"Image format error: {str(e)}") return 'jpeg' 6.2 Format Conversion 6.2.1. Image file to Base64 conversion This invention supports intelligent encoding processing of local image files, automatically adding Data URL prefixes and achieving format-adaptive encoding conversion, ensuring that image files of different formats can be processed efficiently and accurately. Through automated prefix addition and intelligent format recognition, the system can seamlessly process multiple image formats such as JPEG and PNG, providing standardized data format support for subsequent data transmission and display. Example logic is shown in the following code: def image_to_base64(image_path, isfr=True) -> str: try: with open(image_path, "rb") as image_file: image_data = image_file.read() # Read binary data base64_str = base64.b64encode(image_data).decode('utf-8') # Use standard base64 encoding base64_str = ''.join(base64_str.split()) image_format = get_image_format(image_path) # Get the image format #Add Data URL prefix return data_url 6.2.2. Memory Image to Base64 Conversion This module supports flexible input formats, can process PIL Image objects and NumPy arrays simultaneously, and automatically performs necessary color space conversions (such as BGR to RGB) to ensure the consistency of image data formats. Through an optimized memory management mechanism, the module employs an efficient byte stream processing method, minimizing memory consumption while ensuring processing performance, providing stable and reliable data conversion support for image processing tasks. Example code is shown below: def image_array_to_base64(image_data, format='PNG', isfr=True) ->str: try: # If it's a NumPy array, convert it to a PIL image. if isinstance(image_data, np.ndarray): # Assuming it's in BGR format (OpenCV), it needs to be converted to RGB. if len(image_data.shape) == 3 and image_data.shape[2] == 3: image_data = image_data[:, :, ::-1] # BGR to RGB image_data = Image.fromarray(image_data.astype('uint8')) # Throw an exception if it is not a PIL image. if not isinstance(image_data, Image.Image): raise ValueError("Input must be a PIL Image object or a NumPy array") # Use BytesIO to store images in memory buffer = io.BytesIO() image_data.save(buffer, format=format) buffer.seek(0) # Convert to base64 image_bytes = buffer.read() base64_str = base64.b64encode(image_bytes).decode('utf-8') # Add Data URL prefix return data_url 6.2.3. Base64 to Image Conversion This module provides complete Data URL parsing functionality, automatically recognizing and parsing Data URL strings containing prefixes, and supporting decoding of various common image formats such as JPEG and PNG. After parsing, the module returns a standard PILImage object for subsequent image processing and analysis. This end-to-end parsing scheme ensures seamless conversion from Data URL to image object, providing developers with a convenient and efficient image processing interface. Example code is shown below: def base64_to_image(base64_string: str) -> Image.Image: try: # Process base64 strings and remove possible prefixes if ',' in base64 string: # If it contains a prefix like data:image / png;base64, only take the part after it. base64_string = base64_string.split(',')[1] image_data = base64.b64decode(base64_string) image = Image.open(io.BytesIO(image_data)) return image 7. BIM Interaction This module aims to enable rapid retrieval and visual association of Building Information Modeling (BIM) objects based on image segmentation and semantic recognition. By spatially aligning and semantically mapping the reconstructed 3D scene with BIM data, an object-level correspondence can be established between real-world images and BIM models, providing a foundation for subsequent intelligent querying and visual analysis.

[0095] 7.1 Spatial Registration and Geometric Alignment Spatial registration is the foundation for integrating the reality model and the BIM model. Its main goal is to eliminate differences in their coordinate systems and achieve geometric consistency. During implementation, rigid transformations (rotation, translation) or similarity transformations (rotation, translation, and uniform scaling) are typically used to establish the mapping relationship between the two coordinate systems. The specific implementation path consists of the following key steps: 7.1.1 Preliminary Registration The method primarily relies on the control point method (GCP). This method requires selecting at least three non-collinear control points of the same name in both the real-world model and the BIM model. Subsequently, the system uses the least squares method to solve for the spatial transformation matrix (such as seven-parameter or four-parameter), completing the basic geometric alignment.

[0096] 7.1.2 Fine Registration To achieve higher accuracy, we employ the Iterative Closest Point (ICP) algorithm based on the initial registration. ICP is particularly suitable for situations with dense point clouds or no obvious control points. It achieves high-precision alignment by iteratively minimizing the Euclidean distance between the actual point cloud and the BIM model (or its point cloud).

[0097] 7.1.3 Registration Accuracy Evaluation After registration is completed, the registration accuracy must be quantitatively evaluated by calculating the root mean square error (RMS Error). Our standard is to ensure that the positional deviation between the BIM object and the real-world component in three-dimensional space is controlled within 5 centimeters, thereby laying a solid and accurate spatial foundation for subsequent semantic matching and visualization retrieval.

[0098] 7.2 Semantic Mapping and Object Matching After completing spatial alignment, the system achieves precise association between real-world components and BIM objects based on semantic recognition results.

[0099] 7.2.1 Semantic Classification The system first classifies a massive number of points in a real-world 3D model (such as a point cloud) based on a pre-built semantic segmentation module. This module outputs structured building component labels (such as "concrete beam", "brick wall", "air duct", etc.), providing a crucial semantic foundation for subsequent "class-based matching".

[0100] 7.2.2 Dual-Constraint Matching Strategy Subsequently, the system employs a dual-constraint matching strategy to improve matching accuracy and robustness. On one hand, under spatial proximity constraints, the system utilizes spatial indexing structures such as octrees or kd-trees to quickly retrieve candidate objects in the BIM model whose distance from the center point or bounding box of the real-world component is less than a preset threshold Dmax, ensuring that matching occurs only between objects with reasonable spatial proximity. On the other hand, under semantic similarity constraints, the system calculates the semantic similarity S between the real-world component and the candidate BIM object based on the component's semantic labels and feature vectors (such as color, texture, and size ratio) using measurement methods such as cosine similarity.

[0101] 7.2.3 Matching Decision Ultimately, the system makes a comprehensive judgment, and only when a BIM object simultaneously satisfies the conditions that its spatial distance is less than Dmax and its semantic similarity is higher than the threshold S, will the object be identified as a target for matching with the real-world component, thereby achieving dual constraint matching of spatial geometry and semantic features.

[0102] 7.3 Bidirectional Indexing and Visual Interaction After semantic matching is completed, the system establishes a bidirectional index relationship between real-world objects and BIM objects to support efficient object retrieval and visualization interaction.

[0103] 7.3.1 Index Creation The index structure is based on the real-world component ID and the globally unique identifier (GUID) of the BIM object. It achieves a two-way mapping between the two by establishing a lookup table, thereby realizing the linkage between the data layer and the spatial layer.

[0104] 7.3.2 Forward Queries In the forward query (BIM → Real Scene), users can search the BIM database by entering semantic keywords (such as "query concrete beam") or component number. The system locates the corresponding real scene component in the index table based on the obtained GUID and performs visual annotation in the 3D rendering environment using highlighting, realizing direct positioning from information model to real scene.

[0105] 7.3.3 Reverse Lookup In reverse lookup (real-world image → BIM), when a user clicks on a component in the real-world image, the system searches for its corresponding BIM object GUID using an index table. It then instantly extracts and displays the component's detailed attribute information from the BIM database (such as an IFC file or a structured database), including component type (e.g., IfcWall), fire rating, material type, and maintenance records. Through this bidirectional indexing mechanism, the system achieves semantic interconnection and spatial visualization linkage between the real-world image and BIM, providing data support and an operational foundation for subsequent intelligent retrieval, maintenance decisions, and digital twin interaction.

[0106] 8. Application Scenarios 8.1 Comparison of Construction Progress The system automatically identifies constructed components in real-world images through the BIM query module, performs semantic matching with the BIM design model, and generates a construction progress visualization report.

[0107] 8.2 Facility Operation and Maintenance Management During the building operation and maintenance phase, managers can use voice or text commands (such as "show all air conditioning units") to automatically retrieve the corresponding components in the BIM model and highlight them in the real-world 3D environment.

[0108] 8.3 Safety Hazard Investigation By combining structural semantics and spatial topology information, the system can automatically detect deviations between the real scene and the BIM model, identify components that may be deformed or misaligned, and achieve intelligent early warning of safety risks.

[0109] Example 2 A method for querying BIM objects using image segmentation and semantic recognition, comprising: Acquire real-world architectural images, preprocess the images to obtain standardized images, segment the standardized images to extract each architectural component, the region where each component is located, and the two-dimensional coordinate information of the region where each component is located from the real-world architectural images, and generate segmentation masks for each architectural component. The semantic recognition model is used to perform semantic recognition on the segmentation mask of each building component to obtain the category, attribute and bounding box coordinate information of the area where each building component is located; Establish a mapping relationship between the two-dimensional coordinate information of the area where each building component is located and the spatial coordinates of the BIM model, and generate mapping relationship data; Based on the user-selected target area, mapping relationship data, and the category, attributes, and boundary coordinate information of each building component's area, the system queries the corresponding BIM object attribute information in the BIM model space; based on the BIM object attribute information, it queries the BIM database to obtain a visualized real-world image of the building with semantic tags and boundary annotations.

[0110] Example 3 A computer program product includes a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the method described in Embodiment 2.

[0111] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A BIM object query system based on image segmentation and semantic recognition, characterized in that, It includes: The image segmentation module is used to acquire real-world building images, preprocess the real-world building images, and obtain standardized images. Target segmentation is performed on the standardized image to extract the two-dimensional coordinate information of each building component, the area where each building component is located, and the area where each building component is located from the real building image, and at the same time, the segmentation mask of each building component is generated. The semantic recognition module is used to perform semantic recognition on the segmentation mask of each building component through the semantic recognition model, and to obtain the category, attributes and bounding box coordinate information of the area where each building component is located. The coordinate mapping module is used to establish the mapping relationship between the two-dimensional coordinate information of the area where each building component is located and the spatial coordinates of the BIM model, and to generate mapping relationship data; The interactive query output module is used to query the corresponding BIM object attribute information in the BIM model space based on the target area selected by the user, the mapping relationship data, and the category, attributes and boundary coordinate information of the area where each building component is located. Based on the attribute information of the BIM object, a query is performed in the BIM database to obtain a visualized real-world image of the building with semantic tags and boundary annotations.

2. The BIM object query system based on image segmentation and semantic recognition according to claim 1, characterized in that: The building scene image is preprocessed. The preprocessing steps include reading the building scene image and obtaining the coordinate points of the building scene image. If the size of the building image is inconsistent with the size of the mask, a linear transformation is performed on the coordinate points of all building images according to the size ratio of the building image to the mask. The linearly transformed coordinate points are then mapped to the spatial coordinate system of the mask, so that the size of the building image is aligned with the size of the mask, resulting in a standardized image.

3. A BIM object query system based on image segmentation and semantic recognition according to claim 1 or 2, characterized in that: The image segmentation module supports at least one of the following: panoramic segmentation mode, point-hint segmentation mode, box-hint segmentation mode, or text-hint segmentation mode. Among them, the panoramic segmentation mode automatically detects and segments all identifiable objects in the image using a deep learning image segmentation model; The dot-hint segmentation mode generates a target segmentation mask by receiving user click coordinates, foreground labels, or background labels, based on a mask area sorting and positive / negative dot hint fusion strategy. The box suggestion segmentation mode selects the optimal matching mask by calculating the IoU value between the user input bounding box and the candidate mask; The text prompt segmentation mode calculates the similarity between the user-input text description and the cropped image region generated by the preliminary segmentation model by using a visual-language model pre-trained based on contrastive learning, and determines the final segmentation result by combining a mask overlap ratio expansion strategy.

4. The BIM object query system based on image segmentation and semantic recognition according to claim 3, characterized in that: The specific method for the dot-hint splitting mode is as follows: Based on the target area selected by the user and the size ratio of the building scene image and the BIM model mask, the image is linearly transformed to the spatial coordinate system where the mask is located. Obtain all candidate segmentation masks generated by the initial segmentation model and sort them in descending order by area; traverse all candidate masks, and if a mask covers a point that the user has marked as a foreground label, then all the building scene image pixel regions of that mask are marked as selected; If a mask covers a point that the user has marked as a background label, then all the building image pixels of that mask are marked as excluded. All selected and not excluded mask areas are merged to form a precise single object mask as the final result.

5. The BIM object query system based on image segmentation and semantic recognition according to claim 1, characterized in that: The specific method for establishing the mapping relationship between the two-dimensional coordinate information of the area where each building component is located and the spatial coordinates of the BIM model is as follows: After reconstructing the two-dimensional coordinate information of the area where each building component is located in the real building image, a three-dimensional scene is obtained. The reconstructed three-dimensional scene is aligned with the BIM model in space through spatial registration and geometric alignment. Then, the correspondence between specific objects of building components in the reconstructed three-dimensional scene is found from the BIM model through a dual constraint matching strategy of semantics and space.

6. The BIM object query system based on image segmentation and semantic recognition according to claim 1, characterized in that: The specific method for querying the attribute information of the corresponding BIM object in the BIM model space based on the user-selected target area, mapping relationship data, and the category, attributes, and boundary coordinate information of each building component's area is as follows: On the interactive interface provided by the system, users can select one or more target building components from a real-world image with semantic labels and bounding boxes using a set selection method. The system captures the user's interaction and immediately matches it against the category, attribute, and bounding box coordinate information of the area where each building component is located. It then searches for the globally unique identifier of the corresponding BIM model object through mapping relationship data and obtains the BIM object attribute information using the globally unique identifier of the obtained BIM model object.

7. A BIM object query system based on image segmentation and semantic recognition according to claim 1, characterized in that: A specific method for obtaining visualized building reality images with semantic tags and boundary annotations by searching and matching in the BIM database based on the attribute information of the BIM objects: Based on the BIM object attribute information, the corresponding BIM object is found in the BIM database, and the globally unique identifier of the corresponding BIM object is obtained. Based on the globally unique identifier, a forward query is performed through a bidirectional index table generated by the mapping relationship data to find one or more building components associated with the attribute information of the BIM object. The corresponding building scene image is retrieved according to the found one or more building components. Based on the category, attribute and bounding box coordinate information of the area where the building component is located, a highlight box, mask and semantic label are drawn on the area where the corresponding building component is located on the building scene image to obtain a visualized building scene image with semantic labels and boundary annotations.

8. The BIM object query system based on image segmentation and semantic recognition according to claim 1, characterized in that: It also includes a visualization module, which receives the segmentation mask of each building component from the image segmentation module and the category, attribute and bounding box coordinate information of the area where each building component is located from the semantic recognition module, standardizes the format of the segmentation mask, and obtains the height and width of the building real scene image; Depending on the hardware system load and configuration, select to use CPU or GPU for rendering calculations, randomly generate different RGB colors for the segmentation mask of each building component, or specify fixed colors for specific categories of components, set the transparency of the segmentation mask, and sort all segmentation masks by area. Apply the set colors and transparency to the corresponding segmentation mask to generate a mask layer; A white outline is drawn at the edge of each segmentation mask. Based on the height and width of the original image, the generated mask layer and white outline are overlaid with the actual building image to obtain a visualization result.

9. A BIM object query method based on image segmentation and semantic recognition, characterized in that, It includes: Acquire real-world images of buildings, preprocess these images, and obtain standardized images; Target segmentation is performed on the standardized image to extract the two-dimensional coordinate information of each building component, the area where each building component is located, and the area where each building component is located from the real building image, and at the same time, the segmentation mask of each building component is generated. The semantic recognition model is used to perform semantic recognition on the segmentation mask of each building component to obtain the category, attribute and bounding box coordinate information of the area where each building component is located; Establish a mapping relationship between the two-dimensional coordinate information of the area where each building component is located and the spatial coordinates of the BIM model, and generate mapping relationship data; Based on the user-selected target area, mapping relationship data, and the category, attributes, and boundary coordinate information of each building component's area, query the corresponding BIM object attribute information in the BIM model space; Based on the attribute information of the BIM object, a query is performed in the BIM database to obtain a visualized real-world image of the building with semantic tags and boundary annotations.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 9.

Images