Building element identification method, electronic device and computer storage medium

Abstract

This invention discloses a method, electronic device, and computer storage medium for identifying building elements, relating to the field of building element identification technology. The method includes: acquiring a building image to be identified; processing the building image to generate vector contours of building elements and extracting text information from the building image, the text information including text content and its position in the building image; based on the position, associating the text information with the vector contours to generate semantic information of the building elements; and inputting the vector contours of the building elements and the semantic information into a semantic reasoning model to obtain the semantic categories output by the semantic reasoning model corresponding to each building element. This invention can accurately determine the category of building elements.

Description

[0001] Technology Neighborhood This invention relates to the field of building element recognition technology, and more specifically to a building element recognition method, electronic device, and computer storage medium. Background Technology

[0002] Architectural elements, as key components of architectural spatial structure, are widely used in various architectural images. In architectural images, architectural elements are based on standardized lines as their core constituent elements, thus presenting similar geometric shapes in appearance, such as straight line segments and simple closed figures.

[0003] Because architectural elements in architectural images often exhibit similar geometric shapes and lack distinctive features such as texture and color, it is difficult to accurately classify these elements. Summary of the Invention

[0004] This invention aims to address, to a certain extent, one of the technical problems in related technologies. To this end, this invention provides a method for identifying architectural elements, an electronic device, and a computer storage medium, which has the advantage of accurately determining the category of architectural elements.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for identifying architectural elements, comprising: Acquire images of the building to be identified; The building image to be identified is processed to generate vector contours of building elements, and text information in the building image to be identified is extracted, the text information including text content and its position in the building image to be identified; Based on the location, the text information is associated with the vector outline to generate semantic information of the building elements; The vector outlines of the building elements and the semantic information are input into the semantic reasoning model to obtain the semantic categories output by the semantic reasoning model corresponding to each building element.

[0006] Optionally, the semantic reasoning model includes a cascaded feature encoding module, a semantic reasoning module, and an output layer; the feature encoding module is used to encode the vector contour and the text information to output a multimodal feature vector for each building element; the semantic reasoning module is used to perform semantic reasoning based on local context on each of the multimodal feature vectors to output an enhanced feature vector; and the output layer is used to map the enhanced feature vector to a semantic category probability vector to output the semantic category of each building element.

[0007] Optionally, the feature encoding module includes a geometric feature encoding layer, a semantic feature encoding layer, and a multimodal fusion layer, wherein the semantic information includes the association between indicator text and building elements, and the association between room text and building elements; The feature encoding module encodes the vector contour and the text information based on the following steps, outputting a multimodal feature vector for each building element: Geometric feature vectors are extracted from the vector contours of the building elements through the geometric feature encoding layer; The semantic feature encoding layer converts the indicator text corresponding to each building element into the first OneHot encoding, and the room text corresponding to each building element into the second OneHot encoding. The geometric feature vector is fused with the corresponding first OneHot code and second OneHot code through the multimodal fusion layer to obtain the multimodal feature vector.

[0008] Optionally, the semantic reasoning module performs semantic reasoning based on local context on each of the multimodal feature vectors and outputs enhanced feature vectors based on the following steps: Calculate the centroid coordinates of the vector profile of each building element; Using the centroid coordinates of the vector profiles of each of the aforementioned building elements, calculate the centroid distance between all vector profile pairs; Using the centroid distance, the neighborhood vector profiles corresponding to each vector profile are determined; Calculate the relative positional offset between each of the vector contours and the domain vector contour; Multi-head attention is calculated on the multimodal feature vectors of each vector contour, the multimodal feature vectors of the corresponding neighborhood vector contours, and the relative position offset to obtain a single attention output; All the individual attention outputs are concatenated, and a linear transformation is performed on the concatenated individual attention outputs to obtain the enhanced feature vector.

[0009] Optionally, processing the building image to be identified to generate vector contours of building elements includes: The initial vector contour and edge image are extracted from the building image to be identified using a pre-trained vectorization model. The initial vector contour is parameterized into a set of primitives, wherein the primitives are straight lines, circular arcs, or second-order Bézier curves; By optimizing the parameters of the primitives to align them with the edge image, the vector outline of the building element is obtained.

[0010] Optionally, the step of aligning the primitive with the edge image by optimizing the parameters of the primitive to obtain the vector outline of the building element includes: Construct the energy interaction function between the primitive and the edge image; The position and size parameters of the primitive are determined using the energy interaction function. The graphic element is adjusted using the position parameters and the size parameters to obtain the adjusted graphic element; By splicing together the adjusted primitives belonging to the same architectural element, a vector outline corresponding to each architectural element is obtained.

[0011] Optionally, the step of associating the text information with the vector contour based on the location to generate semantic information of the building elements includes: Based on the text content, the text information is classified into instruction text, room text, and invalid text; Filter out the invalid text. The semantic information of the building elements is obtained by associating the indicator text and the room text with the vector outline of the building element that is spatially closest to them, based on their positions.

[0012] Optionally, the method further includes: Generate a building vector image with semantic tags based on the vector outline of each building element and its corresponding semantic category.

[0013] In a second aspect, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the building element recognition method described in any of the preceding claims.

[0014] Thirdly, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the building element recognition method described in any of the preceding claims.

[0015] This invention addresses the technical challenge of accurately classifying architectural elements, which are characterized by standardized lines, similar geometric shapes, and a lack of unique textures and colors. The invention first acquires an image of the building to be identified, processes it to generate vector contours of the architectural elements, and extracts textual information containing location information. Then, it generates semantic information about the architectural elements based on the location-related textual information and vector contours. Finally, it inputs both the vector contours and semantic information into a semantic reasoning model. Since the textual information in the architectural image is associated with key semantics such as the function and type of the architectural elements, location-related textual information is transformed into semantic supplementary features of the architectural elements. This effectively compensates for the difficulty in distinguishing architectural element categories based solely on geometric shapes. The semantic reasoning model can then combine geometric and semantic features for comprehensive judgment, thereby achieving accurate determination of the semantic category of architectural elements and significantly improving the accuracy and reliability of architectural element category recognition.

[0016] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and accompanying drawings. The preferred embodiments or means of the present invention will be shown in detail in conjunction with the accompanying drawings, but are not intended to limit the technical solutions of the present invention. In addition, each of these features, elements and components appearing in the following text and drawings is a plurality of, and different symbols or numbers are used for convenience of representation, but all represent parts with the same or similar construction or function. Attached Figure Description

[0017] The present invention will be further described below with reference to the accompanying drawings: Figure 1 A flowchart illustrating one embodiment of the building element identification method provided by the present invention; Figure 2 This is a schematic diagram of the process for processing a building image to be identified to generate vector outlines of building elements provided by the present invention. Figure 3 This is a schematic diagram of the process provided by the present invention for aligning primitives with edge images by optimizing primitive parameters; Figure 4 A schematic diagram illustrating the optimization of primitives provided by this invention; Figure 5 This is a schematic diagram illustrating the interaction between primitives and edge pixels provided by the present invention; Figure 6 This is a schematic diagram illustrating the interaction between graphic elements provided by the present invention; Figure 7 This is a schematic diagram illustrating how the interactive energy of a primitive in this invention causes it to expand outward. Figure 8 This is a schematic diagram illustrating how the constraints between primitives and edge pixels in an edge image, provided by the present invention, cause them to shrink inward. Figure 9 This is a schematic diagram of the process for generating semantic information of building elements provided by the present invention. Figure 10 A schematic diagram of the model structure of the semantic reasoning model provided by this invention; Figure 11 This is a schematic diagram illustrating the process by which the encoding feature module provided by the present invention encodes vector contours and text information. Figure 12 This is a schematic diagram illustrating the process by which the semantic reasoning module provided by this invention performs semantic reasoning based on the local context of each multimodal feature vector to output enhanced feature vectors. Figure 13 A block diagram illustrating one embodiment of the electronic device provided by the present invention; Figure 14 This is a schematic diagram of a computer-readable medium provided by the present invention.

[0018] Explanation of reference numerals in the attached figures: 101: Processor; 102: Memory; 103: I / O interface; 104: Bus. Detailed Implementation

[0019] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described are intended to explain the present invention and should not be construed as limiting the invention.

[0020] The terms "an embodiment," "example," or "trademark" used in this specification refer to a particular feature, structure, or characteristic described in connection with the embodiment itself that may be included in at least one embodiment disclosed in this invention. The phrase "in an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0021] As a first aspect of the present invention, a method for identifying architectural elements is provided, such as... Figure 1 As shown, the method includes: In step S110, an image of the building to be identified is acquired.

[0022] In step S120, the building image to be identified is processed to generate vector contours of building elements, and text information in the building image to be identified is extracted. The text information includes text content and its position in the building image to be identified.

[0023] After acquiring the image of the building to be identified, it can be processed by denoising, grayscale conversion, and binarization to obtain a preprocessed image, thus eliminating interference factors. For the preprocessed image, the text region can be extracted using the MSER (Maximum Stable Region) algorithm, leveraging the frequent grayscale changes and repetitive texture characteristics of the text region. Then, an OCR algorithm is used to extract the text string from the text region, yielding the text content. Subsequently, text detection models such as EAST and CTPN can be used to obtain the bounding box coordinates of the text region. After obtaining the bounding box coordinates, a two-dimensional Cartesian coordinate system can be constructed using the top-left corner of the building image as the origin. The bounding box coordinates of the text region are then converted to standardized coordinates relative to the image origin, revealing the position of the text content within the building image.

[0024] Meanwhile, for the preprocessed building image to be identified, the contour information of the building elements can be extracted from the preprocessed building image using edge detection and contour finding algorithms, and output in a standard vector format to obtain the vector contour corresponding to each building element.

[0025] Alternatively, as another optional implementation, refer to Figure 2 As shown, Figure 2 A flowchart illustrating the process of processing a building image to generate vector outlines of building elements is provided. Step S120 specifically includes: In step S210, an initial vector contour and edge image are extracted from the building image to be identified using a pre-trained vectorization model.

[0026] In step S220, the initial vector contour is parameterized into a set of primitives, wherein the primitives are straight lines, circular arcs, or second-order Bézier curves.

[0027] In step S230, the parameters of the primitive are optimized to align the primitive with the edge image, thereby obtaining the vector outline of the building element.

[0028] In this embodiment, the pre-trained vectorized model can adopt a Transformer or graph neural network architecture.

[0029] Specifically, after inputting the building image to be identified into a pre-trained vectorization model, the pre-trained vectorization model predicts the initial vector contours describing each building element through forward propagation and outputs the initial vector contours and edge images. The initial vector contours are essentially a set of parameters for lines, arcs, and curves; therefore, the initial vector contours can be parameterized as primitives.

[0030] After obtaining the primitive and edge images, since the primitive shape is roughly correct, but the primitive shape cannot perfectly match the corresponding pixel edge in the edge image, it is necessary to optimize the primitive parameters to align the primitive with the edge image. Therefore, as an optional implementation method, refer to... Figure 3 As shown, Figure 3 This is a flowchart illustrating the process of aligning primitives with edge images by optimizing primitive parameters. Step S230 specifically includes: In step S310, an energy interaction function between the primitive and the edge image is constructed.

[0031] In step S320, the position and size parameters of the primitive are determined using the energy interaction function.

[0032] In step S330, the graphic element is adjusted using the position parameter and the size parameter to obtain the adjusted graphic element.

[0033] In step S340, the adjusted primitives belonging to the same building element are spliced ​​together to obtain the vector outline corresponding to each building element.

[0034] In this embodiment, the set of lines that constitute the vector contour of the building element is called a primitive, and the image predicted by the edge enhancement branch is called an edge pixel.

[0035] Specifically, to enable primitives to adaptively repair themselves based on edge pixels in the edge image and achieve higher positioning accuracy, refer to Figure 4 As shown, Figure 4 This diagram illustrates primitive optimization. During the primitive optimization process, edge pixels in the edge image are assigned positive values, while primitives are assigned negative values. Therefore, the energy interaction function between primitives and the edge image is defined as follows: , , , These represent the indices of the edge pixels in the edge image. and Indicates the index of the graphic element. Indicates the first In the image of primitives and edges, the first... Interaction energy between edge pixels Indicates the first The primitive and the first Interactive energy between primitives Represents the edge image of the first The edge pixel and the first Interaction energy between edge pixels, energy function The sum of the three optimization terms includes the interaction between primitives and edge pixels in the edge pixel image, the interaction between primitives, and the interaction between edge pixels. The energy function can achieve the alignment of primitives with edge pixels in the edge image by fine-tuning the coordinates of the primitives.

[0036] In the energy function, This represents the interaction energy between primitives and edge pixels in the edge image. It measures the degree of overlap between primitives and edge pixels in the edge image. Higher overlap and edge pixel confidence result in lower energy, guiding the alignment of primitives with edge pixels in the edge image and eliminating errors between primitives and edge pixels. Therefore, it can be further defined as: , Represents the edge image of the first Confidence of each edge pixel, Indicates the first The spatial area covered by each graphic element This represents the coordinates of any point within the primitive region. Indicates the first The coordinates of each edge pixel are given. The integrand uses a negative exponential form to represent the coverage relationship between the primitive and the edge pixel, forming a mechanism that decays exponentially with increasing distance. During the interaction between the primitive and the edge pixel, the energy generated by both is negative, and its energy value decreases rapidly as the primitive gradually approaches the edge pixel region.

[0037] exist In the middle, when the primitive size increases so that it can cover more edge pixels, This will be further reduced, thereby adjusting the movement stride of the primitives. This process involves making small adjustments based on the initially predicted vector contours to achieve precise alignment between the primitives and edge pixels. It is the confidence score of edge pixels in an edge image, which ensures that primitives are preferentially aligned with areas of higher edge confidence, and finally aligned with the line areas covered by the primitives. By accumulating these values, we can obtain the energy generated by the primitive on the edge pixels, as shown in the reference. Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the interaction between primitives and edge pixels. Figure 5 In this context, when a primitive is sufficiently close to an edge pixel, the primitive and the edge pixel will generate a large negative energy value, which can be used to offset the energy value generated by the subsequent primitive itself.

[0038] In the energy function, It consists of two parts: the energy generated by the interaction between different primitives, and the energy generated by the expansion or contraction of a single primitive. The energy generated by the interaction between different primitives is positive and decreases as the distance between the two primitives increases. Simultaneously, the energy value also decreases when the size of the primitive shrinks, therefore it can be... Defined as: , Used to describe the The primitive and the first The interaction energy between primitives is used to maintain the spatial independence of each primitive and avoid overlap. If two independent primitives do not overlap, this term is 0. Conversely, if two independent primitives overlap, the overlapping regions will generate a positive energy after accumulation. This phenomenon can be eliminated by minimizing its own energy, see reference. Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the interaction between graphic elements, and further... The definition is as follows: , Indicates the first The spatial area covered by each graphic element Indicates the first The spatial area covered by each graphic element Indicates the first The coordinates of any point within a primitive. Indicates the first The coordinates of any point within a primitive and the interaction energy interaction function between primitives can determine the degree of overlap between primitives. That is, if there is no overlap between primitives, the energy is 0. If there is overlap, and the larger the overlapping area and the closer the distance, the higher the energy.

[0039] This refers to the energy generated by the internal shape changes of the primitive itself. When the primitive does not completely cover the pixel edges in the edge image, the primitive's own energy will drive the primitive to change its size, so that its own energy can offset the energy generated by the interaction between the primitive and the pixel edges in the edge image. (Refer to...) Figure 7 As shown, Figure 7 This is a schematic diagram illustrating how the interaction energy of a primitive causes it to expand outward. Correspondingly, when all regions in an edge image are filled by other primitives, the excess primitives will drive their size towards 0 to eliminate additional positive energy, as shown in the diagram. Figure 8 As shown, Figure 8 This is a schematic diagram illustrating how the constraint between a primitive and edge pixels in the edge image causes it to shrink inward. Based on this, the energy function of the primitive itself can be... Defined as: , , Indicates the first The coordinates of two arbitrary points within a primitive are considered. During the optimization of the primitive's own energy function, the edge pixels in the edge image, serving as a standard reference, remain unchanged. Therefore, the energy function between edge pixels is set to 0, and the optimization is achieved by minimizing the energy function. It can provide guidance for the dynamic adjustment of primitives on edge images, enabling primitives to adaptively adjust their position and size to match edge pixels in the edge image.

[0040] After obtaining the energy interaction function Then, the position and size parameters of the graphic element can be updated in the following ways: , Indicates the first Parameters to be optimized for each element Parameters to be optimized This includes the position parameter (pos) and size parameter (size) of the graphic element. Indicates the learning rate. Represents the total energy with respect to parameters The gradient.

[0041] When updating the position and size parameters of a primitive, the gradient generated by the interaction between the primitive and edge pixels is defined as follows: .

[0042] Meanwhile, the gradient generated by the interaction between different primitives is defined as follows: .

[0043] Correspondingly, the gradient formula generated by the change in the size of the primitive itself is defined as follows: .

[0044] Since changes in the position of the primitive itself do not affect the gradient change, therefore, the following settings will be used: The value is zero, and after merging all location-related sub-terms, the location-related gradient is obtained: , Indicates the first Position parameters of each graphic element Indicates the first The interaction energy between the first primitive and other primitives affects the first primitive. The contribution of the positional parameters of each primitive. Indicates the first The interaction energy between the primitive and the edge pixels in the edge image affects the first primitive. The contribution of each primitive's positional parameters.

[0045] Accordingly, after merging all size-related sub-items, we obtain the size-related gradient: , Indicates the first The size parameters of each graphic element Indicates the first The interaction energy between the first primitive and all primitives is related to the first primitive's interaction energy with all primitives. The contribution of each element's size parameter. Indicates the first The interaction energy between the primitive and the edge pixels in the edge image affects the first primitive. The contribution of each element's size parameter.

[0046] Finally, by minimizing The position and size parameters of the primitives are adjusted to align with the edge pixels in the edge image, resulting in the adjusted primitives. The adjusted primitives belonging to the same building element are then stitched together to obtain the vector contours corresponding to each building element.

[0047] In this embodiment, by constructing an energy interaction function between primitives and edge images, the matching relationship between primitives and edges is quantified based on the real edge image. This allows for clear optimization objectives in determining the position and size parameters of primitives, effectively avoiding the problems of deviation between vectorization results and real image edges, irregular edges, and jagged edges in traditional methods. This ensures that the vector contour edges are clear, accurate in shape, and precise in detail. The energy interaction function takes into account the edge matching degree of primitives, non-overlapping constraints between primitives, and the smoothness of primitive shapes. It can adapt to regular building elements and accurately handle complex elements such as streamlined and curved shapes through methods such as second-order Bézier curve fitting. At the same time, it can resist noise interference such as creases and tilts in scanned images, significantly improving the adaptability to diverse building elements and complex scenes. Through parameter adjustment driven by the energy interaction function, gaps between building elements can be automatically filled, and primitive overlap or omission can be avoided. This ensures that the spliced ​​vector contour forms a closed and complete building structure, reducing the workload of subsequent topology repair and providing high-quality vector contour input for downstream applications such as building information model reconstruction and interior space analysis.

[0048] In step S130, based on the location, the text information is associated with the vector contour to generate semantic information of the building elements.

[0049] When associating text information with vector contours to generate semantic information for architectural elements, to ensure accurate association between the text information and vector contours, further, as an optional real-time method, referencing... Figure 9 As shown, Figure 9 A flowchart illustrating the process of generating semantic information for architectural elements is provided. Step S130 specifically includes: In step S910, the text information is classified according to the text content to obtain instruction text, room text, and invalid text.

[0050] In step S920, the invalid text is filtered out.

[0051] In step S930, the indicator text and the room text are associated with the vector outline of the building element that is spatially closest to them, based on their positions, to obtain the semantic information of the building element.

[0052] In this embodiment, indicator text can provide direct category or property indications for building elements. Indicator text can intuitively reflect the type of the building element itself and is associated only with the most recent building element. Room type text is used to indicate the function or type of a room. Considering that a room may be composed of multiple building elements and there are shared building elements, each building element can be associated with multiple most recent room type texts.

[0053] Specifically, after extracting text information from the building image to be identified, the text information can be categorized into instruction text, room text, and invalid text based on its content, and invalid text can be filtered out. After filtering out invalid text, for instruction text and room text, the center coordinates of their bounding boxes can be determined based on their standardized coordinates, and then the straight-line distance between the center coordinates of their bounding boxes and the centroid coordinates of each vector profile can be calculated.

[0054] After obtaining the straight-line distance between the center coordinates of the bounding boxes of indicator text and room text and the centroid coordinates of each vector contour, for indicator text, the indicator text with the smallest straight-line distance can be bound to the vector contour; for room text, the room text with a straight-line distance less than a preset distance can be bound to the corresponding vector contour, thus obtaining the semantic information of the building elements.

[0055] In this embodiment, by explicitly dividing text information into indicator, room, and invalid categories and filtering out invalid text, valuable semantic information for building element identification can be accurately selected, avoiding classification errors caused by irrelevant text interference. This solves the problems of low semantic information utilization and excessive interference in traditional methods. Secondly, by associating indicator text with the vector contours of the nearest building element based on location, key semantic labels such as category and property can be directly assigned to building elements, helping to accurately distinguish elements with similar visual features but different functions, overcoming the limitation of relying solely on geometric features to identify similar visual elements. At the same time, associating room text with building elements within the corresponding spatial range can clarify the functional attributes of the room to which the building element belongs, providing a reliable basis for the semantic division of building space and improving the structuring degree of the vectorization results. In addition, the association process achieves accurate matching based on spatial distance, ensuring the accuracy of the correspondence between semantic information and building elements, and providing high-quality data support with both geometric accuracy and semantic integrity for subsequent downstream applications such as building information model reconstruction and interior space analysis.

[0056] In step S140, the vector outline of the building element and the semantic information are input into the semantic reasoning model to obtain the semantic category output by the semantic reasoning model corresponding to each building element.

[0057] In this embodiment, refer to Figure 10 As shown, Figure 10 This is a schematic diagram of the semantic reasoning model structure. The semantic reasoning model includes a cascaded feature encoding module, a semantic reasoning module, and an output layer. The feature encoding module is used to encode the vector contour and the text information to output a multimodal feature vector for each building element. The semantic reasoning module is used to perform semantic reasoning based on local context on each of the multimodal feature vectors to output an enhanced feature vector. The output layer is used to map the enhanced feature vector to a semantic category probability vector and output the semantic category of each building element.

[0058] As an optional implementation, the feature encoding module includes a geometric feature encoding layer, a semantic feature encoding layer, and a multimodal fusion layer. The semantic information includes the association between indicator text and building elements, and the association between room text and building elements. Figure 11 As shown, Figure 11 The flowchart illustrates the process of encoding vector contours and text information for the encoding feature module, specifically including the following steps: In step S1110, geometric feature vectors are extracted from the vector contours of the building elements through the geometric feature encoding layer.

[0059] In step S1120, the semantic feature encoding layer converts the indicator text corresponding to each building element into a first OneHot code, and the room text corresponding to each building element into a second OneHot code.

[0060] In step S1130, the geometric feature vector is fused with the corresponding first OneHot code and second OneHot code through the multimodal fusion layer to obtain the multimodal feature vector.

[0061] Specifically, for each architectural element First, the edge sampling sublayer in the geometric feature encoding layer is used to sample along each building element. Bilinear interpolation is performed on the edges of the vector contour to ensure that the sampling points are evenly distributed on the vector contour and to avoid feature omission. Then, based on the coordinates of the sampling points in the feature map of the building image to be identified, high-dimensional features corresponding to each sampling point are extracted to form a contour feature sequence, denoted as . , Indicates the number of sampling points. This represents the dimension of the feature map in the building image to be identified. Subsequently, the high-dimensional features of the sampling points constituting the feature contour sequence are divided into... Each feature group undergoes one-dimensional max-pooling processing through a one-dimensional max-pooling layer to map a contour feature sequence of arbitrary length into a fixed-dimensional feature vector. This fixed-dimensional feature vector is then used as the geometric feature encoding result, denoted as... .

[0062] After extracting the text information from the building image to be identified through the aforementioned operations, the indicator text in the text information can be bound to the nearest building element, and each building element can be bound to the nearest room text. Subsequently, the indicator text and room text bound to each building element can be converted into OneHot vectors, that is, the semantic feature encoding layer converts the indicator text corresponding to each building element into the first OneHot code, and the room text corresponding to each building element into the second OneHot code.

[0063] Finally, the multimodal fusion layer concatenates the geometric feature vectors, the first OneHot code, and the second OneHot code corresponding to each building element along the channel dimension to obtain a multimodal feature vector, denoted as . , This represents the dimension of the flattened geometric eigenvector. The first OneHot encoding of the indicator text. The second OneHot encoding represents the room-type text.

[0064] In this embodiment, geometric feature vectors can accurately capture the inherent geometric features of building elements, such as their outlines and shapes, providing a basic morphological basis for the recognition of building elements. Considering that many building elements in a building plan image have similar visual features, such as ordinary walls versus load-bearing walls, and different types of doors, it is difficult to distinguish them using only geometric features. Therefore, each building element is further bound to its corresponding instruction text and room text, and the geometric feature vectors of each building element are fused with their corresponding first OneHot encoding and second OneHot encoding to obtain a multimodal feature vector. This multimodal feature vector retains both the geometric morphological information such as the outline and shape of the building elements and incorporates semantic attribute information, effectively compensating for the deficiency of single geometric features in distinguishing visually similar building elements, significantly reducing the decision-making difficulty of subsequent recognition tasks, and improving overall recognition accuracy and efficiency.

[0065] It is worth mentioning that the building image to be identified contains multiple building elements. To limit the category association of each building element to its neighboring building elements, a localized attention mechanism can be used, which associates the vector contour of each building element with the nearest neighboring building elements. The vector contours of individual building elements interact, and trainable positional encoding is introduced to reduce computational complexity while preserving the local contextual information of each building element. Therefore, furthermore, as an optional real-time method, referencing Figure 12 As shown, Figure 12 The flowchart illustrates the process by which the semantic reasoning module performs local contextual semantic reasoning on each multimodal feature vector to output enhanced feature vectors, including the following steps: In step S1210, the centroid coordinates of the vector profile of each building element are calculated.

[0066] In step S1220, the centroid distance of all vector profile pairs is calculated using the centroid coordinates of the vector profiles of each of the building elements.

[0067] In step S1230, the neighborhood vector profiles corresponding to each vector profile are determined using the centroid distance.

[0068] In step S1240, the relative position offset between each of the vector contours and the domain vector contour is calculated.

[0069] In step S1250, multi-head attention calculation is performed on the multimodal feature vectors of each vector contour, the multimodal feature vectors of the corresponding neighborhood vector contours, and the relative position offset to obtain a single attention output.

[0070] In step S1260, all the individual attention outputs are concatenated, and a linear transformation is performed on all the concatenated individual attention outputs to obtain the enhanced feature vector.

[0071] Specifically, when extracting the vector contours of each building element through the aforementioned steps, the coordinate information of each building element's vector contour in the coordinate system of the building image to be identified can be used to calculate the centroid coordinates of each vector contour. After obtaining the centroid coordinates of each vector contour, the centroid distance of all vector contour pairs can be calculated using the centroid coordinates of each building element's vector contour. , Indicates the first The centroid coordinates of the vector outline of each architectural element. Indicates the first The centroid coordinates of the vector outline of each architectural element. Indicates the first The centroid coordinates of the vector outline of the first building element, and the first... The centroid distance between the centroid coordinates of the vector profile of each building element.

[0072] After obtaining the centroid distance between the vector contour pairs, based on the model stage of the semantic reasoning model, the formula is used. Filter out the vector outlines of each architectural element. The vector profile of the nearest neighbor building element, i.e., the neighborhood vector profile, where Represents the set of neighborhood vector contours. This represents the set of centroid distances between vector profile pairs of all building elements. This indicates the number of domains in the semantic reasoning model at the current model stage.

[0073] After obtaining the neighborhood vector contours corresponding to each vector contour, the relative positional offset between each vector contour and the neighborhood vector contours can be calculated, and the relative positional offset can be converted into a positional code using MLP. , Indicates position code, The centroid coordinates of the vector profile are represented. The centroid coordinates of the neighborhood vector profile are represented.

[0074] After obtaining the positional encoding between each vector contour and its neighboring vector contours, the multimodal feature vectors of each vector contour are analyzed. Extract the query vector of each vector contour from the multimodal feature vectors of each vector contour. Extracting key vectors from the multimodal feature vectors of neighborhood vector contours Sum value vector Using the formula , Indicates the first The individual attention output of each attention head, i.e., the fusion feature of each vector contour and the neighborhood vector contours. Represents the feature dimension of the query vector With key vector The feature dimensions are calculated. Finally, all individual attention outputs are concatenated, and a linear transformation is performed on the concatenated individual attention outputs to obtain the enhanced feature vector, i.e. , Represents the enhanced feature vector. This represents a vector concatenation operation. This indicates the total number of attention heads. This represents a linear transformation matrix.

[0075] In this embodiment, neighboring vector contours are filtered by calculating the centroid coordinates and centroid distance of the building element vector contours. This not only accurately locates neighboring vector contours with strong spatial correlation with each vector contour, ensuring that the objects of subsequent feature fusion are targeted, but also effectively narrows the scope of feature interaction, avoids redundant calculations caused by indiscriminate feature calculations in the global scope, and greatly reduces the complexity of subsequent multi-head attention calculations.

[0076] In addition, by introducing the relative position offset between the vector contour and the neighboring vector contour, and combining multi-modal feature vectors to carry out multi-head attention calculation, the final enhanced feature vector can not only contain the multi-dimensional information of the building element itself, but also incorporate the association information and spatial position relationship of the neighboring building elements. This significantly improves the feature vector's ability to represent the semantic, geometric and spatial attributes of building elements, laying a high-quality feature foundation for subsequent building element recognition, classification and other tasks.

[0077] After obtaining the semantic categories corresponding to each building element from the semantic reasoning model, as an optional real-time method, a building vector map with semantic labels can be generated based on the vector outline of each building element and its corresponding semantic category. This compensates for the shortcomings of pure vector maps, which only contain geometric information and lack semantic association, making the data more valuable for interpretation.

[0078] In this embodiment, addressing the technical challenge of accurately classifying architectural elements due to their standardized lines as core components, similar geometric shapes, and lack of unique textures and colors, a solution is proposed. This solution involves first acquiring an image of the building to be identified, then processing the image to generate vector contours of the architectural elements and extracting textual information containing location information. Semantic information of the architectural elements is then generated based on the location-related textual information and vector contours. Finally, the vector contours and semantic information are input into a semantic reasoning model. Since the textual information in the architectural image is associated with key semantics such as the function and type of the architectural elements, location-related textual information is transformed into semantic supplementary features of the architectural elements. This effectively compensates for the difficulty in distinguishing architectural element categories based solely on geometric shapes, enabling the semantic reasoning model to combine geometric and semantic features for comprehensive judgment. This achieves accurate determination of the semantic category of architectural elements, significantly improving the accuracy and reliability of architectural element category recognition.

[0079] Meanwhile, this embodiment also provides an electronic device, referring to Figure 13 As shown, Figure 13 This is a schematic diagram of the structure of an electronic device, which includes: One or more processors; A memory having stored one or more computer programs that, when executed by one or more processors, cause the one or more processors to implement the building element identification method according to the first aspect of the invention.

[0080] The electronic device may also include one or more I / O interfaces connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0081] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the first memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically such as SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, enabling information exchange between the processor and the memory, including but not limited to the data bus (Bus).

[0082] In some embodiments, the processor, memory, and I / O interfaces are interconnected via a bus, and thus connected to other components of the computing device.

[0083] As a third aspect of the present invention, a computer-readable medium is provided, with reference to... Figure 14 The above, Figure 14This is a schematic diagram of a computer-readable medium on which a computer program is stored, which, when executed by a processor, implements the building element identification method provided in the first aspect of this disclosure.

[0084] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0085] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A method of identifying a building element, characterized by, include: Acquire images of the building to be identified; The building image to be identified is processed to generate vector contours of building elements, and text information in the building image to be identified is extracted, the text information including text content and its position in the building image to be identified; Based on the location, the text information is associated with the vector outline to generate semantic information of the building elements; The vector outlines of the building elements and the semantic information are input into the semantic reasoning model to obtain the semantic categories output by the semantic reasoning model corresponding to each building element.

2. The method of claim 1, wherein, The semantic reasoning model includes a cascaded feature encoding module, a semantic reasoning module, and an output layer. The feature encoding module is used to encode the vector contour and the text information to output a multimodal feature vector for each building element. The semantic reasoning module is used to perform semantic reasoning based on local context on each of the multimodal feature vectors to output an enhanced feature vector. The output layer is used to map the enhanced feature vector to a semantic category probability vector and output the semantic category of each building element.

3. The method of claim 2, wherein, The feature encoding module includes a geometric feature encoding layer, a semantic feature encoding layer, and a multimodal fusion layer. The semantic information includes the association between indicator text and building elements, and the association between room text and building elements. The feature encoding module encodes the vector contour and the text information based on the following steps, outputting a multimodal feature vector for each building element: Geometric feature vectors are extracted from the vector contours of the building elements through the geometric feature encoding layer; The semantic feature encoding layer converts the indicator text corresponding to each building element into the first OneHot encoding, and the room text corresponding to each building element into the second OneHot encoding. The geometric feature vector is fused with the corresponding first OneHot code and second OneHot code through the multimodal fusion layer to obtain the multimodal feature vector.

4. The method of claim 2, wherein, The semantic reasoning module performs semantic reasoning based on local context on each of the multimodal feature vectors and outputs enhanced feature vectors based on the following steps: Calculate the centroid coordinates of the vector profile of each building element; Using the centroid coordinates of the vector profiles of each of the aforementioned building elements, calculate the centroid distance between all vector profile pairs; Using the centroid distance, the neighborhood vector profiles corresponding to each vector profile are determined; Calculate the relative positional offset between each of the vector contours and the domain vector contour; Multi-head attention is calculated on the multimodal feature vectors of each vector contour, the multimodal feature vectors of the corresponding neighborhood vector contours, and the relative position offset to obtain a single attention output; All the individual attention outputs are concatenated, and a linear transformation is performed on the concatenated individual attention outputs to obtain the enhanced feature vector.

5. The method of claim 1, wherein, The process of processing the building image to be identified to generate vector contours of building elements includes: The initial vector contour and edge image are extracted from the building image to be identified using a pre-trained vectorization model. The initial vector contour is parameterized into a set of primitives, wherein the primitives are straight lines, circular arcs, or second-order Bézier curves; By optimizing the parameters of the primitives to align them with the edge image, the vector outline of the building element is obtained.

6. The method of claim 5, wherein, The step of aligning the primitives with the edge image by optimizing their parameters to obtain the vector outline of the building element includes: Construct the energy interaction function between the primitive and the edge image; The position and size parameters of the primitive are determined using the energy interaction function. The graphic element is adjusted using the position parameters and the size parameters to obtain the adjusted graphic element; By splicing together the adjusted primitives belonging to the same architectural element, a vector outline corresponding to each architectural element is obtained.

7. The method according to any one of claims 1 to 6, characterized in that, The step of associating the text information with the vector contour based on the location to generate semantic information of the building elements includes: Based on the text content, the text information is classified into instruction text, room text, and invalid text; Filter out the invalid text. The semantic information of the building elements is obtained by associating the indicator text and the room text with the vector outline of the building element that is spatially closest to them, based on their positions.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: Generate a building vector image with semantic tags based on the vector outline of each building element and its corresponding semantic category.

9. An electronic device, comprising: include: One or more processors; A memory having stored one or more computer programs thereon, which, when executed by the one or more processors, cause the one or more processors to implement the building element identification method according to any one of claims 1 to 8.

10. A computer readable medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the building element identification method according to any one of claims 1 to 8.

Images