Real estate room semantic information extraction method and system based on building plan

Abstract

The invention discloses a method and system for extracting semantic information of real estate households and rooms based on building plans, and relates to the technical field of three-dimensional cadastral modeling. The method and system comprise the following steps: receiving a building plan, preprocessing the building plan, obtaining a real estate household and room image, inputting the real estate household and room image into a pre-established convolutional neural network model, outputting a multi-scale feature map containing key features of the household and room, flattening the multi-scale feature map to obtain a one-dimensional feature vector; inputting the one-dimensional feature vector into a pre-established Transformer Encoder for depth enhancement, and outputting a vector containing enhanced key features; inputting the vector into a pre-established Transformer Decoder for iterative polygon query refinement, and outputting polygon vertex coordinates representing the household and room and its components, and generating a vector map according to the polygon vertex coordinates representing the household and room and its components.

Description

Technical Field

[0001] The invention relates to the technical field of three-dimensional cadastral modeling, and in particular to a method and system for extracting semantic information of real estate households and rooms based on a building plan. Background Art

[0002] The traditional two-dimensional cadastral management model can no longer fully reflect the ownership and utilization of land in three-dimensional space. Therefore, it is necessary to establish a three-dimensional land management model with three-dimensional cadastral as the core. Three-dimensional cadastral modeling is a key step in achieving the goal of three-dimensional cadastral management.

[0003] A building plan is a drawing that shows the walls, doors, windows, stairs, floors, and internal functional layout of a new building or structure in a horizontal projection method and with corresponding legends. It contains a large amount of geometric, semantic, and relational information, and can be used as a rich information source for 3D cadastral modeling. At present, semantic extraction methods in building plans can be roughly divided into two categories: manual extraction and automated extraction. Manual extraction methods are time-consuming, labor-intensive, and have low economic benefits, and are prone to human errors in complex data processing and semantic extraction processes. Automated extraction methods usually identify building components, lack the extraction of household and room semantic information, and the reconstruction process of building components. The extracted information cannot be directly used as a data source for 3D cadastral modeling. Summary of the invention

[0004] In order to address the deficiencies mentioned in the above-mentioned background technology, the purpose of the present invention is to provide a method and system for extracting semantic information of real estate households and rooms based on building floor plans, which can solve the problems of time-consuming and labor-intensive manual extraction of real estate semantics from building floor plans and the lack of extraction of household and room semantic information and reconstruction of building components in related automated extraction methods.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a method for extracting semantic information of real estate rooms based on a building plan, the method comprising the following steps:

[0006] Receiving a building plan, preprocessing the building plan to obtain a real estate room image, inputting the real estate room image into a pre-established convolutional neural network model, outputting a multi-scale feature map containing key features of the room, and flattening the multi-scale feature map containing the key features of the room to obtain a one-dimensional feature vector;

[0007] Input the one-dimensional feature vector into the pre-established Transformer Encoder for deep enhancement, and output a vector containing enhanced key features;

[0008] The vector containing the enhanced key features is input into the pre-established Transformer Decoder for iterative polygon query refinement, and the coordinates of the polygon vertices representing the household room and its components are output, and a vector map is generated based on the coordinates of the polygon vertices representing the household room and its components.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the preprocessing of the building plan includes removing interference information of the axis, image format conversion and image standardization; the process is as follows:

[0010] The removal of interference information of the notes and axes is as follows: remove the interference information of the text notes and axes in the building plan, retain the important structural information of the walls, doors and windows,

[0011] Image format conversion: Convert architectural floor plans in DXF format to PNG format;

[0012] Image standardization is to standardize and normalize the image size.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: selecting ResNet-101 as the pre-established convolutional neural network model;

[0014] The residual module of the ResNet-101 network generates a multi-scale feature map containing household rooms. The number of channels of the multi-scale feature map is unified through convolution, and then the feature map is flattened to obtain a one-dimensional feature vector composed of several pixel values.

[0015] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the expression of the ResNet-101 network is as follows:

[0016] The ResNet-101 backbone consists of five modules:

[0017] Initial convolutional layer and max pooling layer ;

[0018] Residual Module 1: ;

[0019] Residual Module 2: ;

[0020] Residual Module 3: ;

[0021] Residual Module 4 ;

[0022] :Indicates the use of Convolution operation is performed with convolution kernels of different sizes;

[0023] :Indicates the use of The maximum pooling operation is performed on the window of size;

[0024] : Indicates the step length is , used to describe the distance the window moves each time in a convolution or pooling operation;

[0025] : Represents the number of channels of the convolution kernel in each convolution operation.

[0026] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the pre-established Transformer Encoder includes: a plurality of encoder layers, each layer consisting of two modules: a multi-head self-attention module and a feedforward network;

[0027] The one-dimensional feature vector is mapped to a 256-dimensional vector space through the encoder input embedding layer, and the corresponding position code is added to the feature vector corresponding to each pixel. Then, it is passed through a multi-head attention mechanism and a feedforward network to capture the vector information of the geometric, semantic, and relational features of the room, extract local and global features as well as more complex relationships between pixels, and output a vector containing enhanced key features.

[0028] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the expression of the pre-established Transformer Encoder is as follows:

[0029] Multi-head attention module:

[0030]

[0031]

[0032]

[0033] : represents the final output of the multi-head attention module; : A function that concatenates multiple vectors or matrices together in sequence; : represents the output of all heads in the model; : indicates the total number of heads; : represents the output transformation matrix; It is The output result of each head; They are The query, key, and value transformation matrix of each head;

[0034] is the dimension of the key vector; is the attention calculation function; : represent query vector, key vector and value vector respectively; : probability distribution function, and The dot product result of is converted into a probability distribution;

[0035] Feedforward Network:

[0036]

[0037] : Input vector, which is the input data of the feedforward network, : The weight matrix of the first linear transformation; : The bias vector of the first linear transformation; : Activation function; The weight matrix of the second linear transformation; : Bias vector for the second linear transformation.

[0038] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the pre-established Transformer Decoder includes: a plurality of decoder layers, each layer consisting of three main modules: a multi-head attention module, a multi-head attention module, and a feedforward network;

[0039] The expression of the pre-built Transformer Decoder is as follows:

[0040] Masked Multi-Head Attention Module:

[0041]

[0042]

[0043]

[0044] In the original attention function Add a mask to shield the information of the next position;

[0045] The masked multi-head attention function is expressed as:

[0046]

[0047] : represents bitwise multiplication; : represents the mask matrix;

[0048] As well as multi-head attention modules and feed-forward networks.

[0049] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of inputting the vector containing the enhanced key feature into a pre-established Transformer Decoder for iterative polygon query refinement is as follows:

[0050] Create a polygon query:

[0051] Define a set consisting of several polygons: ; Each polygon ; Each vertex ,in represents the probability of a valid vertex, Represent vertex coordinates, using polygons Represents the prediction of the room and its components, which is passed in the last layer of Transformer Decoder Function predict polygon For a building plan, Set four types of labels: room, door, window, and passage, corresponding to four types of polygon sets respectively

[0052] ;

[0053] The relationship between the predicted set and the true set is established through polygon matching:

[0054] The real coordinates of the ground are Represents that the polygon ;vertex ,for , first Fill to The length of the vertex, and then further expand middle arrive The length of Set each vertex in the polygon .

[0055] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of generating a vector diagram according to the coordinates of the polygonal vertices representing the household room and its components:

[0056] The polygon Four types of polygon sets are matched through type prediction In, they represent the predictions of room, door, window, and passage polygons, respectively;

[0057] For polygon collection Polygons in The vertex coordinates are automatically connected to obtain the line layer, the line layer is topologically reconstructed and a buffer is established to generate the surface layer ; Connect polygon collection Get the line segment from the coordinates of the two points in the image, and create a buffer for the line segment to generate a surface layer. ; Area layer and Perform intersection and clipping operations and match type tags to generate door, window, and wall layers.

[0058] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a real estate room semantic information extraction system based on a building plan, comprising:

[0059] An image processing module is used to receive a building plan, pre-process the building plan to obtain a real estate room image, input the real estate room image into a pre-established convolutional neural network model, output a multi-scale feature map containing key features of the room, and flatten the multi-scale feature map containing the key features of the room to obtain a one-dimensional feature vector;

[0060] The deep enhancement module is used to input the one-dimensional feature vector into the pre-established Transformer Encoder for deep enhancement, and output a vector containing enhanced key features;

[0061] The layer generation module is used to input the vector containing the enhanced key features into the pre-established TransformerDecoder for iterative polygon query refinement, output the coordinates of the polygon vertices representing the household room and its components, and generate a vector map based on the coordinates of the polygon vertices representing the household room and its components.

[0062] Beneficial effects of the present invention:

[0063] The present invention uses deep learning technology of convolutional neural networks and Transformer models through automated and intelligent processing procedures to accurately extract key geometric, semantic and relational features from building plans, and effectively convert these features into polygon vertex coordinates, and finally automatically generate vector maps. The present invention provides a new solution for extracting real estate semantic information from building plans, reduces the errors and costs of manual extraction, and greatly improves the efficiency of information extraction, providing rich and high-quality data for three-dimensional cadastral modeling, which can be directly used as a data source for three-dimensional cadastral modeling, and has broad application prospects and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 It is a schematic flow chart of the method of the present invention;

[0066] Figure 2 This is the building floor plan data diagram of this embodiment;

[0067] Figure 3 This is a flow chart of building plan data preprocessing in this embodiment;

[0068] Figure 4 This is the main diagram of the convolutional neural network ResNet-101 architecture of this embodiment;

[0069] Figure 5 This is the Transformer Encoder architecture diagram of this embodiment;

[0070] Figure 6 This is the Transformer Decoder architecture diagram of this embodiment;

[0071] Figure 7 This is a diagram showing the results of this embodiment;

[0072] Figure 8 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Embodiment 1:

[0075] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0076] Convolutional Neural Networks (CNN) are a type of feedforward neural network that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure. Therefore, they are also called "Shift-Invariant Artificial Neural Networks (SIANN)".

[0077] Multi-scale features refer to features extracted at different spatial scales. In image processing, scale is usually associated with the size of the observed object. By extracting features at different scales, information of different levels and granularity can be captured, thereby improving the understanding and analysis of image content. The importance of multi-scale features lies in the fact that objects in an image may have different sizes, shapes, and structures, and these changes can be better captured by using multi-scale features. In addition, multi-scale features also help solve problems caused by scale changes of objects in images, such as scale changes caused by the distance of the target, occlusion, etc.

[0078] like Figure 1 As shown, a method for extracting semantic information of real estate rooms based on a building plan includes the following steps:

[0079] K101: receiving a building plan, preprocessing the building plan to obtain a real estate room image, inputting the real estate room image into a pre-established convolutional neural network model, outputting a multi-scale feature map containing key features of the room, and flattening the multi-scale feature map containing the key features of the room to obtain a one-dimensional feature vector;

[0080] This method uses the building floor plan as the basic data, such as Figure 2 As shown in the figure, the architectural plan describes in detail the floor plan layout of each floor of the building, including the location and size of the indoor space such as rooms, corridors, stairs, elevators, and the location and size of architectural elements such as walls, doors, windows, and structural components, as well as annotation information such as room type. The original architectural plan is annotated and the interference information such as the axis is removed, the image format is converted, and the automatic preprocessing module is used for processing. The process is as follows: Figure 3 The details are as follows:

[0081] Use CAD drawing software to process the original building plan layer by layer:

[0082] S101: Delete interfering information such as axes, annotations, isolated lines, and pattern fills in the drawing, and retain important structural information of household spaces such as walls, doors, and windows.

[0083] S102: The building plan after removing the interference information is converted into an image format through the export function of the CAD drawing software, and the DXF format graphic file is converted into a high-resolution PNG image format.

[0084] S103: using the OpenCV library to establish an automatic image preprocessing module, and performing image size standardization (adjusting the image size to a uniform size) and normalization processing (scaling the image pixel value to a range of [0, 1]) through scaling functions and normalization functions, to obtain a 224*224 pixel standard real estate room image describing the room structure and semantic information;

[0085] The pre-built convolutional neural network model selected is ResNet-101;

[0086] The residual module of the ResNet-101 network generates a multi-scale feature map, which is reduced through convolution to obtain key features. The key features are flattened to obtain a one-dimensional feature vector composed of several pixel values.

[0087] The main structure of ResNet-101 is as follows Figure 4 As shown in the figure, the architecture is used to extract key features of indoor images, as follows:

[0088] The 224*224 standard real estate room image is input into the ResNet-101 architecture and passed through an initial convolutional layer. and max pooling layer A preliminary feature extraction is performed; the image is then fed into four residual modules to capture feature information at different scales. Three sets of multi-scale feature maps are extracted from the first three residual modules, with sizes and channel numbers of 56×56 pixels, 256 channels; 28×28 pixels, 512 channels; and 14×14 pixels, 1024 channels. A 1×1 convolution operation is applied to all feature maps to reduce their channel numbers to 128.

[0089] The multi-scale feature map obtained by 1×1 convolution kernel processing is flattened: first, the two-dimensional feature map is converted into a continuous one-dimensional pixel sequence, and then each pixel sequence is connected in series to construct a one-dimensional feature vector that integrates rich spatial hierarchical information. Such a feature vector provides a comprehensive multi-scale feature representation for the deep learning model, greatly enhancing the model's ability to parse and understand indoor image data.

[0090] The expression of the ResNet-101 network is as follows:

[0091] The ResNet-101 backbone consists of five modules:

[0092] Initial convolutional layer and max pooling layer ;

[0093] Residual Module 1: ;

[0094] Residual Module 2: ;

[0095] Residual Module 3: ;

[0096] Residual Module 4 ;

[0097] :Indicates the use of The convolution operation is performed with a convolution kernel of size .

[0098] :Indicates the use of The maximum pooling operation is performed on a window of size .

[0099] : Indicates the step length is , used to describe the distance the window moves each time in a convolution or pooling operation.

[0100] : Represents the number of channels of the convolution kernel in each convolution operation.

[0101] K102: Input the one-dimensional feature vector into the pre-established Transformer Encoder for deep enhancement, and output a vector containing the enhanced key features;

[0102] The Transformer Encoder part is as follows Figure 5 As shown in the figure, 6 encoder layers are set, each layer consists of two main modules: Multi-Head Attention module and Feed-Forward Network. The encoder performs deep enhancement on these image features (one-dimensional feature vectors) extracted by the convolutional neural network to capture the vector information of the geometric, semantic and relational features of the room. The details are as follows:

[0103] The one-dimensional feature vector output by the convolutional neural network is mapped to a 256-dimensional vector space through the encoder input embedding layer, and the corresponding position code is added to the feature vector corresponding to each pixel. The formula is:

[0104]

[0105]

[0106] in represents the position code, Indicates the position of the pixel in the sequence (starting from 0), Indicates the model dimension (set to 256 in Transformer Encoder), is the dimension index in the positional encoding vector (starting from 0).

[0107] The feature vector after adding position encoding is passed through a multi-head attention mechanism and a feedforward network to capture the vector information of the geometric, semantic and relational features of the room, extract local features (such as edge detail information, door and window semantic information) and global features (such as overall shape information), and understand the relationship between pixels. The feedforward neural network consists of two fully connected layers, which can learn deeper feature representations and capture more complex pixel relationships. In addition, the model also adds two residual connections and layer normalization (Add&Norm) modules, which helps to avoid the gradient vanishing problem in deep networks. The six encoder layers are connected in sequence to gradually extract the feature representation of the input vector, and finally output the feature vector (encoder output) after deep enhancement from the last layer of encoder, including enhanced key features.

[0108] The expression of the pre-built Transformer Encoder is as follows:

[0109] Multi-head attention module:

[0110]

[0111]

[0112]

[0113] : represents the final output of the multi-head attention module; : A function that concatenates multiple vectors or matrices together in sequence; : represents the output of all heads in the model; : represents the total number of heads, that is, the number of subspaces into which the self-attention mechanism in the model is divided; : represents the output transformation matrix, which is used to linearly transform the concatenated vectors to integrate the information of different heads; It is The output result of each head; They are The query, key, and value transformation matrix of each head;

[0114] is the dimension of the key vector; is the attention calculation function; : represent query vector, key vector and value vector respectively; : probability distribution function, used to convert and The dot product result of is converted into a probability distribution, that is, a weight is assigned to each possible match.

[0115] Feedforward Network:

[0116]

[0117] : Input vector, which is the input data of the feedforward network. : The weight matrix of the first linear transformation. This is a linear mapping from the input space to the intermediate feature space; : The bias vector of the first linear transformation; : activation function, which replaces negative values ​​in the input element by element , while keeping the positive value unchanged; The weight matrix of the second linear transformation. It is The activation function is applied after that to map features from the intermediate feature space to the output space; : Bias vector for the second linear transformation.

[0118] K103: Input the vector containing the enhanced key features into the pre-established Transformer Decoder for iterative polygon query refinement, output the coordinates of the polygon vertices representing the household room and its components, and generate a vector map based on the coordinates of the polygon vertices representing the household room and its components.

[0119] The expression of the pre-built Transformer Decoder is as follows:

[0120] Masked Multi-Head Attention Module:

[0121]

[0122]

[0123]

[0124] The difference is that in the original attention function A mask is added to mask the information of the next position, ensuring that each step of decoding depends only on the output that has been generated. The masked multi-head attention function can be expressed as:

[0125]

[0126] : represents bitwise multiplication; : represents the mask matrix.

[0127] Multi-head attention module: as described in the Transformer Encoder expression;

[0128] Feedforward network: as described in Transformer Encoder expression.

[0129] The Transformer Decoder part is as follows Figure 6 As shown in the figure, 6 decoder layers are set, and each layer consists of three main modules: Masked-Multi-Head Attention Module, Multi-Head Attention Module, and Feed-Forward Network. A polygon query is established to transform the problem of extracting real estate semantic household information from the building plan into a vertex prediction problem of a polygon set. During the training phase, the model can reduce the error between the predicted coordinates and the true coordinates through iterative polygon query refinement. After training, the model can accurately identify the vertex coordinates of polygons representing households and their components. The details are as follows:

[0130] Establish a polygon query. This module consists of several polygons and vertex sets: Define a set consisting of several polygons: ; Each polygon ; Each vertex ,in represents the probability of a valid vertex, Represents vertex coordinates. Indicates that it contains multiple vertices Represents the corner point of a room; doors, windows, and passages in real estate rooms are also represented by Indicates that they are considered as polygons with only two vertices . Several polygons identified in the building plan In the initial stage, the specific type is not specified; in the output embedding of the last layer of Transformer Decoder, extract The average eigenvalues ​​of the vectors representing the vertices in the image are calculated to obtain the aggregated eigenvector of each polygon. The comprehensive eigenvector of each polygon is obtained by averaging the eigenvalues ​​of these vectors and then using a linear projection layer to map these features to possible polygon type labels. function to predict a probability distribution for each polygon. For an example of a building floor plan, There are four types of rooms, doors, windows, and passages, corresponding to four types of polygon sets. .

[0131] The connection between the predicted set and the true set is established through polygon matching. The prediction module of Transformer Decoder outputs a fixed number of polygons, each of which includes a fixed number of polygon vertices. The true value contains an uncertain number of vertices and polygons, which are matched through the following steps:

[0132] ① A set of polygons predicted by the model is used Indicates that the predicted polygon is Indicates that the vertex = (c, [x, y]), where c represents the probability of a valid vertex and [x, y] represents the predicted vertex coordinates.

[0133] ② The real polygon set is represented by V={ , , ,..., } means that the polygon ={ , , ,..., };vertex for , first Fill to The length of the vertex, and then further expand middle arrive The length of Set each vertex in the polygon .

[0134] The dataset FloorPlan is created, which contains 2,000 floor plan data with different floors and household and semantic coordinate annotations, 1,600 pre-defined training samples, 200 validation samples and 200 test samples. The input of Transformer Decoder consists of three parts: the output vector of the encoder, polygon query, and decoder embedding; during model training, the polygon query and decoder embedding (Decoder Embeddings) input to the first layer of decoder are randomly initialized and generated by normal distribution, and they are learned as part of the model parameters. After each layer of decoder polygon query is gradually iterated, gradually reducing the error between the predicted coordinates and the real coordinates; in the prediction stage, we directly load the learned polygon query and decoder embedding, and update them layer by layer, so that the model can accurately predict the vertex coordinates of the polygon representing the house and its semantics.

[0135] Specifically, the present invention is further described below by way of embodiments:

[0136] For a building floor plan example, the polygon predictions of its rooms, doors, windows, and passages are defined as four types of polygon sets. , in each polygon set and They are identified by their unique labels, which are directly related to the predicted attributes they represent, i.e., rooms, doors, windows or passages. Through ArcEngine vector layer editing, spatial analysis and other related functions, vector maps are automatically generated for the vertex coordinates in the predicted polygon set, as follows:

[0137] Room vector map generation: prediction polygon set Room polygons in , the vertex coordinates Connect in sequence to generate a vector line layer of rooms For line layers First, set a certain threshold. When the distance between two line segments is less than the preset threshold, merge them into one line segment; then connect the hanging line segments with isolated vertices to the nearest line segments, thus reconstructing the new topological relationship. Create a buffer surface (radius is 1 / 2 of the wall) to get a vector surface map .

[0138] Wall, door, window vector map generation: predicted polygon collection Polygons in From two point coordinates Composition, connecting each polygon Get several line segments using Indicates that each line segment is expanded to generate a buffer surface (The radius is 1.5 times the wall), a one-to-one correspondence is formed between polygons, line segments and buffer surfaces; The label type matches the buffer surface Then, ask and The intersection of Zhongyu The intersecting parts are divided according to the label attributes Divided into three types of layers

[0139] , respectively, are the vector diagrams of doors, windows, and passages; Cut off Get a new layer for the wall layer; The channel layer is not output, and the final output layer is (Door layer), (Window layer), (Wall layer).

[0140] Embodiment 2: In the second aspect, as Figure 8 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a real estate room semantic information extraction system based on a building plan, comprising:

[0141] The image processing module 11 is used to receive the building plan, pre-process the building plan to obtain the real estate room image, input the real estate room image into a pre-established convolutional neural network model, output a multi-scale feature map containing key features of the room, and flatten the multi-scale feature map containing the key features of the room to obtain a one-dimensional feature vector;

[0142] A depth enhancement module 12 is used to input the one-dimensional feature vector into a pre-established Transformer Encoder for depth enhancement, and output a vector containing enhanced key features;

[0143] The layer generation module 13 is used to input the vector containing the enhanced key features into the pre-established Transformer Decoder for iterative polygon query refinement, output the polygon vertex coordinates representing the household room and its components, and generate a vector map based on the polygon vertex coordinates representing the household room and its components.

[0144] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0145] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0147] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A method for extracting semantic information of real estate rooms based on building plan, characterized in that: The method comprises the following steps: Receiving a building plan, preprocessing the building plan to obtain a real estate room image, inputting the real estate room image into a pre-established convolutional neural network model, outputting a multi-scale feature map containing key features of the room, and flattening the multi-scale feature map containing the key features of the room to obtain a one-dimensional feature vector; Input the one-dimensional feature vector into the pre-established Transformer Encoder for deep enhancement, and output a vector containing enhanced key features; The pre-established Transformer Encoder includes: a plurality of encoder layers, each layer consisting of two modules: a multi-head self-attention module and a feed-forward network; The one-dimensional feature vector is mapped to a 256-dimensional vector space through the encoder input embedding layer, and the corresponding position code is added to the feature vector corresponding to each pixel. Then, the multi-head attention mechanism and feedforward network are used to capture the vector information of the geometric, semantic and relational features of the room, extract local features and global features as well as more complex relationships between pixels, and output a vector containing enhanced key features. The expression of the pre-established Transformer Encoder is as follows: Multi-head attention module: MultiHead(Q,K,V)=Concat(head1,...,head h )W o MultiHead(Q, K, V): represents the final output of the multi-head attention module; Concat: a function that concatenates multiple vectors or matrices in sequence; head1, ..., head h : represents the output of all heads in the model; h: represents the total number of heads; W o : represents the output transformation matrix; head i is the output result of the i-th head; are the query, key, and value transformation matrices of the i-th head respectively; d k is the dimension of the key vector; Attention is the attention calculation function; Q, K, V: represent the query vector, key vector and value vector respectively; softmax: probability distribution function, converts the dot product result of Q and K into probability distribution; Feedforward Network: FFN(x)=max(0,xW1+b1)W2+b2 x: input vector, which is the input data of the feedforward network, W1: weight matrix of the first linear transformation; b1: bias vector of the first linear transformation; max(): ReLU activation function; W2: weight matrix of the second linear transformation; b2: bias vector of the second linear transformation; Input the vector containing the enhanced key features into the pre-established Transformer Decoder for iterative polygon query refinement, output the coordinates of the polygon vertices representing the room and its components, and generate a vector map based on the coordinates of the polygon vertices representing the room and its components; The pre-established Transformer Decoder includes: multiple decoder layers, each layer consisting of three main modules: a masked multi-head attention module, a multi-head attention module, and a feedforward network; The expression of the pre-built Transformer Decoder is as follows: Masked Multi-Head Attention Module: MultiHead(Q,K,V)=Concat(head1,...,head h )W o Add a mask to the original attention function Attention(Q, K, V) to shield the information of the next position; The masked multi-head attention function is expressed as: Represents bitwise multiplication; M: represents the mask matrix; As well as multi-head attention modules and feed-forward networks; The process of inputting the vector containing the enhanced key features into the pre-established Transformer Decoder for iterative polygon query refinement is as follows: Create a polygon query: Define a set consisting of several polygons: Each polygon Each vertex Where c represents the probability of a valid vertex, [x, y] represents the vertex coordinates, and the polygon Represents the prediction of the room and its components. The polygon is predicted by the softmax function in the last layer of Transformer Decoder For a building plan, Set four types of labels: room, door, window, and passage, corresponding to four types of polygon sets respectively The relationship between the predicted set and the true set is established through polygon matching: The real coordinate value of the ground is V = {Q1, Q2, Q3, ..., Q N } means that the polygon Q n = {P1, P2, P3, ..., P M }; Vertex P m =(c,[x,y]), for Q n First, P M Fill to The length of the vertex, and then further expand the Q in V N arrive The length of Set each vertex P in the polygon m =(0,[x,y]).

2. The method for extracting semantic information of real estate rooms based on building plan according to claim 1, characterized in that: The preprocessing of the architectural plan includes annotation, removal of interference information of the axis, image format conversion and image standardization; the process is as follows: The removal of the interference information of the annotations and axes is as follows: removing the interference information of the text annotations and axes of the building plan, retaining the important structural information of the walls, doors and windows, Image format conversion: Convert architectural floor plans in DXF format to PNG format; Image standardization is to standardize and normalize the image size.

3. The method for extracting semantic information of real estate rooms based on building plan according to claim 1, characterized in that: The pre-established convolutional neural network model selects ResNet-101; The residual module of the ResNet-101 network generates a multi-scale feature map containing the key features of the household room. The number of channels of the multi-scale feature map is unified through convolution, and then the feature map is flattened to obtain a one-dimensional feature vector composed of several pixel values.

4. The method for extracting semantic information of real estate rooms based on building plan according to claim 3, characterized in that: The expression of the ResNet-101 network is as follows: The ResNet-101 backbone consists of five modules: Initial convolutional layer (conv_7×7, 64, stride2) and max pooling layer (Maxpooling_3×3, stride2); Residual module 1: [(conv_1×1, 64); (conv_3×3, 64); (conv_1×1, 256)]×3; Residual module 2: [(conv_1×1, 128); (conv_3×3, 128); (conv_1×1.512)]×4; Residual module 3: [(conv_1×1.256); (conv_3×3, 256); (conv_1×1, 1024)]×23; Residual module 4[(conv_1×1, 512); (conv_3×3, 512); (conv_1×1, 2048)]×3; conv n×n: indicates that a convolution operation is performed using a convolution kernel of size n×n; Maxpooling_3×3: indicates that a maximum pooling operation is performed using a 3×3 window; stride=2: indicates a stride of 2, which is used to describe the distance the window moves each time in a convolution or pooling operation; 64.128.256.512.1024.2048: represent the number of channels of the convolution kernel in each convolution operation.

5. The method for extracting semantic information of real estate rooms based on building plan according to claim 1, characterized in that: The process of generating a vector diagram based on the coordinates of the polygonal vertices representing the room and its components: The polygon Four types of polygon sets are matched through type prediction In, they represent the predictions of room, door, window, and passage polygons, respectively; For polygon collection Polygons in The vertex coordinates of the line layer are automatically connected to obtain the line layer, the line layer is topologically reconstructed and a buffer is established to generate the surface layer A; connect the polygon set The line segment is obtained by using the coordinates of the two points in the image, and a buffer is established for the line segment to generate the surface layer B. The surface layers A and B are intersected and clipped, and the type labels are matched to generate the door, window, and wall layers.

6. A real estate room semantic information extraction system based on building plan, characterized in that: include: An image processing module is used to receive a building plan, pre-process the building plan to obtain a real estate room image, input the real estate room image into a pre-established convolutional neural network model, output a multi-scale feature map containing key features of the room, and flatten the multi-scale feature map containing the key features of the room to obtain a one-dimensional feature vector; The deep enhancement module is used to input the one-dimensional feature vector into the pre-established Transformer Encoder for deep enhancement, and output a vector containing enhanced key features; The pre-established Transformer Encoder includes: a plurality of encoder layers, each layer consisting of two modules: a multi-head self-attention module and a feed-forward network; The one-dimensional feature vector is mapped to a 256-dimensional vector space through the encoder input embedding layer, and the corresponding position code is added to the feature vector corresponding to each pixel. Then, the multi-head attention mechanism and feedforward network are used to capture the vector information of the geometric, semantic and relational features of the room, extract local features and global features as well as more complex relationships between pixels, and output a vector containing enhanced key features. The expression of the pre-established Transformer Encoder is as follows: Multi-head attention module: MultiHead(Q,K,V)=Concat(head1,...,head h )W o MultiHead(Q, K, V): represents the final output of the multi-head attention module; Concat: a function that concatenates multiple vectors or matrices in sequence; head1, ..., headh: represents the output of all heads in the model; h: represents the total number of heads; W o : represents the output transformation matrix; head i is the output result of the i-th head; are the query, key, and value transformation matrices of the i-th head respectively; d k is the dimension of the key vector; Attention is the attention calculation function; Q, K, V: represent the query vector, key vector and value vector respectively; softmax: probability distribution function, converts the dot product result of Q and K into probability distribution; Feedforward Network: FFN(x)=max(0,xW1+b1)W2+b2 x: input vector, which is the input data of the feedforward network, W1: weight matrix of the first linear transformation; b1: bias vector of the first linear transformation; max(): ReLU activation function; W2: weight matrix of the second linear transformation; b2: bias vector of the second linear transformation; A layer generation module is used to input the vector containing the enhanced key features into a pre-established TransformerDecoder for iterative polygon query refinement, output the coordinates of the polygon vertices representing the room and its components, and generate a vector map according to the coordinates of the polygon vertices representing the room and its components; The pre-established Transformer Decoder includes: multiple decoder layers, each layer consisting of three main modules: a masked multi-head attention module, a multi-head attention module, and a feedforward network; The expression of the pre-built Transformer Decoder is as follows: Masked Multi-Head Attention Module: MultiHead(Q,K,V)=Concat(head1,...,head h )W o Add a mask to the original attention function Attention(Q, K, V) to shield the information of the next position; The masked multi-head attention function is expressed as: Represents bitwise multiplication; M: represents the mask matrix; As well as multi-head attention modules and feed-forward networks; The process of inputting the vector containing the enhanced key features into the pre-established Transformer Decoder for iterative polygon query refinement is as follows: Create a polygon query: Define a set consisting of several polygons: Each polygon Each vertex Where c represents the probability of a valid vertex, [x, y] represents the vertex coordinates, and the polygon Represents the prediction of the room and its components. The polygon is predicted by the softmax function in the last layer of Transformer Decoder For a building plan, Set four types of labels: room, door, window, and passage, corresponding to four types of polygon sets respectively The relationship between the predicted set and the true set is established through polygon matching: The real coordinate value of the ground is V = {Q1, Q2, Q3, ..., Q N } means that the polygon Q n = {P1, P2, P3, ..., P M }; Vertex P m =(c,[x,y]), for Q n First, P M Fill to The length of the vertex, and then further expand the Q in V N arrive The length of Set each vertex P in the polygon m =(0,[x,y]).

Images