Building cad object recognition method and device, electronic equipment and medium

Abstract

The application discloses a kind of building CAD object identification method, device, electronic equipment and medium, is particularly related to image recognition technical field, method includes: the feature of building CAD object is input into routing network, obtains multiple multi-modal large model respectively corresponding weight coefficient;Building CAD object is identified respectively using multiple multi-modal large model, and the prediction probability that building CAD object output by each multi-modal large model belongs to each candidate class is obtained;According to the weight coefficient corresponding to each multi-modal large model respectively, the prediction probability output by each multi-modal large model is weighted and fused, and the preliminary identification result of building CAD object is determined;According to specification rule base, preliminary identification result is normatively checked, and specification rule base stores the constraint condition of building CAD object extracted from building design specification file;In the case where preliminary identification result meets constraint condition, preliminary identification result is taken as the identification result of building CAD object.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, electronic device, and medium for recognizing architectural CAD objects. Background Technology

[0002] In architectural engineering, architectural drawings contain a wide variety and a large number of CAD objects. Currently, one way to identify architectural CAD objects is to use traditional machine learning models to classify and identify them; however, this method has a low accuracy rate for CAD objects not covered in the training data.

[0003] Another approach is to use a single large model to identify architectural CAD objects; however, when using this approach, the accuracy of the identification results is not high when dealing with CAD objects of various types and formats.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problem of low recognition accuracy when using a single large model to identify architectural CAD objects of various types and formats, and to provide a method, device, electronic device and medium for recognizing architectural CAD objects.

[0006] The first aspect of this invention provides a method for identifying architectural CAD objects. The method includes: inputting features of an architectural CAD object into a routing network to obtain weight coefficients corresponding to multiple multimodal large models; using the multiple multimodal large models to identify the architectural CAD object respectively, obtaining the predicted probability of the architectural CAD object belonging to each candidate category output by each multimodal large model; weighting and fusing the predicted probabilities output by each multimodal large model according to the weight coefficients corresponding to each multimodal large model to obtain the fusion probability of the architectural CAD object belonging to each candidate category, and determining a preliminary identification result of the architectural CAD object based on the fusion probability; performing normative verification on the preliminary identification result according to a normative rule base, wherein the normative rule base stores constraints on the architectural CAD object extracted from architectural design code documents; if the preliminary identification result meets the constraints, using the preliminary identification result as the identification result of the architectural CAD object; if the preliminary identification result does not meet the constraints, correcting the preliminary identification result according to the constraints to obtain the identification result of the architectural CAD object.

[0007] In one embodiment of the present invention, before inputting the features of the architectural CAD object into the routing network, the method further includes: extracting the visual features of the architectural CAD object through an image encoder, and extracting the text features of the architectural CAD object through a text encoder; and then concatenating the visual features and the text features after normalization processing to obtain the features of the architectural CAD object.

[0008] In one embodiment of the present invention, the routing network includes a multilayer perceptron; wherein, the step of inputting the features of the architectural CAD object into the routing network to obtain the weight coefficients corresponding to each of the multiple multimodal large models includes: inputting the features of the architectural CAD object into the multilayer perceptron, and normalizing the output of the multilayer perceptron to obtain the weight coefficients corresponding to each of the multiple multimodal large models.

[0009] In one embodiment of the present invention, the plurality of multimodal large models are obtained by training a plurality of pre-trained multimodal large models using a training set; the training set includes CAD object recognition task data, which is obtained by cropping and text recognition processing of architectural CAD drawings.

[0010] In one embodiment of the present invention, the training set further includes normative rule instruction data, which is obtained by converting the constraints into instruction-response pairs; the constraints include at least one of geometric feature rules, topological relationship rules, and annotation specification rules, wherein the geometric feature rules are constraints on the geometric shape of the architectural CAD object, the topological relationship rules are constraints on the spatial positional relationship between the architectural CAD objects, and the annotation specification rules are constraints on the text annotation format of the architectural CAD object.

[0011] In one embodiment of the present invention, the loss function used in the training includes a task loss term and a rule constraint loss term; the task loss term is determined based on the prediction probability of the trained multimodal large model for the true category; the rule constraint loss term is a penalty term determined based on the prediction probability assigned by the trained multimodal large model to categories that do not meet the constraint conditions.

[0012] In one embodiment of the present invention, the step of correcting the preliminary identification result according to the constraints to obtain the identification result of the architectural CAD object includes: selecting the first candidate category that meets the constraints from the candidate categories according to the fusion probability from high to low, as the identification result of the architectural CAD object; if none of the candidate categories meet the constraints, the reason why the preliminary identification result does not meet the constraints is used as a supplementary condition, and re-inputted into the multiple multimodal large models together with the architectural CAD object for re-identification; if the result obtained from the re-identification still does not meet the constraints, the architectural CAD object is marked as pending manual confirmation.

[0013] A second aspect of the present invention provides a building CAD object recognition device, the device comprising: a weight determination module, configured to input the features of a building CAD object into a routing network to obtain weight coefficients corresponding to each of multiple multimodal large models; a recognition module, configured to use the multiple multimodal large models to recognize the building CAD object respectively, and obtain the predicted probability of the building CAD object belonging to each candidate category output by each of the multimodal large models; a fusion module, configured to perform weighted fusion of the predicted probabilities output by each of the multimodal large models according to the weight coefficients corresponding to each of the multimodal large models, to obtain the fusion probability of the building CAD object belonging to each candidate category, and determine a preliminary recognition result of the building CAD object based on the fusion probability; a verification module, configured to perform normative verification of the preliminary recognition result according to a normative rule base, the normative rule base storing constraints of the building CAD object extracted from building design code documents; and a correction module, configured to, if the preliminary recognition result meets the constraints, use the preliminary recognition result as the recognition result of the building CAD object; if the preliminary recognition result does not meet the constraints, correct the preliminary recognition result according to the constraints to obtain the recognition result of the building CAD object.

[0014] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first aspect.

[0016] Compared with the prior art, the technical effects achieved by the present invention are as follows:

[0017] Multiple multimodal large models are used to identify architectural CAD objects separately, and the predicted probability of the architectural CAD object belonging to each candidate category is obtained from the output of each multimodal large model. According to the weight coefficients corresponding to each multimodal large model, the predicted probabilities output by each multimodal large model are weighted and fused to obtain the fused probability of the architectural CAD object belonging to each candidate category, and the preliminary identification result of the architectural CAD object is determined based on the fused probability. With the help of a routing network, weight coefficients are determined for each of the multiple multimodal large models according to the features of the architectural CAD object, and the predicted probabilities output by each model are weighted and fused accordingly. The identification result is jointly determined by multiple multimodal large models according to the weights corresponding to the features, and does not depend on a single large model.

[0018] The preliminary identification results are validated against a set of normative rules. This rule set stores constraints on architectural CAD objects extracted from architectural design code documents. If the preliminary identification results meet these constraints, they are used as the final identification results for the architectural CAD objects. The rule set stores constraints extracted from architectural design code documents; before being output as the final identification result, the preliminary identification results undergo normative validation against these constraints, and those that do not meet the constraints are corrected accordingly. This approach solves the technical problem of low identification accuracy when using a single large model to identify architectural CAD objects of diverse types and formats. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for identifying architectural CAD objects according to an embodiment of the present invention; Figure 2 This is a frame diagram of a building CAD object recognition device according to an embodiment of the present invention; Figure 3 This is a framework diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0021] The technical solution of the present invention is illustrated below through specific embodiments. It should be understood that the one or more steps mentioned in the present invention do not preclude the existence of other methods and steps before or after the combined steps, or that other methods and steps may be inserted between these explicitly mentioned steps. It should also be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the present invention. Unless otherwise stated, the numbering of each method step is only for the purpose of identifying each method step, and not for limiting the order of each method or limiting the scope of the present invention. Changes or adjustments to their relative relationships, without substantial changes to the technical content, can also be considered as within the scope of the present invention.

[0022] In one embodiment of the present invention, a method for recognizing architectural CAD objects is provided. This method can be executed by electronic devices with data processing capabilities, such as servers or terminals. In this embodiment, architectural CAD objects refer to graphic objects in architectural drawings used to represent architectural components, such as walls, columns, beams, doors, windows, and stairs. These objects can be graphic objects without text annotations or graphic objects with text annotations.

[0023] This method uses the features of architectural CAD objects as the processing objects. The features are characteristic data that characterize architectural CAD objects, and usually exist in the form of feature vectors. For example, the features can be visual features that reflect the graphic form of the object, text features that reflect the text annotations of the object, or features that are a combination of visual features and text features.

[0024] like Figure 1 As shown, the method may specifically include the following steps: S101 inputs the features of the architectural CAD object into the routing network to obtain the weight coefficients corresponding to each of the multiple multimodal large models.

[0025] In this embodiment, the routing network takes the features of the architectural CAD object as input and outputs the weight coefficients corresponding to each multimodal large model in multiple multimodal large models. The weight coefficients represent the proportion of the corresponding multimodal large model in the subsequent fusion. The output of the routing network varies with the features of its input. Therefore, for architectural CAD objects with different features, the weight coefficients of the obtained multimodal large models can be different.

[0026] The multimodal large model mentioned here refers to a large model capable of processing multiple modal inputs such as images and text, and there are multiple of them, i.e., two or more. For example, multiple multimodal large models can be open-source multimodal large models used for architectural CAD object recognition, such as the large language model meta-LLAMA and Qwen.

[0027] Optionally, the routing network can be implemented using neural networks such as multilayer perceptrons. This application does not limit the specific network structure of the routing network.

[0028] S102, the building CAD objects are identified using the multiple multimodal large models respectively, and the predicted probability of the building CAD objects belonging to each candidate category is obtained from the output of each multimodal large model.

[0029] In this embodiment, multiple multimodal large models take the same architectural CAD object as the identification object and output the predicted probability of the architectural CAD object belonging to each candidate category. In this embodiment, the candidate categories are the possible categories of architectural CAD objects, such as walls, columns, beams, doors, windows, stairs, etc. That is, for the same architectural CAD object, each multimodal large model gives a set of predicted probabilities covering each candidate category.

[0030] In one example, the identification of the same architectural CAD object by multiple multimodal large models can be performed in parallel.

[0031] S103, according to the weight coefficients corresponding to each multimodal large model, the predicted probabilities output by each multimodal large model are weighted and fused to obtain the fusion probability of the building CAD object belonging to each candidate category, and the preliminary identification result of the building CAD object is determined based on the fusion probability.

[0032] In this embodiment, the weight coefficients of each multimodal large model are used as weights. For the same candidate category, the predicted probabilities of the candidate category output by each multimodal large model are weighted and summarized to obtain the fusion probability of the architectural CAD object belonging to the candidate category. The above processing is performed on each candidate category to obtain the fusion probability of the architectural CAD object belonging to each candidate category.

[0033] After obtaining the fusion probability of each candidate category, a preliminary identification result is determined based on this fusion probability. For example, the candidate category with the highest fusion probability can be determined as the preliminary identification result of the architectural CAD object.

[0034] The fusion probability of an architectural CAD object x belonging to candidate category c is obtained by weighting and summing the predicted probabilities using the weight coefficients of each multimodal large model: (1) (2) The candidate category with the highest fusion probability is selected as the preliminary identification result: (3) Where M is the number of multimodal large models, P(c│x) The fusion probability is the weighted fusion of M models. P m (c │x) For the first mA multimodal large model output, architectural CAD objects x Belongs to candidate category c The predicted probability; w m For the first m The weight coefficients of a multimodal large model satisfy... w m ≥0 and Σw m =1;

[0035] S104, Perform normative verification on the preliminary identification results according to the normative rule base, wherein the normative rule base stores the constraints of architectural CAD objects extracted from architectural design code documents.

[0036] In this embodiment, the constraints in the normative rule base are used to define the requirements that architectural CAD objects should meet in terms of geometric shape, spatial position relationship, and text annotation format; normative verification is to determine whether the preliminary identification result obtained in S103 meets the constraints in the normative rule base corresponding to the preliminary identification result.

[0037] For example, constraints can be constraints on the geometry of architectural CAD objects, such as a door being rectangular or a column having a square or circular cross-section; constraints can be constraints on the spatial relationships between architectural CAD objects, such as doors and windows being attached to walls or beams connecting to columns; and constraints can be constraints on the text annotation format of architectural CAD objects, such as doors being numbered in the format of "M + number" or windows in the format of "C + number". The specification rule base can store the above constraints in the form of structured entries, for example, recording the correspondence between object categories and their geometry, annotation format, etc., in the form of key-value pairs.

[0038] S105, if the preliminary identification result meets the constraints, the preliminary identification result is used as the identification result of the architectural CAD object; if the preliminary identification result does not meet the constraints, the preliminary identification result is corrected according to the constraints to obtain the identification result of the architectural CAD object.

[0039] In this embodiment, the subsequent processing is determined based on the verification results: when the verification results show that the preliminary identification result meets its corresponding constraints, the preliminary identification result is used as the identification result of the architectural CAD object; when the verification results show that the preliminary identification result does not meet its corresponding constraints, the preliminary identification result is corrected according to the constraints, and the corrected result is used as the identification result of the architectural CAD object.

[0040] For example, the correction of the preliminary identification result can be based on the constraint condition, adjusting the preliminary identification result to an identification result that satisfies the constraint condition.

[0041] It should be noted that the number of multimodal large models, the types of candidate categories, and the specific content of the constraints can all be determined according to the actual needs of architectural CAD object recognition, and this application does not impose any restrictions on them.

[0042] This invention utilizes a routing network to determine weight coefficients for multiple multimodal large models based on the characteristics of architectural CAD objects. The predicted probabilities output by each model are then weighted and fused accordingly. The identification result is jointly determined by the multiple multimodal large models according to their weights corresponding to the described features, and does not depend on a single large model. The preliminary identification results are verified against a regulatory rule base, which stores constraints on architectural CAD objects extracted from architectural design code documents. If the preliminary identification result meets the constraints, it is used as the identification result for the architectural CAD object; otherwise, it is corrected according to the regulatory rule base. This solves the technical problem of low identification accuracy when using a single large model to identify architectural CAD objects of diverse types and formats.

[0043] In one embodiment of the present invention, before inputting the features of the architectural CAD object into the routing network, the invention includes, but is not limited to: extracting the visual features of the architectural CAD object through an image encoder, and extracting the text features of the architectural CAD object through a text encoder. Specifically, the image encoder takes an image of an architectural CAD object as input and outputs visual features representing the object's graphic form; the text encoder takes text annotations of the architectural CAD object as input and outputs text features representing those annotations. The image encoder and text encoder are respectively encoders used to extract image features and text features; for example, the image encoder can be a visual Transformer, such as the paired image encoder ViT-L / 14 (visual Transformer) in Chinese-CLIP, and the text encoder can be RoberTa-wwm.

[0044] Optionally, visual features and text features can be obtained by taking the global feature vectors mapped from the projection layer of the corresponding encoder, and the two can have the same dimension, for example, both being 768-dimensional.

[0045] The visual and textual features are normalized separately and then concatenated to obtain the features of the architectural CAD object.

[0046] Specifically, the extracted visual and text features are normalized separately, and then the normalized visual and text features are concatenated. The resulting feature is used as the feature of the architectural CAD object input into the routing network. The normalization process is used to bring the visual and text features into similar numerical ranges to eliminate differences in numerical scale between them.

[0047] For example, the normalization process can be L2 normalization, which is to divide the feature vector by its L2 norm; the concatenation can be to combine the normalized visual features and the text features end to end to form a feature vector.

[0048] In one embodiment of the present invention, the routing network includes a multilayer perceptron; correspondingly, the process of inputting the features of the architectural CAD object into the routing network to obtain the weight coefficients of each multimodal large model can be implemented in the following manner: The features of the architectural CAD object are input into a multilayer perceptron, and the output of the multilayer perceptron is normalized to obtain the weight coefficients corresponding to each multimodal large model. The weight coefficients are non-negative, and the sum of the weight coefficients corresponding to each multimodal large model is a preset value.

[0049] Specifically, the multilayer perceptron takes the features of the building CAD object as input, performs internal calculations, and outputs numerical values ​​that correspond one-to-one with each multimodal large model. The output is then normalized so that the normalized values ​​satisfy the constraints that they are non-negative and their sum is a preset value. The normalized values ​​corresponding to each multimodal large model are then used as the weight coefficients of the corresponding multimodal large model.

[0050] The multilayer perceptron in this embodiment is a feedforward neural network comprising an input layer, one or more hidden layers, and an output layer. Optionally, the layers of the multilayer perceptron are connected by learnable weight matrices and bias vectors, and nonlinearity is introduced by an activation function, such as ReLU.

[0051] For example, the normalization of the multilayer perceptron output can be achieved using the Softmax function, thereby obtaining non-negative weight coefficients for each multimodal large model, and their sum is a preset value, such as 1.

[0052] The weight coefficients of each multimodal large model are calculated by the routing network: (4) (5) Wherein, f(x) is the d-dimensional feature vector of the architectural CAD object, which is obtained by normalizing and concatenating visual features and text features; W 1. W 2 is the learnable weight matrix. h b1 is the dimension of the hidden layer; b2 are the bias vectors; ReLU is the activation function; Softmax is used to ensure that the weight coefficients of the output are non-negative and that their sum is 1 (ensuring that the output weights satisfy the normalization constraint).

[0053] In one embodiment of the present invention, the multiple multimodal large models used for recognizing architectural CAD objects are obtained by training multiple pre-trained multimodal large models using a training set. The pre-trained multimodal large models refer to large models that have completed pre-training and are capable of processing multiple modal inputs such as images and text. For each pre-trained multimodal large model, a training set is used to train it to make it suitable for recognizing architectural CAD objects. The resulting models are the multiple multimodal large models used for recognition in S102. For example, the multiple pre-trained multimodal large models can be open-source multimodal large models such as LLAMA and Qwen.

[0054] The training set in this embodiment includes CAD object recognition task data, which is obtained by cropping and text recognition processing of architectural CAD drawings. The architectural CAD drawings are cropped to obtain individual architectural CAD objects, and text recognition processing is performed on the cropped architectural CAD objects. The resulting images and text constitute the CAD object recognition task data.

[0055] Optionally, the cropping results of architectural CAD drawings can be processed differently depending on whether they contain text: for architectural CAD objects without text, their images are used as CAD object recognition task data; for architectural CAD objects with text, text recognition processing is performed, for example, using optical character recognition (OCR) to recognize the text.

[0056] In one example, the text annotations obtained from the text recognition process contain coordinate information. Based on the spatial distance between the text annotations and the architectural CAD graphic objects, the text annotations can be associated with the nearest architectural CAD graphic objects. The CAD object recognition task data can be manually labeled with their respective categories to form a training set for training.

[0057] In one embodiment of this application, the training set can be obtained by preprocessing the original architectural CAD drawings. The data preprocessing is used to process the original architectural CAD drawings into images and text with uniform format, and to form an image library and a text library accordingly.

[0058] During data preprocessing, the original architectural CAD drawings are cropped to obtain individual architectural CAD objects, which are then categorized into those with and without text. For architectural CAD objects without text, their images are stored in an image library; for those with text, text recognition is performed, and the recognized text is organized and stored in a text library.

[0059] Optionally, text recognition can be performed using OCR; the text annotations obtained by OCR recognition carry coordinate information, and the text annotations can be associated with the nearest architectural CAD graphic objects based on the spatial distance between the text annotations and the architectural CAD graphic objects, and then the associated text can be stored in a text library.

[0060] Images in the image library and text in the text library are processed to have a unified format. In one example, after manually labeling the architectural CAD objects in the image library and text library with their respective categories, a training set for training and a test set for testing the recognition accuracy of each multimodal large model are formed.

[0061] In one embodiment of the present invention, the routing network used to obtain the weight coefficients of each multimodal large model is trained, and the training process is as follows.

[0062] First, the recognition accuracy of each multimodal large model for different types of architectural CAD objects on the test set is statistically analyzed. Based on this, an object type-model-accuracy mapping matrix is ​​constructed. This mapping matrix reflects the differences in the recognition capabilities of each multimodal large model for different types of architectural CAD objects. The test set and training set can be obtained together by cropping architectural CAD drawings, performing text recognition, and manually annotating them. The test set is used to verify the recognition accuracy of each multimodal large model.

[0063] Based on the above mapping matrix, features are extracted from the architectural CAD objects used as training samples. The extracted features are consistent with the features of the input routing network in step S101 above, that is, they are obtained by normalizing and concatenating the visual features and text features of the architectural CAD objects.

[0064] The extracted features are used as input to train the routing network, enabling it to learn the mapping relationship between the features of architectural CAD objects and the weight coefficients of each multimodal large model. Optionally, the training objective of the routing network is to minimize the cross-entropy between the weighted fusion result and the true class: the fusion probability is obtained by weighting and fusing the predicted probabilities of each multimodal large model with the weight coefficients output by the routing network, so that the class indicated by the fusion probability approaches the true class of the training samples, and the parameters of the routing network are optimized by gradient descent.

[0065] The routing network aims to minimize the cross-entropy between the weighted fusion result and the true class. (6) Where N is the number of training samples, C is the total number of candidate categories, and y ic For the sample i One-hot encoding of the real category, P( c │ x i ) as a sample iThe probability aggregation is (i.e., Equation 1). In this embodiment, the parameters of the routing network are optimized through gradient descent.

[0066] The aforementioned object type-model-accuracy mapping matrix is ​​used to characterize the recognition accuracy of each multimodal large model for different types of architectural CAD objects, reflecting the differences in recognition capabilities among the multimodal large models, and serving as the basis for dynamic weighted fusion. In one implementation, this mapping matrix is ​​used as auxiliary analysis data. The routing network is trained with the goal of minimizing the cross-entropy between the weighted fusion result and the true category (Equation 6 above), and the parameters of the routing network are optimized through gradient descent. This mapping matrix is ​​not used as a direct input for the training of the routing network.

[0067] As another feasible approach, the above mapping matrix can also be used to train the routing network: for each type of architectural CAD object, the recognition accuracy of each multimodal large model under that type is normalized to obtain the target weight distribution corresponding to that type. Then, the target weight distribution is used as a supervision signal to pre-train the routing network or initialize its parameters. After that, the cross-entropy of Formula 6 is used as the target to continue training, so that the routing network converges to the state of dynamically outputting weight coefficients according to features.

[0068] After the above training, the routing network is used in step S101 to dynamically output the weight coefficients of each multimodal large model according to the characteristics of the input architectural CAD object.

[0069] In one embodiment of the present invention, the training set includes not only CAD object recognition task data but also normative rule instruction data, which is obtained by converting constraints into instruction-response pairs. In the instruction-response pair, the instruction is used to provide a question or situation description related to the constraint, and the response is used to provide content that conforms to the constraint. The normative rule instruction data converted from the constraints is incorporated into the training set, and when training the pre-trained multimodal large model, the constraints are injected as prior knowledge.

[0070] The aforementioned constraints originate from architectural design specifications, such as industry-specific architectural design standards, and include at least one of geometric feature rules, topological relationship rules, and annotation specification rules. Geometric feature rules constrain the geometric shape of architectural CAD objects, for example, constraining the shape of a door to be rectangular, or the cross-sectional shape of a column to be square or circular. Topological relationship rules constrain the spatial relationships between architectural CAD objects, such as doors and windows being attached to walls, or beams connecting to columns. Annotation specification rules govern the text annotation format of architectural CAD objects; in one example, the door numbering format is "M + number," and the window numbering format is "C + number," which falls under this category.

[0071] The trained multimodal large model outputs the logit value z corresponding to each candidate class at the class prediction position. c The class probability distribution is obtained after Softmax normalization: (7) in, θ represents the class probability output by the optimized model during the training phase, and θ represents the parameters of the trained model.

[0072] The rule-constrained loss term is the penalty term obtained by averaging the predicted probabilities of the multimodal large model assigned to categories that do not meet the constraints, over the sample size. (8) The task loss term is calculated using the negative log-likelihood of the true category: (9) The total loss is the weighted sum of the two: L total = L task +λ× L rule (10) in, Let be the set of candidate categories in sample i that do not meet the constraints; λ is the weight of the rule constraint loss term, with a value ranging from 0.1 to 1.0, for example, 0.5. L task and L rule Acting on the same probability distribution P θ (c|x).

[0073] In one embodiment of the present invention, the loss function used in the training includes a task loss term and a rule constraint loss term. During training, the trained multimodal large model outputs a logit value at the category prediction position for the input architectural CAD object. This logit value is normalized to obtain the predicted probability of the architectural CAD object belonging to each candidate category. The task loss term is determined based on the predicted probability of the multimodal large model for the true category and is used to increase the predicted probability of the true category. The rule constraint loss term is a penalty term determined based on the predicted probability assigned by the multimodal large model to categories that do not meet the constraints, and is used to decrease the predicted probability of categories that do not meet the constraints.

[0074] Optionally, the task loss term can be cross-entropy loss, which is the negative log-likelihood of the predicted probability of the true class; the rule constraint loss term can be obtained by summing the predicted probabilities assigned to each class that does not meet the constraints by the multimodal large model, and included as a penalty term in the loss function. The task loss term and the rule constraint loss term operate on the same predicted probability distribution.

[0075] For example, the loss function can be obtained by weighted summation of the task loss term and the rule constraint loss term, where the weight of the rule constraint loss term is used to balance the relative strength of the two, and the value of the weight can range from 0.1 to 1.0, for example, 0.5. In one example, a warm-up strategy can be used during training, in which the weight of the rule constraint loss term is linearly increased from 0 to the target value within a certain number of training steps at the beginning of training (e.g., the first 10% of training steps), and then the target value is kept unchanged.

[0076] In one embodiment of the present invention, the process of correcting the preliminary identification result according to constraints to obtain the architectural CAD object identification result includes, but is not limited to: S301. From each candidate category, select the first candidate category that meets the constraints according to the fusion probability from high to low, and use it as the recognition result of the architectural CAD object. Specifically, when the preliminary identification result does not meet the constraints, from the candidate categories of the architectural CAD object, in order of the fusion probability of each candidate category from high to low, it is judged whether each candidate category meets the constraints, and the first candidate category that meets the constraints is taken as the identification result of the architectural CAD object.

[0077] S302, if none of the candidate categories meet the constraints, the reason why the preliminary identification results do not meet the constraints is used as a supplementary condition, and multiple multimodal large models are re-entered together with the architectural CAD objects for re-identification; Optionally, the above supplementary conditions are input into the multimodal large model in the form of natural language text.

[0078] S303 If the result obtained from the re-identification still does not meet the constraints, then mark the building CAD object as pending manual confirmation.

[0079] Another embodiment of the present invention relates to a building CAD object recognition device, such as... Figure 2 As shown, the device includes: The weight determination module 20 is used to input the features of the architectural CAD object into the routing network to obtain the weight coefficients corresponding to each of the multiple multimodal large models. The identification module 21 is used to identify the building CAD object using the multiple multimodal large models respectively, and to obtain the predicted probability of the building CAD object belonging to each candidate category output by each of the multimodal large models; The fusion module 22 is used to perform weighted fusion of the predicted probabilities output by each of the multimodal large models according to the weight coefficients corresponding to each of the multimodal large models, to obtain the fusion probability of the building CAD object belonging to each of the candidate categories, and to determine the preliminary identification result of the building CAD object based on the fusion probability. Verification module 23 is used to perform normative verification on the preliminary identification results according to the normative rule base, wherein the normative rule base stores the constraints of the building CAD objects extracted from building design code documents; The correction module 24 is used to, when the preliminary identification result meets the constraints, use the preliminary identification result as the identification result of the architectural CAD object; and when the preliminary identification result does not meet the constraints, correct the preliminary identification result according to the constraints to obtain the identification result of the architectural CAD object.

[0080] Optionally, in this embodiment, it further includes: The feature extraction module is used to extract the visual features of the architectural CAD object by an image encoder and the text features of the architectural CAD object by a text encoder before the features of the architectural CAD object are input into the routing network. The feature processing module is used to normalize the visual features and the text features respectively and then concatenate them to obtain the features of the architectural CAD object.

[0081] Optionally, in this embodiment, the routing network includes a multilayer perceptron; wherein, the weight determination module 20 includes: The weight determination submodule is used to input the features of the building CAD object into the multilayer perceptron and normalize the output of the multilayer perceptron to obtain the weight coefficients corresponding to each of the multimodal large models.

[0082] Optionally, in this embodiment, the plurality of multimodal large models are obtained by training a plurality of pre-trained multimodal large models using a training set; The training set includes CAD object recognition task data, which is obtained by cropping and text recognition processing of architectural CAD drawings.

[0083] Optionally, in this embodiment, the training set further includes normative rule instruction data, which is obtained by converting the constraints into instruction-response pairs; The constraints include at least one of geometric feature rules, topological relationship rules, and annotation specification rules. The geometric feature rules are constraints on the geometric shape of the architectural CAD object, the topological relationship rules are constraints on the spatial positional relationships between the architectural CAD objects, and the annotation specification rules are constraints on the text annotation format of the architectural CAD object.

[0084] Optionally, in this embodiment, the loss function used for training includes a task loss term and a rule constraint loss term; The task loss term is determined based on the prediction probability of the true class by the trained multimodal large model; The rule-constrained loss term is a penalty term determined based on the predicted probabilities assigned to categories that do not meet the constraints by the trained multimodal large model.

[0085] Correction module 24 includes: The determination submodule is used to select the first candidate category that meets the constraints from the candidate categories according to the fusion probability from high to low, and use it as the recognition result of the building CAD object; The identification submodule is used to, when none of the candidate categories meet the constraints, take the reason why the preliminary identification result does not meet the constraints as a supplementary condition, and re-input it together with the architectural CAD object into the multiple multimodal large models for re-identification; The correction submodule is used to mark the architectural CAD object as requiring manual confirmation if the result obtained from re-identification still does not meet the constraints.

[0086] Another embodiment of the present invention relates to an electronic device, such as... Figure 3 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor 301, the instructions being executed by the at least one processor 301 to enable the at least one processor 301 to perform the building CAD object recognition method in the above embodiments.

[0087] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0088] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0089] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0090] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for recognizing architectural CAD objects, characterized in that, The method includes: The features of architectural CAD objects are input into the routing network to obtain the weight coefficients corresponding to each of the multiple multimodal large models; The building CAD object is identified by using the multiple multimodal large models respectively, and the predicted probability of the building CAD object belonging to each candidate category is obtained from the output of each multimodal large model; Based on the weight coefficients corresponding to each of the multimodal large models, the predicted probabilities output by each of the multimodal large models are weighted and fused to obtain the fusion probability of the building CAD object belonging to each of the candidate categories, and the preliminary identification result of the building CAD object is determined based on the fusion probability. The preliminary identification results are verified for standardization based on the standard rule base, which stores the constraints of the architectural CAD objects extracted from architectural design code documents; If the preliminary identification result meets the constraints, the preliminary identification result is used as the identification result of the architectural CAD object; if the preliminary identification result does not meet the constraints, the preliminary identification result is corrected according to the constraints to obtain the identification result of the architectural CAD object.

2. The method according to claim 1, characterized in that, Before inputting the features of the architectural CAD object into the routing network, the following is also included: Visual features of the architectural CAD object are extracted using an image encoder, and text features of the architectural CAD object are extracted using a text encoder. The visual features and text features are normalized and then concatenated to obtain the features of the architectural CAD object.

3. The method according to claim 1, characterized in that, The routing network includes a multilayer perceptron; wherein... The process of inputting the features of architectural CAD objects into a routing network to obtain the weight coefficients corresponding to each of multiple multimodal large models includes: The features of the architectural CAD object are input into the multilayer perceptron, and the output of the multilayer perceptron is normalized to obtain the weight coefficients corresponding to each of the multimodal large models.

4. The method according to claim 1, characterized in that, The multiple multimodal large models are obtained by training multiple pre-trained multimodal large models using a training set; The training set includes CAD object recognition task data, which is obtained by cropping and text recognition processing of architectural CAD drawings.

5. The method according to claim 4, characterized in that, The training set also includes normative rule instruction data, which is obtained by converting the constraints into instruction-response pairs. The constraints include at least one of geometric feature rules, topological relationship rules, and annotation specification rules. The geometric feature rules are constraints on the geometric shape of the architectural CAD object, the topological relationship rules are constraints on the spatial positional relationships between the architectural CAD objects, and the annotation specification rules are constraints on the text annotation format of the architectural CAD object.

6. The method according to claim 5, characterized in that, The loss function used in the training includes a task loss term and a rule constraint loss term; The task loss term is determined based on the prediction probability of the true class by the trained multimodal large model; The rule-constrained loss term is a penalty term determined based on the predicted probabilities assigned to categories that do not meet the constraints by the trained multimodal large model.

7. The method according to claim 1, characterized in that, The step of correcting the preliminary identification result according to the constraints to obtain the identification result of the architectural CAD object includes: From the candidate categories, the first candidate category that meets the constraints is selected according to the fusion probability from high to low, and this is taken as the recognition result of the architectural CAD object. If none of the candidate categories meet the constraints, the reason why the preliminary identification result does not meet the constraints is used as a supplementary condition, and the result is re-entered into the multiple multimodal large models along with the architectural CAD object for re-identification. If the results obtained from the re-identification still do not meet the constraints, the architectural CAD object is marked as awaiting manual confirmation.

8. A device for recognizing architectural CAD objects, characterized in that, The device includes: The weight determination module is used to input the features of architectural CAD objects into the routing network to obtain the weight coefficients corresponding to each of the multiple multimodal large models. The identification module is used to identify the building CAD object using the multiple multimodal large models respectively, and to obtain the predicted probability of the building CAD object belonging to each candidate category output by each of the multimodal large models; The fusion module is used to perform weighted fusion of the predicted probabilities output by each of the multimodal large models according to the weight coefficients corresponding to each of the multimodal large models, to obtain the fusion probability of the building CAD object belonging to each of the candidate categories, and to determine the preliminary identification result of the building CAD object based on the fusion probability. The verification module is used to perform normative verification on the preliminary identification results according to the normative rule base, which stores the constraints of the building CAD objects extracted from building design code documents; The correction module is used to, if the preliminary identification result meets the constraints, use the preliminary identification result as the identification result of the architectural CAD object; if the preliminary identification result does not meet the constraints, correct the preliminary identification result according to the constraints to obtain the identification result of the architectural CAD object.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images