Professional performance evaluation method and system for large building model

Abstract

The invention discloses a professional performance evaluation method and system for a large building model. The method comprises the following steps: establishing an evaluation category data set of the large building model; evaluating the large building model in each evaluation category from two dimensions of consistency check and accuracy check, performing linear weighted average on evaluation results of the two dimensions to obtain a comprehensive evaluation result, and judging professional performance of the large building model in each evaluation category through the comprehensive evaluation result; when the comprehensive evaluation result is greater than the excellent line, determining that the professional performance of the large building model in the corresponding evaluation category is excellent; when the comprehensive evaluation result is between the good line and the excellent line, determining that the professional performance of the large building model in the corresponding evaluation category is good; and when the comprehensive evaluation result is smaller than the good line, determining that the professional performance of the large building model in the corresponding evaluation category is unqualified. According to the invention, the professional performance of the large building model in each evaluation category can be evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of building engineering, and in particular to a method and system for evaluating the professional performance of a large building model. Background Art

[0002] The architectural large model refers to the architectural professional large language model. The large language model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. The large language model can handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and is an important path to artificial intelligence. The large language model technology has been widely used in finance, the Internet, and biomedicine. In the construction industry, it has also been applied and promoted in text data retrieval, knowledge question answering, etc., but there is currently no effective method or system to evaluate the professional performance of such large models. At this stage, many standard open source data sets can be used to evaluate the professional performance of general large language models, but in the field of construction, there is still no systematic evaluation method and standard for evaluating architectural large models. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for evaluating the professional performance of a large building model to solve the problem that the professional performance of a large building model cannot be evaluated.

[0004] In order to solve the above technical problems, the present invention provides a method for evaluating the professional performance of a large building model, comprising:

[0005] Establishing a data set of evaluation categories for a large building model, wherein the evaluation categories include one or more of standards and specifications, process practices, job knowledge, management information, building regulations, cost budget, safety and quality, and survey and mapping;

[0006] The large building model is evaluated in each evaluation category from the two dimensions of consistency check and accuracy check. The evaluation results of the two dimensions are linearly weighted averaged to obtain a comprehensive evaluation result. The professional performance of the large building model in each evaluation category is judged by the comprehensive evaluation result.

[0007] When the comprehensive evaluation result is greater than the excellent line, the professional performance of the large building model in the corresponding evaluation category is determined to be excellent;

[0008] When the comprehensive evaluation result is between the good line and the excellent line, it is determined that the professional performance of the large building model in the corresponding evaluation category is good;

[0009] When the comprehensive evaluation result is less than the good line, it is determined that the professional performance of the large building model in the corresponding evaluation category is unqualified.

[0010] Furthermore, in the professional performance evaluation method for large building models provided by the present invention, the excellent line is 0.8 and the good line is 0.6.

[0011] Furthermore, the professional performance evaluation method for large building models provided by the present invention performs consistency check on large building models in various evaluation categories through the kappa coefficient.

[0012] Furthermore, the method for evaluating the professional performance of a large building model provided by the present invention includes the following steps:

[0013] Use a test category data set with a sample size of N to ask the building model k questions, and use the k-times answer result of the building model to evaluate it using the kappa coefficient:

[0014] When the kappa value is between 0.9 and 1, the consistency of the building model is determined to be excellent;

[0015] When the kappa value is between 0.65 and 0.9, the consistency of the building model is determined to be good;

[0016] When the kappa value is between 0.4 and 0.65, the consistency of the building model is determined to be qualified;

[0017] When the kappa value is between -1 and 0.4, the consistency of the building model is determined to be unqualified.

[0018] Furthermore, the professional performance evaluation method for large building models provided by the present invention uses the multivariate grammar ROUGE-N score to check the accuracy of large building models in each evaluation category.

[0019] Furthermore, the method for evaluating the professional performance of a large building model provided by the present invention includes the following steps:

[0020] When the decimal value of the ROUGE-N score was greater than 0.8, the accuracy of the building model was determined to be excellent;

[0021] When the decimal value of the ROUGE-N score was between 0.6 and 0.8, the accuracy of the large building model was determined to be good;

[0022] When the decimal value of the ROUGE-N score is less than 0.6, the accuracy of the building model is determined to be unqualified. The decimal values ​​of excellent, good, and unqualified can be adjusted as needed.

[0023] Furthermore, the professional performance evaluation method for large building models provided by the present invention utilizes the common problems extracted by the extraction capabilities of large building models and one or more general large language models to establish evaluation category data sets for standards and specifications, process practices, job knowledge, management information, building regulations, cost budgets, safety quality and / or survey and mapping.

[0024] In order to solve the above technical problems, the present invention also provides a large-scale building model professional performance evaluation system, comprising:

[0025] Evaluation category data set module, including one or more evaluation category data set sub-modules among standards and specifications, process practices, job knowledge, management information, building regulations, cost budget, safety and quality, and survey and mapping;

[0026] The evaluation dimension module includes a consistency check submodule and an accuracy check submodule. The evaluation categories corresponding to the large building model in each evaluation category data set submodule are evaluated from two dimensions through the consistency check submodule and the accuracy check submodule of the evaluation dimension module. The evaluation results of the two dimensions are linearly weighted averaged to obtain a comprehensive evaluation result. The professional performance of the large building model in the evaluation category corresponding to the submodule of the data set of each evaluation category is judged through the comprehensive evaluation result.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The method and system for evaluating the professional performance of large building models provided by the present invention establish an evaluation category data set, extract search terms from each evaluation category data set and input them into the large building model, check whether the output terms of the large building model are the same as the search terms, and evaluate the professional performance of the large building model in each evaluation category from two dimensions: consistency check and accuracy check, thereby avoiding the problem of incorrect evaluation due to a single dimension, thereby improving the accuracy, precision and quality of the evaluation and solving the problem of being unable to evaluate the professional performance of the large building model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the professional performance evaluation method of the building large model;

[0030] Figure 2 It is a schematic diagram of the comprehensive evaluation results of the eight evaluation categories;

[0031] Figure 3 It is a schematic diagram of the block structure of the professional performance evaluation system for large building models;

[0032] As shown in the figure:

[0033] 200. Professional performance evaluation system for large building models, 210. Evaluation category data set module, 211. Standard specification sub-module, 212. Process practice sub-module, 213. Job knowledge sub-module, 214. Management information sub-module, 215. Building specification sub-module, 216. Cost budget sub-module, 217. Safety and quality sub-module, 218. Survey and mapping sub-module, 220. Evaluation dimension module, 221. Consistency check sub-module, 222. Accuracy check sub-module. DETAILED DESCRIPTION

[0034] The present invention is described in detail below in conjunction with the accompanying drawings: The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in very simplified form and in non-precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0035] Please refer to Figure 1 The embodiment of the present invention provides a method for evaluating the professional performance of a large building model, which may include:

[0036] Step 101, establish an evaluation category data set of the building model, wherein the evaluation category includes one or more of standard specifications, process practices, job knowledge, management information, building regulations, cost budget, safety quality and survey and mapping. The evaluation category data set of standard specifications, process practices, job knowledge, management information, building regulations, cost budget, safety quality and / or survey and mapping can be established by using the common problems extracted by the extraction capability of the building model and one or more general large language models. Among them:

[0037] Standard specifications include but are not limited to national specifications, industry specifications, group specifications and local specifications.

[0038] The technical practices include but are not limited to foundation and base engineering, ground engineering, electrical engineering and waterproofing engineering, water supply and drainage and heating engineering, foundation pit support and groundwater control engineering, energy-saving engineering, ventilation and air-conditioning engineering, door and window engineering, roofing engineering, intelligent engineering, decoration and decoration engineering equipment installation engineering, full-professional engineering, masonry, steel, wood structure engineering, curtain wall and decorative panels, brick engineering, industrial pipeline engineering, concrete and steel-concrete composite structure engineering.

[0039] Job knowledge includes but is not limited to the basic knowledge and professional knowledge of 10 positions, including document clerk, quality inspector, construction worker, technician, labor worker, mechanic, material clerk, standard clerk, designer, and supervisor.

[0040] The management materials include but are not limited to seven items: foundation and waterproofing engineering, steel structure and prefabricated concrete engineering, water supply and drainage, heating, ventilation and air-conditioning engineering, concrete engineering, building electrical and intelligent building engineering, masonry and decoration and decoration engineering, and municipal engineering.

[0041] Building regulations include but are not limited to legal provisions, administrative regulations, departmental regulations, and departmental work documents.

[0042] The cost budget includes but is not limited to 10 types of sub-projects, including foundation and base engineering, main engineering, building decoration and renovation engineering, roofing engineering, building drainage and heating engineering, ventilation and air-conditioning engineering, building electrical engineering, intelligent building, building energy conservation and elevators.

[0043] Safety quality includes but is not limited to four dimensions: engineering entity quality control, production safety site control, quality management information, and safety management information.

[0044] Survey and mapping include but are not limited to three aspects of data: engineering measurement, hydrogeological survey, and engineering geological survey.

[0045] Step 102, evaluate the large building model in each evaluation category from the two dimensions of consistency check and accuracy check, perform linear weighted average on the evaluation results of the two dimensions to obtain a comprehensive evaluation result, and judge the professional performance of the large building model in each evaluation category through the comprehensive evaluation result.

[0046] When the comprehensive evaluation result is greater than the excellent line, it is determined that the professional performance of the large building model in the corresponding evaluation category is excellent.

[0047] When the comprehensive evaluation result is between the good line and the excellent line, it is determined that the professional performance of the architectural model in the corresponding evaluation category is good.

[0048] When the comprehensive evaluation result is less than the good line, the professional performance of the large building model in the corresponding evaluation category is determined to be unqualified. The excellent line can be 0.8 and the good line can be 0.6.

[0049] Please refer to Figure 2 , which is a schematic diagram of the comprehensive evaluation results of all the above 8 evaluation categories.

[0050] Please refer to Figure 3 The embodiment of the present invention further provides a large-scale building model professional performance evaluation system 200, which is based on the above evaluation method and includes:

[0051] The evaluation category data set module 210 includes one or more evaluation category data set sub-modules 211-218 of standard specifications, process practices, job knowledge, management information, building regulations, cost budget, safety and quality, and survey and mapping.

[0052] The evaluation dimension module 220 includes a consistency check submodule 221 and an accuracy check submodule 222. The consistency check submodule 221 and the accuracy check submodule 222 of the evaluation dimension module 220 are used to evaluate the evaluation categories corresponding to the large building model in each evaluation category data set submodule from two dimensions, and the evaluation results of the two dimensions are linearly weighted averaged to obtain a comprehensive evaluation result. The professional performance of the large building model in the evaluation category corresponding to the submodule of each evaluation category data set is judged through the comprehensive evaluation result.

[0053] In order to perform consistency check on the large building model in each evaluation category, the large building model professional performance evaluation method and system provided by the embodiment of the present invention can perform consistency check on the large building model in each evaluation category through the kappa coefficient, specifically including:

[0054] Use a test category data set with a sample size of N to ask the building model k questions, and use the k-times answer result of the building model to evaluate it using the kappa coefficient:

[0055] When the kappa value is between 0.9 and 1, the consistency of the building model is determined to be excellent;

[0056] When the kappa value is between 0.65 and 0.9, the consistency of the building model is determined to be good;

[0057] When the kappa value is between 0.4 and 0.65, the consistency of the building model is determined to be qualified;

[0058] When the kappa value is between -1 and 0.4, the consistency of the building model is determined to be unqualified.

[0059] In order to check the accuracy of the large building model in each evaluation category, the large building model professional performance evaluation method and system provided by the embodiment of the present invention can check the accuracy of the large building model in each evaluation category through the multivariate grammar ROUGE-N score, which can specifically include:

[0060] When the decimal value of the ROUGE-N score was greater than 0.8, the accuracy of the building model was determined to be excellent;

[0061] When the decimal value of the ROUGE-N score was between 0.6 and 0.8, the accuracy of the large building model was determined to be good;

[0062] When the decimal value of the ROUGE-N score is less than 0.6, the accuracy of the building model is determined to be unacceptable.

[0063] For example, for a group of large building models, the result "Scaffolding is a common auxiliary equipment for building construction" and the standard answer in the data set is "Scaffolding is a common auxiliary facility for building construction", N is 2, and the ROUGE-2 (bigram) score calculation process and calculation results are as follows:

[0064] Generate the resulting bigram: {Scaffolding is a common building construction auxiliary facility};

[0065] Standard answer binary phrase: {scaffolding, is, a, common, building, construction, auxiliary, equipment};

[0066] Overlapping bigrams: {scaffolding, is, a, common, building, construction, auxiliary}; the overlapping part accounts for 7, and the number of standard answer phrases is 8, so the ROUGE-2 score is 7 / 8=0.875.

[0067] The method and system for evaluating the professional performance of a large building model provided by the embodiment of the present invention establish an evaluation category data set, extract search terms from each evaluation category data set and input them into the large building model, check whether the output terms of the large building model are the same as the search terms, and evaluate the professional performance of the large building model in each evaluation category from two dimensions: consistency check and accuracy check, thereby avoiding the problem of incorrect evaluation due to a single dimension, thereby improving the accuracy, precision and quality of the evaluation and solving the problem of being unable to evaluate the professional performance of the large building model.

[0068] The method and system for evaluating the professional performance of large building models provided in the embodiments of the present invention evaluate the performance of large building models from two aspects, namely, retrieval and answering, thereby solving the problem that large building models cannot be evaluated. The evaluation has multiple dimensions, and evaluates the professional performance of large building models from multiple dimensions, such as the comprehensiveness and accuracy of retrieval, and the consistency, professionalism, and accuracy of answers.

[0069] The present invention is not limited to the above-mentioned specific implementation modes. Obviously, the above-mentioned embodiments are only some embodiments of the embodiments of the present invention, but not all embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention. Those skilled in the art can make other levels of modifications and changes to the present invention. In this way, if these modifications and changes of the present invention fall within the scope of the claims of the present invention, the present invention is also intended to include these changes and changes.

Claims

1. A method for evaluating the professional performance of a large building model, characterized in that: include: Establishing a data set of evaluation categories for a large building model, wherein the evaluation categories include one or more of standards and specifications, process practices, job knowledge, management information, building regulations, cost budget, safety and quality, and survey and mapping; The large building model is evaluated in each evaluation category from the two dimensions of consistency check and accuracy check. The evaluation results of the two dimensions are linearly weighted averaged to obtain a comprehensive evaluation result. The professional performance of the large building model in each evaluation category is judged by the comprehensive evaluation result. When the comprehensive evaluation result is greater than the excellent line, the professional performance of the large building model in the corresponding evaluation category is determined to be excellent; When the comprehensive evaluation result is between the good line and the excellent line, it is determined that the professional performance of the large building model in the corresponding evaluation category is good; When the comprehensive evaluation result is less than the good line, it is determined that the professional performance of the large building model in the corresponding evaluation category is unqualified.

2. The method for evaluating the professional performance of a large building model according to claim 1, characterized in that: The excellent line is 0.8 and the good line is 0.

6.

3. The method for evaluating the professional performance of a large building model according to claim 1, characterized in that: The Kappa coefficient was used to check the consistency of the large building model in each evaluation category.

4. The method for evaluating the professional performance of a large building model according to claim 3, characterized in that: The method for checking the consistency of the large building model in each evaluation category by using the kappa coefficient includes: Use a test category data set with a sample size of N to ask the building model k questions, and use the k-times answer result of the building model to evaluate it using the kappa coefficient: When the kappa value is between 0.9 and 1, the consistency of the building model is determined to be excellent; When the kappa value is between 0.65 and 0.9, the consistency of the building model is determined to be good; When the kappa value is between 0.4 and 0.65, the consistency of the building model is determined to be qualified; When the kappa value is between -1 and 0.4, the consistency of the building model is determined to be unqualified.

5. The method for evaluating the professional performance of a large building model according to claim 1 or 3, characterized in that: The accuracy of the large building model in each evaluation category was checked using the multivariate grammar ROUGE-N score.

6. The method for evaluating the professional performance of a large building model according to claim 5, characterized in that: The method for checking the accuracy of the building model in each evaluation category by using the multivariate grammar ROUGE-N score includes: When the decimal value of the ROUGE-N score was greater than 0.8, the accuracy of the building model was determined to be excellent; When the decimal value of the ROUGE-N score was between 0.6 and 0.8, the accuracy of the large building model was determined to be good; When the decimal value of the ROUGE-N score is less than 0.6, the accuracy of the building model is determined to be unacceptable.

7. The method for evaluating the professional performance of a large building model according to claim 1, characterized in that: The common problems extracted by using the extraction capabilities of the building large model and one or more general large language models are used to establish evaluation category data sets for standards and specifications, process practices, job knowledge, management information, building regulations, cost budgets, safety and quality, and / or surveying and mapping.

8. A professional performance evaluation system for large building models, characterized in that: include: Evaluation category data set module, including one or more evaluation category data set sub-modules among standards and specifications, process practices, job knowledge, management information, building regulations, cost budget, safety and quality, and survey and mapping; The evaluation dimension module includes a consistency check submodule and an accuracy check submodule. The evaluation categories corresponding to the large building model in each evaluation category data set submodule are evaluated from two dimensions through the consistency check submodule and the accuracy check submodule of the evaluation dimension module. The evaluation results of the two dimensions are linearly weighted averaged to obtain a comprehensive evaluation result. The professional performance of the large building model in the evaluation category corresponding to the submodule of the data set of each evaluation category is judged through the comprehensive evaluation result.

Images