Process evaluation method based on HDBSCAN clustering and large language model

Abstract

The invention discloses a process evaluation method based on HDBSCAN clustering and a large language model, and the method comprises the steps: decomposing each expert questionnaire into the data of each evaluation index according to the characteristics of a project and the opinions of experts: the first questionnaire and the first evaluation index of the first questionnaire; the feature vector of each evaluation dimension of each evaluation index is constructed, the evaluation dimension is the first evaluation dimension of the first evaluation index of the first questionnaire, and the evaluation dimensions comprise the expert working age, the traditional process score and the intelligent construction process score. Complicated descriptive texts and a large number of expert questionnaire tables can be integrated to generate scientific and comprehensive process evaluation results and optimization suggestions, the logical reasoning ability of LLM on specific scene complex corpora is enhanced, and the LLM input length limitation is overcome.

Description

Technical Field

[0001] This invention relates to the field of process evaluation technology, and in particular to a process evaluation method based on HDBSCAN clustering and a large language model. Background Technology

[0002] As is well known, traditional process evaluation methods rely heavily on individual expert experience, resulting in significant subjective bias and difficulty in effectively clustering expert opinions, leading to a lack of objectivity in the evaluation results. Furthermore, when using large language models for process evaluation, the input data is not structured, and the model output is prone to problems such as vague descriptions and insufficient logic, making it difficult to meet the accurate evaluation needs in actual engineering.

[0003] The Analytic Hierarchy Process (AHP), proposed by Saaty, is a systematic multi-criteria decision-making method that provides decision support for complex evaluation problems by constructing pairwise comparison matrices. Due to its theoretical simplicity and clear structure, AHP is widely used in fields such as railway route design and scheme evaluation. However, the AHP method primarily relies on a single data source—expert questionnaires—making it difficult to ensure the scientific rigor and comprehensiveness of the evaluation results. Inspired by data-driven decision-making, combining descriptive textual data closely related to new technologies (such as project completion reports) with expert questionnaires has become an important direction for improving evaluation quality. However, descriptive textual data, due to its unstructured nature, is difficult to transform into pairwise comparison matrices, and existing evaluation methods relying on AHP fail because they struggle to integrate multi-source heterogeneous data.

[0004] In theory, LLM can integrate descriptive text data and expert questionnaires, directly generating comprehensive evaluation results through the semantic understanding and reasoning capabilities of pre-trained models. However, the practical implementation faces the following challenges: 1. LLM has limited descriptive textual logical reasoning capabilities for complex semantics in specific domains, potentially leading to logically flawed evaluation results and misleading decisions or ineffective recommendations; 2. LLM has strict limitations on input length. For example, when calling ChatGPT-4, a single context window only supports approximately 6,000 characters of Chinese text. Batch input may result in memory loss, significantly reducing the quality of responses. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a XXXX that can be achieved.

[0006] To solve the above-mentioned technical problems, the present invention provides the following first technical solution: This invention discloses a process evaluation method based on HDBSCAN clustering and a large language model. The original inputs of the evaluation method include suggested engineering processes, quantitative descriptive text, and expert questionnaires, and include the following steps: s1.s1. Prompts the background and field required for the engineering input process evaluation. After decomposing the quantitative descriptive text into various evaluation index data, the expert questionnaire is also decomposed and then generated into sub-questionnaires through clustering. s2. Based on the project characteristics and expert opinions, each expert questionnaire is broken down into data for various evaluation indicators: ,in, For the first One questionnaire, For the first The first questionnaire Each evaluation indicator is used to construct a feature vector for each evaluation dimension. ,in, For the first The first questionnaire The first evaluation indicator The evaluation dimensions include expert seniority, traditional process score, and intelligent construction process score. s3. Calculate the core distance of each questionnaire to a certain evaluation indicator, whereby the core distance is defined as the distance between each data point and the evaluation indicator. To its first The distance between neighbors , yes To its first The distance between neighbors Data points and Euclidean distance between Data points are calculated based on core distance and Euclidean distance. and The reachability distance is used to measure the association between two points. Each data point is treated as an independent cluster. Based on the reachability distance, starting from the smallest distance, clusters with closer distances are gradually merged to generate a clustering hierarchical structure tree. s4. During the construction of the cluster hierarchical structure tree, HDBSCAN calculates the density reachability between each pair of clusters. If the reachability distance between two clusters is small and their core distance is close, the density reachability is high, and they are temporarily merged into one cluster. and The merged quantity is: , Cluster The number of data points is only satisfied if the size of the merged cluster meets the requirement. Only clusters that meet the merging criteria can be considered valid clusters. Min_cluster_size represents the minimum cluster size; clusters that do not meet the merging criteria are discarded as noise. The denoised clustering hierarchy structure tree is traversed from bottom to top to extract stable clusters and output the final clustering result. s5. Extract information for each cluster ClCl, including the number of experts, average years of service, traditional process score, and intelligent construction process score. , For the first Number of experts in each cluster For the first Average length of service of experts in each cluster For the first Traditional craftsmanship scores for each cluster For the first The intelligent construction process score for each cluster; s6. After converting the clustering results into structured text, input them into LLM to generate causal relationships to construct prompt sub-instructions for step-by-step evaluation; s7. The silhouette coefficient is used as an indicator to measure the quality of HDBSCAN clustering. The formula for calculating the silhouette coefficient is as follows: ,in Representing data points The average distance to other data points within the same cluster. This indicates a data point. The average distance to the nearest other clusters is used, and the output of the large language model is evaluated using a four-dimensional scoring system, which includes descriptive accuracy, logicality, analytical depth, and feasibility. The total score is calculated using the following formula: ,in, For the score of the i-th dimension, Let be the weight coefficient of the i-th dimension, and the sum of the weight coefficients is 1.

[0007] S8. Finally, input all the results into LLM to perform the evaluation task, and summarize the results of each evaluation indicator to generate a comprehensive evaluation.

[0008] As a preferred embodiment of the present invention, in step S3, the calculation of the core distance is based on the data point and its first... The Euclidean distance between the nearest neighbors.

[0009] As a preferred technical solution of the present invention, in step S4, the minimum cluster size parameter Min_cluster_size is preset according to the clustering effect.

[0010] As a preferred technical solution of the present invention, in step S7, the contour coefficient has a value range of [−1,1], which is used to measure the compactness and separation of the clustering results.

[0011] As a preferred embodiment of the present invention, in step S7, the weight coefficients of the four-dimensional scoring system... Adjustments will be made dynamically based on evaluation requirements.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention can integrate complex descriptive text and a large number of expert questionnaires to generate scientific and comprehensive process evaluation results and optimization suggestions.

[0013] 2: This invention can enhance the logical reasoning ability of LLM for complex corpora in specific scenarios.

[0014] 3. This invention uses the HDBSCAN clustering method to condense a large number of expert questionnaires, overcoming the input length limitation of LLM. The effectiveness and generalization ability of the proposed method are verified through ablation experiments and robust tests, providing a decision-making basis for the evaluation of intelligent construction processes. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0016] In the attached diagram, all identical reference numerals refer to the same components.

[0017] Furthermore, detailed descriptions of known technologies are omitted if they are unnecessary to illustrate the features of the present invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, while the terms "inner" and "outer" refer to directions toward or away from the geometric center of a particular component, respectively.

[0018] In the attached diagram: Figure 1 This is a flowchart of the semantic perception evaluation method of the present invention; Figure 2 These are clustering results of the present invention under different Min_cluster_sizes; Figure 3 This is a graph showing the word-saving ratio and clustering contour coefficient effect of the present invention under different Min_cluster_sizes; Detailed Implementation

[0019] The following describes preferred embodiments of the present invention. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0020] Example 1 This invention provides a process evaluation method based on HDBSCAN clustering and a large language model. The original inputs of the evaluation method include prompts for the process, quantitative descriptive text, and an expert questionnaire, and include the following steps: s1. Prompt the background and field required for the engineering input process evaluation. After decomposing the quantitative descriptive text into various evaluation index data, the expert questionnaire is also decomposed and then generated into sub-questionnaires through clustering. s2. Based on the project characteristics and expert opinions, each expert questionnaire is broken down into data for various evaluation indicators: ,in, For the first One questionnaire, For the first The first questionnaire Each evaluation indicator is used to construct a feature vector for each evaluation dimension. ,in, For the first The first questionnaire The first evaluation indicator The evaluation dimensions include expert seniority, traditional process score, and intelligent construction process score. s3. Calculate the core distance of each questionnaire to a certain evaluation indicator, whereby the core distance is defined as the distance between each data point and the evaluation indicator. To its first The distance between neighbors , yes To its first The distance between neighbors Data points and Euclidean distance between Data points are calculated based on core distance and Euclidean distance. and The reachability distance is used to measure the association between two points. Each data point is treated as an independent cluster. Based on the reachability distance, starting from the smallest distance, clusters with closer distances are gradually merged to generate a clustering hierarchical structure tree. s4. During the construction of the cluster hierarchical structure tree, HDBSCAN calculates the density reachability between each pair of clusters. If the reachability distance between two clusters is small and their core distance is close, the density reachability is high, and they are temporarily merged into one cluster. and The merged quantity is: , Cluster The number of data points is only satisfied if the size of the merged cluster meets the requirement. Only clusters that meet the merging criteria can be considered valid clusters. Min_cluster_size represents the minimum cluster size; clusters that do not meet the merging criteria are discarded as noise. The denoised clustering hierarchy structure tree is traversed from bottom to top to extract stable clusters and output the final clustering result. s5. Extract information for each cluster ClCl, including the number of experts, average years of service, traditional process score, and intelligent construction process score. , For the first Number of experts in each cluster For the first Average length of service of experts in each cluster For the first Traditional craftsmanship scores for each cluster For the first The intelligent construction process score for each cluster; s6. After converting the clustering results into structured text, input them into LLM to generate causal relationships to construct prompt sub-instructions for step-by-step evaluation; s7. The silhouette coefficient is used as an indicator to measure the quality of HDBSCAN clustering. The formula for calculating the silhouette coefficient is as follows: ,in Representing data points The average distance to other data points within the same cluster. This indicates a data point. The average distance to the nearest other clusters is used, and the output of the large language model is evaluated using a four-dimensional scoring system, which includes descriptive accuracy, logicality, analytical depth, and feasibility. The total score is calculated using the following formula: ,in, For the score of the i-th dimension, Let be the weight coefficient of the i-th dimension, and the sum of the weight coefficients is 1.

[0021] S8. Finally, input all the results into LLM to perform the evaluation task, and summarize the results of each evaluation indicator to generate a comprehensive evaluation.

[0022] In step S3, the core distance is calculated based on the data point and its first... The Euclidean distance between the nearest neighbors.

[0023] In step S4, the minimum cluster size parameter Min_cluster_size is preset according to the clustering effect.

[0024] In step S7, the contour coefficient has a value range of [−1,1] and is used to measure the compactness and separation of the clustering results.

[0025] In step S7, the weighting coefficients of the four-dimensional scoring system Dynamically adjusted according to evaluation needs The most relevant data required for the case analysis of this invention: This project is the first automated production line for box girder steel reinforcement cages. The project duration is one year, and it is planned to produce approximately 300 steel reinforcement cages.

[0026] Before optimization, a single rebar cage contained 100 types and more than 10,000 rebars. After optimization, the rebar cage is modularized into 3 categories and 22 rebar components.

[0027] Before optimization, there were 9 types of rebar spacing; after optimization, the number of rebar spacing types was reduced to 4.

[0028] Through design optimization, the peak stress of the steel reinforcement cage was reduced by approximately 1%.

[0029] Compared with the traditional manual binding process, this intelligent construction process reduces the fatigue stress amplitude of the steel reinforcement cage by about 5%.

[0030] The background of the project is as follows: "Traditional steel reinforcement cage design focuses on meeting stress requirements, production relies on manual binding, and inspection is delayed. In contrast, intelligent construction technology considers stress and construction standardization requirements in the design phase, production relies on automated welding, and inspection is a synchronous process inspection." The field is: "You are a senior expert with rich experience in high-speed rail project management, scheme analysis, and statistics. When answering the questions, please combine language analysis and mathematical statistical methods to ensure your answers are academic and professional." The chain instruction format based on causal relationships is: "Based on the input data, please analyze the advantages and disadvantages of traditional and intelligent construction technologies on a certain topic, combine causal relationships, generate evaluation results, and propose optimization suggestions."

[0031] The participants in the expert questionnaire survey included design institute engineers, construction representatives, equipment manufacturer technicians, and university researchers. The format of the questionnaire is shown in Table 1.

[0032] Table 1 Expert Questionnaire The focus is on evaluating the performance of the key clustering parameter Min_cluster_size in balancing word count savings and clustering effectiveness. Using Table 1, "Degree of consideration for collision avoidance during the design phase," as an example, experiments were conducted to compare the clustering effects when Min_cluster_size was 2, 4, and 6. Figure 2 As shown.

[0033] The results show that when Min_cluster_size is 2, the clustering results can basically reflect the data distribution, but the data distribution is relatively scattered. This indicates that although a smaller parameter can maximize the preservation of data point diversity, it may lead to the clustering results failing to present clear structural characteristics. When Min_cluster_size is increased to 4, some distant points begin to merge, and the originally scattered data tends to be concentrated. This shows that appropriately increasing this parameter helps to filter out noise points and obtain a clearer clustering structure. In addition, experts with different years of experience tend to give higher scores to intelligent construction processes, and this is more pronounced among experts with longer experience, meaning that experienced experts are better able to identify the potential advantages of intelligent processes. However, when Min_cluster_size is 6, the clustering process has stricter requirements for the data, and many expert opinions that do not meet the criteria are regarded as noise and discarded, resulting in a significant decrease in the number and diversity of effective clusters. This indicates that an excessively large parameter will destroy the original subdivision characteristics of the data, thereby weakening the explanatory power of clustering for actual decision-making situations.

[0034] Based on the above results, subsequent experiments on the evaluation index of "the degree of consideration for collision avoidance during the design phase" will use Min_cluster_size as a parameter setting of 4, and the resulting structured text is as follows: "The degree of consideration for collision avoidance during the design phase was determined by clustering the expert questionnaires into 6 groups. The number of experts, average years of experience, scores for traditional construction techniques, and scores for intelligent construction techniques in each group are as follows:" Expert Group 1 (5.0 members, 3.40 years): Traditional craftsmanship 4.40 points, intelligent craftsmanship 7.00 points; Expert Group 2 (5.0 members, 2.30 years): Traditional process 6.00 points, intelligent process 8.00 points; Expert Group 3 (11.0 members, 3.82 years): Traditional craftsmanship 3.55 points, intelligent craftsmanship 8.82 points; Expert Panel 4 (4.0 members, 2.75 years): Traditional craftsmanship 7.00 points, intelligent craftsmanship 8.00 points; Expert Panel 5 (5.0 members, 3.60 years): Traditional craftsmanship 5.00, intelligent craftsmanship 8.00; Expert Panel 6 (8.0 members, 5.69 years of experience): Traditional craftsmanship 7.12 points, intelligent craftsmanship 9.50 points. Furthermore, the main causal relationships extracted using LLM during the design phase in Table 1 are as follows: Modular design can simplify the production process; Standardizing the spacing of reinforcing bars can reduce production complexity; Reducing the peak stress of the reinforcing steel cage can improve its stability.

[0035] Furthermore, this study conducted clustering experiments on each evaluation index in Table 1 sequentially, and then averaged the results to obtain the final result. Figure 3 .

[0036] Figure 3 The results show that as the Min_cluster_size increases, the ability of clustering to compress text redundancy continuously improves, but the clustering quality continuously declines. Therefore, when determining the optimal parameter of Min_cluster_size, both text compression effect and clustering quality need to be considered. Since this study not only needs to reduce redundant descriptions in a large number of expert questionnaires, but also needs to retain the key features of the questionnaires, the optimal value range of this parameter is determined to be [3,4], and [2,3) and (4,5] are considered as alternative ranges. Taking Min_cluster_size = 4 as an example, under the premise of maintaining high clustering quality, the word count reduction ratio reaches 82.08%, and a total of 56,512 words can be saved for 51 questionnaires, effectively overcoming the limitation of LLM input sequences being too long to input.

[0037] Example 2 ablation experiment To ensure the quality of process evaluation, the high-performance ChatGPT-4o was selected as the base model to further verify the effectiveness of mind chain prompting engineering based on causal relationships and HDBSCAN clustering. The experiment was still conducted with the indicator of "the degree of consideration for collision avoidance in the design stage". Four experimental scenarios were designed: (1) using only LLM; (2) LLM combined with causal reasoning; (3) LLM combined with clustering; (4) LLM combined with both causal reasoning and clustering. The results are shown in Table 2.

[0038] Table 2. LLM-based ablation experiments In its overview, the LLM model simply described the two processes as having "each with its own advantages and disadvantages," failing to clearly indicate which process was superior. While introducing causal relationships or clustering, the model pointed out "significant differences" between the two processes, still without specifying which was better. However, by simultaneously introducing causal relationships and clustering, the model precisely elucidated that the intelligent construction process "more thoroughly" considered collision avoidance, significantly improving the accuracy of the analysis. In the process comparison, the LLM model simply mentioned "the inclusion of standardized and modular design," while the LLM combined with clustering provided a similar response, largely restating the original input without in-depth analysis. In contrast, the LLM combined with causal reasoning and the method presented in this paper deeply analyzed how modular design and standardized rebar spacing reduce design complexity, which is the fundamental reason for reducing collision risk, demonstrating stronger logical reasoning ability. In expert scoring, the LLM and LLM combined with causal reasoning only briefly mentioned that the intelligent construction process scored higher, while the LLM combined with clustering and the method presented in this paper revealed that experienced experts had a higher level of acceptance of the intelligent construction process, demonstrating the effectiveness of clustering in distilling expert opinions and extracting key information. In the optimization suggestions, the introduction of both causal reasoning and clustering algorithms slightly improved the quality of the responses and made them more feasible, but the overall differences in the optimization suggestions across the four scenarios were not significant. To quantify the performance of LLM in these four scenarios, the evaluation metrics in Section 1.3 were used for scoring, with a weight of 0.25 for each of the four dimensions. The results are shown in Table 3.

[0039] Table 3 Overall Scores of LLM Responses under Different Module Combinations The results show that LLM, which incorporates both causal reasoning and clustering, improves across all dimensions and increases the overall score to 91.25, an improvement of 11.25 points compared to using LLM alone, thus verifying the effectiveness of the proposed method.

[0040] Example 3 Robustness test To verify the generalization ability of the proposed method, robustness experiments were conducted using multiple language models (ChatGPT-4o, LLAMA 3.1, Alibaba Synonyms Q&A 2.5, and Baidu Wenxin Yiyan 3.5). The test was conducted using "the degree of damage to fatigue performance caused by rebar connections" as an example. The results are shown in Table 4. Due to the length of the optimization suggestions and the small differences between the different models, only the remaining three items are presented.

[0041] Table 4. Robustness Experiments Based on LLM The results show that, in the overview, all models clearly indicate that the traditional process is significantly better in terms of fatigue performance. In the process comparison, all models identified resistance spot welding technology as the fundamental reason why intelligent construction processes affect the fatigue performance of reinforcing steel. In the expert scoring, all models mentioned that experienced experts tend to recognize the fatigue performance of the traditional process. In conclusion, all models can provide accurate, logically rigorous, and comprehensive evaluation results for the same evaluation index, demonstrating the good generalization ability of the proposed method.

[0042] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A process evaluation method based on HDBSCAN clustering and a large language model, characterized in that, The raw inputs to the evaluation method include prompting engineering, quantitative descriptive text, and expert questionnaires, and include the following steps: s1. Prompt the background and field required for the engineering input process evaluation. After decomposing the quantitative descriptive text into various evaluation index data, the expert questionnaire is also decomposed and then generated into sub-questionnaires through clustering. s2. Based on the project characteristics and expert opinions, each expert questionnaire is broken down into data for various evaluation indicators: ,in, For the first One questionnaire, For the first The first questionnaire Each evaluation indicator is used to construct a feature vector for each evaluation dimension. ,in, For the first The first questionnaire The first evaluation indicator The evaluation dimensions include expert seniority, traditional process score, and intelligent construction process score. s3. Calculate the core distance of each questionnaire to a certain evaluation indicator, whereby the core distance is defined as the distance between each data point and the evaluation indicator. To its first The distance between neighbors , yes To its first The distance between neighbors Data points and Euclidean distance between Data points are calculated based on core distance and Euclidean distance. and The reachability distance is used to measure the association between two points. Each data point is treated as an independent cluster. Based on the reachability distance, starting from the smallest distance, clusters with closer distances are gradually merged to generate a clustering hierarchical structure tree. s4. During the construction of the cluster hierarchical structure tree, HDBSCAN calculates the density reachability between each pair of clusters. If the reachability distance between two clusters is small and their core distance is close, the density reachability is high, and they are temporarily merged into one cluster. and The merged quantity is: , Cluster The number of data points is only satisfied if the size of the merged cluster meets the requirement. Only clusters that meet the merging criteria can be considered valid clusters. Min_cluster_size represents the minimum cluster size; clusters that do not meet the merging criteria are discarded as noise. The denoised clustering hierarchy structure tree is traversed from bottom to top to extract stable clusters and output the final clustering result. s5. Extract information for each cluster ClCl, including the number of experts, average years of service, traditional process score, and intelligent construction process score. , For the first Number of experts in each cluster For the first Average length of service of experts in each cluster For the first The traditional craftsmanship score of each cluster For the first The intelligent construction process score for each cluster; s6. After converting the clustering results into structured text, input them into LLM to generate causal relationships to construct prompt sub-instructions for step-by-step evaluation; s7. The silhouette coefficient is used as an indicator to measure the quality of HDBSCAN clustering. The formula for calculating the silhouette coefficient is as follows: ,in Representing data points The average distance to other data points within the same cluster. This indicates a data point. The average distance to the nearest other clusters is used, and the output of the large language model is evaluated using a four-dimensional scoring system, which includes descriptive accuracy, logicality, analytical depth, and feasibility. The total score is calculated using the following formula: ,in, For the score of the i-th dimension, Let be the weight coefficient of the i-th dimension, and the sum of the weight coefficients is 1; S8. Finally, input all the results into LLM to perform the evaluation task, and summarize the results of each evaluation indicator to generate a comprehensive evaluation.

2. The process evaluation method based on HDBSCAN clustering and large language model according to claim 1, characterized in that, In step S3, the core distance is calculated based on the data point and its first... The Euclidean distance between the nearest neighbors.

3. The process evaluation method based on HDBSCAN clustering and large language model according to claim 1, characterized in that, In step S4, the minimum cluster size parameter Min_cluster_size is preset according to the clustering effect.

4. The process evaluation method based on HDBSCAN clustering and large language model according to claim 1, characterized in that, In step S7, the contour coefficient has a value range of [−1,1] and is used to measure the compactness and separation of the clustering results.

5. The process evaluation method based on HDBSCAN clustering and large language model according to claim 1, characterized in that, In step S7, the weighting coefficients of the four-dimensional scoring system Adjustments will be made dynamically based on evaluation requirements.

Images