Method, device, program, and storage medium for monitoring quality of output of large language model

Abstract

The application relates to the technical field of computers and discloses a large language model output quality monitoring method, equipment, a program and a storage medium. The method comprises the following steps: in response to receiving a question text, inputting the question text into a steel industry field identification model to output a target data object; judging whether a preset general rule covers a field to which the question text belongs; if the general rule does not cover the field, inputting the features of a data entity into a field rule generation model to output a field quality audit rule; according to the volume and features of the data entity, splitting the data entity into a plurality of sub-data; taking the field quality audit rule as judgment logic to judge actual numerical values in each sub-data and output a quality audit result set; and inputting the quality audit result set and a data source into the steel industry field identification model to reversely infer abnormal data in the quality audit result set and output a quality monitoring result. The application can improve the accuracy of steel field data quality monitoring.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a method, device, program, and storage medium for monitoring the output quality of a large language model. Background Technology

[0002] The steel industry has a long and complex production process, generating massive amounts of multimodal industrial data. Existing data quality monitoring technologies typically rely on general-purpose natural language processing tools for text parsing. However, when faced with the complex terminology system of the steel industry, they often struggle to accurately identify compound terms, easily leading to word segmentation errors and semantic distortion. Furthermore, general-purpose tools have weak parsing capabilities for combinations of numerical values ​​and units, or complex parameter expressions with context, often resulting in the fragmentation of key values. In addition, traditional data quality rule construction relies mainly on hard-coding based on human experience. This approach is not only time-consuming and inefficient, but also difficult to change once the rule logic is set, failing to adapt to the rapid dynamic changes in steel production line parameters and operating conditions. Consequently, the accuracy and timeliness of data quality monitoring cannot meet actual production needs, and automatic attribution analysis of abnormal data is impossible. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, program, and storage medium for monitoring the output quality of large language models, in order to solve the problem of low accuracy in data quality monitoring in the steel industry in the prior art.

[0004] To achieve the above objectives, the first aspect of this application provides a method for monitoring the output quality of a large language model, the method comprising:

[0005] In response to receiving a query text, the query text is input into the steel industry domain identification model, and the target data object is output; the target data object includes the domain to which the query text belongs, the data entity, and the associated data source;

[0006] Determine whether the preset general rules cover the domain to which the question text belongs;

[0007] If not covered, the characteristics of the data entity are input into the domain rule generation model, and the domain quality audit rules are output.

[0008] Based on the size and characteristics of the data entity, the data entity is split into multiple sub-data;

[0009] Using domain quality audit rules as the judgment logic, the actual values ​​in each sub-data are judged, and the quality audit result set is output.

[0010] The quality audit result set and data source are input into the steel industry domain identification model to perform reverse reasoning on the abnormal data in the quality audit result set and output the quality monitoring results.

[0011] In this embodiment, the steps for constructing a steel industry domain recognition model include: acquiring a steel industry word segmentation corpus; inputting the steel industry word segmentation corpus into a first base model for training; the first base model includes multiple target layers, each of which has been injected with multiple dynamic low-rank adaptation layers; each target layer corresponds one-to-one with a dynamic low-rank adaptation layer; determining the training stage of the first base model based on the number of training steps; the training stage includes a warm-up stage, a dynamic rank training stage, and a fine-tuning stage; when the first base model is in the warm-up stage or the dynamic rank training stage, determining the optimal rank specification for each dynamic low-rank adaptation layer based on preset sampling rules and a preset rank specification set; when the first base model is in the fine-tuning stage, generating an optimal rank combination of multiple dynamic low-rank adaptation layers based on the optimal rank specification of each dynamic low-rank adaptation layer; determining the weight matrix of each of the multiple dynamic low-rank adaptation layers based on the optimal rank combination, so as to construct a steel industry domain recognition model in conjunction with the first base model.

[0012] In this embodiment, the dynamic low-rank adaptation layer includes a trainable weight matrix. The step of determining the optimal rank specification of each dynamic low-rank adaptation layer according to a preset sampling rule and a preset rank specification set includes: when the first base model is in the warm-up stage or the dynamic rank training stage, determining the rank specification corresponding to each dynamic low-rank adaptation layer in the warm-up stage or the dynamic rank training stage from the preset rank specification set according to the preset sampling rule; controlling each dynamic low-rank adaptation layer to run according to the corresponding rank specification and output the prediction result, determining the error between the prediction result and the steel industry word segmentation data, so as to update the weight matrix of the corresponding dynamic low-rank adaptation layer according to the error and the backpropagation algorithm; and determining the optimal rank specification of the corresponding dynamic low-rank adaptation layer based on the error and the computational cost of each dynamic low-rank adaptation layer.

[0013] In this embodiment, the steps for constructing a domain rule generation model include: acquiring a corpus of segmented words related to the steel industry; the corpus includes process rules, which in turn include real numerical values; converting the process rules into feature vectors; inputting the feature vectors into a second base model for training, and outputting a trained intermediate model and rule text; parsing the rule text to extract key numerical values; determining a reward score based on the key numerical values ​​and real numerical values; constructing an objective function based on the reward score, feature vectors, rule text, intermediate model, second base model, and a preset penalty coefficient; determining the model increment parameters using a gradient ascent algorithm with the objective function's value as the maximization goal; and constructing a domain rule generation model based on the model increment parameters and the second base model.

[0014] In this embodiment of the application, determining the reward score based on key values ​​and the steel industry terminology corpus includes: determining the reward score according to the following formula:

[0015]

[0016] in, As a reward point; These are the weighting coefficients; For the reward dimension index; The total number of reward dimensions; For the first The sub-reward scores for each reward dimension are obtained based on key values ​​and the actual values ​​in the steel industry segmentation corpus.

[0017] In this embodiment, constructing the objective function based on the reward score, feature vector, rule text, intermediate model, second base model, and preset penalty coefficient includes: the objective function satisfies the following formula:

[0018]

[0019] in, For feature vectors; For rule text; The objective function value; As a reward point; The preset penalty coefficient; This is an intermediate model; This is the second base model; Glossary of terms for the steel industry.

[0020] In this embodiment of the application, the characteristics of the data entity include production line, steel grade, and equipment number; the step of splitting the data entity into multiple sub-data according to the size and characteristics of the data entity includes: if the size of the data entity exceeds a preset data volume threshold, splitting the data entity into multiple sub-data with the same data volume; if the size of the data entity does not exceed the preset data volume threshold, splitting the data entity into multiple sub-data according to the production line, steel grade, and equipment number.

[0021] A second aspect of this application provides a computer device, comprising:

[0022] The memory is configured to store instructions; and

[0023] The processor is configured to retrieve instructions from memory and to implement the methods described above when executing instructions.

[0024] A third aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0025] A fourth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the methods described above.

[0026] The above technical solution firstly uses a steel industry domain identification model to accurately analyze the query text, identifying the target data object containing related data sources and effectively avoiding errors in the monitoring scope caused by misunderstandings. Secondly, a domain rule generation model dynamically constructs quality audit rules for specific uncovered domains, ensuring accurate matching between the verification logic and specific process characteristics, and solving the problem of false alarms or missed alarms caused by poor applicability of general rules. At the same time, combined with an intelligent splitting strategy based on data volume and features, the stability and logical integrity of verification calculations under massive data are guaranteed. Finally, the model's reverse reasoning capability is used in conjunction with data sources for attribution analysis, realizing in-depth diagnosis from apparent anomalies to the underlying causes. Thus, the accuracy of data quality monitoring in the steel industry is significantly improved throughout the entire process of identifying the scope, constructing rules, executing verification, and attribution analysis.

[0027] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0028] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0029] Figure 1 A flowchart illustrating a method for monitoring the output quality of a large language model according to an embodiment of this application is shown schematically.

[0030] Figure 2 The diagram illustrates a construction flowchart of a multi-source data acquisition system for the entire steel industry process according to an embodiment of this application.

[0031] Figure 3 A flowchart illustrating a dataset word segmentation method according to an embodiment of this application is shown schematically.

[0032] Figure 4 This illustration schematically shows a flowchart of creating a steel industry domain identification model according to an embodiment of this application;

[0033] Figure 5 A flowchart illustrating a method for creating a domain rule generation model according to an embodiment of this application is shown schematically.

[0034] Figure 6 The schematic diagram illustrates a structural diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0036] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0037] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0038] Figure 1 A flowchart illustrating a method for monitoring the output quality of a large language model according to an embodiment of this application is shown schematically. Figure 1 As shown in the figure, this application provides a method for monitoring the output quality of a large language model, which may include the following steps.

[0039] Step 101: In response to receiving the query text, input the query text into the steel industry domain identification model and output the target data object; the target data object includes the domain to which the query text belongs, the data entity, and the associated data source.

[0040] In this embodiment, the query text refers to a sequence of natural language characters input by the user to express data query or monitoring needs, containing the business intent to be analyzed. The steel industry domain identification model refers to an intelligent identification model built based on deep learning algorithms and incorporating professional knowledge of the steel industry. Through pre-training, it can accurately understand industry terminology and contextual logic. The target data object refers to a structured information set generated by the model after parsing the input content, used to define the scope of data monitoring. Here, the domain refers to the specific steel production process or business scope to which the query content belongs; the data entity refers to the basic data table or data set carrying key monitoring indicators within the corresponding domain; and the associated data source refers to the auxiliary data set that is related to the data entity in terms of process flow or equipment status. Through this step, the model, which integrates industry knowledge, can accurately locate the business scenarios and data scope involved in the user's needs, effectively reducing scope errors caused by manual retrieval or misunderstandings of general models, thereby improving the accuracy of data quality monitoring in the steel industry.

[0041] Step 102: Determine whether the preset general rules cover the domain to which the question text belongs.

[0042] Step 103: If not covered, input the features of the data entity into the domain rule generation model and output the domain quality audit rules.

[0043] In this embodiment, the preset general rules can refer to a pre-stored set of basic logic or algorithm library in the system used to perform routine or standardized data verification tasks. Coverage refers to the process by which the system retrieves and determines whether verification logic matching a specific business scenario exists in the current rule base. Data entity characteristics refer to metadata information describing the intrinsic attributes of data objects and their production environment context, including but not limited to production line affiliation, steel grade, equipment number, and the physical meaning of fields. The domain rule generation model refers to a generative artificial intelligence model fine-tuned and trained with specific domain data, capable of understanding complex process constraints and dynamically constructing logical expressions based on input features. Domain quality audit rules refer to executable verification logic or conditional expressions generated by the model based on input information, specifically applicable to the current specific business scenario and data characteristics. Through this step, when general rules cannot meet the needs of a specific domain, large-scale model technology can be used to dynamically construct dedicated verification rules that conform to the specific process background, achieving flexible adaptation to complex and ever-changing steel production data. This effectively compensates for the shortcomings of static rule bases and significantly improves the accuracy of data quality monitoring in the steel industry.

[0044] Step 104: Based on the size and characteristics of the data entity, split the data entity into multiple sub-data.

[0045] Step 105: Using the domain quality audit rules as the judgment logic, judge the actual value in each sub-data and output the quality audit result set.

[0046] In this embodiment, the volume of a data entity refers to the total number of records contained in the data object, its storage footprint, or the estimated computational resource consumption. The characteristics of a data entity refer to the distribution patterns of data in the time, spatial, or business dimensions, including but not limited to key dimension information used for data segmentation such as production line affiliation, steel grade, and equipment number. Sub-data refers to smaller data fragments or partitions with independent computational attributes formed after logically dividing a large-scale dataset according to a specific splitting strategy. Actual values ​​refer to the process parameter values ​​or status quantities actually collected by sensors or data entry systems during the steel production process. The quality audit result set refers to the summary of judgment results generated after applying verification rules to the data, including the identification of abnormal data, violation details, and related statistical information. Through this step, a parallel processing strategy can be adaptively adopted according to the data scale, avoiding the resource bottleneck of single-machine processing of massive steel data. Simultaneously, based on refined rule comparison, strict verification of each specific process parameter is ensured, thereby significantly improving the accuracy of data quality monitoring in the steel industry while maintaining processing efficiency.

[0047] Step 106: Input the quality audit result set and data source into the steel industry domain identification model to perform reverse reasoning on the abnormal data in the quality audit result set and output the quality monitoring results.

[0048] In this embodiment, abnormal data refers to specific data records that are determined in the quality audit result set to be inconsistent with the domain quality audit rules, i.e., deviating from the preset process standards in terms of numerical characteristics, fluctuation range, or business logic relationships. Reverse reasoning refers to the process by which the model, based on its understanding of the steel industry mechanisms and knowledge graph, takes abnormal data as a starting point and proactively performs multi-dimensional cross-analysis and causal chain reconstruction on related information such as process parameters, equipment status, and raw material attributes covered in the data source, thereby logically deriving the fundamental physical causes or systemic defects leading to the data anomalies. Quality monitoring results can refer to comprehensive diagnostic information or visualization reports generated after comprehensive analysis by the model, including an integrated view of abnormal data, root cause inference conclusions, and targeted improvement suggestions. Through this step, the deep cognitive capabilities of the large model are utilized to achieve automated tracing from data appearance to the essence of the problem, effectively solving the problem that traditional methods can only detect anomalies but cannot locate the causes, thereby significantly improving the accuracy of data quality monitoring in the steel industry.

[0049] The above technical solution firstly uses a steel industry domain identification model to accurately analyze the query text, identifying the target data object containing related data sources and effectively avoiding errors in the monitoring scope caused by misunderstandings. Secondly, a domain rule generation model dynamically constructs quality audit rules for specific uncovered domains, ensuring accurate matching between the verification logic and specific process characteristics, and solving the problem of false alarms or missed alarms caused by poor applicability of general rules. At the same time, combined with an intelligent splitting strategy based on data volume and features, the stability and logical integrity of verification calculations under massive data are guaranteed. Finally, the model's reverse reasoning capability is used in conjunction with data sources for attribution analysis, realizing in-depth diagnosis from apparent anomalies to the underlying causes. Thus, the accuracy of data quality monitoring in the steel industry is significantly improved throughout the entire process of identifying the scope, constructing rules, executing verification, and attribution analysis.

[0050] In this embodiment, the steps for constructing a steel industry domain recognition model may include: acquiring a steel industry word segmentation corpus; inputting the steel industry word segmentation corpus into a first base model for training; the first base model includes multiple target layers, each of which has been injected with multiple dynamic low-rank adaptation layers; each target layer corresponds one-to-one with a dynamic low-rank adaptation layer; determining the training stage of the first base model based on the number of training steps; the training stage includes a warm-up stage, a dynamic rank training stage, and a fine-tuning stage; when the first base model is in the warm-up stage or the dynamic rank training stage, determining the optimal rank specification for each dynamic low-rank adaptation layer based on preset sampling rules and a preset rank specification set; when the first base model is in the fine-tuning stage, generating an optimal rank combination of multiple dynamic low-rank adaptation layers based on the optimal rank specification of each dynamic low-rank adaptation layer; determining the weight matrix of each of the multiple dynamic low-rank adaptation layers based on the optimal rank combination, so as to construct a steel industry domain recognition model in conjunction with the first base model.

[0051] In this embodiment, the steel industry word segmentation corpus refers to a set of structured or semi-structured text training data containing professional content such as steel production process specifications, equipment parameter indicators, and quality control standards, which has been cleaned, standardized, and deduplicated. The first base model refers to a deep neural network model pre-trained on a large-scale general corpus, possessing basic natural language understanding and generation capabilities but not yet optimized for a specific industrial field. The target layer refers to the key neural network layer selected for parameter fine-tuning in the first base model's network architecture, such as the query projection layer or value projection layer in the attention mechanism. The dynamic low-rank adaptation layer refers to a parameter-efficient fine-tuning module inserted into the target layer, containing a matrix structure whose rank can be dynamically adjusted, used to capture incremental features of a specific domain while keeping the original parameters of the base model frozen. The training phase refers to the model optimization lifecycle divided according to the number of training iterations, covering a warm-up phase for parameter initialization, a dynamic rank training phase for exploring the adaptability of different network structures, and a fine-tuning phase for locking in the optimal parameter configuration. The preset sampling rule refers to the strategy logic for selecting specific rank values ​​from the candidate set at different training stages. The preset rank specification set refers to a predefined candidate set containing multiple optional integer rank values. The optimal rank combination refers to the final rank specification configuration set determined for all dynamic low-rank adaptation layers after the training process, based on the performance of each layer and computational cost. The weight matrix refers to the numerical parameter array within the dynamic low-rank adaptation layer after iterative updates via the error backpropagation algorithm; it stores specific domain knowledge and logical patterns of the steel industry. The system utilizes dynamic low-rank adaptation technology to achieve domain-adaptive fine-tuning of the base model. While significantly reducing memory usage and computational resource consumption, it enables the model to adaptively learn and store the complex terminology system and structured query generation logic of the steel industry, ensuring the model's accurate parsing ability for user natural language queries, thereby significantly improving the accuracy of data quality monitoring in the steel industry.

[0052] In this embodiment, the dynamic low-rank adaptation layer includes a trainable weight matrix. The step of determining the optimal rank specification of each dynamic low-rank adaptation layer according to a preset sampling rule and a preset rank specification set may include: when the first base model is in the warm-up stage or the dynamic rank training stage, determining the rank specification corresponding to each dynamic low-rank adaptation layer in the warm-up stage or the dynamic rank training stage from the preset rank specification set according to the preset sampling rule; controlling each dynamic low-rank adaptation layer to run according to the corresponding rank specification and output the prediction result, determining the error between the prediction result and the steel industry word segmentation data, so as to update the weight matrix of the corresponding dynamic low-rank adaptation layer according to the error and the backpropagation algorithm; and determining the optimal rank specification of the corresponding dynamic low-rank adaptation layer based on the error and the computational cost of each dynamic low-rank adaptation layer.

[0053] In this embodiment, the rank specification refers to the specific numerical value selected from a preset rank specification set in a single training iteration to constrain the dimension of matrix operations within the dynamic low-rank adaptation layer. It directly determines the number of parameters and expressive power of the current computational path. The prediction result refers to the probability distribution or sequence information output by the first base model after loading the adaptation layer parameters of the current rank specification and performing inference on the input steel industry word segmentation corpus. The error refers to the deviation between the prediction result quantized by the loss function and the standard labels in the corpus. The backpropagation algorithm is a parameter optimization method based on the error gradient backpropagation mechanism, used to drive the numerical update of the weight matrix within the dynamic low-rank adaptation layer to minimize the deviation. The computational overhead refers to the amount of hardware resources consumed when performing forward propagation computation under a specific rank specification, including memory usage and floating-point operations. The optimal rank specification refers to the target dimension configuration that minimizes resource consumption while ensuring recognition accuracy, selected for each adaptation layer based on performance evaluation during training. The system can explore and optimize the adaptability of multiple model structures in parallel during a single training process. Through a joint decision-making mechanism of error feedback and cost evaluation, it automatically selects the parameter structure combination that is most suitable for the complex terminology characteristics of the steel industry. This maximizes computational efficiency while avoiding overfitting and underfitting, and significantly improves the accuracy of data quality monitoring in the steel industry.

[0054] In this embodiment, the steps of constructing a domain rule generation model may include: acquiring a corpus of segmented words related to the steel industry; the corpus includes process rules, which in turn include real numerical values; converting the process rules into feature vectors; inputting the feature vectors into a second base model for training, and outputting a trained intermediate model and rule text; parsing the rule text and extracting key numerical values; determining a reward score based on the key numerical values ​​and real numerical values; constructing an objective function based on the reward score, feature vectors, rule text, intermediate model, second base model, and a preset penalty coefficient; determining the incremental parameters of the model using a gradient ascent algorithm, with the goal of maximizing the objective function value; and constructing a domain rule generation model based on the incremental parameters and the second base model.

[0055] In this embodiment, process rules refer to textual statements contained in the word segmentation corpus that describe the various technical indicators, operating procedures, and quality standards that must be followed in the steel production process, and they contain specific business constraint logic. True values ​​refer to specific parameter values ​​or component ratios explicitly recorded in the process rules, serving as physical truth values ​​or industry standard references. Feature vectors refer to the transformation of unstructured text descriptions into computer-recognizable high-dimensional numerical matrices, used to characterize the semantic features and contextual information of the data. The second base model refers to the selected general-purpose large language model foundation for carrying out the rule generation task; its parameters are typically frozen during training to preserve general language capabilities. The intermediate model refers to the current policy model whose parameters are dynamically updated with the number of training steps during reinforcement learning training iterations, used to generate candidate rule texts. Rule text refers to the natural language description generated by the model based on the input feature vectors, used to express data quality verification logic. Key values ​​refer to specific values ​​or thresholds extracted from the generated rule text through parsing, used to quantify and define quality standards. The reward score is a quantitative evaluation value assigned based on the scoring mechanism, calculated by assessing the degree of agreement between key numerical values ​​and true numerical values, as well as the logical consistency. It serves to evaluate the quality of the generated results. The preset penalty coefficient is a hyperparameter used during optimization to control the weight distribution differences between the current model and the base model, preventing language degradation or overfitting in pursuit of high rewards. The objective function is a mathematical optimization objective that integrates reward maximization and distribution constraints, guiding the direction of model parameter updates. The incremental model parameters are additional weight matrices specifically learned and updated during training using the gradient ascent algorithm, used to store the mapping relationship between the steel industry rule-building logic and parameters. The system utilizes feedback reinforcement learning technology to transform unstructured process standards into executable generation logic, ensuring that the generated quality audit rules are strictly aligned with the industry's physical truth in terms of numerical accuracy and logical rigor. This effectively avoids rule errors caused by model illusions, significantly improving the accuracy of data quality monitoring in the steel industry.

[0056] In this embodiment of the application, determining the reward score based on key values ​​and steel industry terminology may include: determining the reward score according to the following formula:

[0057]

[0058] in, As a reward point; These are the weighting coefficients; For the reward dimension index; The total number of reward dimensions; For the first The sub-reward scores for each reward dimension are obtained based on key values ​​and the actual values ​​in the steel industry segmentation corpus.

[0059] In this embodiment of the application, constructing the objective function based on the reward score, feature vector, rule text, intermediate model, second base model, and preset penalty coefficient may include: the objective function satisfying the following formula:

[0060]

[0061] in, For feature vectors; For rule text; The objective function value; As a reward point; The preset penalty coefficient; This is an intermediate model; This is the second base model; Glossary of terms for the steel industry.

[0062] In this embodiment of the application, the characteristics of the data entity include production line, steel grade, and equipment number; the step of splitting the data entity into multiple sub-data according to the size and characteristics of the data entity may include: if the size of the data entity exceeds a preset data volume threshold, splitting the data entity into multiple sub-data with the same data volume; if the size of the data entity does not exceed the preset data volume threshold, splitting the data entity into multiple sub-data according to the production line, steel grade, and equipment number.

[0063] In this embodiment, a production line refers to a physical work area or logical process set in the steel production process that undertakes specific smelting, processing, or handling tasks, covering all stages from raw material processing to finished product output. A steel grade is a unique code determined according to relevant standards and specifications, used to identify the chemical composition, mechanical properties, and process requirements of steel, and is directly associated with specific quality control standards. An equipment number is a sequence of numbers or characters used to uniquely identify hardware facilities such as sensors, controllers, and production units in an industrial IoT environment. A preset data volume threshold is a critical data size value pre-set by the system based on available memory resources and computing power to determine whether to trigger a load balancing splitting strategy. Splitting a data entity into multiple sub-data with the same data volume refers to a processing method that uses physical slicing or random hashing techniques to evenly distribute data across different computing nodes to prevent memory overflow when facing extremely large datasets. Splitting a data entity into multiple sub-data according to production line, steel grade, and equipment number refers to a processing method that, under the premise of controllable data size, logically groups data based on the enumeration value of the business dimension to ensure that data with the same business attribute is completely verified within the same computing unit. Through this step, the system can dynamically adapt the optimal parallel computing strategy according to the actual scale of steel data. This ensures the load balance and stability of system resources in massive data scenarios, and avoids verification errors caused by data skew or logical fragmentation in normal scenarios, thereby significantly improving the accuracy of data quality monitoring in the steel industry.

[0064] Through the above technical solution, firstly, by constructing a domain recognition model that integrates knowledge of the steel industry, and using industry-specific word segmentation corpus to specifically train the base model, the system can deeply understand complex professional terms such as "continuous casting billet" and "hot rolling coiling temperature" and their contextual logic. This overcomes the word segmentation errors and semantic distortion problems that easily occur when general models are parsed in industrial vertical domains, and achieves accurate locking of target data objects, data entities, and related data sources in the user's natural language requirements. Secondly, by constructing a domain rule generation model based on human feedback reinforcement learning (RLHF), and integrating it with the relevant data, the system can achieve a more comprehensive understanding of complex professional terms such as "continuous casting billet" and "hot rolling coiling temperature," and can also achieve a more comprehensive understanding of complex professional terms such as "continuous casting billet" and "hot rolling coiling temperature." The introduction of truth values ​​and logical constraints into the reward mechanism of the objective function enables the model to dynamically generate verification logic that conforms to physical laws based on the input process characteristics (such as steel grade and production line) when preset general rules cannot cover specific domains. This breaks through the limitations of traditional manual hard-coded rules, which are rigid and difficult to adapt to dynamic changes in production line parameters, ensuring that the generated quality audit rules strictly align with industry standards in terms of numerical accuracy and logical rigor. Furthermore, by implementing an adaptive splitting strategy based on the size and business characteristics of data entities (such as production line and steel grade), the system can automatically match datasets of different sizes. Equipped with the optimal parallel computing mode, it avoids single-machine memory overflow and computational bottlenecks through physical slicing in massive data scenarios, and prevents data skew or verification logic fragmentation caused by blind splitting in normal scenarios through logical grouping. This significantly improves monitoring efficiency while ensuring system load balance. Furthermore, by feeding the quality audit result set back to the domain identification model for reverse reasoning, and utilizing the industry mechanism map mastered by the model, it performs multi-dimensional cross-analysis of abnormal data and related data sources (such as equipment status and raw material attributes). This enables the system to automatically construct a path from "data anomaly" to "physical cause" (e.g., physical causes). The model establishes a logical chain (such as nozzle blockage and raw material fluctuations), thereby enabling data quality monitoring to move from simply "discovering problems" to "locating root causes," providing a valuable reference for production decisions. Finally, by employing Dynamic Low-Rank Adaptation (DyLoRA) technology during model building, the model dynamically explores different rank specifications and generates the optimal rank combination during the training phase. This allows the model to automatically find the parameter structure most suitable for the characteristics of the steel industry under limited computing resource constraints, thus significantly reducing memory usage and training costs while ensuring the model's efficient capture and accurate representation of specific domain knowledge.

[0065] The following is an example of this application:

[0066] S1. Prepare steel industry data, such as relevant knowledge documents and other materials, as well as structured and unstructured data related to the steel industry within the group. Use data governance and other related technologies to clean the data and create high-quality datasets to form high-quality datasets for each business domain of the steel industry.

[0067] S1.1. Data lake object storage technology is used to store unstructured data. Regular incremental collection and updates are carried out every quarter. Data cleaning methods such as invalid value processing (cleaning up blank and duplicate files) are used for documents.

[0068] S1.2. Structured data is stored using Iceberg technology, low-frequency data is collected and updated on a T+1 cycle, and high-frequency data is processed in seconds using streaming processing. Structured data is cleaned using methods such as invalid value handling, format standardization, outlier handling, and data alignment.

[0069] S1.3 Construct entity types through products, materials, equipment, processes, enterprises, standards, and performance indicators.

[0070] S1.4 Construct an integrated data foundation for steel enterprises' lake and warehouse systems.

[0071] S2. The HanLP + industry dictionary + LTP word segmentation strategy was used multiple times to process and adjust the word segmentation of steel industry data.

[0072] S3. Fine-tuning of the identification model for the steel industry sector should be carried out according to the following steps.

[0073] S3.1 Load the Qwen-32b model.

[0074] S3.2 Configure DyLoRA and use the PEFT library for adaptation. PEFT is a fine-tuning framework. Configure DyLoRA through DyLoraConfig.

[0075] S3.2.1 Initialization: Inject DyLoRA layers into the target modules of the model, such as the Query and Value projection layers, and set a rank search space (r=[4,8,16,32], where 1 target module = 1 DyLoRA layer, and each layer has its own rank candidate set).

[0076] S3.2.2, Dynamic Training, including core steps such as rank sampling, forward propagation, and back propagation; In each training step, for each DyLoRA layer, a rank is randomly sampled from its rank candidate set for this forward and back propagation; Each training step scheduler determines the execution of the warm-up phase, dynamic rank training phase, and fine-tuning phase. The warm-up phase samples every 10 steps, the dynamic rank training phase samples every step, and the fine-tuning phase uses the best rank and does not sample.

[0077] S3.2.3, Rank Optimization: Rank optimization is performed by considering rank performance, importance, and adaptive selection.

[0078] S3.2.4, PEFT integration implementation, configures DyLoRA into the model.

[0079] S3.3 Training loop: Fine-tune the model using the training data (the data processed in S2) and save the model.

[0080] S4. The rule generation model for the steel industry is carried out according to the following steps.

[0081] S4.1 Model Initialization and Loading: Load the Qwen-32b model, initialize the model architecture and configure the corresponding word segmenter to prepare for fine-tuning the rule generation task.

[0082] Parameter configuration: Initialize the low-rank adaptation (LoRA) parameters, focusing on the feedforward layer and attention mechanism in the model that are sensitive to rule logic, reducing interference with the original knowledge, as shown in the following formula.

[0083]

[0084] in These are the original parameters for Qwen-32b; These are the incremental parameters for the model. Generate parameters for the domain rules model.

[0085] S4.2 Training data preparation and preprocessing.

[0086] S4.2.1 Prepare a domain-specific supervised fine-tuning dataset.

[0087] S4.2.2 Data Vectorization: Transforming unstructured process requirements into structured feature vectors. For example, the tundish temperature constraint "1540℃~1560℃" is encoded as a range feature and mapped to the steel grade code (such as "SPHC") and process node.

[0088] S4.3 Fine-tuning of the rule generation task: Use a dedicated rule generation dataset to train the model.

[0089] Fine-tuning data iteration: Human feedback reinforcement learning (RLHF) is employed, with experts in the steel industry scoring the generation rules to optimize the objective function.

[0090]

[0091] in, It could be a score from steel industry experts on the enforceability and security of the rules, or a reward score generated by the system; Given an input x, the probability distribution for generating a rule y; This serves as a second base model, acting as a reference distribution to prevent policy deviation. To pre-determine the penalty coefficient, a conservative parameter of 0.2 is recommended for the model. This represents the objective function value (expected function value).

[0092]

[0093] in, For the first The weight coefficients of each evaluation dimension satisfy... =1; For the first Sub-reward scores in each dimension; in the steel industry, only technical feasibility is considered, i.e., set... =1, the rest =0.

[0094] Technical feasibility reward:

[0095]

[0096] in, For the first Sub-reward scores for each reward dimension; The total number of violations; Let be the penalty value for the k-th violation.

[0097] Penalties for violations:

[0098]

[0099] in, The key temperature is the critical value (i.e., the generated value of the model). This represents the actual physical temperature. The content of key components in the key values ​​(i.e., the generated values ​​of the model); The chemical composition standard values ​​are the actual values; This is a logical conflict indicator variable (1 if a conflict exists, 0 otherwise); Weighting for temperature violations; Weighting for non-compliant ingredients; Zero tolerance for logical inconsistencies; reference parameters for the steel industry:

[0100]

[0101]

[0102]

[0103] S4.4 Model Deployment and Rule Engine Integration: The finely tuned rule generation model is deployed to the inference environment and integrated with the MCP rule engine. In practical applications, a dual-path decision mechanism is constructed: pre-defined rules in the rule base are matched first; for uncovered scenarios, the model inference process is initiated, generating audit rules that conform to the characteristics of the steel industry based on the input feature representation, thus achieving dynamic expansion of the rule base.

[0104] S5 and prompt guide data selection, loading metadata information into the audit system. Data matching is then performed using a steel industry sector identification model. If the matching system is an energy system, the energy system's database metadata, such as tables and fields, is loaded into the audit system. The steel industry sector identification model then parses the user's natural language request, outputting an SQL query or a list of table names. For example, if the user inputs, "Please help me find all data and tables related to the stability of 'hot-rolled coiling temperature'," the steel industry sector identification model identifies it as belonging to the steel metallurgical engineering sector, and the parsed data list includes a process data table and a steel coil tracking table.

[0105] S6, prompt guides rule generation, using the graphTD mode, which combines the data objects determined in S5 with the rules generated in S4 to generate domain quality rules.

[0106] S7. Perform quality audits. The LLM-driven dynamic splitting strategy performs parallel audits of the data and summarizes the results of all parallel tasks.

[0107] S7.1, Data volume threshold: When the number of data rows exceeds 1 million, the data will be split.

[0108] S7.2 Time range threshold: For high-frequency real-time data, split by hour; for medium- and long-term historical data analysis, split by day.

[0109] S7.3, field thresholds, and the enumeration of more than 20 values ​​by key dimensions such as production line, steel grade, and equipment number are split.

[0110] S8. Abnormal data analysis.

[0111] The data in S6 was integrated and analyzed. As a senior analyst, I actively correlated and queried relevant data within this time period (e.g., the "coiling temperature" at the exit of the "F7 finishing mill" of the "1# hot rolling line" was consistently lower than the lower limit of the process standard (650±15℃), with an average of 620℃). Through multi-dimensional cross-analysis, I deduced the most likely root cause.

[0112] S8.1, Process parameter correlation analysis and inference: It is possible that the raw material billet was not heated enough, resulting in insufficient heat transfer from the final rolling temperature to the coiling temperature.

[0113] S8.2 Equipment status correlation analysis and inference: The nozzle is "blocked", the cooling system fails locally, resulting in uneven cooling and excessive overall cooling capacity, causing the winding temperature to be too low.

[0114] S8.3, Correlation Analysis and Inference of Raw Material Composition: Small fluctuations in raw material composition may affect the phase transition temperature, exacerbating temperature deviations under inappropriate processes.

[0115] S8.4, Most likely root cause, inference: "Clogged nozzles in the laminar flow cooling system" is the main reason for the persistently low winding temperature. Repair records were found.

[0116] Figure 2 This illustration schematically shows a flowchart of the construction of a multi-source data acquisition system for the entire steel industry process according to an embodiment of this application. For example... Figure 2 As shown in the embodiment of this application, the process first enters the data type judgment stage from the initial data preprocessing, accurately dividing the input data into three categories: structured data (covering L1 / L2 process data, MES, LIMS and ERP system data), unstructured data (including process specifications and work instructions), and external literature (involving academic papers and technical documents). Subsequently, the above-mentioned multi-source heterogeneous data is uniformly collected and enters the data cleaning and standardization stage, performing cleaning operations including removing invalid values, unifying formats, and handling outliers. Finally, the processed high-quality data is aggregated and stored in the integrated data base of the steel enterprise lake warehouse, thus forming a high-quality dataset of various business domains in the steel industry to support subsequent model training and analysis.

[0117] Figure 3 A flowchart illustrating a dataset word segmentation process according to an embodiment of this application is shown schematically. Figure 3 As shown in this embodiment, the process first initiates an intelligent word segmentation task and enters the first round of word segmentation. The HanLP tool is used to perform basic word segmentation on the input high-quality dataset of various business domains in the steel industry. Then, a pre-set industry dictionary is used for forced matching to accurately identify industry-specific terms such as "steel grade", "steel tapping mark" and "heating furnace" to prevent key information from being incorrectly segmented. Next, semantic correction is performed using LTP technology to eliminate word segmentation ambiguity and correct contextual relationships. Then, the word segmentation results are evaluated for quality. If the result is deemed qualified, a high-quality steel industry word segmentation corpus is output. If the result is deemed unqualified, the process enters the problem analysis stage to optimize the strategy and then re-enters the first round of word segmentation, thus forming a closed-loop optimization mechanism to ensure the quality of the corpus.

[0118] Figure 4 A flowchart illustrating an embodiment of this application for creating a steel industry domain identification model is shown. Figure 4As shown in the embodiment of this application, the process of constructing the steel industry domain recognition model first loads the Qwen-32b base model and configures the dynamic low-rank adapter. Then, the dynamic low-rank adaptation layer is initialized and injected into the model target module. Next, a dynamic training process is entered, which includes a warm-up stage (low-frequency sampling), a dynamic rank stage (randomly sampling the rank and updating the weight matrix at each step), and a fine-tuning stage (locking the best rank). After training, rank optimization is performed based on performance evaluation to select the optimal rank combination. The optimized adaptation layer parameters are integrated into the model through the parameter efficient fine-tuning (PEFT) technology. Finally, the steel industry domain recognition model with industry recognition and SQL generation capabilities is saved.

[0119] Figure 5 A flowchart illustrating a method for creating a domain rule generation model according to an embodiment of this application is shown. Figure 5 As shown in this embodiment, the process of constructing a domain rule generation model first involves model initialization and loading. Specifically, this includes loading the Qwen-32b base model and configuring DyLoRA adaptation parameters to focus on the feedforward layer and attention mechanism. Next, the model enters the training data and preprocessing stage, which covers dataset preparation and data vectorization operations to transform unstructured process rules into structured features. Then, the rule generation task is fine-tuned based on the preprocessed corpus, iteratively updating the model's incremental parameters through Human Feedback Reinforcement Learning (RLHF) and the objective function of maximizing expert rating rewards. Finally, the model is deployed and integrated with the rule engine to construct a domain rule generation model capable of working collaboratively with a general rule library.

[0120] Figure 6 A schematic diagram illustrating the structure of a computer device according to an embodiment of this application is provided. Figure 6 As shown in the illustration, this application provides a computer device that may include:

[0121] Memory 610 is configured to store instructions; and

[0122] Processor 620 is configured to retrieve instructions from memory 610 and to implement the methods described above when executing instructions.

[0123] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0124] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described method.

[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0130] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0131] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0132] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0133] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for monitoring the output quality of a large language model, characterized in that, The method includes: In response to receiving a query text, the query text is input into the steel industry domain identification model, and a target data object is output; the target data object includes the domain to which the query text belongs, the data entity, and the associated data source; Determine whether the preset general rules cover the domain to which the question text belongs; If not covered, the features of the data entity are input into the domain rule generation model, and the domain quality audit rules are output. Based on the size and characteristics of the data entity, the data entity is divided into multiple sub-data; Using the aforementioned domain quality audit rules as the judgment logic, the actual values ​​in each sub-data are judged, and a quality audit result set is output. The quality audit result set and the data source are input into the steel industry domain identification model to perform reverse reasoning on the abnormal data in the quality audit result set and output the quality monitoring results. The steps for constructing the steel industry domain identification model include: Obtain segmented corpus of the steel industry; The steel industry word segmentation corpus is input into the first base model for training; the first base model includes multiple target layers, and multiple dynamic low-rank adaptation layers have been injected into the multiple target layers; the target layers and the dynamic low-rank adaptation layers correspond one-to-one; The training phase of the first base model is determined based on the number of training steps of the first base model; the training phase includes a warm-up phase, a dynamic rank training phase, and a fine-tuning phase; When the first base model is in the preheating stage or the dynamic rank training stage, the optimal rank specification of each dynamic low-rank adaptation layer is determined according to the preset sampling rules and the preset rank specification set. When the first base model is in the fine-tuning stage, the optimal rank combination of the multiple dynamic low-rank adaptation layers is generated according to the optimal rank specification of each dynamic low-rank adaptation layer. The weight matrices of each of the multiple dynamic low-rank adaptation layers are determined based on the optimal rank combination, so as to construct the steel industry domain identification model in combination with the first base model.

2. The method according to claim 1, characterized in that, The dynamic low-rank adaptation layer includes a trainable weight matrix. The step of determining the optimal rank specification for each dynamic low-rank adaptation layer based on preset sampling rules and preset rank specification sets includes: When the first base model is in the preheating stage or the dynamic rank training stage, the rank specification corresponding to each dynamic low-rank adaptation layer in the preheating stage or the dynamic rank training stage is determined from the preset rank specification set according to the preset sampling rules. Each dynamic low-rank adaptation layer is controlled to operate according to the corresponding rank specification and output the prediction results. Determine the error between the prediction result and the steel industry word segmentation corpus, and update the weight matrix of the corresponding dynamic low-rank adaptation layer based on the error and the backpropagation algorithm; The optimal rank specification for the corresponding dynamic low-rank adaptation layer is determined based on the error and the computational cost of each dynamic low-rank adaptation layer.

3. The method according to claim 1 or 2, characterized in that, The steps for constructing the domain rule generation model include: Obtain a keyword corpus for the steel industry; the keyword corpus for the steel industry includes process rules, and the process rules include actual numerical values. The process rules are converted into feature vectors; The feature vector is input into the second base model for training, and the trained intermediate model and rule text are output. The rule text is parsed to extract key values; The reward score is determined based on the key values ​​and the actual values; A target function is constructed based on the reward score, the feature vector, the rule text, the intermediate model, the second base model, and the preset penalty coefficient. With the objective function value of the objective function as the goal, the incremental parameters of the model are determined by the gradient ascent algorithm; The domain rule generation model is constructed based on the incremental parameters of the model and the second base model.

4. The method according to claim 3, characterized in that, The determination of the reward score based on the key values ​​and the steel industry keyword corpus includes: The reward score is determined according to the following formula: in, The reward score; These are the weighting coefficients; For the reward dimension index; The total number of reward dimensions; For the first The sub-reward scores for each reward dimension are obtained based on the key values ​​and the actual values ​​in the steel industry corpus.

5. The method according to claim 3, characterized in that, The step of constructing the objective function based on the reward score, the feature vector, the rule text, the intermediate model, the second base model, and the preset penalty coefficient includes: The objective function satisfies the following formula: in, The feature vector; The rule text; The objective function value; The reward score; The preset penalty coefficient; This is the intermediate model; This is the second base model; The terminology for the steel industry is provided.

6. The method according to claim 1 or 2, characterized in that, The characteristics of the data entity include production line, steel grade, and equipment number; The step of splitting the data entity into multiple sub-data based on its size and characteristics includes: If the size of the data entity exceeds a preset data size threshold, the data entity will be split into multiple sub-data with the same data size. If the size of the data entity does not exceed the preset data volume threshold, the data entity is split into multiple sub-data according to the production line, the steel grade, and the equipment number.

7. A computer device, characterized in that, include: The memory is configured to store instructions; as well as A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method according to any one of claims 1 to 6.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the method according to any one of claims 1 to 6.

Images