Comparison method for large model generation integrity evaluation

Abstract

The invention belongs to the technical field of model evaluation, and relates to a comparison method for large model generation integrity evaluation, which comprises the following steps: firstly, collecting related data in the power grid field, carrying out data annotation and standardization processing in combination with expert experience, and constructing a reference data set; the key features of the generated content and the reference data are extracted from the four dimensions of the semantic feature, the technical feature, the structural feature and the timeliness feature, the difference between the generated content and the standard can be accurately recognized through the multi-dimensional comparison, and the integrity score is distributed for each comparison analysis. And finally, performing weighted summation in combination with various scores to obtain a comprehensive integrity score, thereby comprehensively evaluating the integrity of the generated content. The generated content is evaluated from multiple levels and angles, the limitation of single-dimension evaluation is effectively avoided, a scientific, comprehensive and dynamic content integrity evaluation mechanism is provided, and the method is particularly suitable for application requirements of power grids and other complex fields.

Description

Technical Field

[0001] The present application belongs to the technical field of prediction model evaluation, and more specifically, to a comparison method for evaluating the integrity of large model generation. Background Art

[0002] With the rapid development of large-scale generative models (such as GPT-3, GPT-4, etc.), automatic generation technology based on natural language processing has been widely used in many fields, including electricity, finance, medical and other industries. In these industries, the automatically generated content not only needs to be semantically and technically consistent with the knowledge in the professional field, but also needs to maintain high structural integrity and timeliness. Especially in complex fields such as the power grid industry, the generated content must ensure compliance with industry standards, rules and practices.

[0003] However, despite the powerful capabilities of large models in content generation, how to evaluate the integrity of the generated content has become an important technical challenge. In particular, when the generated content is used to assist decision-making, report writing, or data analysis, how to ensure that the content generated by the model is not only semantically accurate, but also meets the expected requirements in terms of technical implementation, structural rationality, and timeliness has become a problem that must be solved.

[0004] In this context, existing evaluation methods often focus on the accuracy and consistency of model-generated content, but lack an effective evaluation mechanism for content integrity (i.e., judging the comprehensiveness and rationality of generated content from multiple dimensions). Therefore, developing a large model-generated content integrity evaluation method based on multi-level feature comparison analysis has become a core technical problem that needs to be solved urgently in power grids and other industries. Summary of the invention

[0005] The present invention provides a comparison method for evaluating the integrity of large model generation, which is intended to solve the current technical problem of lack of an effective evaluation mechanism for the integrity of content (i.e., judging the comprehensiveness and rationality of generated content from multiple dimensions).

[0006] A comparison method for large model generation integrity assessment includes the following steps:

[0007] Step 1: Collect data related to the large model generation content in the power grid, and standardize the collected data after data annotation based on expert experience to obtain a benchmark data set;

[0008] For example, a large-scale model generates a power grid operation status report, which usually contains information about the real-time operation status of the power grid, equipment status, load forecast, and possible risk assessment. To ensure the integrity of the generated content, we need to collect benchmark data related to the report through the following steps.

[0009] Power grid operation data (real-time data)

[0010] The report usually includes detailed information about the current status of the power grid, so it is necessary to obtain real-time operation data of the power grid. The relevant data includes:

[0011] Grid load data: real-time load, load forecast, load fluctuation, etc. For example, the total system load and power consumption in a certain period of time (such as 10:00AM).

[0012] Voltage and current data: voltage and current data of each substation, especially abnormal fluctuations.

[0013] Power flow data: power flow of transmission lines, generator output power, load distribution, etc.

[0014] Equipment operating status: the working status, health status, and whether there are any faults of key equipment (such as transformers, circuit breakers, switches).

[0015] Power grid equipment maintenance and fault data

[0016] In order to ensure that the generated report can accurately reflect the operational risks of the power grid and the health status of the equipment, it is necessary to obtain the historical maintenance records, fault reports and repair records of the power grid equipment. The relevant data includes:

[0017] Equipment maintenance history: Maintenance and repair records of transformers, switchgear, etc., especially data related to equipment failures.

[0018] Fault event records: the time of grid interruption, power facility failure, fault cause, recovery time, etc. For example, a transformer failure occurred in a substation and the repair time was 3 hours.

[0019] Fault diagnosis and repair data: record the cause analysis, handling process and effect of each fault.

[0020] Power grid operation analysis and forecast data

[0021] In order to assess the future load and potential risks of the power grid, the report may involve the forecast and risk assessment of the power grid load. Therefore, the following types of data must be collected:

[0022] Load forecast data: Load forecast data based on historical load data, seasonal fluctuations, weather impacts and other factors.

[0023] Risk assessment data: risk level, potential failure probability, overload risk, etc. of each node in the power grid (substation, transmission line, etc.). For example, through load forecasting and equipment health data, assess which equipment may fail or require preventive maintenance in the future.

[0024] Grid Standards and Specifications Data

[0025] The power grid industry has a series of operating standards and specifications, which are usually compared in reports to ensure that the power grid operation meets the industry's safety, efficiency and other requirements. Relevant data includes:

[0026] Power grid equipment standards: such as specifications, rated voltage, rated power, etc. of transformers and transmission lines.

[0027] Industry safety standards: For example, parameters such as voltage, current, and stability of the power system must be within the safety range specified by the industry.

[0028] Regulatory requirements: For example, power industry regulatory requirements issued by the National Energy Administration or local grid safety regulations.

[0029] Grid historical reporting and benchmark data

[0030] In order to compare with the generated report and ensure the integrity of the generated content, it is also necessary to collect historical report data of the power grid as a benchmark:

[0031] Historical operation reports: Grid operation reports generated in the past, containing content structures similar to the current report for comparison.

[0032] Grid operation log: historical log records, especially data related to past grid failures, load changes, operation optimization, etc.

[0033] Construction of benchmark dataset

[0034] Through the collection of the above data, we can annotate and standardize the data to build a complete benchmark dataset. This benchmark dataset will provide a standard reference for the integrity assessment of the content of the report generated by the large model, including:

[0035] Real-time grid data (such as load, voltage, current, etc.);

[0036] Maintenance and fault records of power grid equipment;

[0037] Load forecasting and risk assessment of power grid;

[0038] Historical operation reports of power grid and industry standards.

[0039] Step 2: Based on the model-generated content and the benchmark dataset, perform semantic feature extraction, technical feature extraction, structural feature extraction, and timeliness feature extraction on the model-generated content and the benchmark dataset respectively; based on the extracted features, perform a detailed comparison between the model-generated content and the benchmark dataset to find out the differences, and calculate the completeness score of each comparative analysis based on the results of the comparative analysis;

[0040] Step 3: Calculate the comprehensive integrity score by taking the weighted sum of the integrity scores of various comparative analyses.

[0041] Preferably, the semantic feature extraction is implemented using a BERT model, and the specific model structure is as follows:

[0042] Input layer: used to input preprocessed text data;

[0043] Lexical feature extraction module: embeds each word through the pre-trained BERT model to obtain the semantic vector representation of each word;

[0044] Sentence-level feature extraction module: Aggregate the semantic vectors at the vocabulary level and extract the semantic representation of each sentence. That is, use the [CLS]token output in the BERT model as the semantic vector of the sentence to obtain the semantic vector of each sentence.

[0045] Paragraph-level feature extraction module: By taking a weighted average of the semantic vectors of multiple sentences, the semantics of the sentences are aggregated into the semantic representation of the paragraphs to obtain the semantic vector of each paragraph;

[0046] Domain knowledge enhancement module: Introducing power grid domain knowledge to enhance the semantic vector of each layer of feature extraction module of the model:

[0047] h enhanced,i =h original,i +h domain ;

[0048] Where: h enhanced represents the semantic vector enhanced by the i-th feature extraction module; h original,u represents the semantic vector output by the i-th feature extraction module; h domain Represents semantic vectors enhanced with domain knowledge;

[0049] Feature aggregation module: aggregates the semantic vectors enhanced by each feature extraction module to obtain the aggregated semantic vector:

[0050]

[0051] Where: h final represents the final semantic feature vector; h k represents the semantic feature vector of the kth layer; w k Represents the weight of the k-th layer feature.

[0052] Preferably, the technical feature extraction comprises the following steps:

[0053] Rule-based feature extraction: Use the re module in Python to perform regular matching, extract technical parameters, and standardize the extracted technical parameters;

[0054] BiLSTM-CRF model feature extraction: The BiLSTM-CRF model is used to extract complex features. The structure of the BiLSTM-CRF model is as follows:

[0055] Input layer: Input a serialized technical document, use Word2Vec pre-trained word vectors to convert each word into a vector;

[0056] BiLSTM layer: performs forward and backward calculations through two LSTM networks to obtain the context information of the sentence;

[0057] CRF layer: further refines the prediction by taking into account the dependencies between predefined labels:

[0058]

[0059] Where: Indicates that from label y t-1 Move to label y t probability; Indicates the current label y t Corresponding input x t The score of ; y′ represents the given input sequence x, representing all possible label combinations;

[0060] Input layer: predicts the label of each word, including technical terms or non-technical terms, and identifies technical terms and device parameters based on this;

[0061] Feature fusion: The features extracted based on the rules are concatenated with the features of the BiLSTM-CRF model.

[0062] Preferably, the structural feature extraction comprises the following steps:

[0063] Graph structure construction: Graph structures are constructed for the content generated by the model and the information in the benchmark dataset, where nodes include unit information and edges represent the relationship between information units;

[0064] Extract the structural features of the graph:

[0065] Node degree:

[0066]

[0067] Where: d(v) represents the degree of node v; A vi An indicator function indicating whether node v is connected to node i; N represents the total number of nodes in the graph;

[0068] Connectivity characteristics of graphs:

[0069]

[0070] Where: d(i,j) represents the shortest path length between node i and node j; N represents the number of nodes on the way;

[0071] The connectivity of a graph refers to the number of subgraphs that are not connected to each other along the way. The number of connected components is used to measure the looseness of its structure.

[0072] Preferably, the timeliness feature extraction comprises the following steps:

[0073] Time-sensitive delay calculation: Calculate the time difference between the generated content and the benchmark dataset to quantify the delay:

[0074] Δt=|t model -t benchmark |;

[0075] Where: Δt represents the time difference between the generated content and the benchmark dataset; t model and t benchmark Respectively represent the time difference between the generated content and the benchmark dataset;

[0076] Calculate the timeliness difference:

[0077]

[0078] Where: S timeliness (i) represents the timeliness score of the i-th data unit; Δt i represents the time difference between the generated content and the benchmark dataset; τ represents the decay factor;

[0079] Data update frequency division:

[0080]

[0081] Where: S frequency (i) represents the update frequency score of the i-th data unit; freq actual Indicates the actual update frequency of the data; freq expected Indicate the expected update frequency of the benchmark data;

[0082] Calculate the Timeliness Completeness Score:

[0083]

[0084] Where: S timeliNess represents the comprehensive timeliness score of the generated content; α1 and α2 represent weight coefficients; N represents the total number of data units.

[0085] Preferably, the semantic feature difference between the model-generated content and the benchmark dataset is calculated as follows:

[0086]

[0087] Where: ||·|| represents the L2 norm of the vector; The final semantic feature vector representing the content generated by the model; represents the final semantic feature vector of the benchmark dataset; represents the dot product of the vectors.

[0088] Preferably, the technical feature difference between the model-generated content and the benchmark data set is calculated as follows:

[0089]

[0090] Where: Represents the i-th technical feature in the model-generated content; represents the i-th technical feature in the benchmark dataset; N represents the total number of features.

[0091] Preferably, the structural feature differences between the model-generated content and the benchmark dataset are compared as follows:

[0092] Node degree difference:

[0093]

[0094] Where: represents the degree of the i-th node in the model-generated content; represents the degree of the i-th node in the benchmark dataset; N represents the total number of nodes;

[0095] Differences in graph connectivity:

[0096]

[0097] Where: d(i,j) model Indicates the shortest path length between node i and node j in the generated content graph; d(i,j) benchmark represents the shortest path length between node i and node j in the benchmark dataset graph; N represents the number of nodes in the graph;

[0098] Connected component differences:

[0099] Connected Components Difference=|Connected(G model )-

[0100] Connected(Gbenchmark )|;

[0101] Where: Connected(G model ) and Connected(G benchmark ) represent the number of connected components in the generated content and benchmark dataset graphs respectively;

[0102] Calculate the structural feature difference score:

[0103] S structural =w1·Degree Difference+w2·Connectivity Difference+w3·

[0104] Connected COmponents DIfference;

[0105] Wherein: w1, w2 and w3 represent weight coefficients respectively, w1+w2+w3=1.

[0106] The beneficial effects of the present invention include:

[0107] This technical solution effectively solves the problem of lack of comprehensive evaluation mechanism by proposing a large model generated content integrity evaluation method based on multi-dimensional feature comparison analysis. The method first collects relevant data in the power grid field, and combines expert experience to label and standardize data, builds a benchmark data set, and ensures that the evaluation process can be connected with industry standards and actual needs. Then, the method extracts the key features of the generated content and the benchmark data from four dimensions: semantic features, technical features, structural features, and timeliness features, to ensure a detailed comparative analysis of the comprehensiveness and rationality of the content. Through this multi-dimensional comparison, the difference between the generated content and the standard can be accurately identified, and an integrity score can be assigned to each comparative analysis. Finally, a weighted sum is performed by combining various scores to obtain a comprehensive integrity score, thereby comprehensively evaluating the integrity of the generated content. This solution effectively avoids the limitations of single-dimensional evaluation by evaluating the generated content from multiple levels and angles, and provides a scientific, comprehensive, and dynamic content integrity evaluation mechanism, which is particularly suitable for application needs in complex fields such as power grids. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0109] Figure 1An overall step block diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0110] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0111] See also Figure 1 As shown, the best embodiment of the present invention is further described;

[0112] A comparison method for large model generation integrity assessment includes the following steps:

[0113] Step 1: Collect data related to the large model generation content in the power grid, and standardize the collected data after data annotation based on expert experience to obtain a benchmark data set;

[0114] Step 2: Based on the model-generated content and the benchmark dataset, perform semantic feature extraction, technical feature extraction, structural feature extraction, and timeliness feature extraction on the model-generated content and the benchmark dataset respectively; based on the extracted features, perform a detailed comparison between the model-generated content and the benchmark dataset to find out the differences, and calculate the completeness score of each comparative analysis based on the results of the comparative analysis;

[0115] Step 3: Calculate the comprehensive integrity score by taking the weighted sum of the integrity scores of various comparative analyses.

[0116] As a possible implementation of this embodiment, the semantic feature extraction is implemented using a BERT model, and the specific model structure is as follows:

[0117] Input layer: used to input preprocessed text data;

[0118] Lexical feature extraction module: embeds each word through the pre-trained BERT model to obtain the semantic vector representation of each word;

[0119] Sentence-level feature extraction module: Aggregate the semantic vectors at the vocabulary level and extract the semantic representation of each sentence. That is, use the [CLS]token output in the BERT model as the semantic vector of the sentence to obtain the semantic vector of each sentence.

[0120] Paragraph-level feature extraction module: By taking a weighted average of the semantic vectors of multiple sentences, the semantics of the sentences are aggregated into the semantic representation of the paragraphs to obtain the semantic vector of each paragraph;

[0121] Domain knowledge enhancement module: Introducing power grid domain knowledge to enhance the semantic vector of each layer of feature extraction module of the model:

[0122] h enhanced,i =h original,i +h domain ;

[0123] Where: h enhanced represents the semantic vector enhanced by the i-th feature extraction module; h origi,al,i represents the semantic vector output by the i-th feature extraction module; h domain Represents semantic vectors enhanced with domain knowledge;

[0124] Feature aggregation module: aggregates the semantic vectors enhanced by each feature extraction module to obtain the aggregated semantic vector:

[0125]

[0126] Where: h finak represents the final semantic feature vector; h k represents the semantic feature vector of the kth layer; w k Represents the weight of the k-th layer feature.

[0127] The above feature extraction model structure allows us to extract richer semantic information from text and introduce domain knowledge to further enhance the performance of the model. This method not only considers different semantic information from fine-grained to high-level, but also integrates features of each layer through weighted aggregation to ensure that the final semantic representation can most effectively support subsequent difference comparison and completeness.

[0128] As a possible implementation of this embodiment, the technical feature extraction includes the following steps:

[0129] Rule-based feature extraction: Use the re module in Python to perform regular matching, extract technical parameters, and standardize the extracted technical parameters;

[0130] BiLSTM-CRF model feature extraction: The BiLSTM-CRF model is used to extract complex features. The structure of the BiLSTM-CRF model is as follows:

[0131] Input layer: Input a serialized technical document, use Word2Vec pre-trained word vectors to convert each word into a vector;

[0132] BiLSTM layer: performs forward and backward calculations through two LSTM networks to obtain the context information of the sentence;

[0133] CRF layer: further refines the prediction by taking into account the dependencies between predefined labels:

[0134]

[0135] Where: Indicates that from label y t-1 Move to label y t probability; Indicates the current label y t Corresponding input x t The score of ; y′ represents the given input sequence x, representing all possible label combinations;

[0136] Input layer: predicts the label of each word, including technical terms or non-technical terms, and identifies technical terms and device parameters based on this;

[0137] Feature fusion: The features extracted based on the rules are concatenated with the features of the BiLSTM-CRF model.

[0138] The generation of labels depends on manual annotation and automatic annotation tools. Manual annotation usually uses labels such as B-TAG (beginning of term), I-TAG (middle of term) and O (non-term) to indicate the beginning, middle and other non-term parts of technical terms.

[0139] These tags are used to train the BiLSTM-CRF model. By learning the contextual dependencies of these tags, the model can automatically identify technical terms in documents and generate corresponding tag sequences.

[0140] The CRF layer uses the dependencies between these labels to optimize the prediction results and ensure the consistency and correctness of the predicted labels.

[0141] In this way, the model can effectively identify and annotate key features in technical documents, and then extract useful technical features.

[0142] Although rule matching is very efficient in certain situations, it may not be flexible enough when faced with complex technical documents and tasks with strong contextual dependencies. To overcome these limitations, rule matching is combined with deep learning models to ensure both efficiency and flexibility in dealing with complex texts. The final feature extraction solution can improve overall accuracy and scalability by combining the advantages of both.

[0143] As a possible implementation of this embodiment, the structural feature extraction includes the following steps:

[0144] Graph structure construction: Graph structures are constructed for the content generated by the model and the information in the benchmark dataset, where nodes include unit information and edges represent the relationship between information units;

[0145] Extract the structural features of the graph:

[0146] Node degree:

[0147]

[0148] Where: d(v) represents the degree of node v; A vi An indicator function indicating whether node v is connected to node i; N represents the total number of nodes in the graph;

[0149] Connectivity characteristics of graphs:

[0150]

[0151] Where: d(i,j) represents the shortest path length between node i and node j; N represents the number of nodes on the way;

[0152] The connectivity of a graph refers to the number of subgraphs that are not connected to each other along the way. The number of connected components is used to measure the looseness of its structure.

[0153] The method of structural feature extraction captures the hierarchical relationship, node dependency and overall connectivity in the generated content and benchmark datasets through graph structure analysis, and uses multi-dimensional features such as node features, connectivity, depth and width of the graph for extraction. By calculating the difference between the two and combining a weighted scoring mechanism, accurate structural information can be provided for subsequent integrity assessment. The innovation of this method is reflected in the use of graph structure analysis for deep structural comparison, which better reflects the organizational structure and information flow of the content.

[0154] As a possible implementation of this embodiment, the timeliness feature extraction includes the following steps:

[0155] Time-sensitive delay calculation: Calculate the time difference between the generated content and the benchmark dataset to quantify the delay:

[0156] Δt=|t model -t benchmark |;

[0157] Where: Δt represents the time difference between the generated content and the benchmark dataset; t model and t benchmark Respectively represent the time difference between the generated content and the benchmark dataset;

[0158] Calculate the timeliness difference:

[0159]

[0160] Where: S timeli,ess (i) represents the timeliness score of the i-th data unit; Δt i represents the time difference between the generated content and the benchmark dataset; τ represents the decay factor;

[0161] Data update frequency division:

[0162]

[0163] Where: S frequency (i) represents the update frequency score of the i-th data unit; freq actual Indicates the actual update frequency of the data; freq expected Indicate the expected update frequency of the benchmark data;

[0164] Calculate the Timeliness Completeness Score:

[0165]

[0166] Where: S timeliness represents the comprehensive timeliness score of the generated content; α1 and α2 represent weight coefficients; N represents the total number of data units.

[0167] The technical solution for extracting timeliness features combines timestamp differences and data update frequency to propose an innovative timeliness evaluation method based on time windows. This method can fully reflect the timeliness differences between the generated content and the benchmark data set through an exponential decay model and frequency matching, while taking into account the update rules of power grid data, and has strong practicality and scalability.

[0168] As a possible implementation of this embodiment, the semantic feature difference between the model-generated content and the benchmark dataset is calculated as follows:

[0169]

[0170] Where: ||·|| represents the L2 norm of the vector; The final semantic feature vector representing the content generated by the model; represents the final semantic feature vector of the benchmark dataset; represents the dot product of the vectors.

[0171] Preferably, the technical feature difference between the model-generated content and the benchmark data set is calculated as follows:

[0172]

[0173] Where: Represents the i-th technical feature in the model-generated content; represents the i-th technical feature in the benchmark dataset; N represents the total number of features.

[0174] As a possible implementation of this embodiment, the structural feature differences between the model-generated content and the benchmark data set are compared as follows:

[0175] Node degree difference:

[0176]

[0177] Where: represents the degree of the i-th node in the model-generated content; represents the degree of the i-th node in the benchmark dataset; N represents the total number of nodes;

[0178] Differences in graph connectivity:

[0179]

[0180] Where: d(i,j) model Indicates the shortest path length between node i and node j in the generated content graph; d(i,j) benchmark represents the shortest path length between node i and node j in the benchmark dataset graph; N represents the number of nodes in the graph;

[0181] Connected component differences:

[0182] Connected COmponents Difference=|Connected(G model )-

[0183] Connected(G benchmark )|;

[0184] Where: Connected(G model ) and Connected(G benchmark ) represent the number of connected components in the generated content and benchmark dataset graphs respectively;

[0185] Calculate the structural feature difference score:

[0186] S structural =w1·Degree Difference+w2·Connectivity Difference+w3·

[0187] Connected Components Difference;

[0188] Wherein: w1, w2 and w3 represent weight coefficients respectively, w1+w2+w3=1.

[0189] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A comparison method for evaluating the integrity of large model generation, characterized in that: The following steps are involved: Step 1: Collect data related to the large model generation content in the power grid, and standardize the collected data after data annotation based on expert experience to obtain a benchmark data set; Step 2: Based on the model-generated content and the benchmark dataset, semantic feature extraction, technical feature extraction, structural feature extraction, and timeliness feature extraction are performed on the model-generated content and the benchmark dataset respectively; Based on the extracted features, the generated content of the model is compared with the benchmark dataset in detail to find out the differences. The completeness score of each comparative analysis is calculated based on the results of the comparative analysis. Step 3: Calculate the comprehensive integrity score by taking the weighted sum of the integrity scores of various comparative analyses.

2. A comparison method for large model generation integrity assessment according to claim 1, characterized in that: The semantic feature extraction is implemented using the BERT model, and the specific model structure is as follows: Input layer: used to input preprocessed text data; Lexical feature extraction module: embeds each word through the pre-trained BERT model to obtain the semantic vector representation of each word; Sentence-level feature extraction module: Aggregate the semantic vectors at the vocabulary level and extract the semantic representation of each sentence. That is, use the [CLS]token output in the BERT model as the semantic vector of the sentence to obtain the semantic vector of each sentence. Paragraph-level feature extraction module: By taking a weighted average of the semantic vectors of multiple sentences, the semantics of the sentences are aggregated into the semantic representation of the paragraphs to obtain the semantic vector of each paragraph; Domain knowledge enhancement module: Introducing power grid domain knowledge to enhance the semantic vector of each layer of feature extraction module of the model: h enhanced,i =h original,i +h domain ; Where: h enhanced represents the semantic vector enhanced by the i-th feature extraction module; h iriginal,i represents the semantic vector output by the i-th feature extraction module; h domain Represents semantic vectors enhanced with domain knowledge; Feature aggregation module: aggregates the semantic vectors enhanced by each feature extraction module to obtain the aggregated semantic vector: Where: h final represents the final semantic feature vector; h k represents the semantic feature vector of the kth layer; w k Represents the weight of the k-th layer feature.

3. The comparison method for large model generation integrity assessment according to claim 1, characterized in that: The technical feature extraction comprises the following steps: Rule-based feature extraction: Use the re module in Python to perform regular matching, extract technical parameters, and standardize the extracted technical parameters; BiLSTM-CRF model feature extraction: The BiLSTM-CRF model is used to extract complex features. The structure of the BiLSTM-CRF model is as follows: Input layer: Input a serialized technical document, use Word2Vec pre-trained word vectors to convert each word into a vector; BiLSTM layer: performs forward and backward calculations through two LSTM networks to obtain the context information of the sentence; CRF layer: further refines the prediction by taking into account the dependencies between predefined labels: Where: Indicates that from label y t-1 Move to label y t probability; Indicates the current label y t Corresponding input x t The score of ; y′ represents the given input sequence x, representing all possible label combinations; Input layer: predicts the label of each word, including technical terms or non-technical terms, and identifies technical terms and device parameters based on this; Feature fusion: The features extracted based on the rules are concatenated with the features of the BiLSTM-CRF model.

4. The comparison method for evaluating the integrity of a large model generation according to claim 1, characterized in that: The structural feature extraction comprises the following steps: Graph structure construction: Graph structures are constructed for the content generated by the model and the information in the benchmark dataset, where nodes include unit information and edges represent the relationship between information units; Extract the structural features of the graph: Node degree: Where: d(v) represents the degree of node v; A vi An indicator function indicating whether node v is connected to node i; N represents the total number of nodes in the graph; Connectivity characteristics of graphs: Where: d(i,j) represents the shortest path length between node i and node j; N represents the number of nodes on the way; The connectivity of a graph refers to the number of subgraphs that are not connected to each other along the way. The number of connected components is used to measure the looseness of its structure.

5. The comparison method for evaluating the integrity of a large model generation according to claim 1, characterized in that: The timeliness feature extraction comprises the following steps: Time-sensitive delay calculation: Calculate the time difference between the generated content and the benchmark dataset to quantify the delay: Δt=|t model -t benchmark |; Where: Δt represents the time difference between the generated content and the benchmark dataset; t model and t benchmark Respectively represent the time difference between the generated content and the benchmark dataset; Calculate the timeliness difference: Where: S timeliness (i) represents the timeliness score of the i-th data unit; Δt i represents the time difference between the generated content and the benchmark dataset; τ represents the decay factor; Data update frequency division: Where: S frequency (i) represents the update frequency score of the i-th data unit; freq actual Indicates the actual update frequency of the data; freq expected Indicate the expected update frequency of the benchmark data; Calculate the Timeliness Completeness Score: Where: S timeliness represents the comprehensive timeliness score of the generated content; α1 and α2 represent weight coefficients; N represents the total number of data units.

6. A comparison method for evaluating the integrity of large model generation according to claim 1, characterized in that: The difference in semantic features between the model-generated content and the benchmark dataset is calculated as follows: Where: ||·|| represents the L2 norm of the vector; The final semantic feature vector representing the content generated by the model; represents the final semantic feature vector of the benchmark dataset; represents the dot product of the vectors.

7. A comparison method for large model generation integrity assessment according to claim 1, characterized in that: The difference in technical characteristics between the model-generated content and the benchmark dataset is calculated as follows: Where: Represents the i-th technical feature in the model-generated content; represents the i-th technical feature in the benchmark dataset; N represents the total number of features.

8. A comparison method for evaluating the integrity of large model generation according to claim 1, characterized in that: The structural feature differences between the model-generated content and the benchmark dataset are compared as follows: Node degree difference: Where: represents the degree of the i-th node in the model-generated content; represents the degree of the i-th node in the benchmark dataset; N represents the total number of nodes; Differences in graph connectivity: Where: d(i, j) model Represents the shortest path length between node i and node j in the generated content graph; d(i, j) benchmark represents the shortest path length between node i and node j in the benchmark dataset graph; N represents the number of nodes in the graph; Connected component differences: Connected Components Difference=|Connected(G model )-Connected(G benchmark )|; Where: Connected(G model ) and Connected(G benchmark ) represent the number of connected components in the generated content and benchmark dataset graphs respectively; Calculate the structural feature difference score: S structural =w1·Degree Difference+w2·Connectivity Difference+w3·ConnectedComponents Difference; Wherein: w1, w2 and w3 represent weight coefficients respectively, W1+W2+W3=1.

Images