A data assessment method, device, system and storage medium based on large model

Abstract

The present application discloses a data assessment method, device, system, and storage medium based on a large model. The method includes: collecting multi-source heterogeneous data; parsing the multi-source heterogeneous data to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple; calculating a priority weight distribution matrix W_p; in a multimodal scoring engine, using the RoBERTa-wwm model to extract a deep semantic feature vector V_t; extracting a spatiotemporal feature vector V_v through a 3D-ResNet; when the cosine similarity Δ between V_t and V_v is lower than a threshold γ, generating a confidence compensation parameter β through a generator; generating a path weight vector based on the confidence compensation parameter β; calculating the early fusion path output, the mid-term fusion path output, or the late fusion path output; calculating the final score vector S_f; and outputting a final assessment score value based on the final score vector.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data assessment method, device, system and storage medium based on a large model. Background Art

[0002] Performance appraisal is a crucial tool for measuring the effectiveness of individuals, departments, or systems within the management processes of various organizations and institutions. Traditional performance appraisal methods rely primarily on manually defined indicators and scoring criteria, typically based on quantitative data (such as KPIs) and qualitative assessments (such as subjective ratings). However, these methods have limitations in practical application.

[0003] First, the setting of performance indicators often relies on managers' experience and is difficult to adapt to the dynamic changes in the external environment. For example, adjustments to policies and regulations may affect performance standards, and traditional methods lack the ability to respond promptly to external changes, resulting in performance indicators lagging behind actual needs.

[0004] Secondly, assessment materials are becoming increasingly diverse, now encompassing unstructured data such as videos and images, in addition to traditional text. However, existing assessment methods primarily focus on structured data and have limited processing capabilities for unstructured data. This leads to insufficient information utilization during the assessment process, impacting the comprehensiveness and accuracy of the evaluation.

[0005] Furthermore, the assessment process typically relies on fixed scoring rules and lacks a dynamic adjustment mechanism for varying data quality. When assessment data contains uncertainty or low confidence, traditional scoring methods struggle to effectively identify and adjust accordingly, potentially impacting the objectivity and reliability of assessment results.

[0006] With the development of information technology, people have begun exploring data-driven approaches to optimize performance appraisal processes, aiming to improve the adaptability and accuracy of appraisal systems. In this context, how to more efficiently collect, analyze, and utilize appraisal data, and how to rationally evaluate it based on different types of appraisal materials, has become a key focus of current research and application. Summary of the Invention

[0007] In order to solve the above technical problems, the present application provides a data assessment method, device, system and storage medium based on a large model.

[0008] The technical solution provided in this application is described below:

[0009] In a first aspect, the present application provides a method for collecting multi-source heterogeneous data from policy websites, industry databases, and institutional systems through a target agent deployed on a data bus;

[0010] Perform regular detection on the multi-source heterogeneous data. When it is detected that the similarity change ΔS of the policy texts in the multi-source heterogeneous data exceeds a preset threshold α, parse the multi-source heterogeneous data to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple;

[0011] Use graph neural networks to decompose department KPIs into a tree-like demand graph and calculate the priority weight distribution matrix W_p;

[0012] Updating the indicator knowledge graph based on the distribution matrix W_p, and inputting the updated indicator knowledge graph into a preconfigured multimodal scoring engine;

[0013] In the multimodal scoring engine, the RoBERTa-wwm model is used to extract deep semantic feature vectors V_t from text-based assessment materials;

[0014] Extracting the spatiotemporal feature vector V_v from the video stream data corresponding to the text-based assessment material through 3D-ResNet;

[0015] When the cosine similarity Δ between the V_t and the V_v is lower than a threshold γ, a confidence compensation parameter β is generated by a generator based on the cosine similarity Δ;

[0016] Generating a path weight vector according to the confidence compensation parameter β by a dynamic routing controller;

[0017] Based on the path weight vector, calculating an early fusion path output, a mid-fusion path output, or a late fusion path output through a weighted aggregation layer in the multimodal scoring engine;

[0018] Calculate the final score vector S_f based on the early fusion path output, the mid-term fusion path output or the late fusion path output;

[0019] A final assessment score value is output based on the final score vector.

[0020] Optionally, the use of a graph neural network to decompose the department KPI into a tree-like demand graph and calculate the priority weight distribution matrix W_p includes:

[0021] Obtain the department's KPI hierarchical relationships, historical assessment data, and business context information;

[0022] Mapping the KPI hierarchical relationship into a directed graph, wherein nodes in the directed graph represent individual KPI targets, edges represent causal relationships between KPI targets, and weights represent the degree of influence between KPI targets;

[0023] Combining the assessment history data and the business context information to enhance the weights in the directed graph and generate a tree-like knowledge graph;

[0024] Calculate the importance score of each KPI target in the directed graph through GNN;

[0025] A weight distribution matrix W_p is constructed based on the importance scores.

[0026] Optionally, the multi-source heterogeneous data is parsed to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple, including:

[0027] Identify policy subjects in the policy text of the multi-source heterogeneous data using BERT-NER;

[0028] Extract keywords from the policy text using the TF-IDF algorithm;

[0029] Extracting indicator items from the policy text through dependency analysis;

[0030] Extracting quantitative standards from the policy text through regular matching and rule parsing methods;

[0031] Constructing the policy subject-indicator item-quantitative standard triple;

[0032] The triples are stored in a knowledge graph to obtain an indicator knowledge graph, in which the nodes include policy subject nodes, indicator item nodes, and quantitative standard nodes, and the edges include edges connecting the nodes.

[0033] Optionally, when the cosine similarity Δ between the V_t and the V_v is lower than a threshold γ, generating a confidence compensation parameter β by a generator based on the cosine similarity Δ includes:

[0034] Calculate the cosine similarity Δ between the text feature vector V_t and the video feature vector V_v;

[0035] If Δ is lower than the threshold γ, the cosine similarity is input into a preconfigured variational autoencoder VAE, and a confidence compensation parameter β is output.

[0036] Optionally, the calculating the final score vector S_f based on the early fusion path output, the mid-term fusion path output, or the late fusion path output includes:

[0037] When the confidence compensation parameter β is approximately equal to a first preset value, an early fusion path is adopted, wherein the early fusion path includes concatenating or weighted summing the text feature vector and the video feature vector to obtain a final score vector;

[0038] When the confidence compensation parameter β is approximately equal to a second preset value, a mid-term fusion path is adopted, wherein the mid-term fusion path includes respectively calculating the attention scores of the text feature vector and the video feature vector, and fusing them in a hidden layer to obtain a final score vector;

[0039] When the confidence compensation parameter β is approximately equal to a third preset value, a late fusion path is adopted, which includes respectively calculating the score vectors of the text feature vector and the video feature vector, and performing weighted averaging on the score vectors to obtain a final score vector.

[0040] Optionally, outputting a final assessment score value based on the final score vector includes:

[0041] Convert the final rating vector into a low-dimensional rating feature through a fully connected layer;

[0042] Processing the low-dimensional scoring features through the scoring prediction layer to obtain a normalized preliminary score;

[0043] Analyze the historical assessment data and obtain discrete level characteristics or demand range characteristics of the demand;

[0044] Constructing a mapping function according to the discrete level characteristics of the demand or the range characteristics of the demand;

[0045] The preliminary score is input into the mapping function to obtain a discrete grade or assessment score value.

[0046] A second aspect of the present application provides a data assessment device based on a large model, the device comprising:

[0047] The data collection unit is used to collect multi-source heterogeneous data from policy websites, industry databases, and institutional systems through the target agent deployed on the data bus;

[0048] An indicator knowledge graph construction unit is configured to periodically detect the multi-source heterogeneous data, and when it is detected that the similarity change ΔS of the policy texts in the multi-source heterogeneous data exceeds a preset threshold α, parse the multi-source heterogeneous data to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple;

[0049] The matrix construction unit is used to decompose the department KPI into a tree-like demand graph using a graph neural network and calculate the priority weight distribution matrix W_p;

[0050] An input unit, configured to update the indicator knowledge graph based on the distribution matrix W_p, and input the updated indicator knowledge graph into a preconfigured multimodal scoring engine;

[0051] A semantic feature extraction unit is used to extract a deep semantic feature vector V_t from text-based assessment materials using the RoBERTa-wwm model in the multimodal scoring engine;

[0052] A spatiotemporal feature extraction unit is used to extract a spatiotemporal feature vector V_v from the video stream data corresponding to the text-based assessment material through 3D-ResNet;

[0053] a parameter compensation unit, configured to generate a confidence compensation parameter β through a generator based on the cosine similarity Δ between the V_t and the V_v when the cosine similarity Δ is lower than a threshold γ;

[0054] a path weight vector generating unit, configured to generate a path weight vector according to the confidence compensation parameter β through a dynamic routing controller;

[0055] a path fusion unit, configured to calculate an early fusion path output, a mid-fusion path output, or a late fusion path output through a weighted aggregation layer in the multimodal scoring engine based on the path weight vector;

[0056] a scoring vector generating unit, configured to calculate a final scoring vector S_f based on the early fusion path output, the mid-term fusion path output, or the late fusion path output;

[0057] A scoring value generating unit is used to output a final assessment scoring value based on the final scoring vector.

[0058] The matrix construction unit is specifically used for:

[0059] Obtain the department's KPI hierarchical relationships, historical assessment data, and business context information;

[0060] Mapping the KPI hierarchical relationship into a directed graph, wherein nodes in the directed graph represent individual KPI targets, edges represent causal relationships between KPI targets, and weights represent the degree of influence between KPI targets;

[0061] Combining the assessment history data and the business context information to enhance the weights in the directed graph and generate a tree-like knowledge graph;

[0062] Calculate the importance score of each KPI target in the directed graph through GNN;

[0063] A weight distribution matrix W_p is constructed based on the importance scores.

[0064] The third aspect of this application provides a data assessment system based on a large model, including:

[0065] processor, memory, input and output units, and buses;

[0066] The processor is connected to the memory, the input and output unit, and the bus;

[0067] The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0069] It can be seen from the above technical solutions that this application has the following advantages:

[0070] 1. By adopting a method based on policy text similarity detection, it is possible to timely perceive changes in policies, regulations, and industry standards, and to parse the mapping relationship between "policy subject-indicator item-quantitative standard" through knowledge graph, so as to dynamically adjust the assessment indicators to ensure that the assessment system always meets the latest requirements and avoid the lag of traditional manual adjustment.

[0071] 2. Through the multimodal scoring engine, it is possible to simultaneously process various types of assessment materials such as text and video, realize cross-modal information fusion, avoid the one-sidedness of assessment caused by relying solely on a single data source, and thus improve the comprehensiveness and objectivity of the assessment results.

[0072] 3. The RoBERTa-wwm model is used to extract deep semantic features of text, and combined with 3D-ResNet to extract spatiotemporal features of video. When there is a large difference between the two, a confidence compensation mechanism is introduced to dynamically adjust the scoring strategy to ensure that reasonable assessment results can still be given when the data confidence is low, thereby improving the accuracy and stability of the scoring.

[0073] 4. Through the dynamic routing controller, the fusion path is automatically adjusted according to the confidence compensation parameter β, and different scoring results of early fusion, mid-term fusion and late fusion are calculated in the weighted aggregation layer, so as to adapt to different assessment scenarios and make the scoring results more accurate and in line with the actual situation.

[0074] 5. This method is based on a large model and graph neural network, and can complete the entire process of data collection, indicator update, assessment and scoring without human intervention, reducing the subjectivity and workload of manual scoring, improving assessment efficiency, and enhancing the uniformity and traceability of assessment standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0076] Figure 1 This is a flow chart of an embodiment of the data assessment method based on a large model provided in this application;

[0077] Figure 2 This is a flowchart of an implementation method of step S102 in the data assessment method based on a large model provided in this application;

[0078] Figure 3 This is a schematic structural diagram of an embodiment of a data assessment device based on a large model provided in this application;

[0079] Figure 4 This is a schematic diagram of the structure of an embodiment of the large model-based data assessment system provided in this application. DETAILED DESCRIPTION

[0080] See also Figure 1 This application first provides an embodiment of a data assessment method based on a large model, which includes:

[0081] S101, collect multi-source heterogeneous data from policy websites, industry databases and institutional systems through the target agent deployed on the data bus;

[0082] The target agent is deployed in the data bus and is responsible for acquiring multi-source heterogeneous data from different data sources. These data sources include but are not limited to:

[0083] Policy website: contains government organization policies, industry standards, regulatory updates, etc.

[0084] Industry database: contains market trends, industry development reports, technical standards and other information;

[0085] Institutional system: involves performance data, operational data, employee assessment materials, etc. within the enterprise or organization.

[0086] The target agent regularly obtains this data through API interface, crawler technology or data subscription mechanism, performs preliminary structural processing on it, and stores it in the data buffer for subsequent analysis.

[0087] In this embodiment, the target Agent serves as a data collection and preprocessing module, which can be implemented using a large model.

[0088] In this solution, the target agent is not only a traditional data acquisition module, but also combines large models (such as LLM, Transformer, etc.) for tasks such as intelligent parsing, denoising, and structured data conversion, thereby improving data quality and accuracy.

[0089] Traditional crawlers face problems such as complex HTML structure and text noise. The target agent combines LLM (such as GPT-4 and Claude) to understand web page content and automatically extract key information.

[0090] The specific steps are:

[0091] Visit the industry association policy website and crawl the HTML page.

[0092] Use LLM to parse the DOM structure of the web page and extract key information such as policy text, release date, and keywords.

[0093] Combine BERT and GPT for text summarization and denoising, removing irrelevant information such as advertisements and navigation.

[0094] TF-IDF+BERT is used to calculate text similarity and remove duplicate policy documents.

[0095] For example, if some policy texts lack implementation details, the large model can be used to supplement the details and enhance data integrity. T5 and GPT are used to classify policies, automatically categorizing them into categories such as "fiscal policy, industrial policy, and environmental regulations."

[0096] The target agent can access open databases based on APIs and integrate data with LLM. For example, the target agent can connect to data sources such as the enterprise credit information disclosure system and the WIND economic database to obtain macroeconomic and market research data. Use large models to analyze industry trends and extract core data points, such as:

[0097] "The new energy vehicle market is expected to grow by 15% in 2025"

[0098] "The semiconductor industry is affected by policies, and the growth rate in 2024H2 will slow down"

[0099] Another example:

[0100] GPT-4+LangChain is used to parse industry reports and extract key indicators (market size, growth rate, policy impact, etc.) from unstructured PDFs.

[0101] Here is a code implementation example:

[0102] fromlangchain.document_loadersimportPyPDFLoader

[0103] fromlangchain.chains.question_answeringimportload_qa_chain

[0104] fromlangchain.chat_modelsimportChatOpenAI

[0105] loader=PyPDFLoader("industry_report.pdf")

[0106] pages=loader.load()

[0107] #Let GPT-4 analyze industry reports

[0108] chain=load_qa_chain(ChatOpenAI(model="gpt-4"), chain_type="stuff")

[0109] result = chain.run({"question":"Please extract the core trends of this industry", "documents":pages})

[0110] print(result).

[0111] Target Agent can also connect directly to the enterprise ERP / CRM system through API to extract department assessment data, such as sales target achievement rate, customer satisfaction, and operating cost optimization rate.

[0112] This embodiment improves the intelligence level of data collection, ensures the accuracy, timeliness and structure of data, and lays a solid foundation for subsequent KPI evaluation and assessment.

[0113] S102: Periodically test the multi-source heterogeneous data. When it is detected that the similarity change ΔS of the policy texts in the multi-source heterogeneous data exceeds a preset threshold α, parse the multi-source heterogeneous data to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple;

[0114] The system regularly checks the collected multi-source heterogeneous data and uses text vectorization methods such as BERT and TF-IDF to calculate the similarity between newly collected policy texts and texts in the existing policy database;

[0115] When the change in similarity (ΔS) between a new policy and an existing policy exceeds a preset threshold (α), the system determines that the policy may contain new assessment criteria or indicator changes. The system then parses the revised policy text, extracts key entities and their relationships, and generates a "policy subject - indicator item - quantitative standard" triple. This triple is then added to the indicator knowledge graph, forming a structured assessment indicator system.

[0116] In a specific implementation, step S102 can use natural language processing (NLP) technology to extract effective policy indicator information from multi-source heterogeneous data. Through methods such as named entity recognition (NER), keyword extraction, dependency syntax analysis, regular expression matching, and rule parsing, the policy text can be parsed to identify the policy subject, indicator items, and quantitative standards. This ultimately forms a structured "policy subject-indicator item-quantitative standard" triple and constructs an indicator knowledge graph. Figure 2 , one implementation of this step includes:

[0117] S1021. Identify the policy subject in the policy text of the multi-source heterogeneous data using BERT-NER;

[0118] Policy entities typically refer to institutions, industry organizations, enterprises, or other decision-making entities. In this step, a BERT-based named entity recognition (NER) model is used to identify policy entities in policy texts.

[0119] The BERT-BiLSTM-CRF architecture was selected. This architecture combines the contextual understanding capabilities of BERT with the sequence modeling capabilities of BiLSTM, and uses the CRF layer to improve the accuracy of named entity recognition.

[0120] S1022. Extract keywords from the policy text using a TF-IDF algorithm;

[0121] In order to extract the core content of the policy document, the TF-IDF (term frequency-inverse document frequency) method is used to extract key concepts from the policy text to provide a basis for subsequent indicator item identification. A policy corpus is set, the frequency of each word in the policy text is counted, and the inverse document frequency (IDF) is calculated.

[0122] S1023. Extracting indicator items from the policy text through dependency analysis;

[0123] In policy texts, indicators often describe specific assessment requirements for a specific area, such as "intelligent manufacturing enterprise penetration rate." Dependency parsing can be used to identify assessment indicators within policy texts. Specifically, Stanford CoreNLP, spaCy, or LTP can be used for dependency analysis to identify relationships between subject, predicate, and object, as well as between attributive and verbal relationships, thereby extracting core indicators.

[0124] S1024. Extracting quantitative standards from the policy text by using regular matching and rule parsing methods;

[0125] Quantitative standards refer to specific numerical values ​​or quantitative indicators in policy requirements, such as "reaching 40%", "no less than 10 billion yuan", "reduced by 30%", etc.

[0126] You can set regular rules:

[0127] (\d+(\.\d+)?\s*(%|100 million yuan|10,000 square meters|kilometers|units));

[0128] An example sentence is as follows:

[0129] "By 2025, the penetration rate of smart manufacturing enterprises will reach 40%."

[0130] Then, the regular matching results are as follows:

[0131] Quantitative standard: 40%.

[0132] If the sentence is complex, you can combine it with large model rules for parsing, such as using GPT-4 / T5 parsing:

[0133] Input: "What is the target penetration rate for smart manufacturing enterprises?"

[0134] Output: "40%".

[0135] S1025. Constructing a triplet of the policy subject, indicator item, and quantitative standard;

[0136] Based on the above analysis, we can construct the “policy subject-indicator item-quantitative standard” triplet.

[0137] For example:

[0138] Enter policy text:

[0139] "An organization has proposed that by 2025, the penetration rate of smart manufacturing enterprises will reach 40%."

[0140] Extraction results:

[0141] (A certain organization, smart manufacturing enterprise penetration rate, 40%).

[0142] S1026. Storing the triples in a knowledge graph to obtain an indicator knowledge graph, wherein the nodes in the indicator knowledge graph include policy subject nodes, indicator item nodes, and quantitative standard nodes, and the edges include edges connecting the nodes.

[0143] Finally, the triples are stored in the knowledge graph to achieve structured storage of policy data and support subsequent queries and reasoning.

[0144] Node Type:

[0145] Policy subject (e.g., “an organization”);

[0146] Indicator items (such as “smart manufacturing enterprise penetration rate”);

[0147] Quantitative standards (e.g., “40%”);

[0148] Edge Type:

[0149] Implementation (policy entities to indicators);

[0150] Goals (from indicators to quantitative standards).

[0151] Finally, Neo4j can be used to build a knowledge graph. An example is as follows:

[0152] “CREATE(p:policy subject {name:"organization"})

[0153] CREATE(i:Indicator{Name:"Intelligent Manufacturing Enterprise Penetration Rate"})

[0154] CREATE(q:quantitative standard {value: "40%"})

[0155] CREATE(p)-[:implementation]->(i)

[0156] CREATE(i)-[:target]->(q)".

[0157] This implementation method can automatically and efficiently parse policy documents, forming a structured policy indicator knowledge graph to provide support for subsequent KPI evaluation and intelligent decision-making.

[0158] S103. Use a graph neural network to decompose the department KPI into a tree-like demand graph and calculate the priority weight distribution matrix W_p;

[0159] Graph neural network (GNN) is used to analyze department KPIs and decompose them into a tree-like demand graph. First, KPI data of different departments are obtained and converted into a standardized representation, such as a hierarchical indicator structure.

[0160] A graph structure is constructed using KPIs and their subitems, with nodes representing indicators and edges representing relationships. A GAT (graph attention network) or GCN (graph convolutional network) is used to calculate the influence weights between KPIs. Combined with historical assessment data, a graph neural network is used to calculate the weight distribution matrix W_p for each KPI node. This matrix is ​​used to measure the assessment priority of different indicators and provide a basis for subsequent updates to the indicator system.

[0161] Specifically, a specific implementation of this step includes:

[0162] Obtain the department's KPI hierarchical relationship, assessment history data, and business context information; map the KPI hierarchical relationship into a directed graph, where nodes in the directed graph represent each KPI target, edges represent the causal relationship between KPI targets, and weights represent the degree of influence between KPI targets; enhance the weights in the directed graph based on the assessment history data and the business context information to generate a tree-like knowledge graph; calculate the importance score of each KPI target in the directed graph through GNN; and construct a weight distribution matrix W_p based on the importance score.

[0163] In this step, we use a graph neural network (GNN) to perform a hierarchical decomposition of each department's key performance indicators (KPIs), constructing a tree-like demand graph. We then calculate the influence weights between KPIs based on historical assessment data, generating a priority weight distribution matrix W_p. This matrix can be used to measure the importance of different assessment indicators, providing a decision-making basis for subsequent updates to the indicator knowledge graph.

[0164] The KPIs of different departments may have differences in data format, hierarchical relationships, and assessment standards. Therefore, the KPI data needs to be standardized first.

[0165] KPI data mainly comes from:

[0166] Enterprise internal management systems (such as ERP, performance management systems);

[0167] Industry assessment standards (such as ISO quality management system, institutional performance assessment system);

[0168] Historical assessment data (past KPI scores, assessment results, and assessment rules).

[0169] During the standardization process, KPI data is converted into a hierarchical indicator structure to make it suitable for subsequent GNN processing. In order to use GNN to calculate the influence weights between KPIs, it is necessary to construct the KPI and its sub-items into a graph structure, where:

[0170] Node: represents KPI indicators at all levels;

[0171] Edge: Indicates the dependency or influence relationship between KPIs (such as causal relationship and weight transfer).

[0172] Assume the KPI system is G=(V, E):

[0173] V (node): represents KPI and its sub-items;

[0174] E (edge): represents the hierarchical relationship and impact relationship between KPIs.

[0175] The edges of a graph can be classified into:

[0176] Hierarchical dependency edges (e.g., "net profit" is a sub-item of "operating efficiency");

[0177] Influence relationship edges (e.g., “customer satisfaction” may affect “operating income”).

[0178] An example of specific representation is as follows:

[0179] G = {

[0180] Nodes: [operating income, net profit, return on assets, service response time, complaint rate, production efficiency, quality pass rate],

[0181] side: [

[0182] (operating benefit to operating income),

[0183] (Operating profit to net profit),

[0184] (operating efficiency to return on assets),

[0185] (customer satisfaction to service response time),

[0186] (Customer satisfaction to complaint rate) ]

[0188] }.

[0189] In this embodiment, past assessment results (such as KPI scores and corporate performance data) may be used to perform regression analysis and enhance the weight between two nodes.

[0190] In the assessment system, the Importance Score is used to measure the weight of each KPI within the overall assessment system. To quantify the relative importance of different KPIs, this step constructs a weight distribution matrix W_p based on the Importance Score to guide indicator integration and subsequent assessment score calculations.

[0191] Importance scores can be calculated in a variety of ways, for example they can be calculated based on the assignment of a hierarchy, for example:

[0192] In a hierarchical KPI system, weights should be allocated from top to bottom:

[0193] Assume that the parent KPI weight is w_p, and its child KPI weight w_c must satisfy:

[0194] ;

[0195] For example, assume that the weight of "operating effectiveness" is 1.0, and it has three sub-items: operating income, net profit, and return on assets.

[0196] If allocated based on experience:

[0197] Operating income: 0.5;

[0198] Net profit: 0.3;

[0199] Return on assets: 0.2;

[0200] The sum of the importance scores of these sub-KPIs is 1.0.

[0201] Once the importance scores S = [s1, s2, ..., sn] of all KPIs are obtained, the weight distribution matrix W_p can be constructed.

[0202] After ensuring that the sum of all scores is 1, construct the weight distribution matrix:

[0203] ;

[0204] The diagonal elements W_p[i, i] represent the self-weight of the KPI; the non-diagonal elements W_p[i, j] represent the relative influence of KPIi on KPIj.

[0205] S104: updating the indicator knowledge graph based on the distribution matrix W_p, and inputting the updated indicator knowledge graph into a preconfigured multimodal scoring engine;

[0206] Based on the calculated results of W_p, the weights of each assessment indicator in the indicator knowledge graph are adjusted. If the weight values ​​of certain KPIs change significantly, the system adjusts the associated assessment indicators. Based on the latest knowledge graph, it determines whether to add, modify, or delete some assessment indicators. This updated indicator system is formed and input into the multimodal scoring engine.

[0207] In the assessment system, the indicator knowledge graph G adopts the triple structure of "policy subject-indicator item-quantitative standard", namely:

[0208] G=(V, E, A);

[0209] in:

[0210] V: node set, including policy subject nodes, indicator item nodes, and quantitative standard nodes;

[0211] E: edge set, representing the relationship between different nodes, such as subordination, dependence, influence, etc.;

[0212] A: Node attributes, including the description of the assessment indicators, current weight, calculation method, etc.

[0213] In the assessment system, each indicator has a corresponding weight value, which is used to measure its importance in the overall assessment. Once the new weight distribution matrix W_p is calculated, the weights of each indicator in the indicator knowledge graph need to be updated to ensure that the assessment system can accurately reflect the latest business needs.

[0214] The system first compares the current weights of the assessment indicators with the newly calculated weights to check the extent of their change. If the weight change of an indicator exceeds a preset threshold (for example, 10%), the indicator is considered to have changed significantly in importance and needs to be adjusted.

[0215] Because assessment systems typically require a certain degree of stability, newly calculated weights are not used directly. Instead, they are adjusted using a smooth update method. For example, the new weights can be weighted averaged with the old weights to make weight changes more stable and avoid frequent changes in assessment standards due to short-term fluctuations.

[0216] If a metric is influenced not only by its own business data but also by correlations with other assessment items (e.g., one assessment item depends on the performance of another metric), the system will consider these factors and make adjustments. For example, if a department's performance review is influenced by multiple KPIs, the final weight of the metric will be adjusted based on the fluctuations of related indicators to better align with the overall assessment objectives.

[0217] After all indicator weights are updated, the system will check the overall weight distribution to ensure that the sum of all indicator weights remains within a reasonable range (e.g., 100%). If the sum of weights exceeds or falls below the expected value, the system will normalize the weights of all indicators to keep them within the reasonable assessment standard.

[0218] Finally, based on the dynamic changes in KPI importance, appropriate actions are taken. If the importance of certain assessment items increases significantly and is not included in the original assessment system, the indicator item will be automatically added and linked to the relevant KPI. If the weight of certain assessment items remains below the threshold for a long time and the assessment data shows that their impact is weak, the indicator item can be removed to simplify the assessment system. If the weight changes of certain indicators lead to adjustments in their priority, such as a significant increase in the weight of a sub-item, the hierarchy may need to be restructured and the indicator attribution adjusted.

[0219] The updated knowledge graph is passed as input to the Multimodal Scoring Engine (MSE). Specifically, the knowledge graph is first converted into a tensor or embedding representation to facilitate model computation. A GNN is then used to extract node features, such as indicator weights, historical assessment data, and assessment text content. Based on the latest weight distribution W_p, the MSE combines text, video, and indicator data to calculate the assessment score.

[0220] The following is further explained through an example:

[0221] Assume that the original assessment system has the following KPIs: production efficiency (weight 0.4), quality pass rate (weight 0.3), production safety accident rate (weight 0.2), energy consumption level (weight 0.1)

[0222] In the new round of assessment:

[0223] The importance of quality pass rate increased to 0.4;

[0224] The effect of energy consumption level drops to 0.05;

[0225] Added carbon emission compliance rate (0.05).

[0226] The updated indicator knowledge graph is adjusted as follows:

[0227] Update the weight value of each indicator;

[0228] Adjust the priority of quality pass rate in the assessment system;

[0229] Add a carbon emission compliance rate node to the knowledge graph and establish a relationship with other KPIs.

[0230] In this step, the accuracy of the assessment system was enhanced to make it more in line with actual business needs; the scoring mechanism was optimized to ensure that the scoring engine can accurately reflect the contribution of each assessment indicator.

[0231] S105. In the multi-modal scoring engine, use the RoBERTa-wwm model to extract the deep semantic feature vector V_t from the text-based assessment materials.

[0232] In this step, use the RoBERTa-wwm (Whole Word Masking) deep learning model to extract features from the text-based assessment materials. This model can capture context semantic relationships and extract the deep semantic feature vector V_t. Preprocess the text data (remove stop words, tokenize, vectorize); calculate the text semantic embedding through the RoBERTa-wwm model to generate the feature vector V_t.

[0233] During the assessment process, text-based assessment materials usually include policy documents, work reports, business data descriptions, etc. These texts often have complex semantic relationships. To accurately understand and analyze the text content, this step uses the RoBERTa-wwm (Whole Word Masking) pre-trained language model to extract deep semantic features from the text data.

[0234] First, remove irrelevant characters, HTML tags, special symbols, etc. to make the text content more standardized. Use Chinese tokenization tools (such as Jieba, PKUSEG) to tokenize the text and retain the complete word structure. Filter out common words with no practical meaning such as "de, le, shi" to reduce noise interference to the model. Finally, convert the processed text into a word vector representation that the model can recognize and prepare to input into the RoBERTa-wwm model.

[0235] In the RoBERTa-wwm model, input the preprocessed text into the RoBERTa-wwm model through the input embedding layer, and use its specific word embedding layer (Embedding Layer) to convert the text into a high-dimensional vector representation. RoBERTa-wwm adopts the Transformer structure and uses the multi-head self-attention mechanism (Multi-Head Attention) to analyze the semantic associations within the sentence and extract important information in the sentence. Compared with the sub-word level masking training of BERT, RoBERTa-wwm adopts a whole-word level masking strategy, enabling the model to better understand the semantics of complete words and improving the understanding ability of texts such as technical terms and policy documents. After multiple layers of Transformer encoding, the model finally outputs a high-dimensional feature vector V_t to represent the overall semantic information of the text. The finally obtained V_t is the deep semantic feature vector of the text and can be used for subsequent multi-modal fusion scoring to ensure that the text information can accurately affect the final assessment result.

[0236] S106, extracting a spatiotemporal feature vector V_v from the video stream data corresponding to the text-based assessment material through 3D-ResNet;

[0237] 3D-ResNet is a spatiotemporal feature extraction network that can simultaneously extract spatial information (frame images) and temporal information (frame sequence changes) from a video. First, the video stream is frame-sampled and converted into 3D input data. 3D-ResNet is then used to extract features and output a spatiotemporal feature vector V_v.

[0238] Assessment materials are not limited to text and may also include video recordings, such as work scene monitoring and task execution process videos. To extract key information from the video, this step uses 3D-ResNet (3D residual network) for spatiotemporal feature extraction.

[0239] Because videos typically contain a large number of frames, directly processing all frames would result in excessive computational overhead. Therefore, we use equal-interval or keyframe sampling to extract representative frame sequences from the video. We also perform normalization on the video frames, such as adjusting brightness and contrast, and normalizing the size, to improve the model's generalization capabilities. We also convert the frame sequence into a 3D tensor, which adapts to the input format of 3D-ResNet for processing video data.

[0240] In the 3D-ResNet model, 3D-ResNet uses 3D convolution, which can simultaneously process spatial information (intra-frame image features) and temporal information (frame sequence changes) to capture the dynamic characteristics of the video.

[0241] Through skip connections, some original information can be directly transmitted in the deep network, solving the gradient vanishing problem of traditional deep networks and improving the stability of the model.

[0242] The extracted spatiotemporal features are reduced in dimensionality through pooling layers, enabling the model to aggregate video information more effectively and reduce computational costs.

[0243] After 3D-ResNet processing, the output video feature vector V_v contains the dynamic spatiotemporal information of the video, such as behavior patterns, scene changes, video content, etc., providing support for subsequent multimodal fusion scoring.

[0244] Through the above-mentioned feature extraction methods of RoBERTa-wwm and 3D-ResNet, the system can extract key features from text data and video data respectively, ensuring that the assessment system can accurately evaluate different types of assessment materials and provide comprehensive support for the final scoring.

[0245] S107, when the cosine similarity Δ between the V_t and the V_v is lower than a threshold γ, generating a confidence compensation parameter β by a generator based on the cosine similarity Δ;

[0246] The cosine similarity Δ between V_t and V_v is calculated to measure the degree of match between the text and the video. When Δ falls below a threshold γ (indicating a significant discrepancy between the text and video information), the system uses a generative adversarial network (GAN) or Bayesian optimization method to generate a confidence compensation parameter β to correct for the scoring bias.

[0247] In an optional implementation, this step includes: calculating the cosine similarity Δ between the text feature vector V_t and the video feature vector V_v; if Δ is lower than a threshold γ, inputting the cosine similarity into a preconfigured variational autoencoder VAE and outputting a confidence compensation parameter β.

[0248] In this optional implementation, the cosine similarity Δ between the text feature vector V_t and the video feature vector V_v is calculated to assess their matching. If the similarity is low (i.e., Δ is below a set threshold γ), it may indicate a weak correlation between the text and the video or uncertainty. Therefore, a variational autoencoder (VAE) is used to compensate for this and generate a confidence compensation parameter β.

[0249] Variational Autoencoder (VAE) is a probabilistic generative model that learns the underlying distribution of data and can compensate for missing information in the presence of uncertainty.

[0250] In this step, VAE is used to learn the joint distribution of text features V_t and video features V_v, and generate compensation parameters β in the case of low similarity to improve the credibility of the fused features.

[0251] Specifically, in this embodiment, the VAE processing flow is as follows:

[0252] The text feature vector V_t and the video feature vector V_v are concatenated to form the input feature X=[V_t, V_v] and sent to VAE for encoding.

[0253] The potential representation Z of the input feature is extracted through a neural network (such as a multi-layer perceptron MLP), and its mean μ and standard deviation σ are learned to form the Gaussian distribution parameters:

[0254] Z~N(μ,σ);

[0255] Here Z is a low-dimensional latent variable representation of the input feature X, which can capture the potential pattern of the data.

[0256] Since Z obeys the normal distribution, in order to achieve differentiable gradient optimization, the reparameterization technique is introduced:

[0257] Z = μ + σ * ϵ;

[0258] Where ε is random noise that obeys the standard normal distribution N(0, 1), which ensures that the gradient can be propagated smoothly.

[0259] Through the decoder, the latent variable Z is remapped back to the original feature space to generate the compensation parameter β:

[0260] β=Decoder(Z);

[0261] β reflects the confidence compensation value between text and video features. If the text and video match poorly, VAE will predict a suitable compensation value, making the final fused features more valuable for reference.

[0262] In this embodiment, the range of β can be between [0, 1], indicating the degree of compensation:

[0263] If β is set to 0, it means that the compensation effect is low and the effect of the original cosine similarity is maintained.

[0264] If β reaches 1, it means that the compensation effect is strong, which improves the credibility of low-similarity data.

[0265] The subsequent dynamic routing controller will adjust the fusion path of the multimodal scoring engine based on β to ensure that the matching information of text and video can be reasonably integrated, ultimately improving the accuracy of the assessment score.

[0266] This optional implementation calculates the cosine similarity Δ between the text feature vector V_t and the video feature vector V_v. When Δ falls below a threshold γ, a variational autoencoder (VAE) is used to compensate for this difference, generating a confidence compensation parameter β. The VAE learns the joint distribution of text and video features to appropriately compensate for low similarity, making the final multimodal score more stable and accurate.

[0267] S108, generating a path weight vector according to the confidence compensation parameter β by a dynamic routing controller;

[0268] The confidence compensation parameter β is used as a dynamic adjustment factor and input into the dynamic routing controller. The attention mechanism is used to calculate the weight of each fusion path (early fusion, mid-term fusion, and late fusion) and generate a path weight vector.

[0269] The Dynamic Routing Controller is responsible for calculating the weights of different fused paths based on β and generating the final path weight vector. Its core components include:

[0270] Input layer: Receives confidence compensation parameter β and multimodal features (V_t, V_v) as input.

[0271] Attention Mechanism: The attention mechanism is used to calculate the weights of different fusion paths (early fusion, mid-term fusion, and late fusion).

[0272] During calculation, different path strategies are set, such as W_e (early fusion weight), W_m (mid-term fusion weight), and W_l (late fusion weight).

[0273] β is used as a dynamic weight coefficient to adjust the fusion weights of different paths. Since the weights of the three fusion paths need to satisfy the probability distribution property (that is, the sum of the weights of all paths is 1), the Softmax function is used for normalization.

[0274] After normalization, the path weight vector W = [W_e', W_m', W_l'] is finally generated to guide subsequent fusion calculations.

[0275] S109. Calculate an early fusion path output, a mid-fusion path output, or a late fusion path output through a weighted aggregation layer in the multimodal scoring engine based on the path weight vector;

[0276] Early fusion: Information fusion is performed during the feature extraction stage; it is suitable for high-confidence data.

[0277] Mid-term fusion: cross-modal fusion is performed during the feature transformation stage (attention mechanism); it is suitable for situations where the data partially matches but there are certain errors.

[0278] Late fusion: Fusion is performed in the final score calculation stage; it is suitable for low-confidence data and adjusts the score through the compensation parameter β.

[0279] In a specific implementation, this step includes:

[0280] When the confidence compensation parameter β is approximately equal to a first preset value, an early fusion path is adopted, wherein the early fusion path includes concatenating or weighted summing the text feature vector and the video feature vector to obtain a final score vector;

[0281] When the confidence compensation parameter β is approximately equal to a second preset value, a mid-term fusion path is adopted, wherein the mid-term fusion path includes respectively calculating the attention scores of the text feature vector and the video feature vector, and fusing them in a hidden layer to obtain a final score vector;

[0282] When the confidence compensation parameter β is approximately equal to a third preset value, a late fusion path is adopted, which includes respectively calculating the score vectors of the text feature vector and the video feature vector, and performing weighted averaging on the score vectors to obtain a final score vector.

[0283] In this embodiment, a fusion strategy is selected based on different confidence compensation parameters. When β is high (strong text-video match), the early fusion path dominates (larger W_e'), directly performing vector concatenation or cross-attention between the text feature vector V_t and the video feature vector V_v to generate fused features. This approach is suitable for scenarios where text and video match closely, such as a speech script and its corresponding video explanation.

[0284] When β is moderate (meaning the text and video match well), the mid-term fusion path dominates (W_m' is large), first extracting high-level features from the text and video separately (such as semantic features encoded by Transformer and deep features extracted by 3D-ResNet), and then fusing them in the feature space. This approach is suitable for scenarios where the content partially matches but there are some modal differences, such as news reports and corresponding video footage.

[0285] When β is low (poor text-video match), the late fusion path dominates (larger W_l'), calculating the text and video scores separately and then performing a weighted combination at the scoring level. This approach is suitable for situations where the text and video are weakly connected, such as when the text describes a product feature while the video displays a product advertisement.

[0286] The calculated path weight vector W = [W_e', W_m', W_l'] is used as the input of the subsequent weighted aggregation layer to guide the calculation of the scoring model.

[0287] In this step, a dynamic routing controller is used, the confidence compensation parameter β is used as an adjustment factor, and the attention mechanism is combined to calculate the weights of different fusion paths. Softmax normalization is used to generate the path weight vector W.

[0288] S110, calculating a final score vector S_f based on the early fusion path output, the mid-term fusion path output, or the late fusion path output;

[0289] Based on the path weight vector, the final score vector S_f is calculated for the early fusion path output, the mid-term fusion path output, or the late fusion path output.

[0290] This step uses the path weight vector W=[W_e', W_m', W_l'] to perform weighted calculation on the output results of different fusion paths to generate the final score vector S_f.

[0291] In S108, the dynamic routing controller calculates the weight distribution of different fusion paths (early, mid, and late) based on the confidence compensation parameter β. Based on this weight information, this step uses a weighted aggregation strategy to calculate the final score vector S_f, ensuring that the score results can integrate information from different modalities and maintain adaptability to data matching.

[0292] In a multimodal scoring system, the output data of different fusion paths are in different forms, mainly including:

[0293] Early fusion path output:

[0294] Fusion is performed directly at the feature level (for example, the text feature vector V_t and the video feature vector V_v are directly concatenated and fed into the scoring network).

[0295] Its output is a fused feature vector, which represents the comprehensive features of the data.

[0296] Mid-term fusion path output:

[0297] First, high-dimensional semantic representations of text and video are calculated separately, and then fused in the latent variable space (for example, fusion features are obtained after information interaction through Transformer and LSTM).

[0298] Its output is a scoring vector S_m after information fusion, which can capture deeper semantic relationships.

[0299] Late fusion pathway output:

[0300] After calculating the scores for text and video separately, they are fused at the score level (for example, calculating the text score and video score separately and then performing a weighted combination in the final stage). The output is a final decision score vector that can be used in cases where the text and video match poorly.

[0301] In another implementation, the system can calculate the final score vector S_f without using weighted fusion. Instead, it can directly select the output of a fusion path as the final score result. This selection process is based on the applicability of different fusion strategies in the current scenario and the impact of the confidence compensation parameter β on the information fusion strategy.

[0302] When the system detects significant differences in the reliability of different fusion paths, it can select the fusion strategy that best suits the current data matching situation. For example:

[0303] If the text and video features are highly matched (cosine similarity Δ is high), the late fusion path can be used directly for output, because the text and video can be scored separately and then directly fused to reduce information loss.

[0304] If the text and video features have a low match (low cosine similarity Δ), the early fusion path output can be selected because early fusion can align data at the feature level and alleviate the information mismatch problem.

[0305] If the amount of text and video data is large and the fusion requires deep feature interaction, you can choose the mid-term fusion path output because this path can use deep learning networks (such as Transformer and LSTM) for feature interaction and improve the fusion effect.

[0306] S111. Output a final assessment score value based on the final score vector.

[0307] In this step, the system needs to convert the final score vector into a specific assessment score value for subsequent processing and decision-making by the assessment system. This process can be achieved through linear transformation or deep neural network mapping, ensuring that the scoring results meet the established assessment standards and accurately reflect the actual performance of the assessed person.

[0308] In order to obtain a clear assessment score from the high-dimensional final score vector S_f, the system can adopt the following two methods:

[0309] When the assessment scoring system is relatively fixed and the weight of each feature's influence on the final score is relatively stable, a linear mapping method can be used to directly convert S_f into a numerical assessment score.

[0310] Use a weight vector W_{out} to perform weighted summation on the final score vector S_f:

[0311] Score=W_{out}*S_f+b;

[0312] W_{out} is a pre-trained weight matrix used to convert the high-dimensional rating vector into an assessment score. b is a bias term used to normalize or offset the score. The calculated score may need to be adjusted to a specified scoring range (e.g., 0-100).

[0313] When the scoring system is more complex and needs to consider the nonlinear relationship of multiple features, the neural network mapping method can be used to predict the assessment score value from the final score vector SfS_fSf.

[0314] A multi-layer perceptron (MLP) is used, with the final score vector SfS_fSf as input, to output the final assessment score.

[0315] Structure example:

[0316] Input layer: accepts the final score vector S_f (e.g. 256 dimensions).

[0317] Hidden layer:

[0318] First layer: 128 neurons, ReLU activation function.

[0319] Second layer: 64 neurons, ReLU activation function.

[0320] The third layer: 32 neurons, ReLU activation function.

[0321] Output layer: a single neuron with Sigmoid or Tanh activation function, outputting a score value between 0 and 1.

[0322] In one implementation, an embodiment of this step includes:

[0323] The final rating vector is converted into a low-dimensional rating feature through a fully connected layer; the low-dimensional rating feature is processed through a rating prediction layer to obtain a normalized preliminary rating; the assessment history data is parsed to obtain a discrete level feature of demand or a range feature of demand; a mapping function is constructed based on the discrete level feature of demand or the range feature of demand; the preliminary rating is input into the mapping function to obtain a discrete level or an assessment rating value.

[0324] In this embodiment, this step uses methods such as a fully connected layer, a score prediction layer, and a mapping function to gradually convert the final score vector into an assessment score value.

[0325] Specifically, the final rating vector S_f may be a high-dimensional representation (such as 256 dimensions or 512 dimensions), and directly using it to calculate the rating may contain redundant information. Therefore, it is necessary to first reduce the dimensionality and retain the most important information. This vector comes from the rating calculation of the fusion path (early fusion / mid-term fusion / late fusion). It may contain features of different modal data (such as rating information of text and video). One or more layers of fully connected neural networks (FC layers) are used to map the high-dimensional rating vector to a low-dimensional space:

[0326] Fscore=ReLU(W1S_f+b1);

[0327] Where W1 is the weight matrix and b_1 is the bias term. After passing through ReLU (or other nonlinear activation functions), the feature expression ability can be guaranteed.

[0328] This vector retains the core information of the final score vector and removes redundant information (such as certain low-correlation dimensions). The dimension can be set to 16, 32, or 64, depending on the complexity of the scoring task.

[0329] Based on the reduced-dimensional score feature Fscore, a normalized score is calculated to ensure that the score value is within a standard range (e.g., between 0 and 1). An additional fully connected layer is used to convert the low-dimensional score feature Fscore into a single score value: Scorenorm = σ(W2Fscore + b2), where W2 is the weight matrix of the score prediction layer and b2 is the bias term. The activation function uses Sigmoid (σ) to ensure that the output range is between 0 and 1. The resulting Scorenorm is a normalized preliminary score value, which will be used to map to a specific assessment level or score value.

[0330] To ensure that assessment scores meet business requirements, you need to reference historical data and determine the scoring criteria (discrete levels or scoring ranges). Read past assessment score data and analyze the score distribution for different indicators and assessment objectives. Compute the mean and standard deviation of historical scores to observe the trends in assessment grade divisions.

[0331] If the assessment score is graded (e.g., A, B, C, D), you need to analyze the boundary values ​​of different grades from historical data. If the assessment score is a continuous value range (e.g., 60-100 points), you need to analyze the distribution of the score range.

[0332] Based on historical data, build a suitable scoring mapping function to ensure that the calculated preliminary score can be correctly converted into the final assessment score value that meets business needs.

[0333] One implementation method is to build a classification mapping function (applicable to discrete scoring) based on the discrete grade characteristics of the requirements. If the scoring system uses fixed grades (such as A / B / C / D), then build interval division rules to ensure that the assessment scores can fall into the predefined grade range and meet business needs.

[0334] If the assessment score is a continuous interval (such as 60-100 points), a linear transformation mapping can be constructed to convert the normalized score between 0 and 1 into the target assessment score range. Finally, a mapping function is used to convert the normalized score into the actual assessment score or grade.

[0335] The above embodiments describe the method in the present application. The following describes embodiments of the device, system, and storage medium in the present application.

[0336] See Figure 3 The present application provides an embodiment of a data assessment device based on a large model, which includes:

[0337] The data collection unit 301 is used to collect multi-source heterogeneous data from policy websites, industry databases and institutional systems through the target agent deployed on the data bus;

[0338] The indicator knowledge graph construction unit 302 is configured to periodically detect the multi-source heterogeneous data. When it is detected that the similarity change ΔS of the policy texts in the multi-source heterogeneous data exceeds a preset threshold α, the multi-source heterogeneous data is analyzed to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple.

[0339] The matrix construction unit 303 is used to decompose the department KPI into a tree-like demand graph using a graph neural network and calculate the priority weight distribution matrix W_p;

[0340] An input unit 304 is configured to update the indicator knowledge graph based on the distribution matrix W_p and input the updated indicator knowledge graph into a preconfigured multimodal scoring engine;

[0341] A semantic feature extraction unit 305 is used to extract a deep semantic feature vector V_t from text-based assessment materials using a RoBERTa-wwm model in the multimodal scoring engine;

[0342] A spatiotemporal feature extraction unit 306 is configured to extract a spatiotemporal feature vector V_v from the video stream data corresponding to the text-based assessment material through a 3D-ResNet;

[0343] a parameter compensation unit 307 configured to generate a confidence compensation parameter β through a generator based on the cosine similarity Δ between the V_t and the V_v when the cosine similarity Δ is lower than a threshold γ;

[0344] A path weight vector generating unit 308 is configured to generate a path weight vector according to the confidence compensation parameter β through a dynamic routing controller;

[0345] a path fusion unit 309 configured to calculate an early fusion path output, a mid-fusion path output, or a late fusion path output through a weighted aggregation layer in the multimodal scoring engine based on the path weight vector;

[0346] A scoring vector generating unit 310 is configured to calculate a final scoring vector S_f based on the early fusion path output, the mid-term fusion path output, or the late fusion path output;

[0347] The scoring value generating unit 311 is configured to output a final assessment scoring value based on the final scoring vector.

[0348] Optionally, the matrix construction unit 303 is specifically configured to:

[0349] Obtain the department's KPI hierarchical relationships, historical assessment data, and business context information;

[0350] Mapping the KPI hierarchical relationship into a directed graph, wherein nodes in the directed graph represent individual KPI targets, edges represent causal relationships between KPI targets, and weights represent the degree of influence between KPI targets;

[0351] Combining the assessment history data and the business context information to enhance the weights in the directed graph and generate a tree-like knowledge graph;

[0352] Calculate the importance score of each KPI target in the directed graph through GNN;

[0353] A weight distribution matrix W_p is constructed based on the importance scores.

[0354] Optionally, the indicator knowledge graph construction unit 302 is specifically used to:

[0355] Identify policy subjects in the policy text of the multi-source heterogeneous data using BERT-NER;

[0356] Extract keywords from the policy text using the TF-IDF algorithm;

[0357] Extracting indicator items from the policy text through dependency analysis;

[0358] Extracting quantitative standards from the policy text through regular matching and rule parsing methods;

[0359] Constructing the policy subject-indicator item-quantitative standard triple;

[0360] The triples are stored in a knowledge graph to obtain an indicator knowledge graph, in which the nodes include policy subject nodes, indicator item nodes, and quantitative standard nodes, and the edges include edges connecting the nodes.

[0361] Optionally, the parameter compensation unit 307 is specifically configured to:

[0362] Calculate the cosine similarity Δ between the text feature vector V_t and the video feature vector V_v;

[0363] If Δ is lower than the threshold γ, the cosine similarity is input into a preconfigured variational autoencoder VAE, and a confidence compensation parameter β is output.

[0364] Optionally, the scoring value generating unit 311 is specifically configured to:

[0365] Convert the final rating vector into a low-dimensional rating feature through a fully connected layer;

[0366] Processing the low-dimensional scoring features through the scoring prediction layer to obtain a normalized preliminary score;

[0367] Analyze the historical assessment data and obtain discrete level characteristics or demand range characteristics of the demand;

[0368] Constructing a mapping function according to the discrete level characteristics of the demand or the range characteristics of the demand;

[0369] The preliminary score is input into the mapping function to obtain a discrete grade or assessment score value.

[0370] See also Figure 4 , the present application also provides a data assessment device based on a large model, comprising:

[0371] Processor 401, memory 402, input and output unit 403, bus 404;

[0372] The processor 401 is connected to the memory 402, the input and output unit 403 and the bus 404;

[0373] The memory 402 stores a program, and the processor 401 calls the program to execute any of the above methods.

[0374] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer is caused to execute any of the above methods.

[0375] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0376] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0377] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0378] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0379] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, and other media that can store program code.

Claims

1. A data assessment method based on a large model, characterized in that: The method comprises: The target agent deployed on the data bus collects multi-source heterogeneous data from policy websites, industry databases, and institutional systems; Perform regular detection on the multi-source heterogeneous data. When it is detected that the similarity change ΔS of the policy texts in the multi-source heterogeneous data exceeds a preset threshold α, parse the multi-source heterogeneous data to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple; Use graph neural networks to decompose department KPIs into a tree-like demand graph and calculate the priority weight distribution matrix W_p; Updating the indicator knowledge graph based on the distribution matrix W_p, and inputting the updated indicator knowledge graph into a preconfigured multimodal scoring engine; In the multimodal scoring engine, the RoBERTa-wwm model is used to extract deep semantic feature vectors V_t from text-based assessment materials; Extracting the spatiotemporal feature vector V_v from the video stream data corresponding to the text-based assessment material through 3D-ResNet; When the cosine similarity Δ between the V_t and the V_v is lower than a threshold γ, a confidence compensation parameter β is generated by a generator based on the cosine similarity Δ; Generating a path weight vector according to the confidence compensation parameter β by a dynamic routing controller; Based on the path weight vector, calculating an early fusion path output, a mid-fusion path output, or a late fusion path output through a weighted aggregation layer in the multimodal scoring engine; Calculate the final score vector S_f based on the early fusion path output, the mid-term fusion path output or the late fusion path output; A final assessment score value is output based on the final score vector.

2. The data assessment method based on a large model according to claim 1 is characterized in that: The graph neural network is used to decompose the department KPI into a tree-like demand graph and calculate the priority weight distribution matrix W_p, which includes: Obtain the department's KPI hierarchical relationships, historical assessment data, and business context information; Mapping the KPI hierarchical relationship into a directed graph, wherein nodes in the directed graph represent individual KPI targets, edges represent causal relationships between KPI targets, and weights represent the degree of influence between KPI targets; Combining the assessment history data and the business context information to enhance the weights in the directed graph and generate a tree-like knowledge graph; Calculate the importance score of each KPI target in the directed graph through GNN; A weight distribution matrix W_p is constructed based on the importance scores.

3. The data assessment method based on a large model according to claim 1 is characterized in that: Parse the multi-source heterogeneous data to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple, including: Identify policy subjects in the policy text of the multi-source heterogeneous data using BERT-NER; Extract keywords from the policy text using the TF-IDF algorithm; Extracting indicator items from the policy text through dependency analysis; Extracting quantitative standards from the policy text through regular matching and rule parsing methods; Constructing the policy subject-indicator item-quantitative standard triple; The triples are stored in a knowledge graph to obtain an indicator knowledge graph, in which the nodes include policy subject nodes, indicator item nodes, and quantitative standard nodes, and the edges include edges connecting the nodes.

4. The data assessment method based on a large model according to claim 1 is characterized in that: When the cosine similarity Δ between the V_t and the V_v is lower than a threshold γ, generating a confidence compensation parameter β based on the cosine similarity Δ by a generator, including: Calculate the cosine similarity Δ between the text feature vector V_t and the video feature vector V_v; If Δ is lower than the threshold γ, the cosine similarity is input into a preconfigured variational autoencoder VAE, and a confidence compensation parameter β is output.

5. The data assessment method based on a large model according to claim 4 is characterized in that: The calculating of the final score vector S_f based on the early fusion path output, the mid-term fusion path output or the late fusion path output comprises: When the confidence compensation parameter β is approximately equal to a first preset value, an early fusion path is adopted, wherein the early fusion path includes concatenating or weighted summing the text feature vector and the video feature vector to obtain a final score vector; When the confidence compensation parameter β is approximately equal to a second preset value, a mid-term fusion path is adopted, wherein the mid-term fusion path includes respectively calculating the attention scores of the text feature vector and the video feature vector, and fusing them in a hidden layer to obtain a final score vector; When the confidence compensation parameter β is approximately equal to a third preset value, a late fusion path is adopted, which includes respectively calculating the score vectors of the text feature vector and the video feature vector, and performing weighted averaging on the score vectors to obtain a final score vector.

6. The data assessment method based on a large model according to claim 2 is characterized in that: Outputting a final assessment score value based on the final score vector includes: Convert the final rating vector into a low-dimensional rating feature through a fully connected layer; Processing the low-dimensional scoring features through the scoring prediction layer to obtain a normalized preliminary score; Analyze the historical assessment data and obtain discrete level characteristics or demand range characteristics of the demand; Constructing a mapping function according to the discrete level characteristics of the demand or the range characteristics of the demand; The preliminary score is input into the mapping function to obtain a discrete grade or assessment score value.

7. A data assessment device based on a large model, characterized in that: The device comprises: The data collection unit is used to collect multi-source heterogeneous data from policy websites, industry databases, and institutional systems through the target agent deployed on the data bus; An indicator knowledge graph construction unit is configured to periodically detect the multi-source heterogeneous data, and when it is detected that the similarity change ΔS of the policy texts in the multi-source heterogeneous data exceeds a preset threshold α, parse the multi-source heterogeneous data to generate an indicator knowledge graph containing a policy subject-indicator item-quantitative standard triple; The matrix construction unit is used to decompose the department KPI into a tree-like demand graph using a graph neural network and calculate the priority weight distribution matrix W_p; An input unit, configured to update the indicator knowledge graph based on the distribution matrix W_p, and input the updated indicator knowledge graph into a preconfigured multimodal scoring engine; A semantic feature extraction unit is used to extract a deep semantic feature vector V_t from text-based assessment materials using the RoBERTa-wwm model in the multimodal scoring engine; A spatiotemporal feature extraction unit is used to extract a spatiotemporal feature vector V_v from the video stream data corresponding to the text-based assessment material through 3D-ResNet; a parameter compensation unit, configured to generate a confidence compensation parameter β through a generator based on the cosine similarity Δ between the V_t and the V_v when the cosine similarity Δ is lower than a threshold γ; a path weight vector generating unit, configured to generate a path weight vector according to the confidence compensation parameter β through a dynamic routing controller; a path fusion unit, configured to calculate an early fusion path output, a mid-fusion path output, or a late fusion path output through a weighted aggregation layer in the multimodal scoring engine based on the path weight vector; a scoring vector generating unit, configured to calculate a final scoring vector S_f based on the early fusion path output, the mid-term fusion path output, or the late fusion path output; A scoring value generating unit is used to output a final assessment scoring value based on the final scoring vector.

8. The data assessment device based on a large model according to claim 7, characterized in that: The matrix construction unit is specifically used for: Obtain the department's KPI hierarchical relationships, historical assessment data, and business context information; Mapping the KPI hierarchical relationship into a directed graph, wherein nodes in the directed graph represent individual KPI targets, edges represent causal relationships between KPI targets, and weights represent the degree of influence between KPI targets; Combining the assessment history data and the business context information to enhance the weights in the directed graph and generate a tree-like knowledge graph; Calculate the importance score of each KPI target in the directed graph through GNN; A weight distribution matrix W_p is constructed based on the importance scores.

9. A data assessment system based on a large model, characterized in that: The system comprises: processor, memory, input and output units, and buses; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and when the program is executed on a computer, the method according to any one of claims 1 to 6 is executed.

Images