Spark and LLM-based field credibility evaluation method and system

Abstract

This invention discloses a field credibility assessment method and system based on Spark and LLM, belonging to the field of data governance and data quality management technology. The method includes: extracting metadata from the business database and storing it in a metadata management table; synchronizing business data to a big data platform to form an ODS layer data table; based on the metadata management table and a predefined multi-dimensional data quality assessment index system, using the Spark distributed computing engine to dynamically construct and execute corresponding index calculation tasks for each target field in the ODS layer data table, obtaining and storing the quantitative index results for each field; constructing structured prompt words based on the quality index results, calling a large language model for processing, and obtaining and parsing the standardized assessment results output by the large language model, which include the quality level, assessment reasons, and improvement suggestions for each field. This invention achieves automated, large-scale, and intelligent assessment of the credibility of massive data fields.

Description

Technical Field

[0001] This invention relates to the field of data governance and data quality management technology, specifically to a field credibility assessment method and system based on Spark and LLM. Background Technology

[0002] As enterprises deepen their digital transformation, data has become a core production factor. However, enterprises accumulate massive amounts of data in their daily operations, which are scattered across different business systems and generally suffer from problems such as inconsistent data standards, varying quality, and missing management processes. Low-quality data directly affects the accuracy and reliability of downstream applications such as data analysis and machine decision-making.

[0003] Traditional data quality inspection methods mainly rely on two types: one is based on predefined rules, such as non-empty checks, format validation, and value range checks. This method lacks flexibility and intelligence, making it difficult to detect complex semantic or distributional anomalies. The second type relies on manual sampling and evaluation by data governance experts. This method is inefficient, cannot handle massive amounts of data, and is highly subjective, making it difficult to scale up.

[0004] Apache Spark, as a high-performance distributed computing engine, can efficiently handle batch computing tasks on massive datasets. Large Language Models (LLMs) possess powerful natural language understanding and reasoning capabilities, enabling them to handle complex, unstructured evaluation tasks. Currently, there is no mature solution that deeply integrates Spark's distributed computing capabilities with LLM's intelligent semantic evaluation capabilities for automated and intelligent evaluation of data field credibility. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a field credibility assessment method and system based on Spark and LLM to solve the problems of low automation, insufficient intelligence, and difficulty in dealing with massive data scales in traditional data quality detection methods.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] Field credibility assessment methods based on Spark and LLM include:

[0008] Metadata is extracted from at least one business database. The metadata includes the names, data types, and comment information of the databases, tables, and fields. The metadata is then stored in a unified metadata management table.

[0009] Data from the business database is synchronized to the data warehouse of the big data platform to form ODS layer data tables;

[0010] Based on the metadata management table and the predefined multi-dimensional data quality assessment index system, the Spark distributed computing engine is used to dynamically generate and execute index calculation tasks for each target field in the ODS layer data table, obtain the quantitative index results of each target field, and store the quantitative index results in the quality index result table.

[0011] Based on the data in the quality indicator results table, structured prompt words are constructed, and a large language model is called for processing to obtain and parse the standardized evaluation results output by the large language model, which include the quality level, evaluation reason, and improvement suggestions for each field.

[0012] Furthermore, metadata is extracted from at least one business database, specifically including: obtaining metadata by querying the system information table of the business database; the metadata management table contains at least column keys for uniquely identifying fields, the database name, the table name, the field name, the data type, and the field comment fields.

[0013] Furthermore, during the process of synchronizing data to the data warehouse of the big data platform, the fields of specified numeric types are converted to strings.

[0014] Furthermore, the multi-dimensional data quality assessment index system includes the following indicators: total number of rows, number of non-empty rows, maximum value, minimum value, number of unique values, high-frequency distribution of field values, and quantile statistics and equal-width bin distribution statistics for numeric fields.

[0015] Furthermore, the Spark distributed computing engine is used to dynamically generate and execute metric calculation tasks, specifically including:

[0016] Read the metadata management table and construct an in-memory dictionary that reflects the correspondence between fields and their data types;

[0017] Based on a multi-dimensional data quality assessment index system, a set of corresponding index calculation logic is predefined for each type of data. The set of index calculation logic consists of Spark calculation expressions used to calculate each index.

[0018] For each target field, the corresponding data type is looked up in the memory dictionary based on the data type of the target field, and the appropriate indicator calculation logic is dynamically selected from the predefined set of indicator calculation logics.

[0019] Using the Spark SQL API, aggregate calculation expressions are generated based on the selected metric calculation logic, forming a unified DataFrame;

[0020] The DataFrame is submitted to the Spark cluster for distributed computation, and the computation results are written to the quality metric results table.

[0021] Furthermore, the indicator calculation logic set includes a common basic logic set applicable to all data type fields and a numeric-specific logic set applicable only to numeric fields; if the field type is numeric, all logic from the common basic logic set and the numeric-specific logic set is dynamically selected for assembly; if it is non-numeric, only the logic from the common basic logic set is selected.

[0022] Furthermore, the common basic logic set includes Spark computation expressions for calculating the total number of rows, the number of non-empty rows, the maximum value, the minimum value, the number of unique values, and high-frequency distribution metrics of field values;

[0023] The Numerical Proprietary Logic Collection includes Spark computational expressions for calculating quantile statistics and equal-width bin distribution statistics.

[0024] Furthermore, structured prompts include the following components:

[0025] The role setting section is used to limit the role of the large language model to a data governance expert.

[0026] The format instructions section is used to force the large language model to output the evaluation results in a predefined JSON format. The JSON format must include at least a level field to indicate the quality level, a reason field to indicate the reason for the evaluation, and a suggestion field to indicate improvement recommendations.

[0027] The semantic interpretation section is used to provide Chinese semantic interpretation of the names of each key in the quantitative index results input to the large language model;

[0028] The data input section is used to encapsulate the data from a single record in the quality indicator results table as the actual input content.

[0029] Furthermore, the large language model is invoked for processing, specifically including: encapsulating the function that invokes the large language model into a user-defined function in Spark, and in the Spark environment, calling the user-defined function to process the records in the quality indicator result table one by one or in batches, and parsing and storing the evaluation results in JSON format returned by the large language model.

[0030] A field credibility assessment system based on Spark and LLM executes the methods described above, including:

[0031] The metadata management module is used to perform metadata extraction and storage functions;

[0032] The data access module is used to synchronize data from the business database to the big data platform;

[0033] The metrics calculation engine module, built on the Spark distributed computing framework, is used to dynamically generate and execute metrics calculation tasks.

[0034] The intelligent evaluation module integrates the ability to call large language models, which is used to construct structured prompt words based on quantitative indicator results and obtain evaluation results;

[0035] The results storage module is used to store metadata, quantitative indicator results, and evaluation results.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. Achieved a deep integration of automated, scalable, and intelligent assessment: Existing technologies largely rely on manual rules or single-machine scripts for data quality checks, which cannot handle massive amounts of enterprise data. This invention combines Spark distributed computing with Large Language Model (LLM) intelligent judgment to form a complete automated pipeline. By using Spark to achieve scalable indicator calculations and LLM to achieve intelligent assessment, it fundamentally solves the efficiency and accuracy problems of assessing the credibility of fields in massive datasets.

[0038] 2. A metadata-driven dynamic task construction mechanism is proposed, significantly improving system flexibility and maintainability: Traditional detection systems based on hard-coded rules require code modification for rule changes, resulting in high maintenance costs. This invention reads the metadata management table, constructs an in-memory dictionary, and dynamically selects and assembles predefined sets of indicator calculation logic based on field data types. This mechanism enables the system to automatically adapt to different table structures, expanding the detection range simply by updating metadata or indicator set definitions, without refactoring the core calculation code, greatly enhancing the system's adaptability and maintainability.

[0039] 3. A comprehensive, multi-dimensional, and quantifiable indicator system for intelligent evaluation was designed: Existing rules often target single issues (such as null value checks), resulting in one-sided evaluation dimensions. This invention constructs a comprehensive indicator system covering multiple dimensions, including completeness (total number of rows, number of non-null rows), uniqueness (number of unique values), value range rationality (maximum / minimum values, quantiles), and content distribution (high-frequency values, equal-width binning). This system provides LLM with comprehensive and structured quantitative inputs, serving as the data foundation for making reliable comprehensive judgments.

[0040] 4. An innovative structured prompting word engineering method for controlling large language models to perform domain-specific tasks: Directly feeding raw data or simple instructions to an LLM (Language Model) cannot produce stable, structured output that meets domain requirements. This invention designs a structured prompting word template comprising four parts: role setting, format instructions, semantic interpretation, and data input. In particular, the semantic interpretation part transforms technical fields into business language, effectively guiding the LLM to simulate the thinking of data experts and forcing it to output standardized evaluation reports with uniform format, including levels, reasons, and suggestions. This solves the controllability and interpretability challenges of LLM in professional domain applications.

[0041] 5. Achieved efficient and seamless system integration of big data engine and AI model: This invention encapsulates LLM calls as Spark UDFs (User-Defined Functions), enabling intelligent evaluation to be a natural step in the Spark data processing pipeline. This design allows for distributed calls to quality indicator results, either individually or in batches, and directly parses and stores the results. It achieves full-process automation and high-performance parallel processing from data computation to intelligent evaluation and result storage, resulting in high system integration and strong engineering practicality.

[0042] 6. The final output possesses high interpretability and direct actionability: Compared to traditional methods that only output "pass / fail" labels or numerical scores, the evaluation results generated by this invention through LLM not only include an overall quality level, but also detail multiple evaluation reasons and specific improvement suggestions. This provides data governance personnel with clear decision-making basis and action guidelines, greatly improving the efficiency and accuracy of data governance work. Attached Figure Description

[0043] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] like Figure 1 As shown, this invention provides a field credibility evaluation method based on Spark and LLM, comprising:

[0046] Metadata is extracted from at least one business database. The metadata includes the names, data types, and comment information of the databases, tables, and fields. The metadata is then stored in a unified metadata management table.

[0047] Data from the business database is synchronized to the data warehouse of the big data platform to form ODS layer data tables;

[0048] Based on the metadata management table and the predefined multi-dimensional data quality assessment index system, the Spark distributed computing engine is used to dynamically generate and execute index calculation tasks for each target field in the ODS layer data table, obtain the quantitative index results of each target field, and store the quantitative index results in the quality index result table.

[0049] Based on the data in the quality indicator results table, structured prompt words are constructed, and a large language model is called for processing to obtain and parse the standardized evaluation results output by the large language model, which include the quality level, evaluation reason, and improvement suggestions for each field.

[0050] This invention constructs an automated closed loop of "metadata-driven perception, distributed quantitative computation, and intelligent qualitative judgment." First, the system establishes a global data index by collecting metadata; then, it uses the Spark distributed engine to dynamically perform multi-dimensional quality indicator calculations based on field data types, transforming massive amounts of data into standardized features; finally, it guides a large language model through designed structured prompts to simulate data governance experts performing comprehensive reasoning on quantitative indicators, outputting standardized evaluation results that include quality level, evaluation reasons, and improvement suggestions.

[0051] Compared to existing technologies, this invention achieves a triple breakthrough in efficiency, intelligence, and practicality: Firstly, it utilizes Spark parallel computing to extract metrics from massive data fields at the millisecond level, improving evaluation efficiency by several orders of magnitude. Secondly, leveraging the semantic understanding and comprehensive reasoning capabilities of LLM, it replaces rigid threshold rules, making evaluation results closer to expert levels and possessing excellent interpretability. Simultaneously, the metadata-driven dynamic construction mechanism enables the system to flexibly adapt to different data sources, significantly reducing operational costs. This solution deeply integrates big data processing and artificial intelligence technologies, providing a practical technical path for data governance to move from manual inspection to intelligent autonomy.

[0052] In a preferred embodiment, the business database is a MySQL database. The specific implementation of obtaining metadata by querying the system information table of the business database is as follows: A metadata extraction program is written to execute the SQL statement `SELECT TABLE_SCHEMA as db, TABLE_NAME as tb, COLUMN_NAME as col_name, DATA_TYPE as data_type, COLUMN_COMMENT as col_comment FROM information_schema.COLUMNS`, and the query results are stored in a metadata management table (e.g., named `bd_col_meta_info`). The metadata management table includes at least a column key for uniquely identifying fields, the database name, the table name, the field name, the data type, and field comment fields. The specific structure of the metadata management table is shown in Table 1.

[0053] Table 1

[0054]

[0055] To ensure efficient processing of massive amounts of data, this method relies on synchronizing data from the business database to a big data platform with distributed computing capabilities (such as a Hadoop-based Hive data warehouse) to form an operational data storage layer (ODS layer). The data synchronization process can be implemented using mature data synchronization tools (such as ETL tools) or data access programs. During the data synchronization process to the big data platform's data warehouse, fields of specified numeric types undergo string type conversion. This operation aims to ensure data accuracy and simplify the subsequent unified quality indicator calculation logic.

[0056] In a preferred embodiment, string type conversion for fields of specified numeric types specifically includes: converting fields with data types of float, double, and decimal in the source business table of the business database to string in the target ODS layer table, and completing the corresponding value conversion and storage during data synchronization. In cross-system (e.g., from MySQL to Hive) data migration and subsequent Spark distributed computing, converting high-precision numeric types (e.g., decimal) to strings can avoid data distortion caused by differences in floating-point implementations or precision truncation between different systems. After conversion to strings, whether it's numeric, text, or any special characters that may appear later, string-level quality indicator calculations (e.g., content distribution analysis) can be performed uniformly, simplifying the type judgment logic in the computing engine. It also ensures the stability of the data storage format in data warehouses such as Hive, avoiding import failures due to type mismatches. Furthermore, ODS layer table names in the data warehouse are typically mapped to source business table names in the business database by adding a unified identifier prefix (e.g., ods_) before the source business table name, facilitating management. For example, the user table in the business database can be named ods_user in the ODS layer.

[0057] The multi-dimensional data quality assessment index system predefined in this invention, in a preferred embodiment, specifically includes the following computable indicators: total number of rows (total_row); number of non-empty rows (col_no_null); maximum value (max_value); minimum value (min_value); number of unique values ​​(unique_cnt); high-frequency distribution of field values ​​(e.g., content_top30, i.e., the top 30 values ​​with the highest frequency and their counts); equal-width bucket distribution statistics for numeric fields (e.g., num_split_30, i.e., the data distribution after dividing the numerical range into 30 equal intervals); quantile statistics for numeric fields: e.g., the 25th percentile (P25), 50th percentile (P50), 75th percentile (P75), and 90th percentile (P90).

[0058] A specific embodiment of this invention that utilizes the Spark distributed computing engine to dynamically generate and execute metric calculation tasks includes:

[0059] (I) SparkSession Initialization and Metadata Preparation:

[0060] First, create the Spark application entry point using the PySpark API. Specifically, in the program, use SparkSession.builder.appName("DataQualityAssessment").enableHiveSupport().getOrCreate() to build a SparkSession instance named spark. This instance serves as the unified channel for communication with the Spark distributed cluster and task scheduling.

[0061] Subsequently, the metadata management table (e.g., a table named bd_col_meta_info) stored in the data warehouse is read using the `spark.sql("SELECT * FROM bd_col_meta_info")` or `spark.read.table("bd_col_meta_info")` methods. The data is then pulled into the driver node's memory using the `collect()` operation, and a Python dictionary (e.g., named `col_type_map`), i.e., an in-memory dictionary, is constructed. This dictionary uses the fully qualified name of the field (e.g., "database_name.table_name.field_name") or a composite key as the key, and the field's `data_type` (e.g., varchar, int, decimal) as the value, thus quickly mapping the relationship between fields and their data types in memory.

[0062] (II) Predefined and dynamic selection of indicator calculation logic:

[0063] Based on a multi-dimensional data quality assessment index system, two sets of core calculation logic are predefined in the program:

[0064] Common underlying logic collection: Fields applicable to all data types, including Spark computation expressions used for calculations:

[0065] count(1) as total_row (total number of rows);

[0066] sum(case when {col_name} is null then 0 else 1 end) as col_no_null (number of non-empty rows);

[0067] max({col_name}) as max_value (maximum value);

[0068] min({col_name}) as min_value (minimum value); count(distinct {col_name}) as unique_cnt (number of unique values);

[0069] By using the groupby, count, orderBy and limit(30) operations, combined with the to_json and map_from_entries functions, a JSON string representing the high-frequency distribution of field values ​​(content_top30) is generated.

[0070] Numeric-specific logical collections: These are only applicable to numeric fields (such as int, bigint, float, double, decimal), and contain Spark computation expressions used for calculations.

[0071] Equal-width bucket distribution statistics (num_split_30): The field value is evenly divided into 30 intervals using the width_bucket function, the number of records in each interval is counted, and the result is formatted as JSON.

[0072] Quantile statistics: The percentile_approx function was used to calculate the 25th (p25), 50th (p50), 75th (p75), and 90th (p90) percentiles respectively.

[0073] Spark computation expressions can be expressed as Spark SQL expressions or DataFrame API operators.

[0074] When the program runs, for each target field in the ODS layer target table (ODS layer data table), the col_type_map dictionary is queried. If the field type is numeric, all logic from the common basic logic set and the numeric-specific logic set is dynamically selected for assembly; if the field type is non-numeric, only the logic from the common basic logic set is selected.

[0075] (III) Dynamic SQL / DataFrame Construction and Execution:

[0076] Iterate through all fields of the target table in the ODS layer. Based on the results of the dynamic selection described above, use Python's string formatting or Spark's `expr()` function to replace the field name `{col_name}` with the actual field name, thereby generating a specific set of aggregation expressions for each field. Then, use Spark SQL's `select` or `agg` operations to combine these expressions for different fields into a unified DataFrame computation plan. The schema of this DataFrame matches the structure of a predefined quality metric results table (e.g., `data_governance_column_stats`), as shown in Table 2 below.

[0077] Table 2

[0078]

[0079] (iv) Distributed computing and result persistence:

[0080] Call the `DataFrame.write.saveAsTable()` or `DataFrame.write.insertInto()` method to submit the constructed DataFrame to the Spark cluster. The Spark cluster executes the computation plan in a distributed parallel manner, scanning the target table in the ODS layer and completing the calculation of all metrics. Finally, the calculation results are written to the specified quality metric results table (e.g., named `data_governance_column_stats`), with each record corresponding to a complete set of quality metrics for one field.

[0081] A preferred embodiment of the structured prompt words of this invention is constructed based on the following four parts:

[0082] Role setting section: Explicitly input "You are a data governance expert. Please score the data quality based on the input JSON content."

[0083] Formatting directives: It is mandatory that "the output data format cannot be Markdown; it must be output in standard JSON format, with the output JSON fields being level, reason, and suggestion. The level is defined as one of four levels: 'Poor,' 'Average,' 'Good,' or 'Reliable.' There must be at least five 'reason' suggestions."

[0084] Semantic explanation section: Provides Chinese explanations for each key in the input JSON.

[0085] Data Input Section: Enter the specific JSON data read from and converted from the data_governance_column_stats table in this section.

[0086] Finally, the above four parts are concatenated in order to form a complete prompt word string, which is then input into the large language model.

[0087] The semantic explanation portion of the structured prompts specifically includes Chinese interpretations of key fields in the input JSON, for example:

[0088] num_split_30 indicates that the numeric field is divided into 30 equally wide buckets;

[0089] p25 represents the 25th percentile;

[0090] p50 represents the 50th percentile (median);

[0091] p75 represents the 75th percentile;

[0092] p90 represents the 90th percentile;

[0093] max_value represents the maximum value;

[0094] min_value represents the minimum value;

[0095] col_name represents the column name;

[0096] col_comment represents a comment indicating the business meaning of the column;

[0097] total_row represents the total number of rows;

[0098] col_no_null represents the number of non-empty rows;

[0099] unique_cnt represents the number of unique values;

[0100] content_top30 indicates the distribution of high-frequency values ​​for a field.

[0101] A specific embodiment of the present invention that uses a large language model for processing includes:

[0102] (I) Encapsulation and registration of user-defined functions (UDFs):

[0103] Define a Python function, for example, named `call_llm`. This function accepts a string parameter `prompt_str`, and its internal logic implements a call to a large language model (such as Qianwen, GPT, etc.) API, returning the API's response (usually a JSON-formatted string) as its return value. Then, register the `call_llm` function as a user-defined function (UDF) available in the Spark SQL environment using the statement `spark.udf.register("call_llm_udf", call_llm)`.

[0104] (II) Preparation of evaluation data and execution of UDF:

[0105] The quality indicator result table storing quantitative indicators is read using Spark SQL (e.g., spark.sql(“SELECT * FROM data_governance_column_stats”)) to obtain a DataFrame (e.g., stat_df). Based on the aforementioned method, a complete structured prompt string is constructed for each row of data in stat_df (corresponding to a complete set of indicators for one field), forming a new data column (e.g., prompt). Subsequently, a DataFrame operation similar to stat_df.withColumn(“llm_result”, call_llm_udf(col(“prompt”))) is executed. This operation will trigger the Spark distributed computing framework to call the UDF in either a row-by-row or batch processing manner.

[0106] Item-by-item processing mode: UDFs are designed to process one prompt word string at a time. The Spark framework automatically treats each row value in the prompt column as an independent parameter and calls the call_llm_udf function in parallel and sequentially on each executor in the cluster.

[0107] Batch processing mode: UDFs are designed to receive a set of structured prompts (such as a list). In this mode, the internal logic of the UDF may combine multiple prompts into a single batch request, call the batch API provided by the LLM, or integrate them into a longer contextual dialogue for more efficient interaction.

[0108] (III) Analysis and persistent storage of intelligent evaluation results:

[0109] The large language model returns a structured JSON object as the evaluation result, based on strict rules of structured prompt words. This JSON object contains at least the following fields:

[0110] The level field: Its value is a level label selected from a predefined set of discrete quality levels (such as "poor", "average", "good", "trustworthy"), used to comprehensively characterize the trustworthiness level of the field.

[0111] The reason field is a JSON array containing at least a predetermined number (e.g., 5) of evaluation reason texts generated by a large language model, each text being derived from inferences based on the input quantitative metrics.

[0112] The suggestion field: A JSON array containing zero or more specific, actionable data governance improvement suggestions from the large language model for the identified issues.

[0113] Subsequently, Spark SQL's built-in functions (such as `from_json`) are used to parse the JSON string in the `llm_result` column, extracting structured subfields such as `level`, `reason`, and `suggestion`. Finally, the complete DataFrame containing the original metrics and intelligent assessment results is persisted to a specified data warehouse table using methods such as `.write.saveAsTable("final_quality_assessment")`, completing the entire process from data quality detection to intelligent assessment.

[0114] This invention also provides a field credibility assessment system based on Spark and LLM, comprising:

[0115] The metadata management module is used to perform metadata extraction and storage functions;

[0116] The data access module is used to synchronize data from the business database to the big data platform;

[0117] The metrics calculation engine module, built on the Spark distributed computing framework, is used to dynamically generate and execute metrics calculation tasks.

[0118] The intelligent evaluation module integrates the ability to call large language models, which is used to construct structured prompt words based on quantitative indicator results and obtain evaluation results;

[0119] The results storage module is used to store metadata, quantitative indicator results, and evaluation results.

[0120] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention used to illustrate the technical solutions of the present invention, and are not intended to limit the invention, nor are they intended to limit the patent scope of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention. That is to say, any changes or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but whose technical problems are still consistent with the present invention, should be included within the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention to other related technical fields are similarly included within the patent protection scope of the present invention.

Claims

1. A method for field trustworthiness evaluation based on Spark and LLM, characterized in that, include: Metadata is extracted from at least one business database. The metadata includes the names, data types, and comment information of the databases, tables, and fields. The metadata is then stored in a unified metadata management table. Data from the business database is synchronized to the data warehouse of the big data platform to form ODS layer data tables; Based on the metadata management table and the predefined multi-dimensional data quality assessment index system, the Spark distributed computing engine is used to dynamically generate and execute index calculation tasks for each target field in the ODS layer data table, obtain the quantitative index results of each target field, and store the quantitative index results in the quality index result table. Based on the data in the quality indicator results table, structured prompt words are constructed, and a large language model is called for processing to obtain and parse the standardized evaluation results output by the large language model, which include the quality level, evaluation reason and improvement suggestions for each field. The multi-dimensional data quality assessment index system includes the following indicators: total number of rows, number of non-empty rows, maximum value, minimum value, number of unique values, high-frequency distribution of field values, and quantile statistics and equal-width bucket distribution statistics for numeric fields; Dynamically generate and execute metric calculation tasks using the Spark distributed computing engine, specifically including: Read the metadata management table and construct an in-memory dictionary that reflects the correspondence between fields and their data types; Based on a multi-dimensional data quality assessment index system, a set of corresponding index calculation logic is predefined for each type of data. The set of index calculation logic consists of Spark calculation expressions used to calculate each index. For each target field, the corresponding data type is looked up in the memory dictionary based on the data type of the target field, and the appropriate indicator calculation logic is dynamically selected from the predefined set of indicator calculation logics. Using the Spark SQL API, aggregate calculation expressions are generated based on the selected metric calculation logic, forming a unified DataFrame; The DataFrame is submitted to the Spark cluster for distributed computation, and the computation results are written to the quality metric results table. 2.The Spark and LLM based field trustworthiness evaluation method according to claim 1, wherein, Metadata is extracted from at least one business database, specifically by: obtaining metadata by querying the system information table of the business database; the metadata management table contains at least column keys for uniquely identifying fields, the database name, the table name, the field name, the data type, and the field comment fields. 3.The Spark and LLM based field trustworthiness evaluation method according to claim 1, wherein, During the process of synchronizing data to the data warehouse of the big data platform, the fields of specified numeric types are converted to strings. 4.The Spark and LLM based field trustworthiness evaluation method according to claim 1, wherein, The indicator calculation logic set includes a common basic logic set applicable to all data type fields and a numeric-specific logic set applicable only to numeric fields; if the field type is numeric, all logic from the common basic logic set and the numeric-specific logic set is dynamically selected for assembly. If it is a non-numerical type, only the logic of the common basic logic set will be selected. 5.The Spark and LLM based field trustworthiness evaluation method according to claim 4, wherein, The common basic logic set includes Spark computation expressions for calculating the total number of rows, the number of non-empty rows, the maximum value, the minimum value, the number of unique values, and high-frequency distribution metrics of field values; The Numerical Proprietary Logic Collection includes Spark computational expressions for calculating quantile statistics and equal-width bin distribution statistics. 6.The Spark and LLM based field trustworthiness evaluation method according to claim 1, wherein, Structured prompts include the following components: The role setting section is used to limit the role of the large language model to a data governance expert. The format instructions section is used to force the large language model to output the evaluation results in a predefined JSON format. The JSON format must include at least a level field to indicate the quality level, a reason field to indicate the reason for the evaluation, and a suggestion field to indicate improvement recommendations. The semantic interpretation section is used to provide Chinese semantic interpretation of the names of each key in the quantitative index results input to the large language model; The data input section is used to encapsulate the data from a single record in the quality indicator results table as the actual input content.

7. The Spark and LLM based field trustworthiness evaluation method according to claim 6, characterized in that, The process involves calling a large language model, specifically encapsulating the function that calls the large language model into a user-defined function in Spark, and then calling the user-defined function in the Spark environment to process the records in the quality indicator result table one by one or in batches. The process also involves parsing and storing the JSON-formatted evaluation results returned by the large language model.

8. A system for field trustworthiness assessment based on Spark and LLM, performing the method of any one of claims 1 to 7, characterized in that, include: The metadata management module is used to perform metadata extraction and storage functions; The data access module is used to synchronize data from the business database to the big data platform; The metrics calculation engine module, built on the Spark distributed computing framework, is used to dynamically generate and execute metrics calculation tasks. The intelligent evaluation module integrates the ability to call large language models, which is used to construct structured prompt words based on quantitative indicator results and obtain evaluation results; The results storage module is used to store metadata, quantitative indicator results, and evaluation results.

Images