A multi-dimensional index data source quality analysis and optimization method and system

Abstract

The present application relates to big data resource service technical field, specifically to a kind of based on multi-dimension index data source quality analysis and optimization method and system, including the following steps, target field set is obtained by parsing calling task, associated field and quality index are detected candidate data source, gather field quality state and form task level quality description, according to this determine calling sequence and disposal path, in order to access and from subsequent data source make up not satisfied field, compare measured field state and before calling state, form deviation and revise quality condition.In the present application, by quality judgment is refined to field and index granularity and is formed task evaluation according to field combination, make calling decision adjust with field state, keep available result when local exception and make up missing data, reduce whole source replacement and repeated access, deviation feedback updates priority and disposal condition, improve the consistency of evaluation and true calling performance, enhance disposal pertinence and data acquisition efficiency.

Description

Technical Field

[0001] This invention relates to the field of big data resource service technology, and in particular to a method and system for analyzing and optimizing data source quality based on multi-dimensional indicators. Background Technology

[0002] The field of big data resource service technology revolves around the technical processing of access, transmission, storage, retrieval, and management of multi-source data resources. Typically, databases, distributed storage systems, business terminals, and servers exchange structured records, file data, interface-returned data, and operational logs through data interfaces or network links. Retrieval, verification, statistics, and storage are performed based on data fields, timestamps, source identifiers, and access status. Traditional data source quality analysis and optimization systems based on multi-dimensional indicators refer to systems that analyze and process the quality status of accessed data sources. They typically obtain data records and access logs from databases, file storage areas, or data interfaces, and perform statistical analysis, comparison, and rule-based judgments based on completeness, accuracy, consistency, timeliness, and availability, corresponding to missing fields, value matching relationships, update time, and access status. The analysis results are written to the server storage area, and the data sources are marked, filtered, have their retrieval order adjusted, or are re-collected and processed based on preset thresholds or sorting results.

[0003] Existing data source quality analysis typically involves overall statistics based on fixed indicators and sorting or labeling according to thresholds. Field quality lacks dynamic correlation with specific invocation tasks, making it difficult to form executable judgments for multi-field combination requirements. The invocation phase relies on static sequential access to data sources, and when some fields are abnormal, it can easily trigger a complete source switch or repeated collection, resulting in the discarding of usable fields and increased access overhead. The actual returned results lack continuous comparison with the judgment before invocation, and deviations are difficult to translate into a basis for subsequent rule adjustments. The quality evaluation deviates from the actual invocation performance, and the stability of anomaly handling and data acquisition efficiency are affected. Summary of the Invention

[0004] This invention aims to provide a method and system for data source quality analysis and optimization based on multi-dimensional indicators. It determines target fields and corresponding quality indicators for data call tasks, performs field-level quality checks and task-level quality status aggregation on candidate data sources, and forms a data source call sequence and anomaly handling path based on the target field's satisfied status and abnormal quality indicators. This enables different candidate data sources to collaboratively complete data acquisition according to field quality status. Simultaneously, based on the deviation between the actual call result and the quality status before the call, the priority of quality indicator judgment, verification trigger conditions, and data source handling conditions are adjusted to form quality conditions suitable for subsequent data call tasks.

[0005] To achieve the above objectives, this invention provides a method for quality analysis and optimization of data sources based on multi-dimensional indicators, comprising the following steps:

[0006] The system acquires the target field set corresponding to the data retrieval task and, based on a pre-defined correspondence between fields and quality indicators, determines the set of participating indicators required for this task. For each candidate data source, the target field set is tested according to the participating indicator set, generating a field quality status record. This record is associated with at least the data source identifier, field identifier, quality indicator identifier, and test result. Based on the combination relationships of the target field set in the current task, the field quality status records of each candidate data source are aggregated to generate a data source quality status description corresponding to the current task. Based on the data source quality status description and according to the satisfaction status of the target fields and the abnormal quality indicators corresponding to abnormal fields, the system determines the retrieval sequence and exception handling path for each candidate data source.

[0007] Candidate data sources are accessed sequentially according to the call sequence. When an accessed data source meets all target field requirements, the corresponding data is output; when some target fields do not meet the requirements, the corresponding data for the unmet fields is obtained from subsequent candidate data sources. After the candidate data source is actually accessed, quality checks are performed on the data records and access status returned by the accessed candidate data source according to the target field set and the participating indicator set, generating an actual quality status record; according to the data source identifier, field identifier, and quality indicator identifier, the actual quality status record is compared item by item with the field quality status record generated before the call, generating a deviation record, and the judgment priority, verification triggering condition, or data source handling condition of the corresponding quality indicators in subsequent data call tasks are adjusted based on the deviation record.

[0008] Furthermore, the determination of the target field set and the participating indicator set includes: parsing the data structure description, returned field constraints, and field combination relationships of the data retrieval task; extracting field names, field types, and mandatory statuses; converting field names into standardized field identifiers based on a unified field directory, deleting duplicate standardized field identifiers, and recording common return constraints among standardized field identifiers to form the target field set. Reading the quality indicator identifiers corresponding to each standardized field identifier from the field quality indicator mapping table, filtering basic quality indicator identifiers according to mandatory statuses, supplementing associated quality indicator identifiers according to common return constraints, merging and deduplicating the quality indicator identifiers, and arranging the retained quality indicator identifiers according to judgment priority to form the participating indicator set.

[0009] Furthermore, the generation of field quality status records includes: obtaining data records, field metadata, and access status returned by each candidate data source for the target field set; matching the quality indicator identifiers in the participating indicator set with the target field to generate field indicator detection pairs; extracting detection input values ​​from data records, field metadata, or access status based on the detection conditions associated with each field indicator detection pair; converting the detection input values ​​into a data format consistent with the detection conditions to generate indicator detection inputs; comparing the indicator detection inputs with the detection conditions to determine the pass status, abnormal status, or undetectable status; and associating the data source identifier, field identifier, and quality indicator identifier to form a field quality status record.

[0010] Furthermore, the generation of the data source quality status description includes: grouping field quality status records according to the data source identifier; establishing field status positions based on field identifiers in the target field set; writing the detection results of quality indicators associated with the same field into the corresponding field status positions to form a field indicator status sequence; sorting abnormal states, undetectable states, and pass states according to the judgment priority of quality indicators in the participating indicator set to determine the field satisfaction status of each target field and generate a field quality aggregation result; judging the field quality aggregation result according to the combination relationship of the target field set, and associating it with the data source identifier, field satisfaction status, abnormal quality indicators, and unsatisfied field identifiers to form a data source quality status description.

[0011] Furthermore, determining the call sequence and exception handling path includes: classifying data source status into categories based on whether all target fields are satisfied, partially satisfied, or all target fields are not satisfied; within the same data source status category, determining the order of candidate data sources based on the number of satisfied fields, the priority of exception quality indicators, and the access status to form a call sequence. For each candidate data source in the call sequence, identifying the unsatisfied field identifier and its corresponding exception quality indicator identifier, and determining the handling action—continuing to supplement, skipping fields, switching data sources, or terminating the call—based on the mapping relationship between the exception quality indicator identifier and the handling action, and associating the corresponding handling action according to the order of candidate data sources to form an exception handling path.

[0012] Furthermore, when accessing candidate data sources according to the call sequence, a list of fields to be retrieved is established based on the target field set. The first candidate data source in the call sequence is determined as the current access data source, and a data access request carrying the identifier of the field to be retrieved is sent to the current access data source. The returned field results are compared with the field satisfaction status in the data source quality status description. Fields that meet the quality requirements are written into the task result record, and their corresponding field identifiers are removed from the list of fields to be retrieved. Field identifiers of fields that were not returned or did not meet the quality requirements are retained. When the list of fields to be retrieved is not empty, the next candidate data source is selected according to the call sequence, and a data access request carrying only the identifiers of the remaining fields is sent to the next candidate data source until the list of fields to be retrieved is empty or all candidate data sources have been accessed, generating the data retrieval result.

[0013] When selecting the next candidate data source, the system queries the data source quality status descriptions of candidate data sources that have not yet been accessed in the call sequence, filters candidate data sources with at least one remaining field in a satisfactory state, forms a set of supplementary data sources, and determines the next candidate data source according to its order in the call sequence. When no supplementary data source exists, the remaining field identifier is determined as a skipped field identifier or a call failure field identifier according to the exception handling path, and subsequent access to the corresponding remaining field is stopped.

[0014] Furthermore, the generation of deviation records includes: performing quality checks on the data records and access status returned by the candidate data sources for this actual access according to the target field set and the participating indicator set, and generating actual quality status records; matching and comparing the actual quality status records with the field quality status records generated before the call item by item according to the data source identifier, field identifier, and quality indicator identifier, and recording the pre-call detection result and actual detection result for each matching item to generate quality status comparison items; when the pre-call detection result and the actual detection result are the same, the deviation direction is determined to be no deviation; when the two are different, a positive deviation, negative deviation, or detection reachability deviation is determined according to the change direction of the detection result, and a deviation record is generated.

[0015] Furthermore, deviation records are grouped according to data source identifier, field identifier, and quality indicator identifier. Within a preset observation period, the continuous occurrence of positive deviation, negative deviation, and no deviation is statistically analyzed to form a deviation stability record. This deviation stability record is compared with preset adjustment conditions to determine the priority adjustment items, verification trigger condition adjustment items, and data source handling condition adjustment items, forming a quality condition adjustment list. The judgment priority, verification trigger rule, or anomaly handling rule of the corresponding quality indicator is updated according to the quality condition adjustment list, and the conditions before, after, and effective time of the update are recorded, forming the quality condition version for subsequent data retrieval tasks.

[0016] The present invention also provides a multi-dimensional indicator data source quality analysis and optimization system for performing the above-described method, including a data source access module, a task parsing module, an indicator association module, a field quality analysis module, a quality status generation module, a data source scheduling module, a call execution module, and a feedback correction module. The data source access module establishes communication connections with each candidate data source to obtain data records, source identifiers, and access status. The task parsing module parses data call tasks to obtain the target field set required by the task. The indicator association module stores the correspondence between fields and quality indicators, used to determine the set of participating indicators for this task based on the target field set. The field quality analysis module detects the target fields of candidate data sources according to the participating indicator set and generates field quality status records. The quality status generation module aggregates the field quality status records based on the combination relationship of the target field set of the current task, generating a data source quality status description corresponding to the current task. The data source scheduling module determines the call sequence and exception handling path of candidate data sources based on the data source quality status description. The call execution module accesses candidate data sources according to the call sequence and, if some target fields of the preceding data source do not meet the requirements, obtains data for the unmet fields from subsequent data sources. The feedback correction module compares the actual call result with the data source quality status description before the call, generates a deviation record, and adjusts the quality conditions of subsequent data call tasks based on the deviation record. All of the above modules are implemented by the processor in the server executing program instructions stored in memory.

[0017] The beneficial effects of the technical solution provided by the embodiments of the present invention include at least the following: establishing a correspondence between fields and quality indicators around the task target fields, enabling quality detection to be implemented at the field and indicator granularity, and aggregating multiple field combinations to form a task-level quality status description. The calling order and exception handling path are jointly determined by the field satisfaction status and the exception indicators. When some fields in the preceding data source are unqualified, qualified results are retained and missing fields are supplemented from subsequent data sources, reducing the replacement of the entire source due to the anomaly of some fields, as well as the repeated requests and repeated collections of fields that have been satisfied. After the call, the actual results are compared with the predicted status to form a deviation record, thereby correcting the indicator judgment priority, verifying the trigger conditions and data source handling conditions, so that the quality rules are updated with the actual call performance, improving the accuracy of data source selection and exception handling for unsatisfied fields, reducing invalid field requests, and improving call stability and data acquisition efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is the main flowchart for data source quality analysis and optimization in this invention;

[0020] Figure 2 Generate a data flow diagram for the target field set and the participating indicator set of this invention;

[0021] Figure 3 This is a logic diagram for the aggregation of field quality status and determination of the calling path in this invention;

[0022] Figure 4 This is a state transition diagram of candidate data source access and unsatisfied field completion in this invention;

[0023] Figure 5 This is a closed-loop diagram of deviation feedback and system module collaboration in this invention. Detailed Implementation

[0024] The following embodiments describe the technical solutions of this invention based on the scope of protection text. The original input did not provide any drawings, drawing numbers, drawing names, drawing labels, or specific illustrated content that could confirm their existence; therefore, subsequent drawings are derived from… Figure 1 The starting number is used to describe the data retrieval task parsing, field quality detection, data source quality status aggregation, call supplementation, deviation feedback, and system module collaboration relationships. To maintain consistency in technical objects, this document uses the following terms uniformly: "Data Request Task" to represent a data access task that requires retrieving target fields from a candidate data source; "Target Field Set" to represent the set of fields parsed by the data request task and deduplicated by standardized field identifiers; "Participating Indicator Set" to represent the set of quality indicator identifiers corresponding to the target field set and arranged according to judgment priority; "Field Quality Status Record" to represent the detection result record formed for a single data source, a single field, and a single quality indicator; "Data Source Quality Status Description" to represent the task-level description formed by aggregating the field quality status records of the same candidate data source according to field combination relationships for the current data request task; "Request Sequence" to represent the access order of candidate data sources; "Abnormal Handling Path" to represent the chain of actions for handling fields that do not meet the requirements of their abnormal quality indicators, such as continuing to retrieve, skipping fields, switching data sources, or terminating the request; "Deviation Record" to represent the difference record between the quality detection result after the actual request and the quality status before the request; and "Quality Condition Version" to represent the effective record of the judgment priority, verification trigger conditions, or data source handling conditions updated based on the deviation record.

[0025] Example 1

[0026] Please see Figures 1 to 4This embodiment provides a method for data source quality analysis and optimization based on multi-dimensional indicators. In this embodiment, the method takes a data retrieval task as the starting input, the field-level quality status of candidate data sources as the intermediate basis, determines the retrieval sequence and anomaly handling path based on the data source quality status description, and corrects subsequent quality conditions through deviation records after the actual retrieval, so that data source selection, field supplementation, and quality condition adjustment form a traceable closed loop.

[0027] S1: Obtain the target field set corresponding to the data retrieval task and determine the participating indicator set based on the preset field-quality indicator correspondence. When executing this step, the data retrieval task serves as the input object. The task parsing process first reads the data structure description, return field constraints, and field combination relationships. The data structure description is used to determine the field name and field type of the field to be returned in the task; the return field constraints are used to determine whether a field is mandatory; and the field combination relationships are used to determine whether multiple fields need to be returned together. After the field name enters the unified field directory, it is converted into a standardized field identifier; when the same standardized field identifier appears repeatedly, only one standardized field identifier is retained, along with its common return constraints with other standardized field identifiers. The standardized field identifier set, field types, mandatory status, and common return constraints formed by the above processing together constitute the target field set.

[0028] S101, parse the data structure description of the data call task, converting field names that can match the unified field directory into standardized field identifiers; when field names are duplicated, use the standardized field identifier as the deduplication basis, retaining the field types and required statuses related to the task return. If a field name cannot be mapped to the unified field directory, set the standardized field identifier corresponding to that field name to a mapping failure status. This status is not included in the valid target field set, but is retained as a task parsing exception status for subsequent exception handling paths to determine whether to terminate the call or skip the corresponding field.

[0029] S102: Read the field quality indicator mapping table and query the quality indicator identifiers according to the identifiers of each specified field in the target field set. For required fields, retain the basic quality indicator identifiers; for field groups with common return constraints, supplement with related quality indicator identifiers that can reflect the completeness, consistency, or common return status of the field combination. After merging and deduplication, the queried quality indicator identifiers are arranged according to a preset judgment priority to form the participating indicator set.

[0030] In one possible implementation, each quality indicator record in the field quality indicator mapping table includes at least a quality indicator identifier, applicable field type, detection input source, detection rules, judgment direction, and judgment parameters. For quality indicators that can be statistically determined through sample records, valid detection records within the current detection batch or a preset statistical window are used as the statistical objects.

[0031] For field completeness metrics, the completeness rate is used. express:

[0032] ;

[0033] in, This counts the number of records in the target object whose corresponding field has a non-nullable valid value. This represents the total number of records that participated in the integrity check for this field. >0, It is a dimensionless proportion and satisfies 0 ≤ ≤1.

[0034] For field rule compliance metrics, use compliance rate express:

[0035] ;

[0036] in, This specifies the number of records whose field values ​​satisfy the corresponding data type, length, value range, or format validation rules. This represents the total number of records that participate in the corresponding rule-based detection. It is a dimensionless proportion and satisfies 0≤V≤1.

[0037] For the timeliness index of the field, calculate the time difference. :

[0038] ;

[0039] in, The evaluation time is for performing quality inspection. For the generation or update time corresponding to the field data, and Using the same time base and time unit; The maximum allowable time difference corresponding to this field Comparison, and Using the same time unit, when 0 ≤ ≤ When the corresponding timeliness indicators meet the testing conditions, > When it is determined that the corresponding timeliness indicator does not meet the testing conditions.

[0040] For data source access availability metrics, in terms of availability rate express:

[0041] ;

[0042] in, This is the preset count of requests that were successfully accessed and received a valid response within the statistics window. This represents the total number of requests sent to the corresponding data source within the same statistics window. >0, It is a dimensionless proportion and satisfies 0 ≤ ≤1.

[0043] The judgment thresholds corresponding to each quality indicator are stored as part of the quality conditions. Different fields can be configured with corresponding judgment thresholds. The field quality analysis module reads the quality condition version used by the current data call task and determines the pass status, abnormal status, or undetectable status based on the detection rules and judgment thresholds of the corresponding quality indicators.

[0044] Priority determination serves as a common basis for subsequent ranking of test results, identification of abnormal quality indicators, and determination of data source order.

[0045] S2, for each candidate data source, detects the target field set according to the participating indicator set and generates field quality status records.

[0046] The field quality status record used to determine the candidate data source call sequence before the call is executed does not require a complete business data access to all candidate data sources after each data call task begins. In one possible implementation, the system establishes a field quality snapshot for the data source identifier, field identifier, and quality indicator identifier. The field quality snapshot is updated from the actual quality status record formed during the historical actual call process, the periodic quality test results, or the lightweight quality test results performed on the candidate data sources.

[0047] Each field quality snapshot records at least the data source identifier, field identifier, quality indicator identifier, most recent detection result, detection time, and validity status. When a data retrieval task begins, the field quality analysis module first queries for field quality snapshots that match the current target field set and participating indicator set. If the corresponding field quality snapshot is valid, the detection result in that snapshot is used as the pre-request field quality status record. If no corresponding field quality snapshot exists or the snapshot has exceeded its preset validity period, the corresponding quality indicator is determined to be undetectable, or a pre-request quality check is performed according to the corresponding quality indicator's verification trigger conditions.

[0048] The validity period of a field quality snapshot is expressed in terms of time length, and its time unit is consistent with the time unit of the detection time. Different data sources, fields, or quality indicators can be configured with corresponding validity periods. Therefore, the data source field quality status for the current data retrieval task can be obtained without needing to pre-access all candidate data sources to determine the call order.

[0049] The candidate data source serves as the detection object, and its returned data records, field metadata, and access status serve as the detection input sources. The data records are used to detect whether the field values ​​themselves meet the corresponding quality indicators; the field metadata is used to detect whether the field exists, whether the type matches, or whether the field source can be identified; and the access status is used to detect whether the candidate data source is accessible and whether the returned status supports the current task call.

[0050] S201: Obtain the data records, field metadata, and access status returned by each candidate data source for the target field set. The field quality analysis process matches the quality indicator identifiers in the participating indicator set with the standardized field identifiers in the target field set item by item, generating field indicator detection pairs. Each field indicator detection pair includes at least a data source identifier, a field identifier, a quality indicator identifier, and detection conditions, ensuring that each detection can be traced back to a specific data source, a specific field, and a specific quality indicator.

[0051] S202, for each field indicator detection pair, extract the detection input value from the data record, field metadata, or access status based on its associated detection conditions. If the detection condition corresponds to a field value, extract the field return value from the data record; if the detection condition corresponds to a field structure, extract the field type, field existence status, or field source status from the field metadata; if the detection condition corresponds to access availability, extract the access success status, return failure status, or inaccessible status from the access status. Before the detection input value enters the detection process, it undergoes format conversion according to the data format required by the detection conditions; detection input values ​​that cannot complete the format conversion are marked as undetectable and are not mistakenly judged as passing.

[0052] S203 compares the indicator detection input with the detection conditions. If the detection input value meets the detection conditions, the detection result is "pass"; if the detection input value does not meet the detection conditions, the detection result is "abnormal"; if the detection input value is missing, inconsistent in format, the access status cannot support the detection, or the field metadata is insufficient to complete the judgment, the detection result is "undetectable". The field quality status record must at least be associated with the data source identifier, field identifier, quality indicator identifier, and detection result, and can also carry the detection input source, detection trigger conditions, and detection time, ensuring a consistent data key for subsequent aggregation and deviation comparison.

[0053] S3: Based on the combination relationship of the target field set in the current data retrieval task, aggregate the field quality status records of each candidate data source and generate a data source quality status description corresponding to the current task. This step takes the field quality status record as input, the data source identifier as the first grouping condition, and the field identifier as the second positioning condition, and writes the detection results of multiple quality indicators belonging to the same field within the same candidate data source into the field status position.

[0054] S301. Group the field quality status records according to the data source identifier. Each group corresponds to a candidate data source. Within the candidate data source group, establish field status positions based on the field identifiers in the target field set, and write the quality indicator detection results associated with the same field to that field status position, forming a field indicator status sequence. Each detection result in the field indicator status sequence retains its corresponding quality indicator identifier and judgment priority to avoid losing the source of anomalies during the aggregation process.

[0055] S302, based on the judgment priority of quality indicators in the participating indicator set, sort the abnormal states, undetectable states, and pass states under the same field. If there is an abnormal state with higher priority for the same field, the field's satisfying state is determined to be unsatisfied, and the corresponding quality indicator is marked as an abnormal quality indicator; if there is no abnormal state but there is an undetectable state, the field's satisfying state is determined to be pending action, and the undetectable quality indicator is marked as an abnormal quality indicator; if the detection results of all quality indicators associated with the field meet the corresponding detection conditions, the field's satisfying state is determined to be satisfied. The field's satisfying state and abnormal quality indicators together form the field quality aggregation result.

[0056] S303, determine the quality aggregation results of the fields according to the field combination relationship of the target field set.

[0057] In one possible implementation, the required target fields in the current data retrieval task are used to form a set of required fields. The corresponding candidate data source The target fields that are in a satisfied state constitute the set of satisfied fields. Then the corresponding candidate data source Unsatisfied field set Determined according to the following formula:

[0058] ;

[0059] Corresponding candidate data sources Field coverage ratio for the current task Calculate according to the following formula:

[0060] ;

[0061] in, Indicates the number of fields in the collection. It is a dimensionless proportion and satisfies 0 ≤ ≤1.

[0062] For a field combination G with a common return constraint, a condition is met if and only if all required fields in G belong to the same category. When the combination of fields is satisfied, the state is determined; when at least one required field in G does not belong to the state. When this condition is met, the combination of fields is determined to be in a state where the combination is not satisfied, and those fields that do not belong to this state are excluded. The field is added to the set of unsatisfied fields. .

[0063] Field coverage ratio is used to determine the order of candidate data sources within the same data source status category, and does not replace the quality indicator detection results corresponding to each field.

[0064] For a group of fields with common return constraints, if any field that must be returned jointly is not satisfied, the group of fields is deemed to have an unsatisfactory combination under the corresponding candidate data source. For fields not affected by common return constraints, the field satisfaction status is used to determine whether the field can be provided by the candidate data source. After the determination is completed, the data source identifier, field satisfaction status, abnormal quality indicators, and unsatisfactory field identifiers are associated to generate a data source quality status description. This description is not an abstract evaluation of the global quality of the candidate data source, but rather a task-level status formed by the field combination relationships of the current data retrieval task.

[0065] S4, based on the data source quality status description, determines the call sequence and exception handling path for each candidate data source. The data source scheduling process first reads the data source quality status description of each candidate data source, and then determines the call order and handling actions based on the satisfaction status of the target field and the exception quality index corresponding to the exception field. The output of this step includes the call sequence and the exception handling path; the former provides the access order, and the latter provides the executable actions when there is an access exception, a missing field, or the field quality is not satisfied.

[0066] S401 categorizes candidate data sources into three status classes: all target fields satisfied, partially satisfied, and none satisfied. Candidate data sources that satisfy all target fields are prioritized for the preceding call position. Candidate data sources that partially satisfy target fields are prioritized based on the number of satisfied fields, anomaly quality indicators, and access status. Candidate data sources that fail to satisfy any target fields are only considered as subsequent calls when the anomaly handling path requires verification of the access failure reason or when there are no other alternative data sources. For candidate data sources within the same status class, the number of satisfied fields is compared first, then the anomaly quality indicators associated with the unsatisfied fields are compared to determine priority, and finally, the access status is considered to determine the candidate data source order, generating the call sequence.

[0067] When there are more than two candidate data sources within the same data source status category, the order of invocation is determined according to a preset multi-level sorting rule. The multi-level sorting rule includes at least the following steps: first, comparing the number of fields or field coverage ratios that satisfy the current task for each candidate data source; second, comparing the judgment priority of the abnormal quality indicators associated with the unsatisfied fields; and third, comparing the access availability status of the candidate data sources. If the order of candidate data sources still cannot be distinguished according to the aforementioned conditions, the order is determined according to the pre-configured stable order of data sources.

[0068] Multi-level sorting adopts a step-by-step comparison method, and only proceeds to the next level of comparison when the comparison results of the current level are the same. This avoids direct weighted summation of field quantities, quality index levels and access statuses with different dimensions, and ensures that inputs with the same quality status can obtain a definite sequence of candidate data source calls.

[0069] S402, for each candidate data source in the call sequence, identify the unmet field identifiers and corresponding abnormal quality indicator identifiers in their data source quality status descriptions. After the abnormal quality indicator identifiers are entered into the mapping relationship between abnormal quality indicators and handling actions, the corresponding handling action is determined. If the abnormal quality indicator points to a missing field but a subsequent candidate data source has a field that is satisfied, the handling action is determined to continue to retrieve the missing field; if the abnormal quality indicator points to a non-mandatory field and the task return field constraint allows for a default value, the handling action is determined to skip the field; if the abnormal quality indicator points to the current candidate data source being unavailable, but a subsequent candidate data source is accessible, the handling action is determined to switch data sources; if a mandatory field cannot be satisfied in any unaccessed candidate data source within the call sequence, the handling action is determined to terminate the call or mark the field as a call failure field. The handling actions are associated according to the order of the candidate data sources to form an abnormal handling path.

[0070] S403: When multiple actions simultaneously hit the same unmet field, the priority is determined by the field's mandatory status and abnormal quality indicators. Continuing to retrieve or terminating calls for mandatory fields takes precedence over skipping fields; switching data sources due to unavailability takes precedence over continuing requests to that data source; when multiple abnormal quality indicators exist for the same field, the highest priority abnormal quality indicator determines the primary action for that field in the abnormal handling path, while the remaining abnormal quality indicators are retained in the data source quality status description as the basis for handling, for comparison in subsequent deviation records.

[0071] S5 accesses candidate data sources sequentially according to the call sequence. If the target fields in a preceding candidate data source do not meet the requirements, it retrieves the corresponding data for the unmet fields from subsequent candidate data sources. This step takes the call sequence, exception handling path, and target field set as input, and the task result record and the list of fields to be retrieved as the running state. The list of fields to be retrieved initially contains all the standardized field identifiers in the target field set, and the task result record is initially empty.

[0072] S501: Based on the target field set, a list of fields to be retrieved is established, and the first candidate data source in the call sequence is determined as the current access data source. During the call execution process, a data access request is sent to the current access data source, carrying the identifier of the fields to be retrieved, so that the candidate data source only returns the fields that the current task still needs to retrieve. After the candidate data source returns the field return results and access status, the field return results enter the current call detection, and the access status enters the exception handling path judgment.

[0073] S502 compares the returned field results with the field satisfaction status in the data source quality status description. If the returned field results exist and meet the current quality requirements, the corresponding field is written to the task result record, and the corresponding field identifier is removed from the list of fields to be retrieved. If the field is not returned, the field is returned but the quality is not met, the field format cannot support the detection, or the access status indicates that the field cannot be retrieved, the corresponding field identifier is retained in the list of fields to be retrieved. When writing field data into the task result record, the source data source identifier of the field is also recorded, so that subsequent deviation comparisons can locate the actual data source providing the field.

[0074] S503, when the list of fields to be retrieved is not empty, select the next candidate data source according to the call sequence. During the call execution process, query the data source quality status descriptions corresponding to the candidate data sources that have not yet been accessed in the call sequence, filter the candidate data sources with at least one remaining field in a satisfied state, and generate a set of supplementary data sources; determine the next candidate data source from the set of supplementary data sources according to the order in the call sequence, and send a data access request carrying only the remaining field identifier to the next candidate data source.

[0075] In this embodiment, a data access request carrying only the remaining field identifiers means that the field projection parameters used to limit the range of fields returned by the candidate data source only contain the remaining field identifiers within the current list of fields to be retrieved. To ensure that data returned by different candidate data sources can correspond to the same target business record, the data access request can also carry the entity association key or query conditions used to locate the target record, as well as the authentication parameters, pagination parameters, or communication control parameters necessary to complete the access; the entity association key, query conditions, authentication parameters, pagination parameters, and communication control parameters are not part of the target field set requested to be returned.

[0076] When different candidate data sources use different entity identification methods, the entity association key corresponding to the current data retrieval task is converted into a query key that the next candidate data source can recognize through a preset entity identification mapping relationship. Only when the records returned by subsequent candidate data sources can establish a correspondence with the target business record in the current task result record through the entity association key or its mapping result, the supplementary fields are written into the task result record; returned fields that cannot establish a correspondence do not undergo cross-data source merging, and the corresponding fields are recorded as association failure status.

[0077] Fields returned by the next candidate data source that meet the quality requirements are written into the task result record, and the corresponding field identifiers are deleted from the list of fields to be retrieved, thus forming the field retrieval results.

[0078] S504, when no data source can be retrieved, the execution process handles the remaining fields according to the exception handling path. For remaining fields determined to be skipped by the exception handling path, the task result record retains the skipped status of the field; for remaining fields determined to be failed fields, the task result record retains the failed status of the field and the corresponding exception quality index; for remaining fields that trigger termination of the call, subsequent access to the corresponding field is stopped, and the data call result containing the retrieved fields, skipped fields, failed fields, and relevant data source identifiers is output.

[0079] When different target fields in the task result record come from different candidate data sources, for field combinations configured with time consistency constraints or version consistency constraints, the execution module is called to further read the data generation time, update time, or data version identifier corresponding to each field.

[0080] For a combination of fields configured with time consistency constraints, calculate the absolute value of the time difference between the timestamps of any two fields that need to be returned jointly (let this absolute value be...). ), and the corresponding allowable time difference when combined with this field. The comparisons are made, with each timestamp being compared to... Use the same time unit.

[0081] When the following conditions are met:

[0082] At that time, determine whether the corresponding field meets the time consistency constraint;

[0083] When the following conditions are met:

[0084] When this happens, the corresponding field combination will be marked as not satisfying the combination consistency requirement.

[0085] For field combinations configured with version consistency constraints, it is determined whether fields from different sources belong to data versions that can be used together, based on the data version identifier or business version mapping relationship. Fields that do not meet the corresponding version relationship do not directly form the final field combination, but enter the exception handling path to continue to retrieve or are marked as combination failure.

[0086] The output directly reflects the satisfaction status and reasons for non-satisfaction of the target field.

[0087] S6: Based on the target field set and participating indicator set, perform quality checks on the data records and access status returned by the candidate data sources for this actual access, generating an actual quality status record. Then, based on the data source identifier, field identifier, and quality indicator identifier, compare the actual quality status record with the field quality status record generated before the call, generating a deviation record. The inputs for this step include the data records returned by the candidate data sources for this actual access, the access status, the return time, and the field quality status record generated before the call. Deviation comparison is not solely based on the overall success of the data call task; rather, it compares the actual quality status record with the field quality status record before the call, assuming the data source identifier, field identifier, and quality indicator identifier are consistent.

[0088] S601: Obtain the data records, access status, and return time returned by the candidate data source for this actual access. Based on the target field set and participating indicator set, perform quality checks on the data records and access status corresponding to the pre-call quality checks, generating an actual quality status record. The actual quality status record must at least be associated with the data source identifier, field identifier, quality indicator identifier, actual test result, and return time. The actual quality status record is associated with the data source identifier, field identifier, quality indicator identifier, and actual test result, and retains the return time, enabling the status changes of the same candidate data source in different calling tasks to be distinguished by time sequence.

[0089] S602, according to the data source identifier, field identifier, and quality indicator identifier, match the actual quality status record with the field quality status record generated before the call item by item. For each successfully matched item, record the detection result before the call and the actual detection result respectively, and generate a quality status comparison item; when the detection result before the call and the actual detection result are the same, determine the deviation direction of the quality status comparison item as no deviation; when the detection result before the call and the actual detection result are different, determine the corresponding deviation direction according to the change direction of the detection result.

[0090] S603: When the pre-call detection result is in an abnormal or undetectable state, and the actual detection result is a pass state, the deviation direction is determined as a positive deviation; when the pre-call detection result is a pass state, and the actual detection result is in an abnormal or undetectable state, the deviation direction is determined as a negative deviation; when the pre-call detection result and the actual detection result are the same, the deviation direction is determined as no deviation; when the pre-call detection result is an undetectable state, and the actual detection result can be determined as a pass state or an abnormal state, the deviation type is determined as a detection reachability deviation. The deviation record must at least be associated with the data call task identifier, data source identifier, field identifier, quality indicator identifier, pre-call detection result, actual detection result, deviation type, deviation direction, and occurrence time.

[0091] S7, based on the deviation record, adjust the judgment priority, verification triggering conditions, or data source handling conditions of the corresponding quality indicators in subsequent data retrieval tasks. This step uses the deviation record as feedback input and the quality condition version as output. The feedback correction process does not change the already completed data retrieval results, but rather affects the detection order, detection triggering timing, or data source handling actions in subsequent data retrieval tasks by updating the quality conditions.

[0092] S701: Group deviation records according to data source identifier, field identifier, and quality indicator identifier, and statistically analyze the continuous occurrence of positive deviation, negative deviation, and no deviation within a preset observation period to generate a deviation stability record. The deviation stability record must include at least a grouping key, the continuous state of the deviation direction, the deviation type, and the chronological order of occurrence within the observation period. If positive and negative deviations alternate within the observation period, the grouping key is marked as unstable, and priority adjustment or relaxation of handling conditions is not directly triggered.

[0093] For grouping keys consisting of the same data source identifier, field identifier, and quality indicator identifier, an ordered deviation state sequence is established according to the deviation occurrence time. The observation window uses the K most recent actual test results or the most recent... The actual detection results within the time period, where K is a positive integer. This is a time length parameter; when using time length as the observation window, Use the same time unit as the time when the deviation occurs.

[0094] For an ordered deviation state sequence, count the number of records with consecutive deviation directions starting from the latest term and moving backward. When the following conditions are met:

[0095] ;

[0096] When (i.e., when) Reaching the preset number of consecutive triggers for the corresponding quality indicator When), the corresponding grouping key is determined as the condition for reaching a stable deviation trigger, where, It is a positive integer.

[0097] When positive and negative biases alternate within the observation window, and have not yet reached a certain level... When there is a continuous same-direction deviation, the corresponding grouping key will be kept in an unstable state, and no operation of lowering the judgment priority, raising the judgment priority, or relaxing the handling conditions will be performed.

[0098] For unbiased states, the number of consecutive unbiased records is counted, and the corresponding verification trigger frequency is maintained or reduced only after the corresponding stability condition is reached, thereby avoiding a single accidental return result from directly changing the quality conditions of subsequent data call tasks.

[0099] S702 compares the deviation stability record with preset adjustment conditions to determine the judgment priority adjustment item, verification trigger condition adjustment item, and data source handling condition adjustment item. When continuous negative deviations reach the preset adjustment conditions, the judgment priority of the corresponding quality indicator is increased, or the corresponding data source handling condition is adjusted to switch the data source earlier; when continuous positive deviations reach the preset adjustment conditions, the trigger frequency of the corresponding quality indicator can be reduced or the verification trigger condition can be adjusted, but the basic verification related to the required fields is still retained; when the no-deviation state is stable, the existing quality conditions are maintained. Mapping failure, no query results, duplicate records, missing return status, or missing match status are classified as abnormal return statuses and enter the handling condition adjustment items, without directly changing the quality status of the field itself;

[0100] To avoid frequent adjustments to quality conditions corresponding to the same data source, field, and quality indicator due to changes in positive and negative deviations within a short period of time, in one possible implementation, the effective time and effective task number of each effective quality condition adjustment are recorded, and adjustment retention conditions are configured.

[0101] Before the adjustment hold conditions are met, no opposite quality condition adjustments are performed on the same data source identifier, field identifier, and quality indicator identifier, but deviation records continue to be generated and saved. Adjustment hold conditions can be adopted after... A subsequent data retrieval task or after The time length is determined in at least one of the following ways, wherein It is a positive integer. This is the time length parameter.

[0102] Once the adjustment conditions are met, the decision to execute the next quality condition adjustment is made based on the deviation stability record in the latest observation window. This isolates the deviation detection process from the quality condition activation process in time, reducing frequent changes in the call sequence and anomaly handling paths caused by short-term data fluctuations.

[0103] S703 updates the judgment priority, verification trigger rules or exception handling rules of the corresponding quality indicators according to the quality condition adjustment list, and records the conditions before the update, the conditions after the update and the effective time, and generates the quality condition version for subsequent data call tasks.

[0104] Each quality condition version must record at least the version identifier, effective time, corresponding data source identifier, field identifier, quality indicator identifier, pre-adjustment judgment priority, post-adjustment judgment priority, pre-adjustment verification trigger condition, post-adjustment verification trigger condition, pre-adjustment data source handling condition, post-adjustment data source handling condition, and deviation stability record identifier that triggered this adjustment.

[0105] When a data retrieval task begins execution, it reads the quality condition version that has already taken effect at the start time of the task and maintains the same quality condition version during the target field parsing, quality inspection, data source scheduling, and call execution of the task. The quality condition version generated during task execution does not retroactively modify the currently executing data retrieval task, but takes effect from the start of subsequent data retrieval tasks.

[0106] By associating quality condition versions with data retrieval tasks, any data retrieval result can be traced back to the quality indicators used to determine priority, verification trigger conditions, and data source handling conditions based on the task identifier.

[0107] Subsequent data retrieval tasks read the latest effective quality condition version during the execution of S1 and S2, enabling the arrangement of participating indicator sets, the triggering of field indicator detection pairs, and the generation of anomaly handling paths to receive deviation feedback. This creates a closed loop between the actual call return result, deviation records, and the quality condition version, preventing candidate data sources from using pre-call descriptions inconsistent with the actual return status for extended periods.

[0108] Through S1 to S7, the data retrieval task first forms a target field set and a set of participating indicators, then generates field quality status records for candidate data sources, and aggregates them into a data source quality status description under the constraints of target field combination relationships. Subsequently, the call sequence and anomaly handling path jointly determine the field acquisition and supplementation process, and the actual call results are used to generate deviation records and correct quality conditions. This closed loop ensures that the quality judgment of each field, the access priority of each candidate data source, each anomaly handling action, and each feedback correction have traceable data sources and judgment criteria.

[0109] To verify the technical effect of retaining fields that meet the quality status and supplementing the remaining fields according to the field quality status in this embodiment, a comparative test can be conducted using the same set of data retrieval tasks, the same set of candidate data sources, and the same data snapshots of data sources.

[0110] The first comparison method adopts the whole source switching method, that is, if any of the required target fields of the current data source does not meet the quality requirements, the fields that have been returned by the current data source that meet the quality requirements are not retained, and all target fields of the current task are re-requested from the next candidate data source.

[0111] The second comparison method uses a fixed data source order, which means that the access is performed according to a pre-fixed data source order, without adjusting the data source call order based on the field quality status of the current task and the deviation record formed by the actual call.

[0112] In this embodiment, the calling sequence is determined according to the field quality status of the current task. When some fields of the preceding candidate data source meet the quality requirements, the fields that meet the quality requirements are written into the task result record, and field projection requests are only executed on the remaining fields in the list of fields to be obtained to the subsequent candidate data sources.

[0113] Record the actual number of data source accesses for each test method. Request the cumulative number of the target field Cumulative number of duplicate requests Application layer returns business data volume The processing time from the initiation of a single data retrieval task to the formation of the final retrieval result. Number of required target fields to meet quality requirements And the total number of required target fields for the task. .

[0114] Duplicate field request ratio Calculate according to the following formula:

[0115] ;

[0116] Required fields meet the ratio Calculate according to the following formula:

[0117]

[0118] in, and All are dimensionless proportions and their values ​​range from 0 to 1 (i.e., 0 ≤ 1). ≤1 and 0≤ ≤1), Using bytes as the unit of statistics Records are kept using a uniform unit of time.

[0119] During the comparative testing, each testing method used the same data retrieval task input, the same candidate data source data snapshots, and the same quality judgment conditions, and reported separately. , , , , and This allows for the simultaneous evaluation of data source access overhead, field duplication requests, data transfer volume, processing time, and task field satisfaction, avoiding the evaluation of access overhead reduction solely based on lowering the field satisfaction rate.

[0120] Example 2

[0121] Please see Figure 1 and Figure 5 This embodiment provides a system for quality analysis and optimization of data sources based on multi-dimensional indicators. The difference between this embodiment and Embodiment 1 lies in that this embodiment uses a module-based collaborative relationship within a server to implement the method flow of Embodiment 1. The key technical difference is that data source access, task parsing, indicator association, field quality analysis, quality status generation, data source scheduling, call execution, and feedback correction are each implemented in their respective modules. The processor executes program instructions from the memory to achieve data transfer and status updates between modules. The input-output connection is as follows: the data call task enters the task parsing module, the candidate data source returns content to the data source access module, and finally, the call execution module outputs the data call result, while the feedback correction module generates a quality condition version.

[0122] The data source access module establishes communication connections with each candidate data source, receiving data records, source identifiers, and access status from the candidate data sources. The data source access module organizes the source identifiers into a data source identifier, transmits the data records and access status to the field quality analysis module, and, when the execution module initiates a data access request, passes the field identifier to be retrieved to the currently accessed data source. The access status returned by the data source access module serves as a common input source for the field quality status record, the data source quality status description, and the deviation record.

[0123] The task parsing module receives data call tasks, parses the data structure description, return field constraints, and field combination relationships, and outputs the target field set. The standardized field identifiers, field types, mandatory states, and common return constraints in the target field set are passed to the indicator association module, the field quality analysis module, and the call execution module. The indicator association module stores the correspondence between fields and quality indicators, and queries the field quality indicator mapping table based on the target field set, outputting the participating indicator set. The quality indicator identifiers, judgment priorities, and detection trigger conditions in the participating indicator set are passed to the field quality analysis module, enabling the field quality analysis module to form field indicator detection pairs.

[0124] The field quality analysis module performs detection on target fields of candidate data sources according to the participating indicator set, generating field quality status records. This module receives data records, field metadata, and access status from the data source access module, as well as the target field set from the task parsing module and the participating indicator set from the indicator association module. The field quality analysis module extracts detection input values ​​according to the field indicator detection pairs, performs format conversion and detection condition comparison, and writes the status (abnormal status, or undetectable status) into the field quality status record. The quality status generation module receives the field quality status records and aggregates them into a data source quality status description according to the field combination relationships in the target field set.

[0125] The data source scheduling module reads the data source quality status description and categorizes candidate data sources into three status classes: all target fields are satisfied, some target fields are satisfied, and none of the target fields are satisfied. Within each class, it prioritizes and generates a call sequence based on the number of satisfied fields, abnormal quality indicators, and access status. The data source scheduling module also identifies unsatisfied field identifiers and abnormal quality indicator identifiers, generating an exception handling path through the mapping relationship between abnormal quality indicators and handling actions. The call sequence and exception handling path are then passed as output to the call execution module.

[0126] The execution module creates a list of fields to be retrieved based on the target field set, accesses candidate data sources according to the call sequence, and writes fields that meet the quality requirements into the task result record based on their status. The currently accessed data source only receives field identifiers from the list of fields to be retrieved; when a field's returned result does not meet the quality requirements or the field is not returned, the execution module retains the corresponding field identifier and selects the next available data source to continue accessing. When the list of fields to be retrieved is empty, candidate data sources have been accessed, or the exception handling path indicates termination of the call, the execution module outputs the data retrieval result.

[0127] The feedback correction module receives the data call results output by the execution module, as well as the data records, access status, and return time returned by the candidate data source actually accessed. It also reads the target field set, the set of participating indicators, and the field quality status records generated before the call. The feedback correction module performs quality checks on the actually returned data records and access status according to the target field set and the set of participating indicators, generating an actual quality status record. It then matches and compares the actual quality status record with the field quality status record before the call, item by item, according to the data source identifier, field identifier, and quality indicator identifier, generating a deviation record. The feedback correction module then groups and statistically analyzes the deviation records according to a preset observation window, generating a deviation stability record, and generates a quality condition adjustment list based on preset adjustment conditions. After the quality condition adjustment list is updated, a quality condition version is formed, which is read by the indicator association module, the field quality analysis module, and the data source scheduling module in subsequent data call tasks.

[0128] Each of the above modules is implemented by the processor in the server executing program instructions stored in memory. When executing program instructions, the processor sequentially completes task parsing, indicator association, field detection, quality status aggregation, data source scheduling, call execution, and feedback correction according to the data input / output relationships between modules. The memory stores the field quality indicator mapping table, field quality status records, data source quality status descriptions, call sequences, anomaly handling paths, deviation records, deviation stability records, quality condition adjustment lists, and quality condition versions. The system implementation corresponds to the method flow of Embodiment 1. Details of field parsing, quality indicator determination, anomaly handling, and deviation feedback that are not repeated can be executed using the same data objects and processing rules as in Embodiment 1.

[0129] Summary of Embodiments and Declaration of Protection: The above embodiments illustrate the feasible implementation methods of the present invention in a multi-candidate data source scenario for field-level quality analysis, task-level status aggregation, sequential calling, field supplementation, and feedback correction through the continuous transmission between the target field set, participating indicator set, field quality status record, data source quality status description, call sequence, anomaly handling path, deviation record, and quality condition version. The specific processes, module collaboration, field status, judgment order, anomaly handling, and feedback update relationships described in the embodiments are used to explain the technical solutions recorded in the present invention and should not limit the present invention to the specific implementation methods listed. Without departing from the scope of support of the technical solutions recorded in the present invention, any equivalent substitution, equivalent modification, equivalent combination, order adjustment, module corresponding replacement, equivalent transformation of field names, equivalent acceptance of execution subjects, or equivalent changes in carrier forms that can be conceived by those skilled in the art should fall within the protection scope of this patent; however, it shall not be extended to unclaimed topics, nor shall the substantive correspondence of technical objects be changed through name changes.

Claims

1. A method for quality analysis and optimization of data sources based on multi-dimensional indicators, characterized in that, include: Obtain the target field set and field combination relationship corresponding to the current data call task, and determine the set of participating indicators required for this task based on the preset field-quality indicator correspondence. For each candidate data source, according to the set of participating indicators, quality checks are performed on each target field in the set of target fields to generate field-level and quality indicator-level field quality status records; The field quality status record is associated with at least the data source identifier, field identifier, quality indicator identifier, and test result; Based on the combination relationship of the target field set described in the current task, the field quality status records of each candidate data source are aggregated to generate a data source quality status description for the current task. Based on the data source quality status description, and according to the field satisfaction status of each candidate data source for each target field and the abnormal quality indicators corresponding to the unsatisfied fields, the call sequence and exception handling path of each candidate data source are determined. Candidate data sources are accessed sequentially according to the call sequence. When a data source that has been accessed meets all the requirements of the target fields, the data is output. When some target fields do not meet the requirements, the fields that have met the quality requirements are retained, and the corresponding data of the unmet fields are obtained from the subsequent candidate data sources. The field projection parameter used to limit the range of returned fields in the data access request sent to the subsequent candidate data sources only contains the field identifier of the unmet fields. Based on the target field set and the participating indicator set determined before the call, perform quality checks on the data records and access status returned by the candidate data source in this actual access, and generate an actual quality status record; based on the data source identifier, field identifier and quality indicator identifier, compare the actual quality status record with the field quality status record generated before the call item by item, and record the detection results before the call and the actual detection results to generate a corresponding deviation record; Based on the continuous occurrence of the deviation records within the preset observation period, adjust the judgment priority, verification trigger conditions, or data source handling conditions of the corresponding quality indicators in subsequent data retrieval tasks.

2. The method for quality analysis and optimization of data sources based on multi-dimensional indicators according to claim 1, characterized in that, The process of obtaining the target field set and the process of determining the participating indicator set specifically include: The data structure description, return field constraints, and field combination relationships of the data retrieval task are parsed, and the field names, field types, and required states are extracted. The field names are converted into standardized field identifiers according to the unified field directory, duplicate standardized field identifiers are deleted, and the common return constraints between standardized field identifiers are recorded to obtain the target field set. Read the quality indicator identifiers corresponding to each standardized field identifier in the target field set from the field quality indicator mapping table, filter the basic quality indicator identifiers according to the mandatory status, supplement the associated quality indicator identifiers according to the common return constraints, merge and delete duplicate quality indicator identifiers, arrange the retained quality indicator identifiers according to the judgment priority, and determine the participating indicator set.

3. The method for quality analysis and optimization of data sources based on multi-dimensional indicator data as described in claim 1, characterized in that, The process of generating the field quality status record specifically includes: Obtain the data records, field metadata, and access status returned by each candidate data source for the target field set; match the quality indicator identifiers in the participating indicator set with the target fields to generate field indicator detection pairs. Based on the detection conditions associated with each of the field indicators, detection input values ​​are extracted from the data records, the field metadata, or the access status, and the detection input values ​​are converted into a data format consistent with the detection conditions to generate indicator detection inputs; The indicator detection input is compared with the detection conditions to determine the pass status, abnormal status, or undetectable status, and the data source identifier, the field identifier, and the quality indicator identifier are associated to generate the field quality status record.

4. The method for quality analysis and optimization of data sources based on multi-dimensional indicators according to claim 1, characterized in that, The process of generating the data source quality status description specifically includes: The field quality status records are grouped according to the data source identifier, the field status position is established according to the field identifier in the target field set, the quality indicator detection results associated with the same field are written into the corresponding field status position, and a field indicator status sequence is generated. Based on the judgment priority of the quality indicators in the set of participating indicators, the abnormal state, undetectable state and pass state are sorted to determine the field satisfaction state of each target field and generate the field quality aggregation result. The quality aggregation results of the fields are judged according to the combination relationship of the target field set, and the data source identifier, field satisfaction status, abnormal quality indicators and unsatisfied field identifiers are associated to generate a description of the data source quality status.

5. The method for quality analysis and optimization of data sources based on multi-dimensional indicators according to claim 1, characterized in that, The process of determining the call sequence and the exception handling path specifically includes: Obtain the data source quality status description corresponding to each candidate data source, classify the data source status categories according to whether all target fields are satisfied, some target fields are satisfied, and none target fields are satisfied, and within the same data source status category, determine the order of candidate data sources based on the number of fields satisfied in the current task, the judgment priority of abnormal quality indicators, and the access status, and generate the call sequence; For each candidate data source in the call sequence, identify the field identifier that does not meet the requirements and the corresponding abnormal quality indicator identifier. Based on the mapping relationship between the abnormal quality indicator identifier and the handling action, determine the handling action of continuing to supplement, skipping fields, switching data sources, or terminating the call, and generate the abnormal handling path by associating the handling actions according to the order of the candidate data sources.

6. The method for quality analysis and optimization of data sources based on multi-dimensional indicator data as described in claim 1, characterized in that, The process of accessing candidate data sources and supplementing unsatisfied fields according to the aforementioned call sequence specifically includes: A list of fields to be acquired is established based on the target field set. The first candidate data source in the call sequence is determined as the current access data source. A data access request carrying the identifier of the field to be acquired is sent to the current access data source to obtain the field return result and access status. The returned results of the fields are compared with the field satisfaction status in the quality status description of the data source. The returned fields that meet the quality requirements and their corresponding data source identifiers are written into the task result record. The corresponding field identifiers are deleted from the list of fields to be obtained, while the field identifiers of fields that are not returned or do not meet the quality requirements are retained. When the list of fields to be retrieved is not empty, the next candidate data source is selected according to the calling sequence and a data access request carrying only the identifiers of the remaining fields in the list of fields to be retrieved is sent until the list of fields to be retrieved is empty or the candidate data source has been accessed, and a data call result is generated.

7. The method for quality analysis and optimization of data sources based on multi-dimensional indicators according to claim 6, characterized in that, The process of selecting the next candidate data source according to the aforementioned call sequence specifically includes: Obtain the remaining field identifiers from the list of fields to be obtained, query the data source quality status descriptions corresponding to the candidate data sources that have not yet been accessed in the call sequence, filter at least one unaccessed candidate data source whose current remaining field is in a satisfied state, and generate a set of data sources that can be retrieved. The next candidate data source is determined from the set of available data sources according to the order in the call sequence. When there is no available data source, the remaining field identifier is determined as a skipped field identifier or a call failure field identifier according to the exception handling path, and subsequent access to the corresponding remaining field is stopped. Write the fields returned by the next candidate data source that meet the quality requirements into the task result record, update the list of fields to be obtained, and generate the field retrieval results.

8. The method for quality analysis and optimization of data sources based on multi-dimensional indicators according to claim 1, characterized in that, The process of generating the deviation record specifically includes: Obtain the data records, access status, and return time returned after the candidate data source is actually called. Perform quality checks on the actual returned data records and access status according to the target field set and the participation indicator set used before the call. Associate the data source identifier, field identifier, and quality indicator identifier to generate an actual quality status record. According to the data source identifier, field identifier, and quality indicator identifier, the actual quality status record is matched item by item with the field quality status record generated before the call, the matching items that are different between the detection results before the call and the actual detection results are extracted, and the detection results before the call and the actual detection results are recorded respectively to generate quality status difference items; Based on the quality status difference item, determine the deviation type, deviation direction and occurrence time, and associate it with the corresponding data retrieval task identifier to generate the deviation record.

9. The method for quality analysis and optimization of data sources based on multi-dimensional indicators according to claim 1, characterized in that, The process of determining priorities, verifying trigger conditions, and adjusting data source handling conditions specifically includes: The deviation records are grouped according to the data source identifier, field identifier, and quality indicator identifier. The continuous occurrence of positive deviation, negative deviation, and no deviation is statistically analyzed within a preset observation period according to the occurrence time of the deviation records to generate a deviation stability record. When positive deviation and negative deviation occur alternately, the corresponding deviation is determined to be an unstable state. The deviation stability record is compared with the preset adjustment conditions to determine the priority adjustment item, the verification trigger condition adjustment item, and the data source disposal condition adjustment item, and a quality condition adjustment list is generated. Update the judgment priority, verification trigger rules or anomaly handling rules of the corresponding quality indicators according to the quality condition adjustment list, and record the conditions before update, the conditions after update and the effective time to generate the quality condition version for subsequent data call tasks.

10. A system for quality analysis and optimization based on multi-dimensional indicator data sources, characterized in that, The method for quality analysis and optimization based on multi-dimensional indicator data sources according to any one of claims 1-9 includes the following modules: The data source access module establishes communication connections with each candidate data source to obtain the data records, source identifiers, and access status of the candidate data sources. The task parsing module is used to parse data to call tasks and obtain the set of target fields required by the task. The indicator association module stores the correspondence between fields and quality indicators, and is used to determine the set of participating indicators for this task based on the target field set. The field quality analysis module is used to detect the target fields of the candidate data source according to the set of participating indicators and generate field quality status records. The quality status generation module is used to aggregate the quality status records of each field according to the combination relationship of the target field set of the current task, and generate a data source quality status description corresponding to the current task. The data source scheduling module is used to determine the calling sequence and exception handling path of the candidate data source based on the description of the data source quality status. The execution module is invoked to access candidate data sources according to the invocation sequence, retain fields that meet the quality requirements in the preceding data sources, and obtain data of the unmet fields from subsequent data sources when some target fields do not meet the requirements. The feedback correction module is used to generate an actual quality status record corresponding to the candidate data source actually accessed this time. According to the data source identifier, field identifier and quality indicator identifier, the actual quality status record is compared with the field quality status record generated before the call item by item to generate a deviation record, and the quality conditions of subsequent data call tasks are adjusted according to the deviation record. The data source access module, task parsing module, indicator association module, field quality analysis module, quality status generation module, data source scheduling module, call execution module, and feedback correction module are implemented by the processor in the server executing program instructions stored in the memory.

Images