Data quality inspection method and device

Abstract

The application discloses a data quality inspection method and device, relates to the technical field of data processing, and a master control agent in a multi-agent collaborative system acquires metadata of a data source to be inspected, performs semantic matching on a data table name in the metadata, a field name set in the metadata and a preset business label, and an operator label of each preset template in a preset SOP flow template library, determines a target template, so that the target template can be accurately matched in a case where a data source name is not standardized, all data sources of the same type can automatically reuse the target template through semantic matching, templateization and reusability of a quality inspection flow are realized, parameter instantiation is performed on the target template according to the metadata, a quality inspection workflow is generated, that is, structural differences of specific data sources are automatically adapted through parameter instantiation, reconfiguration or copying and modification are not needed, and problems such as parameter filling errors, omissions or format mismatches caused by negligence in manual operation are avoided, so that quality inspection task efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data quality inspection method and device. Background Technology

[0002] Currently, quality control of data sources typically involves developers writing SQL (Structured Query Language) queries or Python / Java scripts to validate specific data tables based on business needs. Alternatively, data governance platforms often provide a visual rule configuration interface that supports simple process orchestration, allowing users to execute multiple rules in sequence.

[0003] However, all of the above methods heavily rely on manual understanding of data metadata. When faced with hundreds or thousands of data tables or frequently changing data sources, the process of manually matching rules and configuring parameters one by one is extremely time-consuming, resulting in long deployment cycles for quality inspection tasks and failing to meet the needs of agile development. Furthermore, when business scenarios change, the process orchestration based on fixed linear logic lacks a deep understanding of business tags and operator semantics, leading to poor quality inspection results.

[0004] Therefore, there is a technical problem of low efficiency in quality inspection tasks in related technologies. Summary of the Invention

[0005] The main purpose of this application is to provide a data quality inspection method and equipment, which aims to solve the technical problem of low efficiency in quality inspection tasks.

[0006] To achieve the above objectives, this application proposes a data quality inspection method applied to the master agent in a multi-agent cooperative system. The data quality inspection method includes: Obtain the metadata of the data source to be inspected, wherein the metadata includes the data table name and the set of field names; The data table name, the set of field names, and the preset business tags are semantically matched with the operator tags of each preset template in the preset SOP process template library to determine the target template. The target template is instantiated with parameters based on the metadata to generate a quality inspection workflow.

[0007] In one embodiment, the metadata further includes the number of field data types. Each template in the preset SOP process template library predefines multiple atomic operators, the execution order of each atomic operator, the dependencies between each atomic operator, and an exception handling strategy. The step of instantiating the target template with parameters based on the metadata to generate a quality inspection workflow includes: Obtain the parameter configuration interface for each atomic operator in the target template; Based on the set of field names and the number of types of field data types, assign values ​​to the parameter configuration interfaces of each atomic operator; Based on the execution order, dependencies, and exception handling strategy, the assigned atomic operators are arranged into the quality inspection workflow.

[0008] In one embodiment, the metadata further includes the number of field data types, the number of fields, and the field nesting hierarchy. After the step of instantiating the target template based on the metadata to generate the quality inspection workflow, the method further includes: Obtain the number of records in the data source to be inspected, the idle time of the MapReduce engine, and the average load of the agent pool; Based on the number of fields, the number of types of field data, the nesting hierarchy of fields, and the number of records, determine the data scale, data complexity, and data stability of the data source; The current resource availability is determined based on the MapReduce engine idle time and the average load of the agent pool; Based on the data size, data complexity, data stability of the data source and the current resource availability, calculate the comprehensive decision score of the quality inspection task corresponding to the quality inspection workflow; Based on the comprehensive decision score, a quality inspection strategy is determined.

[0009] In one embodiment, the quality inspection strategy includes a process-intensive quality inspection strategy and a data-intensive quality inspection strategy. Each template in the preset SOP process template library predefines multiple atomic operators, the execution order of each atomic operator, and the dependencies between each atomic operator. The step of determining the quality inspection strategy based on the comprehensive decision score includes: If the overall decision score is less than a preset decision threshold and the idle degree of the MapReduce engine is less than a preset idle degree threshold, then the quality inspection strategy is determined to be a process-intensive quality inspection strategy. The process-intensive quality inspection strategy is to sequentially call the quality inspection agent corresponding to each atomic operator to execute the quality inspection task according to the execution order and the dependency relationship. If the overall decision score is greater than or equal to the preset decision threshold, or the MapReduce engine idle time is greater than or equal to the preset idle time threshold, then the quality inspection strategy is determined to be a data-intensive quality inspection strategy. The data-intensive quality inspection strategy is to split the quality inspection task into multiple parallel subtasks and perform distributed processing on the multiple parallel subtasks through the MapReduce engine.

[0010] In one embodiment, the step of distributing the processing of the multiple parallel subtasks using the MapReduce engine includes: The MapReduce engine divides the data source to be inspected into multiple logical shards, assigns multiple parallel Mapper tasks to each logical shard, and outputs intermediate key-value pairs respectively. Each Mapper task executes different types of quality inspection operators, including statistical detection, density detection, semantic detection, and algorithm detection. The intermediate key-value pairs use the shard identifier as the key and the detection result containing the anomaly score and anomaly label as the value. The intermediate key-value pairs are partitioned and sorted, and the various quality inspection results belonging to the same logical partition are aggregated into the same Reducer node to generate an input dataset organized by partition. The Reducer task is invoked to read the input dataset, perform weighted calculation or logical fusion of multiple anomaly scores under the same logical segment, generate a comprehensive quality inspection conclusion for the same logical segment, summarize the comprehensive quality inspection conclusions of all segments, and output the final quality inspection report.

[0011] In one embodiment, each template in the preset SOP process template library predefines multiple atomic operators. During the execution of each atomic operator in the quality inspection workflow, when any of the atomic operators needs to call a large language model, the data quality inspection method further includes: In response to the current model call request, obtain the amount of currently available resources in the rate-limited resource pool; If the current available resources meet the resource requirements of the current model call request, then the available resources corresponding to the resource requirements are used, and candidate models that meet the availability conditions are selected from the preset model set, and a qualified model set is constructed based on the selection results. Determine the target model from the set of qualified models; Obtain the input context corresponding to the current model call request, and determine whether the input context meets the context processing conditions of the target model; If the context processing conditions are not met, the input context is compressed to obtain a compressed context, and the compressed context is checked again to see if it meets the context processing conditions. If the conditions are met, the atomic operator is executed through the target model.

[0012] In one embodiment, the rate-limiting resource pool includes a first token bucket for controlling the request rate and a second token bucket for controlling the token consumption rate. The currently available resource quantity includes the current first token count of the first token bucket and the current second token count of the second token bucket. The resource requirement includes a preset single request consumption amount and an estimated token consumption amount of the current model call request. The step of occupying the available resources corresponding to the resource requirement if the current available resources meet the resource requirement of the current model call request includes: If the current first token count is greater than or equal to the preset single request consumption, and the current second token count is greater than or equal to the estimated token consumption, then it is determined that the current available resource quantity meets the resource requirement, and the tokens of the first token count corresponding to the preset single request consumption are consumed, and the tokens of the second token count corresponding to the estimated token consumption are consumed.

[0013] In one embodiment, the step of determining whether the input context meets the context processing conditions of the target model includes: The sum of the current word length of the input context and the estimated output length of the target model is calculated to obtain the estimated total length. Obtain the maximum context window length preset for the target model; If the estimated total length is greater than the maximum context window length, then the input context is determined to be inconsistent with the context processing conditions.

[0014] In one embodiment, the step of semantically matching the data table name, the set of field names, and preset business tags with the operator tags of each preset template in the preset SOP process template library to determine the target template includes: The data table name and the set of field names are processed into text vectors to obtain the first feature vector; The operator labels of each preset template are processed into text vectors to obtain the second feature vector corresponding to each preset template; Calculate the similarity between the first feature vector and each of the second feature vectors, and determine the preset template corresponding to the second feature vector with the highest similarity as the target template.

[0015] In addition, to achieve the above objectives, this application also proposes a data quality inspection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data quality inspection method as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: In this embodiment, the master control agent in the multi-agent collaborative system acquires metadata of the data source to be inspected, including data table names and a set of field names. The data table names, the set of field names, and preset business tags are semantically matched with the operator tags of each preset template in the preset SOP (Standard Operating Procedure) process template library to determine the target template. This ensures accurate matching of the applicable target template even when the data source naming is not standardized. All data sources of the same type can automatically reuse the target template through semantic matching, realizing the templating and reusability of the quality inspection process. Furthermore, the target template is parameterized based on the metadata to generate a quality inspection workflow. That is, parameter instantiation automatically adapts to the structural differences of specific data sources without reconfiguration or copying, avoiding problems such as parameter errors, omissions, or format mismatches caused by negligence in manual operation; thereby improving the efficiency of quality inspection tasks. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the data quality inspection method of this application in Embodiment 1. Figure 2 This is a first architecture diagram of the multi-agent cooperative system provided in Embodiment 1 of this application; Figure 3 This is a second architecture diagram of the multi-agent cooperative system provided in Embodiment 1 of this application; Figure 4 This is a schematic diagram of the first scenario of the multi-agent cooperative system provided in Embodiment 2 of this application; Figure 5 This is a schematic diagram of a second scenario of the multi-agent cooperative system provided in Embodiment 2 of this application; Figure 6 This is a schematic diagram of a third scenario of the multi-agent cooperative system provided in Embodiment 2 of this application; Figure 7 This is a schematic diagram of the hardware operating environment involved in the data quality inspection method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device, data quality inspection device, etc. that can realize the above functions.

[0024] Based on this, embodiments of this application provide a data quality inspection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data quality inspection method of this application.

[0025] In this embodiment, the data quality inspection method includes steps S10 to S30: Step S10: Obtain the metadata of the data source to be inspected, wherein the metadata includes the data table name and the set of field names; Step S20: Semantically match the data table name, the set of field names, and the preset business tags with the operator tags of each preset template in the preset SOP process template library to determine the target template; Step S30: Instantiate the target template with parameters based on the metadata to generate a quality inspection workflow.

[0026] It should be noted that the data quality inspection method in this embodiment can be applied to the master control agent in a multi-agent cooperative system, referring to... Figure 2 and Figure 3 The logical flow of the multi-agent collaborative system is divided into an intelligent engine layer, a multi-agent collaborative layer, a standard operating procedure (SOP) layer, and an application layer from bottom to top.

[0027] The underlying intelligent engine layer includes an LLM (Large Language Model) intelligent scheduling module and a tool execution module. The LLM intelligent scheduling module is configured with multiple parallel-deployed large language model instances (e.g., the Qwen series models) to provide context understanding, complex logical reasoning, and enhanced retrieval generation capabilities. The tool execution module encapsulates an SQL interpreter, code executor, and knowledge base interface, serving as an atomic tool for upper-layer agents to call, used to translate natural language instructions into concrete data operations.

[0028] The multi-agent collaboration layer comprises an orchestration master agent and an agent execution layer. The orchestration master agent, acting as the task scheduling hub, is responsible for task parsing, state management, resource scheduling, and SOP adaptive orchestration. It performs decision-making reasoning by calling the underlying LLM intelligent scheduling module and distributes instructions downwards. The agent execution layer comprises a multi-dimensional detection agent cluster and a repair agent. The multi-dimensional detection agent cluster includes statistical detection agents, density detection agents, algorithm detection agents, and semantic detection agents, each responsible for executing quality verification tasks in different dimensions. The repair agent generates repair suggestions based on the detection results.

[0029] The upper-level SOP standardization layer establishes a standardized process control mechanism. This SOP standardization layer includes an SOP process template library and an exception handling strategy library. The SOP process template library defines standardized work processes composed of multiple atomic operators, providing a basis for orchestrating the master control agent. The exception handling strategy library defines alarm thresholds, exception level standards, and intelligent attribution logic for various types of data anomalies.

[0030] The application layer is used to output standardized quality inspection results. Based on the execution results of the lower-layer intelligent agents and the policy judgment of the SOP layer, the application layer generates multi-dimensional delivery results, including quality reports, anomaly lists, intelligent alarms, anomaly work orders, anomaly repair solutions, and optimization suggestions.

[0031] In this embodiment, when the main control agent receives an access request from the data source to be inspected, it can first perform a structural exploration of the data source. By reading the system directory table or metadata storage area of ​​the data source, it can obtain the identification information and internal structure description information of the data source. The identification information includes the data table name, which is used to uniquely identify the current data set to be inspected; the internal structure description information includes a set of field names, which is used to record the names of all fields in the data table.

[0032] Furthermore, the master control agent can perform text concatenation or combination processing on the data table name and the field names in the field name set to obtain a set of text vectors describing the structural characteristics of the current data source; at the same time, the preset business tags are incorporated into the composition of the text vectors as supplementary text describing the business semantics of the current data source.

[0033] For each preset template in the preset SOP process template library, the main control agent obtains the operator tags corresponding to each atomic operator in the template (the operator tags are pre-labeled with the data features or business scenarios to which the atomic operator is applicable), merges or aggregates all operator tags under the same preset template, and obtains the overall semantic description vector of the preset template.

[0034] The controlling agent performs semantic matching between the overall semantic description vector and the operator labels of each preset template in the preset SOP process template library to determine the target template.

[0035] The master control agent calculates the similarity between the semantic description vector of the current data source and the overall semantic description vector of each preset template, sorts the preset templates according to the similarity, and determines the preset template with the highest similarity as the target template.

[0036] In one feasible implementation, the method for determining the target template by semantically matching the data table name, the set of field names, and the preset business tags with the operator tags of each preset template in the preset SOP process template library can be as follows: The data table name and the set of field names are processed into text vectors to obtain a first feature vector; the operator labels of each preset template are processed into text vectors to obtain a second feature vector corresponding to each preset template; the similarity between the first feature vector and each second feature vector is calculated, and the preset template corresponding to the second feature vector with the highest similarity is determined as the target template.

[0037] The controlling agent combines the data table name and the field names from the set of field names to form a text sequence describing the current data source structure. This text sequence is then converted into a numerical feature vector using a pre-defined vectorization algorithm. This feature vector is the first feature vector, used to characterize the semantic features of the current data source. The dimension of the first feature vector is determined by the dimension of the vector space model used by the vectorization algorithm, and the value of each dimension reflects the weight of the corresponding semantic dimension.

[0038] Each preset template in the preset SOP process template library contains the definition of one or more atomic operators. Each atomic operator is pre-labeled with an operator tag to describe the data characteristics or business scenarios to which the operator is applicable. The master control agent reads all operator tags in the preset template to be processed, combines these operator tags to form a text sequence describing the overall applicable scenario of the template, and converts this text sequence into a numerical feature vector. This feature vector is the second feature vector corresponding to the preset template. The master control agent performs the above processing on each preset template in the preset SOP process template library to obtain the second feature vector corresponding to each preset template.

[0039] For each preset template in the preset SOP process template library, the master control agent calls a preset similarity calculation function, taking the first feature vector and the second feature vector corresponding to the template as input, and calculates the similarity value between them. This value reflects the degree of matching between the semantic features of the current data source and the applicable scenario of the template. After calculating the similarity values ​​for all preset templates, the master control agent compares all similarity values, finds the maximum value, and determines the preset template corresponding to the maximum value as the target template of the current data source.

[0040] By comparing the first feature vector with each of the second feature vectors one by one and determining the target template based on the comparison results, the template selection process is based entirely on numerical mathematical calculations, eliminating the need for manual reading and judgment of the template's applicability conditions, thus automating the template selection process.

[0041] Furthermore, the target template defines a general quality inspection process framework applicable to a certain type of data source. This general quality inspection process framework contains abstract definitions of multiple quality inspection steps, and each abstract definition has several parameter placeholders set to receive the actual structural information of the specific data source. The master control agent fills the specific parameter values ​​(at least including the actual names of each field in the field name set) into the corresponding parameter placeholders in the target template, transforming the target template from an abstract process framework into an executable process definition for the current data source. This executable process definition is the quality inspection workflow.

[0042] The specific field names and other information in the metadata can be filled into the abstract placeholders of the target template to automatically generate the quality inspection workflow, thus automating the generation of the quality inspection process without manual intervention.

[0043] In one feasible implementation, the metadata further includes the number of field data types. Each template in the preset SOP process template library predefines multiple atomic operators, the execution order of each atomic operator, the dependencies between each atomic operator, and an exception handling strategy. The method of instantiating the target template with parameters based on the metadata to generate the quality inspection workflow can be: Obtain the parameter configuration interface of each atomic operator in the target template; assign values ​​to the parameter configuration interface of each atomic operator according to the set of field names and the number of types of field data types; arrange the assigned atomic operators into the quality inspection workflow according to the execution order, dependency relationship and exception handling strategy.

[0044] The target template defines a quality inspection process framework applicable to the business type of the current data source. This framework is composed of multiple atomic operators combined according to a specific execution order and dependencies. Each atomic operator has a parameter configuration interface set during definition to receive specific parameter values ​​passed from external sources. The main control agent identifies all atomic operators contained in the target template by parsing its structured definition and extracts the parameter configuration interface information of each atomic operator one by one. The parameter configuration interface information includes attributes such as interface name, interface data type, whether it is required, and default value.

[0045] Because atomic operators adopt a standardized parameter configuration interface design, the master control agent can obtain the list of parameters that all operators need to be configured in a unified way without manually consulting the operator documentation or understanding the internal implementation logic of the operator. This provides a standardized operation entry point for subsequent automated parameter assignment.

[0046] The master control agent traverses each atomic operator in the target template. For each parameter configuration interface of the current atomic operator, it searches for a matching field name from the field name set as the parameter value based on the definition of the interface (including the interface name, expected data type and business meaning description), or determines the specific value configuration of the parameter based on the number of field data types.

[0047] For example, when the parameter configuration interface of an atomic operator is named "Field to be Validated", the expected data type is string, and the description is "Target field name to be inspected", the master control agent semantically matches the most relevant field name in the set of field names and assigns it a value. After the master control agent completes the above assignment operation for all parameter configuration interfaces of all atomic operators, each atomic operator is transformed from an abstract general operator into an executable operator instance configured for the specific structure of the current data source.

[0048] Based on the execution order, dependencies, and exception handling strategies, and combined with the atomic operator instances whose parameters have been assigned, a complete and executable quality inspection workflow is constructed. This workflow includes not only the execution logic and parameter configuration of each atomic operator, but also the flow control relationships between atomic operators (sequential execution, parallel execution, or conditional triggering) and exception handling strategies, ensuring that the quality inspection process runs correctly according to the expected logic and handles any possible exceptions during actual execution.

[0049] Among them, the types of atomic operators include format validation, null value rate detection, range checking, logical verification, statistical detection, and correlation verification.

[0050] The format validation operator is configured to receive the field value of the field to be checked as input, perform matching and judgment through preset regular expressions or validation rules, and output a boolean pass or error code to verify the standardization of the data.

[0051] The null value rate detection operator is configured to receive a complete list of fields as input, perform non-null statistics on the specified set of fields, and output the percentage of null values ​​to assess the integrity of the data.

[0052] The range check operator is configured to receive field values ​​and preset thresholds or enumeration sets as input, determine whether the value falls within the specified range or belongs to a valid enumeration set, and output the judgment result of whether it is within the range.

[0053] The logical verification operator is configured to receive multiple field values ​​with business dependencies as input, perform cross-field consistency comparisons according to predefined business logic rules, and output the verification result of whether the logical relationship is satisfied.

[0054] The statistical detection operator is configured to receive a continuous numerical column as input, calculate its mean, variance, quantiles or identify outliers and other statistical features, and output specific statistical indicators or anomaly markers to detect anomalies at the data distribution level.

[0055] The association verification operator is configured to receive data from the main table and the relationships between tables as input, perform cross-table consistency comparisons or matching checks, and output a set of inconsistent records to ensure the integrity of references between multiple tables.

[0056] In one feasible implementation, the configuration structure of specific operators in the preset SOP process template library can be defined using serialized data in key-value pair format. This configuration structure clarifies the identity, execution logic, and parameter constraints of each atomic operator. For business semantic verification scenarios, operator objects containing operator identifiers are defined. These operator objects describe specific business rules through description fields and encapsulate the dynamic parameters required for execution through parameter fields. This allows the operators to not only possess general verification capabilities but also adapt to specific business contexts through parameter injection. For data standardization and integrity verification scenarios, pattern matching operators based on regular expressions or mutual exclusion verification operators based on multi-field constraints can be used.

[0057] This embodiment establishes a standardized atomic operator system covering six categories: format verification, null value rate detection, range checking, logical verification, statistical detection, and correlation verification. This encapsulates various common quality inspection actions into independent units with standardized input, output, and parameter configuration interfaces. Because the input and output formats of each atomic operator are unified and their execution logic is clear, the main control agent can combine and orchestrate these atomic operators through standardized interfaces without needing to understand their internal implementation details. This provides reusable basic components for the construction and adaptive orchestration of SOP process templates. When a new quality inspection process needs to be built, the main control agent only needs to select appropriate operators from the atomic operator system and configure parameters according to business requirements, without needing to redevelop the quality inspection logic code, thus significantly reducing the construction cost of the quality inspection process.

[0058] Furthermore, the master control agent can accurately locate and automatically fill in the parameter items that need to be assigned values ​​based on the field type identifier during parameter instantiation, avoiding parameter parsing errors or assignment omissions caused by unclear configuration structures. The standardized configuration structure eliminates the need for the master control agent to write separate parsing logic for each operator. It can read the configuration information of all operators and extract the parameter configuration interfaces of each atomic operator through a unified configuration parser, thereby further reducing the implementation complexity of the system.

[0059] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. In addition, the metadata also includes the number of field data types, the number of fields, and the field nesting hierarchy, as referred to... Figure 4 After the step of instantiating the target template with parameters based on the metadata and generating the quality inspection workflow, the following can also be done: Obtain the number of records, MapReduce engine idle time, and average agent pool load of the data source to be inspected; determine the data size, data complexity, and data stability of the data source based on the number of fields, the number of field data types, the field nesting hierarchy, and the number of records; determine the current resource availability based on the MapReduce engine idle time and the average agent pool load; calculate the comprehensive decision score for the quality inspection task corresponding to the quality inspection workflow based on the data size, data complexity, data stability, and current resource availability of the data source; and determine the quality inspection strategy based on the comprehensive decision score.

[0060] It should be noted that the MapReduce engine is deployed in the intelligent engine layer of the multi-agent collaborative system and communicates and interacts with the master intelligent agent through an application programming interface.

[0061] The master agent obtains the number of records in the data source to be inspected, the idle time of the MapReduce engine, and the average load of the agent pool. Specifically, the master agent can read the number of records (denoted as R) in the data source's metadata, which reflects the size of the current dataset to be processed. Simultaneously, the master agent can send a query request to the system's resource monitoring module to obtain the current idle time of the MapReduce computing engine (denoted as U_MR) and the average load of the agent pool in the multi-agent collaborative system (denoted as U_pool).

[0062] The controlling agent determines the data size, data complexity, and data stability of the data source based on the number of fields (denoted as F), the number of field data types (denoted as T_type), the field nesting hierarchy (denoted as L), and the number of records R. Specifically, the controlling agent uses the number of records R as a quantified value of the data size, uses the number of fields F, the field nesting hierarchy L, and the number of field data types T_type to jointly characterize the data complexity, and uses the degree of recent structural changes in the data source (i.e., the degree of recent drift Δ_drift) as a quantified value of data stability.

[0063] The recent drift level is calculated by comparing the metadata of the current data source with the historical data structure snapshot retained during the last quality inspection. If the current data source has historical records, the recent drift level Δ_drift is calculated as the proportion of the number of fields added or removed and the number of data type changes to the total number of fields. If the current data source is being accessed for the first time and has no historical records, Δ_drift is set to the default value, for example, 0.5.

[0064] The master agent determines the current resource availability based on the MapReduce engine idle time (U_MR) and the agent pool average load (U_pool). The MapReduce engine idle time (U_MR) reflects the remaining parallel processing capacity of the distributed computing framework; a higher idle time indicates a stronger ability to execute data-intensive tasks. The agent pool average load (U_pool) reflects the activity level of the agent cluster; a lower average load indicates a faster response time for the master agent to directly schedule specialized agents to execute process-intensive tasks. Together, the MapReduce engine idle time and the agent pool average load reflect the system's resource availability.

[0065] The controlling agent calculates the comprehensive decision score for the quality inspection task corresponding to the quality inspection workflow based on the data size R, data complexity, data stability, and current resource availability of the data source. In actual calculation, the controlling agent first normalizes each original indicator, mapping the number of records R to the 0-1 range; maps the field nesting level L to the 0-1 range; and calculates the comprehensive decision score according to a weighted fusion formula.

[0066] The calculated comprehensive decision score ranges from 0 to 1. The closer the value is to 1, the more data-intensive the current task is, and the closer the value is to 0, the more process-intensive the task is.

[0067] In one feasible implementation, the quality inspection strategy includes a process-intensive quality inspection strategy and a data-intensive quality inspection strategy, and the method for determining the quality inspection strategy based on the comprehensive decision score can be: If the overall decision score is less than a preset decision threshold and the idle degree of the MapReduce engine is less than a preset idle degree threshold, then the quality inspection strategy is determined to be a process-intensive quality inspection strategy. The process-intensive quality inspection strategy is to sequentially call the quality inspection agent corresponding to each atomic operator to execute the quality inspection task according to the execution order and the dependency relationship. If the overall decision score is greater than or equal to the preset decision threshold, or the MapReduce engine idle time is greater than or equal to the preset idle time threshold, then the quality inspection strategy is determined to be a data-intensive quality inspection strategy. The data-intensive quality inspection strategy is to split the quality inspection task into multiple parallel subtasks and perform distributed processing on the multiple parallel subtasks through the MapReduce engine.

[0068] This embodiment abstracts quality inspection tasks into two types of computing modes: one is process-intensive tasks, represented by dynamic rule adaptive closed loops, characterized by multiple process branches and complex logical judgments, but the amount of data processed in a single run is relatively small, requiring high logical reasoning ability of the agent while having low requirements for data throughput; the other is data-intensive tasks, represented by multi-dimensional parallel detection, characterized by multi-dimensional parallel scanning and calculation of massive amounts of data, large data throughput, and high requirements for the parallel processing capability of computing resources.

[0069] Reference Figure 4 The master control agent has a built-in decision-maker. Upon receiving a quality inspection task, the decision-maker extracts a four-dimensional feature vector of the task, including: data size (number of records R), data complexity (number of fields F, field nesting level L, number of data types T_type), data stability (recent drift Δ_drift, value range 0~1), and resource availability (MapReduce engine idle time U_MR, average agent pool load U_pool). The decision-maker calculates a comprehensive decision score based on the above four-dimensional feature vector and determines whether the current task should use a process-intensive or data-intensive quality inspection strategy.

[0070] The main control agent computes data complexity metric Ccomplex. This data complexity metric is used to comprehensively characterize the structural complexity of the current data source. The more fields there are (baseline 50 fields), the deeper the nesting level (baseline 3 levels), and the more diverse the data types (baseline 10 types), the higher the data complexity metric.

[0071] The formula for calculating the data complexity Ccomplex can be:

[0072] Specifically, the formula for calculating the comprehensive decision score can be: It is understandable that the larger the product of the number of records R and the data complexity Ccomplex, the more obvious the data-intensive characteristics of the task, the higher the process-intensive adaptation score, and the less suitable it is for direct scheduling by the master (the larger the Score_strat1, the lower the relative advantage of direct scheduling by the master). When the recent drift Δ_drift is larger, the data stability is worse, and direct scheduling by the master needs to frequently process rule adaptation, thus reducing the adaptation score. When the data volume cost increases, the adaptation score decreases accordingly. Among them, D_max is the maximum tolerable execution delay of a single atomic task under direct scheduling by the master agent. Only when D_max (the maximum tolerable execution delay of a single atomic task under direct scheduling by the master) is less than 5 seconds, the value of the indicator function D_max is 1; otherwise, it is 0. When D_max < 5s, it means that the maximum expected delay of each atomic task is within an acceptable range, and the master control agent is capable of scheduling and executing a single task within 5 seconds. When D_max ≥ 5s, it means that the expected delay of some atomic tasks has reached or exceeded 5 seconds. Direct scheduling by the master control agent may cause the overall process to time out. In this case, it is necessary to switch to using the distributed parallel processing of the MapReduce engine to shorten the total time.

[0073] Furthermore, the calculated Score_strat1 is compared with a preset decision threshold θ_th (e.g., θ_th = 0.45), and the current MapReduce engine idle time U_MR is obtained and compared with a preset idle time threshold ρ_th (e.g., ρ_th = 0.8). If Score_strat1 < θ_th and U_MR < ρ_th, then the current task is determined to be more suitable for a process-intensive quality inspection strategy; otherwise, the current task is determined to be more suitable for a data-intensive quality inspection strategy.

[0074] That is, if Score_strat1 < θ_th and U_MR < ρ_th, then the master control agent determines the quality inspection strategy as a process-intensive quality inspection strategy. Under this process-intensive quality inspection strategy, the master control agent sequentially calls the quality inspection agents corresponding to each atomic operator to execute the quality inspection tasks according to the execution order and dependencies defined in the quality inspection workflow.

[0075] Specifically, refer to Figure 6 The controlling agent can generate a Directed Acyclic Graph (DAG) of atomic tasks using a Standard Operating Procedure (SOP) template. Each node in the DAG represents an instance of an atomic operator, and the directed edges in the DAG represent execution dependencies between operators. The controlling agent performs a topological sorting algorithm on the DAG to generate a linear execution sequence that satisfies all dependency constraints, and then triggers the execution of each atomic operator sequentially according to this sequence.

[0076] In terms of communication mode, the master control agent invokes various specialized quality inspection agents via synchronous remote procedure calls (RPC). After completing the quality inspection task, the invoked agent directly returns structured results to the master control agent. All intermediate execution states (including the list of completed operators, the return results of each operator, and the current execution progress) are stored in the master control agent's memory and are simultaneously backed up to disk to prevent state loss due to unexpected restarts of the master control agent.

[0077] Regarding fault tolerance mechanisms, the master control agent sets timeout parameters for each atomic task. When the execution time of an atomic task exceeds the preset timeout threshold, the master control agent determines that the task has timed out and immediately triggers a retry mechanism, reissuing the execution instruction for the atomic task. If the atomic task still fails after three consecutive retries, the master control agent marks the entire quality inspection process as failed and triggers an alarm, notifying operations and maintenance personnel to intervene. Simultaneously, the master control agent periodically persists a snapshot of the current state machine to external storage (such as a distributed file system or object storage), ensuring that even if the master control agent itself crashes, it can recover its execution state from the most recent snapshot after restarting and continue executing unfinished quality inspection tasks without having to start from scratch.

[0078] In one feasible implementation, the distributed processing of the multiple parallel subtasks via the MapReduce engine can be: The MapReduce engine divides the data source to be inspected into multiple logical shards. Multiple parallel Mapper tasks are assigned to each logical shard, and intermediate key-value pairs are output. Each Mapper task executes different types of quality inspection operators, including statistical detection, density detection, semantic detection, and algorithm detection. The intermediate key-value pairs use the shard identifier as the key and the detection result containing anomaly scores and anomaly labels as the value. The intermediate key-value pairs are partitioned and sorted, and the quality inspection results belonging to the same logical shard are aggregated to the same Reducer node, generating an input dataset organized by shard. The Reducer task reads the input dataset, performs weighted calculations or logical fusion on the multiple anomaly scores under the same logical shard, generates a comprehensive quality inspection conclusion for the same logical shard, and summarizes the comprehensive quality inspection conclusions of all shards to output the final quality inspection report.

[0079] If Score_strat1 ≥ θ_th or U_MR ≥ ρ_th, the master agent determines the quality inspection strategy to be a data-intensive strategy. Under this strategy, the master agent breaks down the quality inspection task into multiple parallel subtasks and performs distributed processing on these subtasks through the MapReduce engine. That is, the multiple parallel subtasks are processed in a distributed manner through the Map phase, Shuffle phase, and Reduce phase.

[0080] Specifically, after the master control agent submits the data-intensive quality inspection task to the MapReduce engine, the MapReduce engine first divides the original dataset to be inspected into multiple logical partitions according to the preset partitioning strategy, and assigns a unique partition identifier to each logical partition.

[0081] For each logical shard, the MapReduce engine launches multiple parallel Mapper tasks, which simultaneously perform different types of quality inspection calculations on that shard.

[0082] Specifically, refer to Figure 5 During the Map phase, each Mapper task can execute four types of quality control operators: statistical detection, density detection, semantic detection, and algorithm detection. Each type of operator evaluates the quality of the same logical partition from different dimensions. After each Mapper task completes its computation, it outputs an intermediate key-value pair, where the key is the partition identifier of the logical partition, and the value is the detection result of the partition in that quality control dimension (including anomaly score and anomaly label). In this way, each logical partition corresponds to four intermediate key-value pairs, which come from four different types of Mapper tasks. They carry the same partition identifier but have different values.

[0083] Furthermore, after all Mapper tasks have completed execution, the MapReduce engine automatically enters the Shuffle phase. During the Shuffle phase, the engine first partitions the intermediate key-value pairs stored locally on each Mapper node. Specifically, it calculates the partition number corresponding to the key (shard identifier) ​​in the intermediate key-value pair and assigns intermediate key-value pairs with the same shard identifier to the same partition. After partitioning, the four intermediate key-value pairs belonging to the same logical shard (from the statistical detection, density detection, semantic detection, and algorithm detection Mapper tasks, respectively) are grouped into the same partition.

[0084] The MapReduce engine sorts the intermediate key-value pairs within each partition according to their keys, ensuring that all intermediate key-value pairs belonging to the same logical shard within the same partition are arranged contiguously in physical storage. After partitioning and sorting, the engine remotely copies the intermediate key-value pairs from the Mapper nodes to the corresponding Reducer nodes. Each Reducer node receives data from one or more partitions, thereby generating an input dataset organized by shard. The input dataset for each partition contains various quality control results from all Mapper tasks under the same logical shard, i.e., complete detection information for that shard across four dimensions: statistical detection, density detection, semantic detection, and algorithm detection.

[0085] During the Reduce phase, the MapReduce engine calls the input dataset organized by shards, performs weighted calculations or logical fusion of multiple anomaly scores under the same logical shard, generates a comprehensive quality inspection conclusion for that logical shard, and summarizes the comprehensive quality inspection conclusions of all shards to output the final quality inspection report.

[0086] Specifically, the Reducer task running on each Reducer node reads the complete intermediate dataset for each logical shard allocated to that node. For each logical shard, the Reducer task extracts the anomaly scores and anomaly labels for that shard across four dimensions: statistical detection, density detection, semantic detection, and algorithm detection. These scores are then weighted and summed according to preset weighting coefficients to calculate the shard's overall anomaly score. Simultaneously, based on preset logical fusion rules (e.g., if two or more of the four dimensions are identified as anomalies, the shard is considered anomaly overall), the comprehensive anomaly label for that shard is derived. The Reducer task performs the above processing on each logical shard, generating a comprehensive quality inspection conclusion for that shard.

[0087] After a Reducer node completes the comprehensive evaluation of all logical shards assigned to it, it summarizes the comprehensive quality inspection conclusions of all shards to generate the quality inspection results for the shard range under its responsibility. Finally, the output results of all Reducer nodes are summarized to the master control agent, which integrates them into a final quality inspection report covering all the original data.

[0088] This embodiment constructs a dual-strategy collaborative architecture with a master intelligent agent as the sole central hub. First, it introduces four-dimensional feature vectors: data scale (number of records), data complexity (number of fields, nesting levels, and number of data types), data stability (recent drift), and resource availability (MapReduce engine idle time and average load of the agent pool). These vectors are then fused to calculate a comprehensive decision score. Based on a quantitative comparison of the decision score with a preset threshold, the system adaptively switches between process-intensive and data-intensive strategies. This achieves dynamic optimal matching between execution strategies, task characteristics, and system resources. On the one hand, in process-intensive scenarios, synchronous remote procedure calls by the master intelligent agent enable low-latency, highly controllable pipeline-style collaborative scheduling. On the other hand, in data-intensive scenarios, the MapReduce engine divides massive amounts of data into multiple logical shards and assigns parallel Mapper tasks to perform multi-dimensional quality inspection calculations. After partitioning, sorting, and aggregation during the shuffling stage, Reducer tasks perform weighted fusion to output a comprehensive quality inspection conclusion. This distributed parallel processing effectively overcomes the performance bottleneck of single-point processing and significantly improves system throughput.

[0089] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description and will not be repeated hereafter. On this basis, each template in the preset SOP process template library predefines multiple atomic operators. During the execution of each atomic operator of the quality inspection workflow, when any of the atomic operators needs to call a large language model, the data quality inspection method further includes: In response to the current model call request, the system obtains the current available resources in the rate-limited resource pool. If the current available resources meet the resource requirements of the current model call request, the system occupies the available resources corresponding to the resource requirements and selects candidate models that meet the availability conditions from a preset model set. Based on the selection results, a qualified model set is constructed. The system determines the target model from the qualified model set. The system obtains the input context corresponding to the current model call request and determines whether the input context meets the context processing conditions of the target model. If it does not meet the context processing conditions, the system compresses the input context to obtain a compressed context and determines again whether the compressed context meets the context processing conditions. If it does meet the conditions, the system completes the execution of the atomic operator through the target model.

[0090] It should be noted that intelligent data quality inspection is characterized by a large volume of requests, numerous token requirements, and long contexts, which places high demands on the LLM intelligent engine. During the implementation of the solution, issues such as request rate limiting on the model platform, token limits, and exceeding model context limits were encountered, which constrained the system's stability, throughput, and availability when handling large-scale quality inspection tasks.

[0091] This embodiment aims to use the dual token bucket rate limiting scheduling algorithm as the core application strategy, and rely on the configurable management and control mode to uniformly realize the cross-platform quota verification, model random scheduling and context intelligent compression full-process management of intelligent computing platform, extreme computing platform and custom model.

[0092] Specifically, during the execution of each atomic operator in the quality inspection workflow, when any atomic operator needs to call a large language model, the master control agent or each quality inspection agent initiates a current model call request to the LLM intelligent scheduling module in the intelligent engine layer. The LLM intelligent scheduling module is configured with multiple parallel-deployed large language model instances (e.g., the Qwen series models) to provide context understanding, complex logical reasoning, and enhanced retrieval generation capabilities. When any atomic operator needs to call a large language model for rule generation or semantic determination, it enters the LLM flow control process.

[0093] In the LLM flow control process, in response to the current model call request, the amount of currently available resources in the flow-limiting resource pool is obtained.

[0094] In the LLM flow control process, in response to the current model call request, the current available resource quantity in the rate-limiting resource pool is obtained. The rate-limiting resource pool is implemented using a dual-token bucket architecture, including a first token bucket for controlling the request rate and a second token bucket for controlling the token consumption rate. The first token bucket generates first-type tokens at a fixed rate, and each model call request consumes one first-type token, used to control the total number of requests that can be processed per unit time, preventing overload caused by sudden traffic surges. The second token bucket generates second-type tokens at a fixed rate, and each model call request consumes a number of second-type tokens equal to its estimated token consumption, used to control the total number of tokens that can be consumed per unit time, preventing subsequent requests from being blocked due to insufficient token quotas caused by excessive token consumption in a single request. First, the current number of first tokens in the first token bucket and the current number of second tokens in the second token bucket are obtained as the current available resource quantity.

[0095] That is, the rate-limiting resource pool includes a first token bucket for controlling the request rate and a second token bucket for controlling the token consumption rate. The current available resource quantity includes the current first token count of the first token bucket and the current second token count of the second token bucket. The resource requirement includes the preset single request consumption and the estimated token consumption of the current model call request.

[0096] In one feasible implementation, if the currently available resources do not meet the resource requirements of the current model call request, the current model call request is placed in a waiting queue, and the currently available resources are reacquired when a preset retry condition is met. Specifically, if the current first token count is less than a preset single request consumption (e.g., 1 token), or the current second token count is less than the estimated token consumption of the current model call request, it is determined that the currently available resources do not meet the resource requirements. In this case, the request is not immediately rejected, but is temporarily placed in a waiting queue, and a retry timer is set. After a preset waiting interval (e.g., 100 milliseconds), the current available resources are reacquired. If the resource requirements still cannot be met after multiple retries, a request failure response is returned.

[0097] In one feasible implementation, if the currently available resources meet the resource requirements of the current model call request, the available resources corresponding to the resource requirements are occupied; specifically, the implementation method for occupying the available resources corresponding to the resource requirements if the currently available resources meet the resource requirements of the current model call request can be: If the current first token count is greater than or equal to the preset single request consumption, and the current second token count is greater than or equal to the estimated token consumption, then it is determined that the current available resource quantity meets the resource requirement, and the tokens of the first token count corresponding to the preset single request consumption are consumed, and the tokens of the second token count corresponding to the estimated token consumption are consumed.

[0098] Specifically, an atomic operation for resource occupation can be completed by consuming a first token and a second token equal to the estimated token consumption.

[0099] After resource allocation is complete, candidate models that meet availability conditions can be selected from a preset model set, and a qualified model set can be constructed based on the selection results. Specifically, the remaining quota information of each candidate model in the preset model set is obtained. The preset model set includes multiple large language model instances deployed on multiple different platforms (such as intelligent computing platforms and extreme computing platforms).

[0100] For each candidate model, check whether the current remaining quota of its platform is greater than zero, whether the current load is lower than a preset threshold, and whether the model instance is in a healthy and available state. Based on this, calculate the available quota score corresponding to each candidate model. Remove candidate models with available quota scores lower than the preset available threshold from the preset model set. Construct the qualified model set from the remaining candidate models after removal.

[0101] In a feasible implementation, the task queue T_task, the set of available models M_model, and their corresponding performance parameters can be obtained first. The performance parameters include the inference latency L_latency(i) of each model (i.e., the average response time required for the model to process one inference request), the accuracy A_acc(i) (i.e., the prediction accuracy of the model on the quality inspection task), and the cost C_cost(i) (i.e., the resource cost required to call the model).

[0102] The LLM intelligent scheduling module calls the token bucket state update function to update the token bucket state for each model m (where m∈M). Specifically, for each model instance, an independent token bucket is maintained, which continuously generates tokens at a preset generation rate r_m. The number of tokens B_m represents the computational power metric currently available to the model. At the beginning of each scheduling cycle, the system replenishes the tokens according to the time difference Δt between the current time and the last update time, according to the generation rate r_m (i.e., replenishment amount = r_m×Δt), and updates the current token bucket capacity B_m of the model, thereby reflecting the latest available computational power status of each model instance in real time. When the token bucket capacity exceeds a preset upper limit threshold, the system truncates the number of tokens to the upper limit to prevent tokens from accumulating indefinitely.

[0103] Simultaneously, based on the characteristics of the current quality inspection task (such as task type, data volume, and timeliness requirements), the urgency or business value of the task is determined, and a minimum performance threshold τ_min is set accordingly. Subsequently, the LLM intelligent scheduling module calls the candidate model filtering function to select a set of candidate models M_avail from the available model set M that meet the minimum performance threshold. The filtering conditions include: the model's current token bucket capacity B_m is greater than zero (i.e., the model currently has available computing power), the model's inference latency is lower than the maximum latency requirement allowed by the task, and the model's accuracy is higher than the minimum accuracy threshold required by the task. After filtering, models that do not meet the above conditions are removed from the candidate set, resulting in a qualified model set.

[0104] The target model is determined from the set of qualified models. Specifically, a random scheduling strategy is adopted to randomly select a candidate model from the set of qualified models as the target model. This random scheduling strategy can achieve natural load balancing in a multi-model environment, avoiding the situation where a fixed selection of a certain model leads to overheating of that model instance while other instances remain idle, thereby efficiently balancing the cluster's business load.

[0105] In one feasible implementation, the LLM intelligent scheduling module can first check whether the computing power requirement R_req of the current highest priority task t_curr exceeds the total remaining tokens of all candidate models. Specifically, the system iterates through the candidate model set M_avail, accumulates the current token bucket capacity B_m of each model, and obtains the total number of available tokens. If the total number of available tokens is less than the computing power requirement R_req of the task, it indicates that the sum of the remaining computing power of all currently available models is insufficient to support the execution of the task. At this time, a waiting or degradation strategy is triggered: the current task is put back into the queue to wait for the next scheduling cycle, or under the degradation strategy, a preset fallback rule engine is used to replace the large model call.

[0106] If the total number of available tokens is greater than or equal to the computing power requirement R_req for the task, then the model selection logic is entered. Each candidate model m_k in the candidate model set M_avail is traversed. For each candidate model, it is determined whether its current token bucket capacity B_mk is sufficient to support task execution (i.e., whether B_mk is greater than or equal to R_req). If the current model's token bucket capacity is sufficient to support task execution, then the model is added to the temporary feasible set S_feasible. After the traversal is complete, the feasible set S_feasible contains all candidate models with sufficient current computing power that meet the minimum performance threshold requirements.

[0107] Building upon this, a stochastic exploration mechanism is introduced to probabilistically select a target model from the feasible set S_feasible. Specifically, a selection probability is calculated for each model in the feasible set, which is positively correlated with the model's current performance score (e.g., the higher the performance score, the greater the probability of being selected). Simultaneously, a random factor is introduced to prevent uneven load distribution caused by deterministic selection. Through this probabilistic stochastic selection, both overloading of a single model and global load balancing are achieved.

[0108] After determining the target model, the input context corresponding to the current model call request is obtained, and it is determined whether the input context meets the context processing conditions of the target model. Specifically, the sum of the current token length of the input context and the estimated output length of the target model is calculated to obtain the estimated total length; the maximum allowed length corresponding to the current remaining tokens in the second token bucket (i.e., the upper limit of the token capacity corresponding to the current remaining second type of tokens) is obtained; if the estimated total length is less than or equal to the maximum allowed length, the input context is determined to meet the context processing conditions; if the estimated total length is greater than the maximum allowed length, the input context is determined to not meet the context processing conditions.

[0109] If the context processing conditions are not met, the input context is compressed to obtain a compressed context, and the compressed context is then judged again to determine whether it meets the context processing conditions. Specifically, when the estimated total length is greater than the maximum allowed length, a context intelligent compression mechanism is triggered. The compression mechanism calculates the attention weight values ​​of each word in the input context, selects the target words most important to the current quality inspection task from the input context based on the attention weight values, reconstructs the compressed context based on the selected target words, inputs the compressed context, recalculates the estimated total length, and compares it with the maximum allowed length again. If the estimated total length after compression is still greater than the maximum allowed length, the current model call request is abandoned, and a request failure response is returned; if the estimated total length after compression is less than or equal to the maximum allowed length, the compressed context is determined to meet the context processing conditions.

[0110] If the context processing conditions are met (i.e., the original input context meets the conditions, or the compressed context meets the conditions), the content that meets the conditions (original input context or compressed context) is input into the target model, the target large language model is called to perform inference, the execution result output by the target model is obtained, and the execution result is returned to the atomic operator that initiated the call to complete the execution of the atomic operator.

[0111] In one feasible implementation, determining whether the input context meets the context processing conditions of the target model can be: Calculate the sum of the current word length of the input context and the estimated output length of the target model to obtain the estimated total length; obtain the maximum context window length preset by the target model; if the estimated total length is greater than the maximum context window length, then determine that the input context does not meet the context processing conditions.

[0112] You can call the tokenizer corresponding to the target model, inputting the raw text of the input context into the tokenizer. The tokenizer will then segment the text into a sequence of tokens and return the number of tokens. Different models use different tokenizers (for example, the Qwen model uses a different token encoding method than the GPT series). Therefore, when calculating the token length, you need to use a tokenizer instance that matches the target model to ensure the accuracy of the calculation results. For example, for the Qwen series models, you can call the `encode` method of `QwenTokenizer` to encode the input text and obtain the length of the token list; for other models, you can use their corresponding tokenizer.

[0113] After determining the target model, its prediction interface is invoked. This interface outputs a predicted number of output tokens based on features of the input context (such as input token length, task type, etc.). If the target model does not provide a prediction interface, an average output length statistics table grouped by task type and input length range can be maintained. After each call, the actual output length is recorded and the average value of the corresponding group is updated. When prediction is needed, the average value of the corresponding group is retrieved from the statistics table based on the current task type and input length as the predicted output length.

[0114] The maximum context window length preset for the target model can be obtained by reading the model configuration file during the system initialization phase. Specifically, the LLM intelligent scheduling module maintains a configuration information table for each model instance, which records the upper limit of the context window length for each model instance.

[0115] The calculated estimated total length is compared with the maximum context window length read from the configuration information table. If the estimated total length is less than or equal to the maximum context window length, a result that meets the context processing conditions is returned, and the system directly uses the original input context to call the target model. If the estimated total length is greater than the maximum context window length, a result that does not meet the context processing conditions is returned, and the system triggers the context intelligent compression process. If the compressed context still cannot meet the conditions, a degradation process can be performed: fall back to the static rule engine for quality inspection, or switch to a backup model that supports a larger context window and re-execute the above judgment process, so that basic quality inspection capabilities can still be provided when the LLM service is unavailable or the context is severely exceeded.

[0116] After selecting the target model m_target, the task distribution operation is immediately executed, assigning the current task to the selected target model instance for inference computation. Simultaneously, the corresponding number of tokens in the model's token bucket is deducted, i.e., the token bucket capacity is updated to B_m_target = B_m_target - R_req, ensuring that the remaining number of tokens for the model accurately reflects the available computing power status after resource consumption.

[0117] Remove processed tasks from the task queue and return to the beginning of the scheduling loop to check if the task queue is empty. If the queue is empty, the loop terminates and all requests have been responded to; if the queue is not empty, the system continues to process the next task, repeating the core scheduling decision and execution and status update phases described above until all requests have been responded to.

[0118] In this embodiment, a dual dynamic token bucket rate limiting mechanism is constructed to perform fine-grained dual verification of model call traffic from two dimensions: request rate and token consumption rate. Combined with cross-platform quota verification, the model is included in the candidate scheduling range only after the availability verification of the remaining quota on each model platform is passed. Then, the target model is randomly selected from the qualified candidate set through the model random scheduling strategy to achieve natural load balancing among multiple models. With the above dual token verification and context intelligent compression working together, the high-concurrency request rate limiting, token limit and context window over-limit of large models in data quality inspection scenarios can be effectively solved.

[0119] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data quality inspection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0120] This application provides a data quality inspection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data quality inspection method in the first embodiment described above.

[0121] The following is for reference. Figure 7 The diagram illustrates a structural schematic of a data quality inspection device suitable for implementing embodiments of this application. The data quality inspection device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions and desktop computers. Figure 7 The data quality inspection equipment shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0122] like Figure 7As shown, the data quality inspection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data quality inspection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the data quality inspection equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show data quality inspection equipment with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0123] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0124] The data quality inspection equipment provided in this application, employing the data quality inspection method described in the above embodiments, can solve the technical problem of low efficiency in quality inspection tasks. Compared with the prior art, the beneficial effects of the data quality inspection equipment provided in this application are the same as those of the data quality inspection method described in the above embodiments, and other technical features of this data quality inspection equipment are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0125] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0127] The above descriptions are merely some embodiments of this application and do not limit the scope of protection of this application. Any equivalent structural transformations made based on the technical concept of this application and the content of this specification and drawings, or direct / indirect applications in other related technical fields, are included within the scope of protection of this application. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and policies of the country where the application is located and with authorization from the owner of the corresponding device.

Claims

1. A data quality inspection method, characterized in that, The data quality inspection method for the master control agent in a multi-agent cooperative system includes: Obtain the metadata of the data source to be inspected, wherein the metadata includes the data table name and the set of field names; The data table name, the set of field names, and the preset business tags are semantically matched with the operator tags of each preset template in the preset SOP process template library to determine the target template. The target template is instantiated with parameters based on the metadata to generate a quality inspection workflow.

2. The data quality inspection method as described in claim 1, characterized in that, The metadata also includes the number of field data types. Each template in the preset SOP process template library predefines multiple atomic operators, the execution order of each atomic operator, the dependencies between each atomic operator, and an exception handling strategy. The step of instantiating the target template with parameters based on the metadata to generate a quality inspection workflow includes: Obtain the parameter configuration interface for each atomic operator in the target template; Based on the set of field names and the number of types of field data types, assign values ​​to the parameter configuration interfaces of each atomic operator; Based on the execution order, dependencies, and exception handling strategy, the assigned atomic operators are arranged into the quality inspection workflow.

3. The data quality inspection method as described in claim 1, characterized in that, The metadata also includes the number of field data types, the number of fields, and the field nesting hierarchy. After the step of instantiating the target template based on the metadata and generating the quality inspection workflow, the method further includes: Obtain the number of records in the data source to be inspected, the idle time of the MapReduce engine, and the average load of the agent pool; Based on the number of fields, the number of types of field data, the nesting hierarchy of fields, and the number of records, determine the data scale, data complexity, and data stability of the data source; The current resource availability is determined based on the MapReduce engine idle time and the average load of the agent pool; Based on the data size, data complexity, data stability of the data source and the current resource availability, calculate the comprehensive decision score of the quality inspection task corresponding to the quality inspection workflow; Based on the comprehensive decision score, a quality inspection strategy is determined.

4. The data quality inspection method as described in claim 3, characterized in that, The quality inspection strategy includes process-intensive and data-intensive strategies. Each template in the preset SOP process template library predefines multiple atomic operators, the execution order of each atomic operator, and the dependencies between each atomic operator. The step of determining the quality inspection strategy based on the comprehensive decision score includes: If the overall decision score is less than a preset decision threshold and the idle degree of the MapReduce engine is less than a preset idle degree threshold, then the quality inspection strategy is determined to be a process-intensive quality inspection strategy. The process-intensive quality inspection strategy is to sequentially call the quality inspection agent corresponding to each atomic operator to execute the quality inspection task according to the execution order and the dependency relationship. If the overall decision score is greater than or equal to the preset decision threshold, or the MapReduce engine idle time is greater than or equal to the preset idle time threshold, then the quality inspection strategy is determined to be a data-intensive quality inspection strategy. The data-intensive quality inspection strategy is to split the quality inspection task into multiple parallel subtasks and perform distributed processing on the multiple parallel subtasks through the MapReduce engine.

5. The data quality inspection method as described in claim 4, characterized in that, The step of distributing the processing of the multiple parallel subtasks using the MapReduce engine includes: The MapReduce engine divides the data source to be inspected into multiple logical shards, assigns multiple parallel Mapper tasks to each logical shard, and outputs intermediate key-value pairs respectively. Each Mapper task executes different types of quality inspection operators, including statistical detection, density detection, semantic detection, and algorithm detection. The intermediate key-value pairs use the shard identifier as the key and the detection result containing the anomaly score and anomaly label as the value. The intermediate key-value pairs are partitioned and sorted, and the various quality inspection results belonging to the same logical partition are aggregated into the same Reducer node to generate an input dataset organized by partition. The Reducer task is invoked to read the input dataset, perform weighted calculation or logical fusion of multiple anomaly scores under the same logical segment, generate a comprehensive quality inspection conclusion for the same logical segment, summarize the comprehensive quality inspection conclusions of all segments, and output the final quality inspection report.

6. The data quality inspection method as described in claim 1, characterized in that, Each template in the pre-defined SOP process template library predefines multiple atomic operators. During the execution of each atomic operator in the quality inspection workflow, when any of the atomic operators needs to call a large language model, the data quality inspection method further includes: In response to the current model call request, obtain the amount of currently available resources in the rate-limited resource pool; If the current available resources meet the resource requirements of the current model call request, then the available resources corresponding to the resource requirements are used, and candidate models that meet the availability conditions are selected from the preset model set, and a qualified model set is constructed based on the selection results. Determine the target model from the set of qualified models; Obtain the input context corresponding to the current model call request, and determine whether the input context meets the context processing conditions of the target model; If the context processing conditions are not met, the input context is compressed to obtain a compressed context, and the compressed context is checked again to see if it meets the context processing conditions. If the conditions are met, the atomic operator is executed through the target model.

7. The data quality inspection method as described in claim 6, characterized in that, The rate-limiting resource pool includes a first token bucket for controlling the request rate and a second token bucket for controlling the token consumption rate. The currently available resource quantity includes the current first token count of the first token bucket and the current second token count of the second token bucket. The resource requirement includes the preset single request consumption and the estimated token consumption of the current model call request. The step of occupying the available resources corresponding to the resource requirement if the current available resources meet the resource requirement of the current model call request includes: If the current first token count is greater than or equal to the preset single request consumption, and the current second token count is greater than or equal to the estimated token consumption, then it is determined that the current available resource quantity meets the resource requirement, and the tokens of the first token count corresponding to the preset single request consumption are consumed, and the tokens of the second token count corresponding to the estimated token consumption are consumed.

8. The data quality inspection method as described in claim 7, characterized in that, The step of determining whether the input context meets the context processing conditions of the target model includes: The sum of the current word length of the input context and the estimated output length of the target model is calculated to obtain the estimated total length. Obtain the maximum context window length preset for the target model; If the estimated total length is greater than the maximum context window length, then the input context is determined to be inconsistent with the context processing conditions.

9. The data quality inspection method as described in claim 1, characterized in that, The step of semantically matching the data table name, the set of field names, and the preset business tags with the operator tags of each preset template in the preset SOP process template library to determine the target template includes: The data table name and the set of field names are processed into text vectors to obtain the first feature vector; The operator labels of each preset template are processed into text vectors to obtain the second feature vector corresponding to each preset template; Calculate the similarity between the first feature vector and each of the second feature vectors, and determine the preset template corresponding to the second feature vector with the highest similarity as the target template.

10. A data quality inspection device, characterized in that, The data quality inspection device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data quality inspection method as described in any one of claims 1 to 9.

Images