An artificial intelligence-based data quality intelligent management and control system

Abstract

The application discloses an intelligent data quality management and control system based on artificial intelligence and relates to the technical field of data quality management and control.The application comprises a database, an encryption and decryption module, a file compatibility module, a classification module, an extraction module, a search module, an evaluation module and a cache module, executes a data preprocessing and format compatibility program, stores original data into the cache after format conversion, reads and executes classification by the classification module and writes the results back to the database, executes a data classification program, executes multi-stage distillation on data retrieved from the database by the extraction module, stores the distilled data set persistently after transfer in the cache, executes a data extraction and distillation program, and predicts data similarity in advance by scheduling idle intelligent agents.The application prevents data leakage during inter-module transmission by full-link encryption, realizes multi-modal data input through file format conversion, prevents intelligent agents from mistakenly deleting data by a grace period and whitelist mechanism, and matches departmental use requirements through hit rate feedback.

Description

Technical Field

[0001] This invention relates to the field of data quality management technology, specifically to an intelligent data quality management system based on artificial intelligence. Background Technology

[0002] Enterprise data management involves data storage, deduplication, classification, encryption, and access control. The continuous accumulation of similar data and redundant files in databases consumes storage space and requires identification and cleanup. Data from different departments within an enterprise has varying sensitivity levels; financial, human resources, and R&D data need to be isolated to prevent cross-departmental data leaks. Artificial intelligence models offer high accuracy in data feature extraction, similarity assessment, and automatic classification, and can replace manual data quality control. However, due to trade secrets or privacy concerns, enterprise data cannot utilize large-scale language model APIs and requires local deployment of large-scale models for data management, equipped with appropriate data security mechanisms.

[0003] Existing technologies suffer from the following problems: 1. Enterprise data contains trade secrets and privacy information, making data quality management impossible through cloud-based large language model application programming interfaces. Local deployment solutions lack encryption mechanisms across the entire data transmission chain, posing a risk of plaintext leakage during inter-module data transmission; 2. Enterprise databases contain diverse file formats, and large language models cannot directly recognize non-standard formats such as project files, configuration files, and dependency files, limiting the scope of AI data processing; 3. AI-driven data cleaning lacks a human review buffer, leading to model misjudgments that result in the direct and irrecoverable deletion of critical data; 4. Differences in data usage needs across departments mean that a single scoring system cannot reflect departmental preferences, causing data classification and cleaning strategies to deviate from the actual needs of each department. Therefore, designing an AI-based intelligent data quality management system to address these deficiencies is a technical problem that those skilled in the art need to solve. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an intelligent data quality management and control system based on artificial intelligence, which solves the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an intelligent data quality management and control system based on artificial intelligence, applied to enterprise data management, comprising a database, an encryption / decryption module, a file compatibility module, a classification module, an extraction module, a search module, an evaluation module, and a caching module. The database port establishes bidirectional communication with the encryption / decryption module port. The encryption / decryption module port establishes bidirectional communication with the ports of the file compatibility module, classification module, extraction module, search module, and caching module, respectively. The extraction module port establishes bidirectional communication with the evaluation module port. The output of the search module is connected to the input of the evaluation module, and the output of the evaluation module is connected to the input of the caching module.

[0006] The file compatibility module and the encryption / decryption module work together to perform data preprocessing and format compatibility procedures. The original data is converted to a compatible format and then stored in the cache. The classification module reads the data, performs classification, and writes the results back to the database. The cache is then cleared. The classification module and the encryption / decryption module work together to perform data classification procedures. The extraction module retrieves data from the database and performs multi-level distillation. The distilled dataset is persistently stored after being transferred through the cache. The cache is then cleared. The extraction module, evaluation module, cache module, and encryption / decryption module work together to perform data extraction and distillation procedures. The evaluation module schedules idle agents to predict data similarity and makes a comprehensive decision based on department weights. Data to be cleared is transferred to a grace period for administrator review. The evaluation module integrates several agents. The search module and the evaluation module work together to perform agent scoring and weight iteration procedures, triggered by user search behavior. The search module records user access behavior, generates a hit rate, and feeds it back to the evaluation module. After scoring by a loss function, the independent weight sequence of each department is updated.

[0007] The data preprocessing and format compatibility program is executed sequentially with the data classification program. The data extraction and distillation program can be executed in parallel with the data classification program. The agent scoring and weight iteration program is executed independently after being triggered by the search module.

[0008] Furthermore, when the data preprocessing and format compatibility program is executed, the file compatibility module reads the original data file from the database through the encryption and decryption module, parses the file header identifier bytes, identifies the file format type, and obtains the format identification result;

[0009] The file compatibility module determines the format recognition result. If it is a project file, configuration file, or dependency file, the file compatibility module calls the corresponding format parser to convert the file binary stream into a text sequence or image matrix to obtain standardized data. If it is a recognizable format, the format conversion is skipped.

[0010] The encryption / decryption module receives standardized data, calls the RSA public key to perform encryption operations, and transmits the ciphertext to the cache module for temporary storage.

[0011] This program addresses the limitation of AI processing scope caused by heterogeneous file formats in enterprises. The file compatibility module, acting as a format abstraction layer, converts various proprietary formats into standardized inputs that the model can process, eliminating the need for subsequent classification and extraction modules to consider differences in the original file formats.

[0012] Furthermore, when the data classification program is executed, the classification module reads the decrypted plaintext data from the cache module through the encryption / decryption module;

[0013] The classification module constructs a classification tree, with the root node representing the overall data class. The classification module extracts the data feature vector, inputs it into the classification function, and traverses the intermediate nodes from the root node downwards until it reaches the leaf node to obtain the classification path. If a label-based total-score structure is adopted, the classification module first assigns a top-level label to the data, and then adds sub-labels level by level to obtain a label sequence.

[0014] The classification module binds the classification path or label sequence to the original data identifier, encrypts it through the encryption / decryption module, and writes it into the database index table to establish an associated index.

[0015] The cache module releases the memory area occupied by this classification operation, completing the cache clearing;

[0016] This program establishes a hierarchical data organization system. The tree structure or the tag-based sub-structure allows the classification granularity to be flexibly set according to the needs of the enterprise, progressing from coarse to fine at each level. The classification logic is executed in plaintext, but the classification results are encrypted when written to the database to ensure storage security. After classification is completed, cached data is cleared to prevent plaintext residue.

[0017] Furthermore, when the data extraction and distillation program is executed, the extraction module retrieves plaintext data from the database through the encryption and decryption module, and the data is loaded into the pipeline input buffer of the extraction module in batches;

[0018] Execute the first branch process, start the three-stage pipeline to extract features and reduce dimensionality in parallel, output a multi-parameter scale distillation dataset, encrypt it and write it to the database after caching;

[0019] Execute the second branch process, call the idle agent to calculate cosine similarity, make a weighted comprehensive decision to mark the data to be cleared, and directly filter the whitelist data;

[0020] After calculating the cosine similarity, the agent submits a list. The evaluation module performs a weighted summation and compares it with a decision threshold, marking the items to be cleared or retained.

[0021] The evaluation module sends a data transfer instruction to the cache module's grace zone controller. The cache module copies the data marked to be cleared from the database to the grace zone and deletes the original references in the database. Data marked to be retained does not trigger an operation. Whitelisted data is filtered when the second branch process starts and does not enter the similarity calculation.

[0022] If the administrator confirms the deletion or restoration within the grace period, the data will be automatically deleted if no action is taken after the timer reaches zero.

[0023] The pipeline input buffer and cache module temporary storage area of ​​the extraction module are both cleared to release memory resources;

[0024] This program achieves data simplification and intelligent deduplication, distilling and extracting to generate datasets of different parameter sizes to adapt to the different data accuracy needs of different departments. Idle agents perform similarity prediction in parallel during the extraction stage without consuming additional computing resources. A grace zone provides a window for human review, and a whitelist provides an exemption mechanism. Together, these two measures prevent artificial intelligence from accidentally deleting critical data.

[0025] Furthermore, when the intelligent agent scoring and weight iteration program is executed, the user inputs search conditions through the search module. The search module encapsulates the search conditions into a query request message, which is then forwarded to the database or cache module via the encryption / decryption module. The database or cache module returns a set of matching data identifiers. The search module obtains the corresponding plaintext data via the encryption / decryption module and presents it to the user.

[0026] The search module listens to user actions on search results. Each time a user clicks or expands a data entry, the search module increments the access count. After the user closes the search results page, the search module calculates the actual hit rate H using the formula H = Fw ÷ Fhs, where H is the hit rate, Fw is the number of accesses, and Fhs is the total number of returned entries. The search module encapsulates the actual hit rate H into a feedback message and sends it to the evaluation module.

[0027] The third branch process is executed, where the evaluation module calculates the loss value and the specific score, updates the department weight sequence after normalization, and performs parameter fine-tuning on low-weight agents.

[0028] The fourth branch process is executed. The evaluation module marks the agent that has completed fine-tuning as a state to be evaluated. It waits for the next agent scoring and weight iteration program to be triggered and then re-enters the third branch process. The fourth branch process is in a waiting state.

[0029] This program establishes an agent evaluation system that is linked to the actual usage behavior of departments. The hit rate reflects the degree to which users adopt search results and is an external indicator for measuring the quality of data classification and distillation. The loss function quantifies the difference in hit rate into an optimizable numerical signal. The scoring and weight sequence enable high-quality agents to gain more influence, while low-quality agents are automatically adjusted.

[0030] Furthermore, when the first branch process is executed, the extraction module starts a three-stage pipeline. The first-stage pipeline unit reads the raw data from the input buffer and outputs it to the second-stage pipeline unit. The second-stage pipeline unit performs feature extraction and dimensionality reduction operations. The third-stage pipeline unit encapsulates the dimensionality-reduced data into a distillation dataset. The three-stage pipeline units run in parallel. The extraction module outputs multiple distillation datasets with different parameter scales by adjusting the compression ratio parameter of the dimensionality reduction operation.

[0031] The distillation dataset is encrypted by the encryption / decryption module and then written sequentially to the temporary storage area of ​​the cache module. The cache module then writes the dataset in batches to the persistent storage area of ​​the database via the encryption / decryption module.

[0032] Furthermore, when the second branch process is executed, the extraction module sends an agent invocation request message to the evaluation module. The message contains a list of data identifiers. The evaluation module scans the agent status table, marks agents that are currently not assigned tasks as occupied, and returns the list of agent identifiers to the extraction module. The first branch process and the second branch process are executed in parallel.

[0033] Further, after calculating the cosine similarity, the agent submits a list. The evaluation module performs a weighted summation and compares it with a decision threshold, marking the data to be cleared or retained. This includes each called agent reading the converted data from the output buffer of the file compatibility module, extracting feature embedding vectors, and calculating the cosine similarity between the vectors. The cosine similarity formula is: cos(θ)=(A·B) / (||A||×||B||), where A and B are the feature embedding vectors of the two data, A·B is the vector dot product, and ||A|| and ||B|| are the vector magnitudes. The cosine similarity value ranges from -1 to 1. The closer the value is to 1, the more similar the two data are. The agent sends the list of data pairs with similarity exceeding a preset threshold θ0 to the evaluation module.

[0034] The evaluation module summarizes the similarity lists submitted by each agent, performs a weighted summation on each pair of data to obtain a comprehensive score, Score, calculated as Score=Σ(Wi×Sij), where Wi is the weight of the i-th agent in the current department, and Sij is the similarity score of the i-th agent for the j-th pair of data. Data pairs with a comprehensive score exceeding the decision threshold are marked as to be removed, and the rest are marked as to be retained.

[0035] Furthermore, during the grace period, if the administrator confirms the clearing or restoration, and no operation is performed when the timer reaches zero, the data will be automatically cleared. This includes the administrator being able to send an operation command to the grace area controller at any time after the grace period timer starts. If a clearing command is received, the clearing process is executed, and the grace area controller completely deletes the physical storage block of the corresponding data. If a restore command is received, the grace area controller writes the data back to the database and deletes the grace area copy. If the grace period countdown reaches zero and no operation command is received, the grace area controller automatically jumps to the clearing process.

[0036] Furthermore, when the third branch process is executed, the evaluation module reads the actual hit rate H in the feedback message, sequentially traverses the list of agents corresponding to the department, and for the i-th agent, the evaluation module reads its predicted hit rate Hi, calculates the loss value Li, the calculation formula is Li=|H-Hi|, and then calculates the exclusive score Si, the calculation formula is Si=1 / (1+Li), where H is the actual hit rate fed back by the search module, Hi is the predicted hit rate of the i-th agent for this search, Li is the loss value, and Si is the exclusive score;

[0037] The evaluation module normalizes the exclusive scores of all agents in the current department using the normalization formula Wi=Si / (S1+S2+…+Sn), resulting in an updated department weight sequence. The new weight sequence overwrites the old sequence and is saved to the department weight table of the evaluation module.

[0038] The evaluation module scans the department weight sequence and filters out agents with weights lower than a preset threshold Wmin. For each low-weight agent, the evaluation module inputs its current parameters, predicted hit rate Hi, and actual hit rate H into a parameter fine-tuning function. The parameter fine-tuning function aims to minimize the loss value Li and updates the agent's inference parameters along the gradient direction of the loss function. The update direction is for the predicted hit rate Hi to approach the actual hit rate H.

[0039] The present invention has the following beneficial effects:

[0040] 1. By using the encryption and decryption module as an encryption black box bus, data is kept in ciphertext state when transmitted between modules, and is in plaintext state only within the database, during extraction operations, classification operations, and read / write operations. The RSA asymmetric algorithm is used for encryption. Compared with the shortcomings of existing local deployment solutions that do not cover the entire link encryption mechanism, this invention achieves the effect of preventing data leakage during the transmission between modules.

[0041] 2. By using a file compatibility module, project files, configuration files, dependency files, and other formats that cannot be directly recognized are converted into text, images, or data types that can be recognized by artificial intelligence models. Compared with the shortcomings of existing large language models that cannot directly handle heterogeneous file formats, this invention achieves the effect of expanding the range of artificial intelligence data processing formats and realizing multimodal data input.

[0042] 3. By calling the idle intelligent agent of the evaluation module through the extraction module to perform similarity prediction on the data, and establishing a grace zone and whitelist mechanism in the cache module, compared with the shortcomings of existing artificial intelligence data cleaning that lacks human review, this invention achieves the effect of preventing artificial intelligence from accidentally deleting key data and providing a configurable human review window.

[0043] 4. By generating a hit rate through the search module and feeding it back to the evaluation module as a loss function, each agent is scored and a department-independent weight sequence is generated. Compared to the shortcomings of existing single scoring systems that cannot reflect department-level preferences, this invention achieves the effect of matching departmental data classification and cleaning strategies with actual usage needs. Of course, any product implementing this invention does not necessarily need to achieve all the advantages described above simultaneously. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a block diagram of an intelligent data quality control system based on artificial intelligence, according to the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Please see Figure 1This invention provides a technical solution: an intelligent data quality management and control system based on artificial intelligence, comprising a database, an encryption / decryption module, a file compatibility module, a classification module, an extraction module, a search module, an evaluation module, and a caching module. The port of the database establishes bidirectional communication with the port of the encryption / decryption module. The port of the encryption / decryption module establishes bidirectional communication with the ports of the file compatibility module, the classification module, the extraction module, the search module, and the caching module, respectively. The port of the extraction module establishes bidirectional communication with the port of the evaluation module. The output end of the search module is connected to the input end of the evaluation module, and the output end of the evaluation module is connected to the input end of the caching module.

[0048] The file compatibility module and the encryption / decryption module work together to perform data preprocessing and format compatibility procedures. The raw data is converted to a compatible format and then stored in the cache. The classification module reads the data, performs classification, and writes the results back to the database. The cache is then cleared. The classification module and the encryption / decryption module work together to perform data classification procedures. The extraction module retrieves data from the database and performs multi-level distillation. The distilled dataset is persistently stored after being transferred through the cache. The cache is then cleared. The extraction module, evaluation module, cache module, and encryption / decryption module work together to perform data extraction and distillation procedures. The evaluation module schedules idle agents to predict data similarity and makes a comprehensive decision based on department weights. Data to be cleared is transferred to a grace area for administrator review. The evaluation module integrates several agents. The search module and the evaluation module work together to perform agent scoring and weight iteration procedures, triggered by user search behavior. The search module records user access behavior, generates a hit rate, and feeds it back to the evaluation module. After scoring by a loss function, the independent weight sequences of each department are updated.

[0049] The artificial intelligence model adopts the open-source Transformer architecture Decoder-only model with a parameter size of 7B. The 7B model can complete full-precision inference on a single GPU with 80GB of video memory without the need for tensor parallel partitioning, reducing deployment complexity. INT8 quantization is performed before the model is loaded to compress video memory usage and accelerate inference.

[0050] During the execution of the data preprocessing and format compatibility program, the file compatibility module reads the original data file from the database through the encryption and decryption module, parses the file header identifier bytes, identifies the file format type, and obtains the format identification result. End-to-end encryption prevents data leakage during inter-module transmission.

[0051] The file compatibility module determines the format recognition result. If it is a project file, configuration file, or dependency file, the file compatibility module calls the corresponding format parser to convert the file binary stream into a text sequence or image matrix to obtain standardized data. If it is a recognizable format, the format conversion is skipped. File format conversion enables multimodal data input.

[0052] The encryption / decryption module receives standardized data, calls the RSA public key to perform encryption operations, and transmits the ciphertext to the cache module for temporary storage.

[0053] During the execution of the data classification program, the classification module reads the decrypted plaintext data from the cache module through the encryption / decryption module.

[0054] The classification module constructs a classification tree, with the root node representing the overall data class. The classification module extracts the data feature vectors, inputs them into the classification function, and traverses the intermediate nodes from the root node downwards until it reaches the leaf node to obtain the classification path. If a label-based total-score structure is adopted, the classification module first assigns a top-level label to the data, and then adds sub-labels level by level to obtain the label sequence.

[0055] The classification module binds the classification path or label sequence to the original data identifier, encrypts it through the encryption / decryption module, and writes it into the database index table to establish an associated index.

[0056] The cache module releases the memory area occupied by this classification operation, completing the cache clearing.

[0057] During the execution of the data extraction and distillation process, the extraction module retrieves plaintext data from the database through the encryption and decryption module, and the data is loaded into the pipeline input buffer of the extraction module in batches.

[0058] Execute the first branch process, start the three-stage pipeline to extract features and reduce dimensionality in parallel, output a multi-parameter scale distillation dataset, encrypt it and write it to the database after caching;

[0059] Execute the second branch process, call the idle agent to calculate cosine similarity, make a weighted comprehensive decision to mark the data to be cleared, and directly filter the whitelist data;

[0060] After calculating the cosine similarity, the agent submits a list. The evaluation module performs a weighted summation and compares it with the decision threshold, marking the items to be cleared or retained.

[0061] Each agent is an independent model inference instance, created by the evaluation module, and includes the following components: base model reference, pointing to the INT8 quantized version of the 7B base model in the model repository; LoRA adapter weights, a set of trainable A and B matrices with approximately 2 × 16 × 4096 = 131072 parameters; similarity threshold θ0, the lowest cosine similarity threshold for the agent to determine the similarity between two data points; prediction hit rate Hi, the agent's predicted hit rate for the next search; department weight table, recording the agent's current weight Wi in each department; and historical score queue, storing the Si scores of the K most recent searches, where K defaults to 100.

[0062] The evaluation module sends a data transfer instruction to the cache module's grace zone controller. The cache module copies the data marked as to be cleared from the database to the grace zone and deletes the original references in the database. Data marked as to be retained does not trigger any operation. Whitelisted data is filtered when the second branch process starts and does not enter the similarity calculation.

[0063] During the grace period, the administrator confirms the deletion or restoration. If no action is taken after the timer reaches zero, the data will be automatically deleted. The grace zone and whitelist mechanism prevent the agent from accidentally deleting data.

[0064] The pipeline input buffer and cache module temporary storage area of ​​the extraction module are cleared to release memory resources.

[0065] When the agent scoring and weight iteration program is executed, the user inputs search conditions through the search module. The search module encapsulates the search conditions into a query request message, which is then forwarded to the database or cache module via the encryption / decryption module. The database or cache module returns a set of matching data identifiers. The search module obtains the corresponding plaintext data through the encryption / decryption module and presents it to the user.

[0066] The search module listens for user actions on search results. Each time a user clicks or expands a data entry, the search module increments the access count. After the user closes the search results page, the search module calculates the actual hit rate H using the formula H = Fw ÷ Fhs, where H is the hit rate, Fw is the number of accesses, and Fhs is the total number of returned entries. The search module encapsulates the actual hit rate H into a feedback message and sends it to the evaluation module.

[0067] The third branch process is executed, the evaluation module calculates the loss value and the exclusive score, updates the department weight sequence after normalization, performs parameter fine-tuning for low-weight agents, and the hit rate feedback makes the cleanup strategy match the department's usage needs.

[0068] The fourth branch process is executed. The evaluation module marks the agents that have completed fine-tuning as pending evaluation states. When the next agent scoring and weight iteration program is triggered, the process re-enters the third branch process. The fourth branch process is in a waiting state.

[0069] The department-independent weight sequence ensures that the data usage preferences of different departments, such as finance and R&D, are reflected separately, avoiding strategy bias caused by a uniform weight sequence.

[0070] When the first branch process is executed, the extraction module starts a three-stage pipeline. The first-stage pipeline unit reads the raw data from the input buffer and outputs it to the second-stage pipeline unit. The second-stage pipeline unit performs feature extraction and dimensionality reduction operations. The third-stage pipeline unit encapsulates the dimensionality-reduced data into a distillation dataset. The three-stage pipeline units run in parallel. The extraction module outputs multiple distillation datasets with different parameter sizes by adjusting the compression ratio parameter of the dimensionality reduction operation.

[0071] The distillation dataset is encrypted by the encryption / decryption module and then written to the temporary storage area of ​​the cache module. The cache module then writes the dataset to the persistent storage area of ​​the database in batches via the encryption / decryption module.

[0072] During the execution of the second branch process, the extraction module sends an agent call request message to the evaluation module. The message contains a list of data identifiers. The evaluation module scans the agent state table, marks agents that are currently not assigned tasks as occupied, and returns the list of agent identifiers to the extraction module. The first branch process and the second branch process are executed in parallel.

[0073] The process involves the agent calculating cosine similarity and submitting a list. The evaluation module performs a weighted summation and compares it with a decision threshold, marking data as to be cleared or retained. This includes each called agent reading the converted data from the output buffer of the file compatibility module, extracting feature embedding vectors, and calculating the cosine similarity between vectors. The cosine similarity formula is: cos(θ)=(A·B) / (||A||×||B||), where A and B are the feature embedding vectors of the two data points, A·B is the vector dot product, and ||A|| and ||B|| are the vector magnitudes. The cosine similarity value ranges from -1 to 1, with values ​​closer to 1 indicating greater similarity between the two data points. The agent then sends a list of data pairs with similarity exceeding a preset threshold θ0 to the evaluation module.

[0074] The evaluation module summarizes the similarity lists submitted by each agent, performs a weighted summation on each pair of data to obtain a comprehensive score, Score, calculated as Score=Σ(Wi×Sij), where Wi is the weight of the i-th agent in the current department, and Sij is the similarity score of the i-th agent for the j-th pair of data. Data pairs with a comprehensive score exceeding the decision threshold are marked as to be removed, and the rest are marked as to be retained.

[0075] Within the grace period, if the administrator confirms the clearing or restoration, and no action is taken when the timer reaches zero, the data will be automatically cleared. This includes the following: after the grace period timer starts, the administrator can send an operation command to the grace area controller at any time. If a clearing command is received, the clearing process is executed, and the grace area controller completely deletes the physical storage block of the corresponding data. If a restore command is received, the grace area controller writes the data back to the database and deletes the grace area copy. If the grace period countdown reaches zero and no operation command is received, the grace area controller automatically jumps to the clearing process.

[0076] When the third branch process is executed, the evaluation module reads the actual hit rate H in the feedback message, and iterates through the list of agents corresponding to the department. For the i-th agent, the evaluation module reads its predicted hit rate Hi, calculates the loss value Li, and calculates the specific score Si, which is calculated as Si = 1 / (1+Li). Here, H is the actual hit rate fed back by the search module, Hi is the predicted hit rate of the i-th agent for this search, Li is the loss value, and Si is the specific score.

[0077] The evaluation module normalizes the exclusive scores of all agents in the current department using the normalization formula Wi=Si / (S1+S2+…+Sn), resulting in an updated department weight sequence. The new weight sequence overwrites the old sequence and is saved to the department weight table of the evaluation module.

[0078] The evaluation module scans the department weight sequence and filters out agents with weights lower than the preset threshold Wmin. For each low-weight agent, the evaluation module inputs its current parameters, predicted hit rate Hi, and actual hit rate H into the parameter fine-tuning function. The parameter fine-tuning function aims to minimize the loss value Li and updates the agent's inference parameters along the gradient direction of the loss function. The update direction is for the predicted hit rate Hi to approach the actual hit rate H.

[0079] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A data quality intelligent management and control system based on artificial intelligence, characterized in that: It includes a database, an encryption / decryption module, a file compatibility module, a classification module, an extraction module, a search module, an evaluation module, and a caching module. The port of the database establishes bidirectional communication with the port of the encryption / decryption module. The port of the encryption / decryption module establishes bidirectional communication with the ports of the file compatibility module, the classification module, the extraction module, the search module, and the caching module, respectively. The port of the extraction module establishes bidirectional communication with the port of the evaluation module. The output terminal of the search module is connected to the input terminal of the evaluation module, and the output terminal of the evaluation module is connected to the input terminal of the caching module. The system executes a data preprocessing and format compatibility program, converting the raw data to a compatible format and storing it in a cache. The classification module reads the data, performs classification, and writes the results back to the database. The data classification program is then executed. The extraction module retrieves data from the database and performs multi-level distillation. The distilled dataset is then persistently stored after being cached. The data extraction and distillation program is executed. Idle agents are scheduled to predict data similarity and make comprehensive decisions based on department weights. Data to be cleared is transferred to a grace period for administrator review. The system executes an agent scoring and weight iteration program, records user access behavior to generate a hit rate, and feeds it back to the evaluation module. After scoring using a loss function, the independent weight sequences for each department are updated.

2. The data quality intelligent control system based on artificial intelligence according to claim 1, characterized in that, When the data preprocessing and format compatibility program is executed, the file compatibility module reads the original data file from the database through the encryption and decryption module, parses the file header identifier bytes, identifies the file format type, and obtains the format identification result. The file compatibility module determines the format recognition result. If it is a project file, configuration file, or dependency file, the file compatibility module calls the corresponding format parser to convert the file binary stream into a text sequence or image matrix to obtain standardized data. If it is a recognizable format, the format conversion is skipped. The encryption / decryption module receives standardized data, calls the public key to perform encryption operations, and transmits the ciphertext to the cache module for temporary storage.

3. The data quality intelligent control system based on artificial intelligence according to claim 1, characterized in that, When the data classification program is executed, the classification module reads the decrypted plaintext data from the cache module through the encryption / decryption module. The classification module constructs a classification tree, with the root node representing the overall data class. The classification module extracts the data feature vector, inputs it into the classification function, and traverses the intermediate nodes from the root node downwards until it reaches the leaf node to obtain the classification path. If a label-based total-score structure is adopted, the classification module first assigns a top-level label to the data, and then adds sub-labels level by level to obtain a label sequence. The classification module binds the classification path or label sequence to the original data identifier, encrypts it through the encryption / decryption module, and writes it into the database index table to establish an associated index. The cache module releases the memory area occupied by this classification operation, completing the cache clearing.

4. The data quality intelligent control system based on artificial intelligence according to claim 1, characterized in that, When the data extraction and distillation program is executed, the extraction module retrieves plaintext data from the database through the encryption and decryption module, and the data is loaded into the pipeline input buffer of the extraction module in batches. Execute the first branch process, start the three-stage pipeline to extract features and reduce dimensionality in parallel, output a multi-parameter scale distillation dataset, encrypt it and write it to the database after caching; Execute the second branch process, call the idle agent to calculate cosine similarity, make a weighted comprehensive decision to mark the data to be cleared, and directly filter the whitelist data; After calculating the cosine similarity, the agent submits a list. The evaluation module performs a weighted summation and compares it with a decision threshold, marking the items to be cleared or retained. The evaluation module sends a data transfer instruction to the cache module's grace zone controller. The cache module copies the data marked to be cleared from the database to the grace zone and deletes the original references in the database. Data marked to be retained does not trigger an operation. Whitelisted data is filtered when the second branch process starts and does not enter the similarity calculation. If the administrator confirms the deletion or restoration within the grace period, the data will be automatically deleted if no action is taken after the timer reaches zero. The pipeline input buffer and cache module temporary storage area of ​​the extraction module are both cleared to release memory resources.

5. The data quality intelligent control system based on artificial intelligence according to claim 1, characterized in that, When the intelligent agent scoring and weight iteration program is executed, the user inputs search conditions through the search module. The search module encapsulates the search conditions into a query request message, which is then forwarded to the database or cache module via the encryption / decryption module. The database or cache module returns a set of matching data identifiers. The search module obtains the corresponding plaintext data through the encryption / decryption module and presents it to the user. The search module listens to user actions on search results. Each time the user clicks or expands a data entry, the access count is incremented. After the user closes the search results page, the search module calculates the actual hit rate and encapsulates the actual hit rate into a feedback message and sends it to the evaluation module. The third branch process is executed, where the evaluation module calculates the loss value and the specific score, updates the department weight sequence after normalization, and performs parameter fine-tuning on low-weight agents. The fourth branch process is executed. The evaluation module marks the agent that has completed fine-tuning as a state to be evaluated. When the agent scoring and weight iteration program is triggered again, it will re-enter the third branch process. The fourth branch process is in a waiting state.

6. The data quality intelligent control system based on artificial intelligence according to claim 4, characterized in that, When the first branch process is executed, the extraction module starts a three-stage pipeline. The first-stage pipeline unit reads the raw data from the input buffer and outputs it to the second-stage pipeline unit. The second-stage pipeline unit performs feature extraction and dimensionality reduction operations. The third-stage pipeline unit encapsulates the dimensionality-reduced data into a distillation dataset. The three-stage pipeline units run in parallel. The extraction module outputs multiple distillation datasets with different parameter sizes by adjusting the compression ratio parameter of the dimensionality reduction operation. The distillation dataset is encrypted by the encryption / decryption module and then written sequentially to the temporary storage area of ​​the cache module. The cache module then writes the dataset in batches to the persistent storage area of ​​the database via the encryption / decryption module.

7. The data quality intelligent control system based on artificial intelligence according to claim 4, characterized in that, When the second branch process is executed, the extraction module sends an agent call request message to the evaluation module. The message contains a list of data identifiers. The evaluation module scans the agent status table, marks agents that are currently not assigned tasks as occupied, and returns the list of agent identifiers to the extraction module. The first branch process and the second branch process are executed in parallel.

8. The data quality intelligent control system based on artificial intelligence according to claim 4, characterized in that, After calculating the cosine similarity, the agent submits a list. The evaluation module performs a weighted summation and compares it with a decision threshold, marking the data to be cleared or retained. This includes each called agent reading the converted data from the output buffer of the file compatibility module, extracting feature embedding vectors, calculating the cosine similarity between vectors (the cosine similarity ranges from -1 to 1), and sending the list of data with similarity exceeding a preset threshold to the evaluation module. The evaluation module summarizes the similarity lists submitted by each agent, performs a weighted summation on each pair of data to obtain a comprehensive score, and marks the data pairs with comprehensive scores exceeding the decision threshold as to be removed, while the rest are marked as to be retained.

9. The data quality intelligent control system based on artificial intelligence according to claim 4, characterized in that, During the grace period, if the administrator confirms the clearing or restoration, and no operation is performed when the timer reaches zero, the data will be automatically cleared. This includes the following: after the grace period timer starts, the administrator can send an operation command to the grace area controller at any time. If a clearing command is received, the clearing process is executed, and the grace area controller completely deletes the physical storage block of the corresponding data. If a restore command is received, the grace area controller writes the data back to the database and deletes the grace area copy. If the grace period countdown reaches zero and no operation command is received, the grace area controller automatically jumps to the clearing process.

10. The data quality intelligent control system based on artificial intelligence according to claim 5, characterized in that, When the third branch process is executed, the evaluation module reads the actual hit rate in the feedback message, iterates through the list of agents corresponding to the department, and for the i-th agent, the evaluation module reads its predicted hit rate, calculates the loss value, and then calculates the exclusive score. The evaluation module normalizes the exclusive scores of all agents in the current department to obtain an updated department weight sequence. The new weight sequence overwrites the old sequence and is saved to the department weight table of the evaluation module. The evaluation module scans the department weight sequence and filters out agents with weights lower than a preset threshold. For each low-weight agent, the evaluation module inputs its current parameters, predicted hit rate, and actual hit rate into a parameter fine-tuning function. The parameter fine-tuning function aims to minimize the loss value and updates the agent's inference parameters along the gradient direction of the loss function. The update direction is to make the predicted hit rate approach the actual hit rate.

Images