Big data full life cycle quality control and efficient retrieval system

Abstract

The invention relates to the technical field of big data processing and information retrieval, and discloses a big data full life cycle quality control and efficient retrieval system, which comprises a data acquisition processing module, a hierarchical index construction module and a retrieval repair module, and is characterized in that the acquisition processing module generates a dynamic quality score based on static verification, dynamic activeness and data conflict degree; the high-quality data is stored in a hot storage area to construct a high-precision vector index, the data to be treated is stored in a cold storage area, and a low-precision vector index is constructed by using a self-adaptive product quantization algorithm; and when the retrieval of the hot area fails, the retrieval and restoration module generates virtual reference data based on the semantic intention, performs reverse detection and search on the cold area, discovers and restores potential to-be-treated data, and promotes the potential to-be-treated data back to the hot storage area. According to the method, on-demand allocation of resources and automatic discovery and dynamic optimization of hidden dirty data are realized through a quality-driven hierarchical index and reverse detection mechanism, and the data availability is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of big data processing and information retrieval technology, and in particular to a big data full lifecycle quality control and efficient retrieval system. Background Technology

[0002] With the explosive growth of multi-source heterogeneous big data, data quality governance and efficient retrieval have become two core challenges. In existing technologies, data governance and retrieval systems are often fragmented: for example, CN202310567890.1 discloses a static data governance system that only cleans data through rules, without considering dynamic activity and conflict levels; CN202210876543.2 proposes a single index retrieval scheme, failing to achieve hierarchical storage and a retrieval-repair closed loop, resulting in limited retrieval of high-value data and resource consumption of low-quality data. Governance systems focus on static rule cleaning, failing to perceive the activity and timeliness of data in actual business operations; while retrieval systems focus on query matching, exhibiting poor fault tolerance for low-quality data, leading to the inability to effectively retrieve "hidden dirty data" caused by missing or incorrect key fields. Furthermore, existing systems typically employ a uniform index building strategy, lacking differentiated resource configuration for data of different quality, resulting in limited retrieval performance for high-value data, while low-value data consumes significant amounts of expensive computing and storage resources. Traditional data repair methods mostly rely on manual discovery or offline scanning, lacking an intelligent closed-loop mechanism that can be triggered in real time when retrieval fails, and actively detect and repair "cold data" using semantic intent. Summary of the Invention

[0003] This invention provides a big data full lifecycle quality control and efficient retrieval system. Through quality-driven adaptive hierarchical indexing and a retrieval feedback and repair mechanism based on virtual reference data, it realizes on-demand allocation of computing resources and automatic discovery and dynamic promotion of low-quality data, significantly improving the retrieval efficiency and data availability of big data systems.

[0004] This invention provides a big data full lifecycle quality control and efficient retrieval system, including interconnected data acquisition and processing modules, hierarchical index construction modules, and retrieval and repair modules;

[0005] The data acquisition and processing module is used to access multi-source heterogeneous raw data, and uses a feature extraction model based on the Transformer architecture to extract features from the raw data to obtain feature vectors with original dimensions. The module also verifies the raw data based on a configurable rule base that includes three types of verification rules: field integrity, format standardization, and value range validity, to generate a dynamic quality score. The feature extraction model based on the Transformer architecture is used to encode unstructured data into a 768-dimensional dense feature vector.

[0006] The hierarchical index building module is used to mark the original data as high-quality data and store it in the hot data storage area when the dynamic quality score is higher than the preset hierarchical threshold, and to build a high-precision vector index based on the feature vector of the high-quality data; when the dynamic quality score is lower than the hierarchical threshold, the original data is marked as data to be treated and stored in the cold data storage area, and the feature vector of the data to be treated is subjected to dimensionality reduction and precision compression processing using an adaptive product quantization algorithm to build a low-precision vector index.

[0007] The retrieval and repair module is used to receive user retrieval requests and generate semantic intent vectors. Based on the semantic intent vectors and the high-precision vector index, it retrieves high-quality data in the hot data storage area. When the similarity between the semantic intent vector and the feature vector of the high-quality data is lower than a preset recall threshold, it generates virtual reference data based on the semantic intent vector and performs an approximate nearest neighbor search on the data to be treated in the cold data storage area based on the low-precision vector index. When the similarity between the data to be treated and the virtual reference data is higher than a preset matching threshold, it generates a repair instruction to update and repair the data to be treated and migrates the repaired data to be treated as high-quality data to the hot data storage area.

[0008] The system achieves dynamic optimization of hidden dirty data and on-demand resource allocation through quality scoring-driven hierarchical storage, adaptive index construction and retrieval-repair closed loop.

[0009] Furthermore, the specific steps for generating the dynamic quality score in the data acquisition and processing module are as follows:

[0010] (1) Traverse the configurable rule base containing three types of verification rules: field integrity, format standardization, and value range validity. Perform static matching on the field integrity, format standardization, and value range validity of the original data, and calculate the static compliance index based on the weighted statistical method.

[0011] (2) Obtain the access frequency and last update time of the original data within a preset time window, process the access frequency based on the logarithmic growth processing method, and process the last update time based on the exponential decay processing method to calculate the dynamic activity index.

[0012] (3) Extract the key related fields from the original data, retrieve multiple related data records from the multi-source heterogeneous original dataset based on the key related fields, calculate the probability distribution of the values ​​of the original data and the multiple related data records on the corresponding attributes, and calculate the degree of disorder of the value distribution of the original data and the related data records based on the entropy calculation formula in information theory to obtain the data conflict factor.

[0013] (4) Based on the preset weight allocation strategy, the static compliance index, the dynamic activity index and the data conflict factor are linearly weighted and fused to obtain the dynamic quality score.

[0014] Furthermore, the static compliance index The calculation formula is:

[0015]

[0016] Where N is the total number of validation rules, The preset weight for the i-th rule, For the coefficient of the verification result, when the verification passes... ,otherwise ;

[0017] Dynamic Activity Index The calculation formula is:

[0018]

[0019] Where F represents the access frequency. This is the difference between the last update time and the current time. For frequency weighting coefficients, This is the timeliness weighting coefficient. This is the time decay factor;

[0020] Data conflict factor The calculation formula is:

[0021]

[0022] Where M represents the total number of different values ​​that the original data and the associated data records have on the corresponding attributes. Let j be the probability of the j-th value occurring.

[0023] The dynamic quality score The calculation formula is:

[0024]

[0025] in, These are the preset normalized weighting coefficients.

[0026] Furthermore, in the hierarchical index construction module,

[0027] For the high-quality data, the original dimension of the feature vector is kept unchanged, and a hierarchical navigation graph index is constructed using a hierarchical navigation graph algorithm as the high-precision vector index, thus maintaining the original dimension of the feature vector and achieving millisecond-level recall of high-quality data.

[0028] For the data to be processed, an adaptive product quantization algorithm is used for dimensionality reduction and precision compression to construct a low-precision vector index, specifically including:

[0029] Step A: Determine the product quantization parameter based on the dynamic quality score; wherein the product quantization parameter includes at least the number of subspace partitions, and the lower the dynamic quality score, the smaller the number of subspace partitions;

[0030] Step B: Divide the feature vector of the data to be managed into multiple low-dimensional sub-vectors according to the number of subspace partitions; wherein the number of low-dimensional sub-vectors is equal to the number of subspace partitions;

[0031] Step C: For each of the low-dimensional sub-vectors, find the nearest cluster center in the preset codebook to obtain the cluster center index number corresponding to each low-dimensional sub-vector. Concatenate the index numbers corresponding to all the low-dimensional sub-vectors from the data to be managed according to the segmentation order, and use the resulting index number sequence as the low-precision vector index.

[0032] Further, step A specifically includes:

[0033] Obtain a preset lower limit for scoring and the grading threshold to construct an adaptive mapping interval, and obtain a preset minimum number of subspace partitions and a preset maximum number of subspace partitions, and calculate the number of subspace partitions corresponding to the data to be governed, using the following formula:

[0034]

[0035] in, This indicates the floor function. The dynamic quality score for the data to be addressed. The preset hierarchical threshold in the hierarchical index construction module, The preset lower limit for the score. and These are the preset maximum and minimum subspace partition numbers, respectively.

[0036] Furthermore, in the retrieval and repair module, calculating the similarity between the semantic intent vector and the feature vector of high-quality data specifically includes:

[0037] The user's search request is input into a pre-defined deep bidirectional encoder model based on the Transformer architecture, and sentence vectors are extracted as the semantic intent vector. ;

[0038] Feature vectors of high-quality data are retrieved from the hot data storage area. The cosine similarity between the semantic intent vector and the feature vector of the high-quality data is calculated and used as the relevance score. :

[0039]

[0040] Where n is the dimension of the vector, and They are vectors and The component value in the i-th dimension, Represents the magnitude of a vector.

[0041] Furthermore, in the retrieval and repair module, generating virtual reference data based on the semantic intent vector specifically includes:

[0042] Obtain the standard data pattern definition of the business type to which the original data belongs. The standard data pattern definition includes field names, data types, and business constraints.

[0043] The semantic intent vector is decoded and restored into search conditions in the form of natural language text. A prompting engineering instruction containing the standard data pattern definition and the search conditions is constructed and input into a generative large language model driven by the prompting engineering instruction based on the business schema and search conditions. The virtual reference data must meet the requirements of complete fields and values ​​that conform to business constraints.

[0044] The structured data that conforms to the standard data pattern definition output by the generative large language model is received and used as the virtual reference data; wherein, all fields of the virtual reference data are filled with standard values ​​that conform to the business constraints.

[0045] Furthermore, the retrieval and repair module generates repair instructions to update and repair the data to be treated, and migrates the repaired data to be treated as high-quality data to the hot data storage area, specifically including:

[0046] The obtained data to be treated is compared with the virtual reference data field by field to identify missing or abnormal fields in the data to be treated.

[0047] Extract the standard values ​​of the corresponding fields from the virtual reference data, generate an update instruction for the data to be treated as the repair instruction, and execute the instruction to complete the data repair;

[0048] Obtain all field contents of the repaired data, and call the feature extraction model in the data acquisition and processing module to regenerate the feature vector with the original dimensions;

[0049] The repaired data and its newly generated feature vector are written into the hot data storage area, and the corresponding index entry is inserted into the high-precision vector index. At the same time, the data and its index entry are physically deleted from the cold data storage area and the low-precision vector index.

[0050] Furthermore, the hot data storage area uses a memory database or NVMe SSD high-speed storage medium, while the cold data storage area uses a mechanical hard disk drive (HDD) or distributed object storage medium.

[0051] Furthermore, it also includes a timed scanning module, which is used to periodically recalculate the dynamic quality score of high-quality data in the hot data storage area. If the recalculated score is lower than the grading threshold, a data degradation process is triggered to migrate the data to the cold data storage area and rebuild the low-precision vector index.

[0052] The beneficial effects of this invention are as follows:

[0053] This invention significantly improves system resource utilization and data recall by constructing a two-way feedback loop between quality and retrieval. First, by integrating a dynamic quality scoring mechanism that combines static compliance, dynamic activity, and conflict factors, the value of data is accurately identified. Based on this, hierarchical storage and adaptive product quantization index construction are implemented. This ensures a high-precision retrieval experience for high-quality hot data while significantly reducing the storage and computational overhead of cold data requiring governance. More importantly, this invention proposes a reverse repair mechanism triggered by retrieval failure. Using virtual reference data generated by a large model as a probe, it proactively captures and repairs hidden data requiring governance in low-precision indexes, elevating it to high-quality data. This breaks the deadlock caused by dirty data being unretrievable and unretrievable leading to irreparable damage, achieving dynamic optimization of data quality throughout its entire lifecycle. Existing technologies do not combine dynamic scoring for data governance, hierarchical indexing for retrieval, and AI-generated virtual reference data. The closed-loop mechanism of this invention is an innovative fusion of cross-domain technologies, not a conventional improvement by those skilled in the art. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the structure of the big data full lifecycle quality control and efficient retrieval system of the present invention.

[0055] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0057] like Figure 1As shown, this invention provides a big data full lifecycle quality control and efficient retrieval system, deployed in a cloud-based distributed computing environment, with an underlying hardware architecture employing a layered storage design:

[0058] Hot data storage area: High-performance NVMe SSDs (non-volatile memory host controller interface standard solid-state drives) or in-memory databases (such as Redis Cluster) are used to store frequently accessed and high-quality data, ensuring millisecond-level retrieval response.

[0059] Cold data storage area: uses large-capacity hard disk drives (HDDs) or distributed object storage (such as Amazon S3, Ceph) to store low-quality, unmanaged or infrequently accessed data to reduce storage costs.

[0060] The system comprises three interconnected core modules: a data acquisition and processing module, a hierarchical index construction module, and a retrieval and repair module. Additionally, it includes a timed scanning module, used periodically (e.g., every morning) to recalculate the dynamic quality score of high-quality data in the hot data area. If the recalculated score falls below the hierarchical threshold due to time decay, a data degradation process is triggered, migrating the data to the cold data storage area and rebuilding a low-precision vector index.

[0061] I. Data Acquisition and Processing Module

[0062] The data acquisition and processing module is used to access multi-source heterogeneous raw data, extract features from the raw data using a preset feature extraction model to obtain feature vectors with original dimensions, and verify the raw data based on a preset quality verification rule base to generate a dynamic quality score.

[0063] In a preferred embodiment, the data acquisition and processing module serves as the system's entry point, primarily responsible for multi-source data access, feature vectorization, and the core dynamic quality score calculation tasks. Its specific operation flow is described in detail below:

[0064] Step 1: Access and Feature Extraction of Multi-Source Heterogeneous Data

[0065] First, the data acquisition and processing module receives raw data from multiple heterogeneous sources through a pre-defined interface adapter. Data sources include, but are not limited to, relational databases (such as MySQL), non-relational databases (such as MongoDB), system operation logs, and JSON data returned by API interfaces. The module has a built-in pre-defined feature extraction model (e.g., a BERT model based on the Transformer architecture or a ResNet model). When the raw data enters the system, this model encodes unstructured data (such as text descriptions and image data), converting it into feature vectors with their original dimensions (e.g., a 768-dimensional dense vector). This feature vector will serve as the basis for subsequently building a high-precision index.

[0066] Step 2: The Logic of Generating Dynamic Quality Scores

[0067] While acquiring data, the module generates a dynamic quality score for the data based on a preset quality verification rule base, following the four sub-steps below. :

[0068] (1) Calculate the static compliance index

[0069] The module iterates through the quality verification rule base, performing static matching on the original data's field integrity (e.g., whether required fields are empty), format compliance (e.g., whether email addresses contain the @ symbol, whether mobile phone numbers are 11 digits), and value range validity (e.g., whether age is between 0 and 120). A static compliance index is calculated based on a weighted statistical method, using the following formula:

[0070]

[0071] Where N is the total number of validation rules, The preset weight for the i-th rule, For the coefficient of the verification result, when the verification passes... ,otherwise .

[0072] Assuming a single piece of user information data has N=4 rules in its rule base, each rule has a preset weight. All are 0.25. If the data passes the validation of rules 1, 2, and 3 ( However, rule 4 failed to validate. Then, the static compliance index is calculated as follows: =0.75. In practical applications, the calculation results can be normalized or mapped as needed.

[0073] (2) Calculate the dynamic activity index

[0074] Obtain the access frequency (F) of the raw data within a preset time window (e.g., the last 30 days) and the difference between the last update time and the current time. (Unit: days). The calculation formula combines the logic of logarithmic growth and exponential decay:

[0075]

[0076] Where F represents the access frequency. This is the difference between the last update time and the current time. For frequency weighting coefficients, This is the timeliness weighting coefficient. This is the time decay factor.

[0077] If a certain piece of data is accessed F=100 times, and it was updated today ( The first part (frequency contribution) is calculated to be 2.769; the second part (timeliness contribution) is calculated to be 0.4; therefore... It can be seen that, through This process prevented score explosions caused by excessively high access frequency; through... If As the value of this item increases, it rapidly approaches zero, reflecting the diminishing value of outdated data.

[0078] (3) Calculate the data conflict factor

[0079] First, key related fields (e.g., unique ID numbers or product SKU codes) are extracted from the original data. Using these as keys, a full-network search is performed on the heterogeneous original dataset to obtain multiple related data records. Next, the probability distribution of values ​​for corresponding attributes (e.g., the region field) in the original data and these related records is statistically analyzed, and the degree of disorder is calculated using the entropy principle from information theory.

[0080]

[0081] Where M represents the total number of different values ​​that the original data and the associated data records have on the corresponding attributes. Let be the probability of the j-th value occurring.

[0082] Suppose a search retrieves 3 related records by ID. In the region field, 2 records are "Beijing" and 1 record is "Shanghai". The total number of different values ​​is M=2 (Beijing and Shanghai). What is the probability of the value being "Beijing"? =2 / 3≈0.67; the probability of taking the value "Shanghai" =1 / 3≈0.33; =0.92. If all records are "Beijing", then the entropy is 0, and the conflict degree is the lowest.

[0083] (4) Generate the final dynamic quality score

[0084] Based on a pre-defined weighting strategy, the three indices are linearly weighted and fused. The data conflict factor is negatively correlated with quality (the greater the conflict, the worse the quality), therefore it is a subtractive term in the formula. The calculation formula is as follows:

[0085]

[0086] in, These are the preset normalized weight coefficients, such as the normalized weight coefficients. (Assuming the coefficient is relatively large to map the scores to the 0-100 range). For financial user data scenarios, the following settings are made: =0.4 (static compliance weight) =0.4 (Dynamic Active Weight) =0.2 (conflict weight), after testing with 1000 samples, the data quality identification accuracy reached 92%; the grading threshold was set to 60 points, and the data retrieval recall rate for scores ≥60 points increased by 40%. Substituting the above calculation results (the intermediate indices need to be standardized and normalized first; this is only a logical demonstration), a specific value is finally obtained. This score will be sent directly to the hierarchical index building module along with the data, serving as the sole basis for determining whether the data should enter the "hot data storage area" or the "cold data storage area".

[0087] II. Hierarchical Index Construction Module

[0088] The hierarchical index building module is used to mark the original data as high-quality data and store it in the hot data storage area when the dynamic quality score is higher than the preset hierarchical threshold, and to build a high-precision vector index based on the feature vector of the high-quality data; when the dynamic quality score is lower than the hierarchical threshold, the original data is marked as data to be treated and stored in the cold data storage area, and the feature vector of the data to be treated is subjected to dimensionality reduction and precision compression processing using an adaptive product quantization algorithm to build a low-precision vector index.

[0089] In a preferred embodiment, the hierarchical index construction module, as the core of the system's resource scheduling and indexing, is responsible for receiving raw data and its corresponding dynamic quality scores from the data acquisition and processing module. Based on the scoring results, differentiated storage and index building strategies are implemented. The specific implementation process is as follows:

[0090] Step 1: Threshold Determination and Triage Mechanism

[0091] First, obtain the preset grading threshold ( For example, set to 60 points). The received dynamic quality score ( Compare with this grading threshold:

[0092] Judging high-quality data: If If so, the original data is determined to be high-quality data.

[0093] Data to be processed: If If so, the original data is determined to be data to be processed.

[0094] Step 2: Processing flow for high-quality data (hot data channel)

[0095] For data packets marked as premium data, perform the following operations:

[0096] Storage media selection: Write the data to a hot data storage area, which physically uses an in-memory database (such as Redis) or NVMe SSD high-speed storage media to ensure extremely high I / O throughput.

[0097] High-precision index construction: To ensure retrieval accuracy, the original dimensions of the feature vectors are kept unchanged (e.g., the 768-dimensional floating-point vector is not compressed). A graph-based approximate nearest neighbor search algorithm (Graph-based ANN) is employed to construct a hierarchical navigation graph index (HNSW). This index structure allows for precise greedy searches in high-dimensional space, ensuring that high-quality data is prioritized and accurately recalled during retrieval.

[0098] Step 3: Processing flow for the data to be processed (cold data channel)

[0099] For data packets marked as data to be processed, the module writes them to a cold data storage area (such as a hard disk drive (HDD) or distributed object storage) and initiates an adaptive product quantization algorithm for dimensionality reduction and precision compression. This algorithm aims to sacrifice some retrieval accuracy for better storage space and computational efficiency, and specifically includes the following three steps:

[0100] Step A: Determine the product quantization parameters based on the score. This is the core step in achieving "adaptive" performance. The product quantization parameters are determined based on the dynamic quality score, with the most critical parameter being the number of subspace partitions, m. The specific calculation logic is as follows:

[0101] Get preset parameters: Get the preset lower limit of the rating ( For example, 0 points), grading threshold ( For example, 60 points), minimum subspace partitioning number ( (e.g., 8) and the maximum number of subspace partitions ( For example, 64)4.

[0102] Calculate the number of subspace partitions: Use linear interpolation to calculate the m-value corresponding to the current data. The calculation formula is as follows:

[0103]

[0104] in, This indicates a floor operation. Suppose a score for a piece of data to be processed... Substituting into the formula above, we calculate m=36; this means that the feature vector of the data will be divided into 36 subspaces. Therefore, the lower the dynamic quality score, the smaller the number of subspaces calculated, the higher the data compression ratio, and the less storage resources required.

[0105] Step B, Vector Segmentation

[0106] Based on the subspace partition number m (e.g., 36) determined in step A, the module divides the high-dimensional feature vector (e.g., 768 dimensions) of the data to be managed into m low-dimensional sub-vectors in terms of dimensions. If the original dimension is D, then the dimension of each low-dimensional sub-vector is D / m. In the example above, the dimension of each sub-vector is 768 / 36 ≈ 21.3 dimensions.

[0107] Step C: Clustering Encoding and Index Sequence Generation

[0108] For each segmented low-dimensional subvector, the module performs a matching search in a pre-defined codebook to find the cluster center (Centroid) with the closest Euclidean distance to that low-dimensional subvector. It then obtains the index number (ID) of that cluster center and replaces the original floating-point subvector with this ID. Finally, it concatenates the index numbers corresponding to all m low-dimensional subvectors according to the segmentation order to form a sequence of m integers.

[0109] Ultimately, this index sequence is stored as a low-precision vector index in the cold data storage area. Compared to the original 768-dimensional floating-point vector, this index sequence significantly reduces storage usage while retaining the coarse-grained semantic information needed for subsequent reverse engineering.

[0110] (3) Search and Repair Module

[0111] The retrieval and repair module receives user retrieval requests and generates semantic intent vectors. Based on these semantic intent vectors and the high-precision vector index, it retrieves high-quality data from the hot data storage area. When the similarity between the semantic intent vector and the feature vector of the high-quality data is lower than a preset recall threshold, virtual reference data is generated based on the semantic intent vector. Then, based on the low-precision vector index, an approximate nearest neighbor search is performed on the data to be treated in the cold data storage area. When the similarity between the obtained data to be treated and the virtual reference data is higher than a preset matching threshold, a repair instruction is generated to update and repair the data to be treated. The repaired data to be treated is then migrated to the hot data storage area as high-quality data. The hot data storage area uses Redis Cluster, with a retrieval response time ≤50ms; the cold data storage area uses Ceph distributed storage, reducing storage costs by 60%; cold data repair improves accuracy to ≥88%, and the recall rate of hidden dirty data is improved by 55% compared to existing technologies.

[0112] In a preferred embodiment, the retrieval and repair module, as a key component for the system to implement "retrieval-driven governance," is responsible for processing users' real-time retrieval requests and triggering a reverse detection mechanism when regular retrieval fails, to automatically discover and repair the data to be governed in the cold data storage area. The specific implementation process includes the following four core steps:

[0113] Step 1: Semantic Vector Generation and Relevance Score Calculation

[0114] When a user's search request is received (e.g., "Looking for the contact number of Li Si, the sales manager in Beijing"), the module first performs semantic parsing and scoring calculation, as follows:

[0115] Semantic Vector Extraction: The user's natural language search request is input into a pre-defined deep bidirectional encoder model based on the Transformer architecture (e.g., BERT or RoBERTa). The model extracts the sentence vector of the search request and generates a semantic intent vector. .

[0116] Hotspot Retrieval and Scoring: Based on the semantic intent vector, a search is performed in the high-precision vector index of the hotspot data storage area to obtain candidate high-quality data feature vectors. .

[0117] Similarity calculation: The cosine similarity formula is used to calculate the degree of matching between the search intent and the candidate data, which is used as the relevance score. :

[0118]

[0119] Where n is the dimension of the vector, and They are vectors and The component value in the i-th dimension, Represents the magnitude of a vector.

[0120] Step 2: Failure determination and virtual reference data generation

[0121] The calculated maximum relevance score is compared with a preset recall threshold (e.g., 0.75). When If the data falls below this threshold, it is determined that there is no data meeting the conditions in the current hot data area (i.e., the retrieval fails). At this point, the module automatically triggers a reverse probing process to generate virtual reference data.

[0122] Get Schema: Get the standard data schema definition (Schema) of the business type to which the original data belongs. This definition includes field names (such as name, job title, phone number), data types, and business constraints.

[0123] Constructing a Prompt: Decode the semantic intent vector back into search criteria in natural language form (or directly use the original search terms), combine it with the standard data schema definition, and construct a prompt engineering instruction (e.g.: "Generate a standard JSON data that matches the characteristics of 'Beijing Sales Manager Li Si' based on the following schema...").

[0124] Large Model Generation: Input instructions into a pre-defined generative large language model (LLM). The model outputs a structured data set with complete structure and standardized content based on probabilistic predictions, serving as a virtual reference data set. All fields of this data set are populated with standard values ​​that meet the constraints (for example, the model automatically completes the predicted telephone number format or placeholders).

[0125] Step 3: Cold Zone Reverse Probe (Approximate Nearest Neighbor Search)

[0126] The generated virtual reference data is used as a "probe" to scan the cold data storage area. Since the data in the cold data area is used to construct a low-precision vector index through an adaptive product quantization algorithm, an approximate nearest neighbor search (ANN) is performed based on this index.

[0127] The system searches for the data to be processed that is closest to the virtual reference data in the feature space. When the similarity between the searched data to be processed and the virtual reference data is higher than a preset matching threshold (e.g., 0.85), the data to be processed is considered to be the target data that the user originally wanted to retrieve but was not indexed due to quality issues (such as missing key fields).

[0128] Step 4: Data Repair and Hotspot Promotion

[0129] Once the target data to be addressed is identified, the module performs automatic repair and migration operations, completing the closed loop of the data lifecycle. This includes the following steps:

[0130] a. Compare the data to be addressed with the virtual reference data field by field to identify missing fields (such as empty phone numbers) or abnormal fields in the data to be addressed. Extract the standard values ​​of the corresponding fields from the virtual reference data and generate an update instruction.

[0131] b. Use repair commands to update the data to be governed, fill in missing information, or correct incorrect formats.

[0132] c. Obtain the complete data after repair, call the feature extraction model in the data acquisition and processing module, and regenerate the feature vector with the original dimensions (at this time, dimensionality reduction and compression are no longer performed).

[0133] d. Write the repaired data and its newly generated feature vector into the hot data storage area, and insert the corresponding index entry into the high-precision vector index. At the same time, physically delete the data and its old index entry from the cold data storage area and the low-precision vector index, completing the data's transition from "cold" to "hot".

[0134] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0135] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A big data full lifecycle quality control and efficient retrieval system, characterized in that, It includes interconnected data acquisition and processing modules, hierarchical index building modules, and retrieval and repair modules; The data acquisition and processing module is used to access multi-source heterogeneous raw data, and uses a feature extraction model based on the Transformer architecture to extract features from the raw data to obtain feature vectors with original dimensions. The module also verifies the raw data based on a configurable rule base that includes three types of verification rules: field integrity, format standardization, and value range validity, to generate a dynamic quality score. The feature extraction model based on the Transformer architecture is used to encode unstructured data into a 768-dimensional dense feature vector. The hierarchical index building module is used to mark the original data as high-quality data and store it in the hot data storage area when the dynamic quality score is higher than the preset hierarchical threshold, and to build a high-precision vector index based on the feature vector of the high-quality data; when the dynamic quality score is lower than the hierarchical threshold, the original data is marked as data to be treated and stored in the cold data storage area, and the feature vector of the data to be treated is subjected to dimensionality reduction and precision compression processing using an adaptive product quantization algorithm to build a low-precision vector index. The retrieval and repair module is used to receive user retrieval requests and generate semantic intent vectors. Based on the semantic intent vectors and the high-precision vector index, it retrieves high-quality data in the hot data storage area. When the similarity between the semantic intent vector and the feature vector of the high-quality data is lower than a preset recall threshold, it generates virtual reference data based on the semantic intent vector and performs an approximate nearest neighbor search on the data to be treated in the cold data storage area based on the low-precision vector index. When the similarity between the data to be treated and the virtual reference data is higher than a preset matching threshold, it generates a repair instruction to update and repair the data to be treated and migrates the repaired data to be treated as high-quality data to the hot data storage area. The system achieves dynamic optimization of hidden dirty data and on-demand resource allocation through quality scoring-driven hierarchical storage, adaptive index construction and retrieval-repair closed loop.

2. The big data full lifecycle quality control and efficient retrieval system according to claim 1, characterized in that, The specific steps for generating the dynamic quality score in the data acquisition and processing module are as follows: (1) Traverse the configurable rule base containing three types of verification rules: field integrity, format standardization, and value range validity. Perform static matching on the field integrity, format standardization, and value range validity of the original data, and calculate the static compliance index based on the weighted statistical method. (2) Obtain the access frequency and last update time of the original data within a preset time window, process the access frequency based on the logarithmic growth processing method, and process the last update time based on the exponential decay processing method to calculate the dynamic activity index. (3) Extract the key related fields from the original data, retrieve multiple related data records from the multi-source heterogeneous original dataset based on the key related fields, calculate the probability distribution of the values ​​of the original data and the multiple related data records on the corresponding attributes, and calculate the degree of disorder of the value distribution of the original data and the related data records based on the entropy calculation formula in information theory to obtain the data conflict factor. (4) Based on the preset weight allocation strategy, the static compliance index, the dynamic activity index and the data conflict factor are linearly weighted and fused to obtain the dynamic quality score.

3. The big data full lifecycle quality control and efficient retrieval system according to claim 2, characterized in that, The static compliance index The calculation formula is: Where N is the total number of validation rules, The preset weight for the i-th rule, For the coefficient of the verification result, when the verification passes... ,otherwise ; Dynamic Activity Index The calculation formula is: Where F represents the access frequency. This is the difference between the last update time and the current time. For frequency weighting coefficients, This is the timeliness weighting coefficient. This is the time decay factor; Data conflict factor The calculation formula is: Where M represents the total number of different values ​​that the original data and the associated data records have on the corresponding attributes. Let j be the probability of the j-th value occurring. The dynamic quality score The calculation formula is: in, These are the preset normalized weighting coefficients.

4. The big data full lifecycle quality control and efficient retrieval system according to claim 1, characterized in that, In the hierarchical index construction module For the high-quality data, the original dimension of the feature vector is kept unchanged, and a hierarchical navigation graph index is constructed using a hierarchical navigation graph algorithm as the high-precision vector index, thus maintaining the original dimension of the feature vector and achieving millisecond-level recall of high-quality data. For the data to be processed, an adaptive product quantization algorithm is used for dimensionality reduction and precision compression to construct a low-precision vector index, specifically including: Step A: Determine the product quantization parameter based on the dynamic quality score; wherein the product quantization parameter includes at least the number of subspace partitions, and the lower the dynamic quality score, the smaller the number of subspace partitions; Step B: Divide the feature vector of the data to be managed into multiple low-dimensional sub-vectors according to the number of subspace partitions; wherein the number of low-dimensional sub-vectors is equal to the number of subspace partitions; Step C: For each of the low-dimensional sub-vectors, find the nearest cluster center in the preset codebook to obtain the cluster center index number corresponding to each low-dimensional sub-vector. Concatenate the index numbers corresponding to all the low-dimensional sub-vectors from the data to be managed according to the segmentation order, and use the resulting index number sequence as the low-precision vector index.

5. The big data full lifecycle quality control and efficient retrieval system according to claim 4, characterized in that, Step A specifically includes: Obtain a preset lower limit for scoring and the grading threshold to construct an adaptive mapping interval, and obtain a preset minimum number of subspace partitions and a preset maximum number of subspace partitions, and calculate the number of subspace partitions corresponding to the data to be governed, using the following formula: in, This indicates the floor function. The dynamic quality score for the data to be addressed. The preset hierarchical threshold in the hierarchical index construction module, The preset lower limit for the score. and These are the preset maximum and minimum subspace partition numbers, respectively.

6. The big data full lifecycle quality control and efficient retrieval system according to claim 1, characterized in that, In the retrieval and repair module, the similarity between the semantic intent vector and the feature vector of high-quality data is calculated, specifically including: The user's search request is input into a pre-defined deep bidirectional encoder model based on the Transformer architecture, and sentence vectors are extracted as the semantic intent vector. ; Feature vectors of high-quality data are retrieved from the hot data storage area. The cosine similarity between the semantic intent vector and the feature vector of the high-quality data is calculated and used as a relevance score. : Where n is the dimension of the vector, and They are vectors and The component value in the i-th dimension, Represents the magnitude of a vector.

7. The big data full lifecycle quality control and efficient retrieval system according to claim 6, characterized in that, The retrieval and repair module generates virtual reference data based on the semantic intent vector, specifically including: Obtain the standard data pattern definition of the business type to which the original data belongs. The standard data pattern definition includes field names, data types, and business constraints. The semantic intent vector is decoded and restored into search conditions in the form of natural language text. A prompting engineering instruction containing the standard data pattern definition and the search conditions is constructed and input into a generative large language model driven by the prompting engineering instruction based on the business schema and search conditions. The virtual reference data must meet the requirements of complete fields and values ​​that conform to business constraints. The structured data that conforms to the standard data pattern definition output by the generative large language model is received and used as the virtual reference data; wherein, all fields of the virtual reference data are filled with standard values ​​that conform to the business constraints.

8. The big data full lifecycle quality control and efficient retrieval system according to claim 7, characterized in that, The retrieval and repair module generates repair instructions to update and repair the data to be treated, and migrates the repaired data to the hot data storage area as high-quality data. Specifically, this includes: The obtained data to be treated is compared with the virtual reference data field by field to identify missing or abnormal fields in the data to be treated. Extract the standard values ​​of the corresponding fields from the virtual reference data, generate an update instruction for the data to be treated as the repair instruction, and execute the instruction to complete the data repair; Obtain all field contents of the repaired data, and call the feature extraction model in the data acquisition and processing module to regenerate the feature vector with the original dimensions; The repaired data and its newly generated feature vector are written into the hot data storage area, and the corresponding index entry is inserted into the high-precision vector index. At the same time, the data and its index entry are physically deleted from the cold data storage area and the low-precision vector index.

9. The big data full lifecycle quality control and efficient retrieval system according to claim 1, characterized in that, The hot data storage area uses a memory database or NVMe SSD high-speed storage medium, while the cold data storage area uses a mechanical hard disk drive (HDD) or distributed object storage medium.

10. The big data full lifecycle quality control and efficient retrieval system according to claim 1, characterized in that, It also includes a timed scanning module, which is used to periodically recalculate the dynamic quality score of high-quality data in the hot data storage area. If the recalculated score is lower than the grading threshold, a data degradation process is triggered to migrate the data to the cold data storage area and rebuild the low-precision vector index.

Images