Intelligent data quality evaluation method for large-scale data management platform

Abstract

The invention discloses a data quality intelligent evaluation method for a large-scale data management platform, and relates to the related technical field of data processing, and the method comprises the steps: receiving real-time uploading data of a data uploading party; collecting a historical uploading data stream, and extracting a plurality of historical downloading data streams of a plurality of data downloading parties associated with the data uploading party; removing redundant data based on the duplicate checking result to obtain updated uploading data; performing quality evaluation on the real-time uploaded data in combination with the historical uploaded data stream, the plurality of historical downloaded data streams and the service information of the data uploader, and outputting a data quality evaluation result; and compressing the updated and uploaded data according to a data quality evaluation result, and uploading the compressed updated and uploaded data to a data management platform. The technical problems of low data quality evaluation efficiency, untimely redundant data processing and lack of real-time intelligent management in the prior art are solved, and the technical effects of improving data quality management efficiency and accuracy and optimizing data storage and transmission performance are achieved.

Description

Technical Field

[0001] The present application relates to the technical field related to data processing, and in particular to a method for intelligently evaluating data quality for a large-scale data management platform. Background Art

[0002] With the continuous development of information technology, the generation, storage and transmission of data have gradually shown an explosive growth trend. Especially in the application scenarios of large-scale data management platforms, data quality management and effective utilization have become a key link. Data quality directly affects the effect and results of data processing, especially for platforms that need to efficiently manage and analyze massive data. The guarantee of data quality is particularly important. However, large-scale data management platforms usually need to process a large number of data streams from multiple data uploaders and data downloaders. The data generated by each uploader and downloader has different timeliness, accuracy, completeness and consistency. How to automatically identify and eliminate redundant data to avoid unnecessary burden on the data platform and ensure the accuracy of data quality assessment is a problem that needs to be solved urgently.

[0003] At the current stage, relevant technologies have technical problems such as low efficiency of data quality assessment, untimely processing of redundant data, and lack of real-time intelligent management. Summary of the invention

[0004] This application solves the technical problems of low data quality assessment efficiency, untimely processing of redundant data, and lack of real-time intelligent management in the prior art by providing a data quality intelligent assessment method for large-scale data management platforms, thereby achieving the technical effect of improving data quality management efficiency and accuracy and optimizing data storage and transmission performance.

[0005] The present application provides a data quality intelligent assessment method for a large-scale data management platform, comprising: receiving real-time uploaded data from a data uploader; collecting historical uploaded data streams of the data uploader, and extracting multiple historical downloaded data streams of multiple data downloaders associated with the data uploader; checking for duplicates on the real-time uploaded data according to the historical uploaded data streams, removing redundant data based on the duplicate checking results, and obtaining updated uploaded data; evaluating the quality of the real-time uploaded data in combination with the historical uploaded data streams, the multiple historical downloaded data streams, and the business information of the data uploader, and outputting a data quality evaluation result; and compressing the updated uploaded data according to the data quality evaluation result and uploading it to a data management platform.

[0006] In a possible implementation, the quality of the real-time uploaded data is evaluated in combination with the historical uploaded data stream, the multiple historical downloaded data streams, and the business information of the data uploader, and the data quality evaluation result is output. The following processing is also performed: data upload quality analysis is performed based on the historical uploaded data stream to generate a first quality indicator; download behavior quality analysis is performed based on the multiple historical downloaded data streams to generate a second quality indicator; weighted improvement weights are configured based on the business information of the data uploader; after fusing the first quality indicator and the second quality indicator, a weighted improvement calculation of the fused indicator is performed using the weighted improvement weights to generate the data quality evaluation result.

[0007] In a possible implementation, data upload quality analysis is performed based on the historical uploaded data stream to generate a first quality indicator, and the following processing is also performed: data integrity identification is performed on the historical uploaded data stream to generate an integrity indicator; consistency identification is performed on the historical uploaded data stream according to a predefined format and a predefined logical relationship to generate a consistency indicator; the integrity indicator and the consistency indicator are standardized and then weighted to generate the first quality indicator.

[0008] In a possible implementation, a download behavior quality analysis is performed based on the multiple historical download data streams to generate a second quality indicator, and the following processing is also performed: an access frequency distribution analysis is performed based on the multiple historical download data streams, a significant value analysis is performed based on the access frequency distribution information, and a significant value indicator is generated; download feedback is extracted based on the multiple historical download data streams, and an actual application value analysis is performed based on the download feedback to generate an actual application value indicator; the significant value indicator and the actual application value indicator are integrated to generate the second quality indicator.

[0009] In a possible implementation, the real-time uploaded data is checked for duplicates according to the historical uploaded data stream, and redundant data is eliminated based on the checking result to obtain updated uploaded data. The following processing is also performed: a feature hash checking model is introduced to check for duplicates on the historical uploaded data stream and the real-time uploaded data to generate the checking result; a dynamic threshold elimination strategy is adopted based on the checking result to determine high redundancy data; the high redundancy data is deleted from the real-time uploaded data to generate the updated uploaded data.

[0010] In a possible implementation, a feature hash duplication detection model is introduced to perform duplication detection on the historical uploaded data stream and the real-time uploaded data to generate the duplication detection result, and the following processing is also performed: the historical uploaded data stream and the real-time uploaded data are respectively divided into data blocks and hash value calculations are performed to generate a historical data block hash value set and a real-time data block hash value set; the historical data block hash value set and the real-time data block hash value set are subjected to local hash value similarity analysis through the feature hash duplication detection model, and the real-time data blocks whose similarity with the historical data block hash value is greater than the preset similarity are output to generate the duplication detection result.

[0011] In a possible implementation, the updated uploaded data is compressed and uploaded to a data management platform according to the data quality evaluation result, and the following processing is also performed: a hot storage area and a cold storage area, as well as a hot storage data quality threshold, are configured on the data management platform; when the data quality evaluation result meets the hot storage data quality threshold, the updated uploaded data is compressed and uploaded to the hot storage area, and stored according to data type classification; when the data quality evaluation result does not meet the hot storage data quality threshold, the updated uploaded data is compressed and uploaded to the cold storage area, and stored according to data type classification.

[0012] In a possible implementation, the data quality intelligent assessment method for a large-scale data management platform also performs the following processing: obtaining the data download requirements of any real-time data downloader; matching the corresponding data storage location and data storage capacity whose data quality is higher than the hot storage data quality threshold based on the data download requirements; and recommending the corresponding data storage location and the data storage capacity to any real-time data downloader.

[0013] The intelligent data quality assessment method for large-scale data management platforms proposed in this application is intended to receive real-time uploaded data from data uploaders; collect historical uploaded data streams from data uploaders, and extract multiple historical downloaded data streams from multiple data downloaders associated with the data uploader; remove redundant data based on duplicate checking results to obtain updated uploaded data; evaluate the quality of real-time uploaded data in combination with historical uploaded data streams, multiple historical downloaded data streams, and business information of the data uploader, and output data quality evaluation results; compress the updated uploaded data according to the data quality evaluation results and upload it to the data management platform. This solves the technical problems of low data quality assessment efficiency, untimely redundant data processing, and lack of real-time intelligent management in the prior art, and achieves the technical effect of improving data quality management efficiency and accuracy and optimizing data storage and transmission performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0015] Figure 1 A flow chart of a data quality intelligent assessment method for a large-scale data management platform provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of a process for outputting data quality evaluation results in a data quality intelligent evaluation method for a large-scale data management platform provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0018] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0019] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0020] The present application embodiment provides a data quality intelligent assessment method for a large-scale data management platform, such as Figure 1 As shown, the method includes:

[0021] Step S100: receiving real-time uploaded data from a data uploader.

[0022] Preferably, the data management platform receives data streams transmitted in real time by data uploaders (i.e., data providers or sources), where uploaders refer to entities that send data to the platform. Uploaders can be various sources, such as IoT devices (such as sensors, smart hardware, etc.), mobile applications, enterprise business systems, online services, etc. Real-time uploaded data may be data generated or updated in real time from devices, users, applications or other management systems, and are usually constantly changing dynamically, such as sensor data, user operation data, transaction data, log data, multimedia content in transmission, etc. The uploaded data may include data in various formats, such as text data, pictures, videos, sensor data (such as temperature, humidity, location, etc.), log data, etc.

[0023] Step S200: collecting the historical upload data stream of the data uploader, and extracting multiple historical download data streams of multiple data downloaders associated with the data uploader.

[0024] Preferably, the historical upload data stream of the data uploader is collected, that is, the data records that the data uploader (i.e., the data provider) has uploaded to the data management platform in the past period of time are obtained. These data are a collection of data generated by the data uploader and sent to the platform in a historical period of time, including various types of data uploaded by the data uploader at different time points, which may include data from the past few days, weeks or even months. By analyzing the historical data, we can better understand the upload rules of the data uploader, data quality changes and other characteristics, thereby helping to identify potential problems in real-time uploaded data. For example, a seller on an e-commerce platform has uploaded various types of data about products (such as product inventory, price, sales volume, etc.) in the past year. These historical data are the historical upload data stream; then extract the historical data related to the data uploader. The historical download data streams of multiple data downloaders associated with the platform, that is, the data that multiple data downloaders (that is, various users or applications that obtain data from the platform) have downloaded from the platform in the past period of time. Among them, the data downloaders may be different systems, terminals, applications or users, relying on the platform's open interface or other methods to obtain data. The downloaded data of each data downloader may contain different data types, time intervals and data contents. These historical download data streams are extracted in order to obtain the upstream and downstream data flows related to the data uploader. Specifically, different downloaders may have used the uploader's data. Based on the historical download data streams of these downloaders, it is possible to understand the usage of the uploaded data in different users or systems, and whether these download behaviors will affect the data quality.

[0025] Step S300: Check the real-time uploaded data for duplication according to the historical uploaded data stream, remove redundant data based on the duplication check result, and obtain updated uploaded data.

[0026] Preferably, the real-time uploaded data is checked for duplicates using the acquired historical uploaded data stream, i.e., the real-time uploaded data is compared with the historical uploaded data stream to identify whether there is duplicate data, wherein the historical uploaded data stream refers to the historical data record that has been uploaded to the platform, and contains all the data content previously uploaded by the data uploader. Specifically, when a new data upload (i.e., real-time uploaded data) occurs, the platform will perform a duplicate check on the data to check whether it is duplicated with the historical data uploaded previously, i.e., match it based on the data content, such as by comparing the data ID, timestamp, data features (such as text, numbers, hash values ​​of files, etc.). If the real-time uploaded data is consistent with the historical uploaded data content (or very similar), then the data can be considered to be duplicated; if The duplicate check identifies which real-time uploaded data is duplicate with the historical uploaded data (i.e., redundant data), and then eliminates the redundant data to avoid storing duplicate and unnecessary data on the platform, save storage space and improve data management efficiency. Specifically, the method of eliminating redundant data usually includes directly deleting the duplicate data, or marking the duplicate data as no longer needing to be uploaded and no longer needing further processing. The data management platform decides whether to update or delete the data based on the duplicate check results to ensure that the uploaded data is new and non-duplicate. The updated uploaded data is finally obtained, that is, the valid data after duplicate checking and redundancy elimination. It may be the part of the original data retained after redundant elimination, or it may be the data that has been corrected, supplemented or updated, representing the latest and most useful data, thereby improving the quality of data evaluation.

[0027] Furthermore, step S300 also includes step S310, introducing a feature hash duplication detection model to perform duplication detection on the historical uploaded data stream and the real-time uploaded data to generate the duplication detection result; step S320, determining high redundancy data based on the duplication detection result by adopting a dynamic threshold elimination strategy; step S330, deleting the high redundancy data from the real-time uploaded data to generate the updated uploaded data.

[0028] Preferably, a feature hash duplication checking model is used to check the historical uploaded data stream and the real-time uploaded data for duplication, so as to identify and mark duplicate data, wherein the feature hash duplication checking model is a model used for duplication checking, and performs similarity analysis based on hash values, and has higher efficiency and lower computational cost. Specifically, features that can represent the data content are extracted from the historical uploaded data stream, and feature hash processing is performed, that is, a hash function is applied to map these features to a fixed hash space, and corresponding hash values ​​are generated and stored in a hash table. Similarly, for each piece of data in the real-time uploaded data, feature hash processing is also performed, and a hash value is generated. The hash table is searched to see whether there is historical data matching the hash value. If so, the real-time data is considered to be duplicated; if not, If it exists, it is considered to be new data, and a hash result is generated. Based on the duplicate checking results, a dynamic threshold elimination strategy is used to determine the high redundancy data, where the redundancy rate refers to the proportion of duplicate data to the total data. There are multiple historical data. If the real-time data is only similar to one of the data blocks, the redundancy rate is relatively low. Specifically, the proportion of historical data blocks in the historical data whose similarity to any real-time data block is greater than a certain threshold is calculated as the redundancy rate. Only when the redundancy rate reaches a certain threshold will it be deleted. The redundancy rate threshold can be dynamically adjusted by the management personnel. For example, if it is found that there are too many similar data in the current database, the threshold can be appropriately lowered. After being processed by the elimination strategy, updated uploaded data is generated, which helps to optimize data quality, reduce storage costs, and improve the efficiency of subsequent data analysis and processing.

[0029] Furthermore, step S310 also includes step S311, performing hash value calculations on the historical uploaded data stream and the real-time uploaded data after data segmentation, thereby generating a historical data block hash value set and a real-time data block hash value set; and step S312, performing local hash value similarity analysis on the historical data block hash value set and the real-time data block hash value set through the feature hash duplication detection model, outputting real-time data blocks whose similarity to the historical data block hash value is greater than a preset similarity, thereby generating the duplication detection result.

[0030] Preferably, data segmentation refers to dividing the historical upload data stream and the real-time upload data into smaller data blocks to reduce the complexity of a single hash calculation and improve the accuracy and efficiency of duplicate checking. The size of the data block can be set according to the characteristics and requirements of the data, for example, according to factors such as the file size and content complexity of the data, and then a hash function is applied to each data block to generate a corresponding hash value. The hash values ​​of all historical data blocks are combined into a set, called a historical data block hash value set; similarly, the hash values ​​of all real-time data blocks are combined into a set, called a real-time data block hash value set; and then feature hashing is used to generate a hash value set. The duplicate checking model performs similarity analysis of local hash values ​​on the hash value sets of historical data blocks and the hash value sets of real-time data blocks, that is, analyzes the similarity of the hash values ​​of a specific part or key feature in the data block, further improving the accuracy and efficiency of duplicate checking, and outputs real-time data blocks whose similarity with the hash values ​​of historical data blocks is greater than a preset similarity, where the preset similarity is a threshold used to determine whether two hash values ​​are close enough. All real-time data blocks whose similarity with the hash values ​​of historical data blocks is greater than the preset similarity are aggregated to generate the final duplicate checking results, which helps to identify and process duplicate or highly similar data, thereby improving the quality and uniqueness of the data.

[0031] Step S400 , performing a quality evaluation on the real-time uploaded data in combination with the historical uploaded data stream, the multiple historical downloaded data streams and the business information of the data uploader, and outputting a data quality evaluation result.

[0032] Preferably, a comprehensive quality assessment is performed on the real-time uploaded data by comprehensively analyzing multiple data sources (including historical uploaded data, historical downloaded data, and business background information of the data uploader), including evaluating the quality of the uploaded data from different dimensions, thereby obtaining a data quality evaluation result. Specifically, by comparing and combining the historical uploaded data stream with the real-time uploaded data, it is possible to analyze whether the real-time uploaded data meets the uploader's past data quality standards. For example, if the data previously uploaded by the uploader has certain specific quality problems (such as missing data, inconsistent format, etc.), it is possible to evaluate whether the real-time uploaded data continues to have similar problems by comparing with historical data. The quality evaluation detects the consistency of real-time data with historical data, including format, completeness, accuracy, etc. If the real-time uploaded data is found to have obvious deviations or problems, a lower quality score can be given.

[0033] Preferably, by analyzing the historical download data stream, the actual usage of the uploaded data, as well as the dependence and concerns of different downloaders on the data, can be understood. Different downloaders have different requirements for data quality. Through the historical download data stream, the platform can understand the performance of the uploaded data in different users or systems. If certain data is downloaded frequently and there are no quality problems, it means that the quality of these data is high; conversely, if certain data downloaders frequently complain or report data problems, it means that the data quality is poor. By combining this method with the historical download data stream, the platform can have a more comprehensive understanding of the quality of real-time uploaded data, especially the actual impact and value of the uploaded data on different downloaders. For example, if a certain real-time uploaded data (such as commodity price information) is used by multiple data downloaders, the platform can analyze the feedback of these downloaders. If some downloaders often encounter situations where the downloaded data is incomplete, formatted incorrectly, or cannot be correctly parsed, it can be inferred that the data quality is poor.

[0034] Preferably, the business information of the data uploader includes the business background, industry situation, upload history, product characteristics, operation specifications, etc. of the uploader. The business information of the uploader can provide important context for the evaluation of data quality. For example, some industries (such as finance, e-commerce, etc.) have high requirements for the accuracy and timeliness of data. The business characteristics of the data uploader may affect the quality standards of its uploaded data. Combined with the business information of the uploader, the platform can judge the quality of the real-time uploaded data according to the business characteristics and requirements of the uploader. If the business requirements of the uploader are high, the uploaded data should have higher quality standards, and strict quality evaluation is performed based on these standards. By integrating the historical upload data flow, the historical download data flow and the business information of the uploader, the platform will conduct a comprehensive quality evaluation of the real-time uploaded data. The data quality evaluation results usually include a score for the data quality (such as high, medium, and low levels), and may be accompanied by specific descriptions of data quality problems (such as missing, redundant, inconsistent, etc.). The output evaluation results can help the data uploader understand the quality level of its data upload and make improvements based on the evaluation results. It can also perform corresponding data processing based on the results, such as optimizing storage, cleaning redundant data, repairing erroneous data, etc., thereby improving the reliability and value of the data.

[0035] Further, such as Figure 2As shown, step S400 also includes step S410, performing data upload quality analysis based on the historical upload data stream to generate a first quality indicator; step S420, performing download behavior quality analysis based on the multiple historical download data streams to generate a second quality indicator; step S430, configuring weighted improvement weights based on the business information of the data uploader; step S440, after fusing the first quality indicator and the second quality indicator, performing weighted improvement calculation of the fused indicator through the weighted improvement weights to generate the data quality evaluation result.

[0036] Preferably, a data upload quality analysis is performed on the historical upload data stream, that is, the performance of the uploaded data in terms of accuracy, completeness, consistency, etc. is evaluated, such as checking whether the data is lost, repeated or wrong, and a first quality indicator is generated based on the evaluation result to measure the overall quality of the historical data of the uploader; a download behavior quality analysis is performed on multiple historical download data streams, that is, with the historical download data as the core, the actual application of the download data stream is analyzed to evaluate its business value, such as analyzing whether the data is frequently downloaded or used, whether error feedback is generated after the data is downloaded (such as the downloader reports data missing or format problems), or whether the data successfully supports the downloader. The actual use effect of the business decision-making is then calculated; then, the weighted improvement weight is configured in combination with the business attributes of the data uploader (such as industry type, authority of data source, business needs, data importance, etc.), that is, the data quality weights of different business scenarios are adjusted, and finally the first quality indicator and the second quality indicator are integrated, and the calculation is performed based on the configured weighted improvement weight to generate the final fusion indicator as the data quality evaluation result, which is used to reflect the overall quality level of the real-time uploaded data, which can be provided to the data uploader for optimizing its data quality, and can also provide a decision-making basis for the data management platform in subsequent operations (such as storage, transmission, analysis, etc.).

[0037] Furthermore, step S410 also includes step S411, identifying the data integrity of the historical uploaded data stream and generating an integrity indicator; step S412, identifying the consistency of the historical uploaded data stream according to a predefined format and a predefined logical relationship, and generating a consistency indicator; step S413, standardizing the integrity indicator and the consistency indicator and then weighting them to generate the first quality indicator.

[0038] Preferably, the data integrity of the historical uploaded data stream is identified, specifically including field checking, i.e., analyzing each field of the uploaded data to confirm whether there is a null value or an illegal value, and record matching, i.e., comparing the historical uploaded records to ensure that key data (such as transaction ID, timestamp) is not lost or tampered with, thereby generating an integrity index to reflect the degree of data integrity; then, the consistency of the historical uploaded data stream is identified according to the predefined format and predefined logical relationship, including checking whether the data meets the specific format requirements and whether the relationship between the data is valid (such as whether the foreign key reference is valid), i.e., format verification is performed to check whether the uploaded data meets the predefined template or format (such as time format, value range), and logical relationship check is performed to verify whether there is a logical contradiction between the data, for example, the transaction end time should not be earlier than the transaction start time, thereby generating a consistency index to reflect the degree of data consistency; then, the integrity index and the consistency index are standardized so that they have the same dimension or comparability, and then different weights are assigned according to the relative importance or influence of the index for weighted calculation to obtain a first quality index to more accurately reflect the quality of the data.

[0039] Furthermore, step S420 also includes step S421, performing access frequency distribution analysis based on the multiple historical download data streams, performing significant value analysis based on the access frequency distribution information, and generating a significant value indicator; step S422, extracting download feedback based on the multiple historical download data streams, performing actual application value analysis based on the download feedback, and generating an actual application value indicator; step S413, fusing the significant value indicator and the actual application value indicator to generate the second quality indicator.

[0040] Preferably, an access frequency distribution analysis is performed on multiple historical download data streams, that is, the number of accesses, access time periods, etc. of each data stream are counted and analyzed to form an access frequency distribution, such as analyzing the time distribution of access frequency (such as high-frequency distribution), wherein access frequency refers to the number of requests for data by the downloader, reflecting the intensity of demand for data in the platform, and based on the access frequency distribution information, statistical methods (such as hypothesis testing, variance analysis, etc.) or machine learning algorithms (such as classification, clustering, etc.) are used to deeply mine and analyze the access frequency distribution information, identify data streams with significant value, evaluate the significant value of different data streams, that is, their importance to users or businesses, and then generate significant value indicators to reflect the significance of different data streams in terms of user access frequency; extract multiple historical Download feedback of historical download data streams, that is, extracting user comments, error reports, ratings, usage time and other feedback data from multiple historical download data streams to understand the actual use value of the data streams, such as practicality and satisfaction. Based on the download feedback, the download feedback is quantified and analyzed using technical means such as text analysis and sentiment analysis to identify data streams with practical application value, evaluate the value of different data streams in practical applications, that is, their actual contribution to users or businesses, and then generate practical application value indicators to reflect the value of different data streams in practical applications; merge the significant value indicators and the practical application value indicators, such as weighted averaging, to obtain the second quality indicator, which comprehensively reflects the quality status of the download data stream in terms of significant value and practical application value.

[0041] Step S500: compress the updated uploaded data according to the data quality evaluation result and upload it to the data management platform.

[0042] Preferably, after quality assessment of the real-time uploaded data, the data is compressed according to the assessment results to optimize data storage and transmission efficiency by reducing the data volume. Specifically, data compression is usually performed when the data quality assessment is high and there is less redundant data to reduce storage space usage and improve subsequent data processing efficiency. Data compression is not just a simple compression algorithm processing of the data, but may also include optimizing the data according to the quality evaluation results, such as deleting unnecessary redundant data, repairing data of poor quality, format standardization, etc. After these processing, the data retains its valid information while compressing the data volume, improving the storage and transmission efficiency. The compressed data is then uploaded to the data management platform for further storage, processing, analysis or use, such as for access by other users or applications. The uploaded data must not only meet the storage format requirements of the platform, but also minimize redundant and invalid data, thereby improving data storage and management efficiency and ensuring data management reliability.

[0043] Furthermore, step S500 also includes step S510, configuring a hot storage area and a cold storage area, as well as a hot storage data quality threshold on the data management platform; step S520, when the data quality evaluation result meets the hot storage data quality threshold, compressing the updated uploaded data and uploading it to the hot storage area, and storing it according to data type classification; step S530, when the data quality evaluation result does not meet the hot storage data quality threshold, compressing the updated uploaded data and uploading it to the cold storage area, and storing it according to data type classification.

[0044] Preferably, on the data management platform, hot storage areas, cold storage areas and hot storage data quality thresholds are configured, wherein the hot storage data quality threshold is used to determine whether the data meets specific quality requirements, thereby determining whether the data should be stored in the hot storage area or the cold storage area. Specifically, hot storage is used to store data with high access frequency and high business value, and this area usually relies on high-performance storage devices or technologies (such as SSD and memory database) to achieve fast access and low-latency read and write operations; cold storage is used to store data with low access frequency or long-term archiving, and this area is usually based on low-cost, large-capacity storage devices (such as disk storage or cloud storage), which is simply archived to reduce storage costs; if the data quality evaluation result meets the hot storage data quality threshold, that is, the data is considered to be of high quality, then the data will be compressed and uploaded to the hot storage area, and classified and stored according to the data type to improve the accessibility and processing efficiency of the data; if the data quality evaluation result does not meet the hot storage data quality threshold, that is, the data is considered to be of low quality, then the data will be compressed and uploaded to the cold storage area, and also classified and stored according to the data type to improve the accessibility and processing efficiency of the data, while ensuring the long-term retention and security of the data.

[0045] Furthermore, step S520 also includes step S521, obtaining the data downloading requirements of any real-time data downloader; step S522, matching the corresponding data storage location and data storage capacity whose data quality is higher than the hot storage data quality threshold based on the data downloading requirements; step S523, recommending the corresponding data storage location and the data storage capacity to any real-time data downloader.

[0046] Preferably, the real-time data downloader refers to the user or application that is currently requesting to download data, and obtains its data download requirements, which may include data type, data volume, data format, data time range, etc., and then filters out data whose data quality is higher than the hot storage data quality threshold, and then performs a query or search operation on the storage system to find data blocks or data sets that meet the requirements, determine the storage location of the selected data in the storage system (that is, which storage device it is stored in) and their storage capacity (that is, the size and quantity of the data), this information may include the physical location of the data (such as the identifier of the storage device, the name of the storage pool, etc.), the logical location of the data (such as the name of the data set, the number of the data block, etc.), the size or quantity of the data, etc. After determining the data storage location and data storage capacity, this information is recommended to the real-time data downloader to ensure the accuracy and efficiency of data downloading, thereby improving data availability, optimizing storage resource utilization, and enhancing user experience.

[0047] The above specific implementation manner does not constitute a limitation to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and can still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. An intelligent data quality assessment method for a large-scale data management platform, characterized in that: include: Receive real-time uploaded data from the data uploader; Collecting the historical upload data stream of the data uploader, and extracting multiple historical download data streams of multiple data downloaders associated with the data uploader; Check the real-time uploaded data for duplication according to the historical uploaded data stream, remove redundant data based on the duplication check result, and obtain updated uploaded data; Performing a quality evaluation on the real-time uploaded data in combination with the historical uploaded data stream, the multiple historical downloaded data streams, and the business information of the data uploader, and outputting a data quality evaluation result; The updated uploaded data is compressed according to the data quality evaluation result and then uploaded to the data management platform.

2. The data quality intelligent assessment method for a large-scale data management platform according to claim 1, characterized in that: The quality of the real-time uploaded data is evaluated in combination with the historical uploaded data stream, the multiple historical downloaded data streams, and the business information of the data uploader, and a data quality evaluation result is output, including: Performing data upload quality analysis based on the historical upload data stream to generate a first quality indicator; Performing download behavior quality analysis based on the multiple historical download data streams to generate a second quality indicator; Configuring weighted boosting weights based on the business information of the data uploader; After the first quality indicator and the second quality indicator are fused, a weighted lifting calculation of the fused indicator is performed using the weighted lifting weight to generate the data quality evaluation result.

3. The data quality intelligent assessment method for a large-scale data management platform as claimed in claim 2, characterized in that: Performing data upload quality analysis based on the historical upload data stream to generate a first quality indicator includes: Performing data integrity identification on the historical uploaded data stream and generating an integrity indicator; Perform consistency identification on the historical uploaded data stream according to a predefined format and a predefined logical relationship, and generate a consistency index; The integrity index and the consistency index are standardized and then weighted to generate the first quality index.

4. The data quality intelligent assessment method for a large-scale data management platform as claimed in claim 2, characterized in that: Performing download behavior quality analysis based on the multiple historical download data streams to generate a second quality indicator includes: Performing access frequency distribution analysis based on the multiple historical download data streams, performing significant value analysis based on the access frequency distribution information, and generating a significant value indicator; Extracting download feedback based on the multiple historical download data streams, performing actual application value analysis based on the download feedback, and generating an actual application value indicator; The significant value indicator and the practical application value indicator are integrated to generate the second quality indicator.

5. The data quality intelligent assessment method for a large-scale data management platform according to claim 1, characterized in that: Checking the real-time uploaded data for duplication according to the historical uploaded data stream, removing redundant data based on the duplication checking result, and obtaining updated uploaded data, including: Introducing a feature hash duplication checking model to check the duplication of the historical uploaded data stream and the real-time uploaded data, and generating the duplication checking result; Based on the duplicate checking results, a dynamic threshold elimination strategy is used to determine high redundancy rate data; The high redundancy rate data is deleted from the real-time upload data to generate the update upload data.

6. The data quality intelligent assessment method for a large-scale data management platform according to claim 5, characterized in that: Introducing a feature hash duplication checking model to check the historical uploaded data stream and the real-time uploaded data for duplication, and generating the duplication checking result, including: The historical uploaded data stream and the real-time uploaded data are divided into blocks and then hash value calculations are performed to generate a historical data block hash value set and a real-time data block hash value set; The characteristic hash duplication checking model is used to perform similarity analysis of local hash values ​​on the historical data block hash value set and the real-time data block hash value set, and the real-time data blocks whose similarity to the historical data block hash value is greater than a preset similarity are output to generate the duplication checking result.

7. The data quality intelligent assessment method for a large-scale data management platform according to claim 1, characterized in that: Compressing the updated uploaded data according to the data quality evaluation result and uploading it to the data management platform includes: Configuring hot storage areas and cold storage areas, and hot storage data quality thresholds on the data management platform; When the data quality evaluation result meets the hot storage data quality threshold, the updated uploaded data is compressed and uploaded to the hot storage area, and stored according to data type classification; When the data quality evaluation result does not meet the hot storage data quality threshold, the updated uploaded data is compressed and uploaded to the cold storage area, and stored according to data type classification.

8. The data quality intelligent assessment method for a large-scale data management platform according to claim 7, characterized in that: Also includes: Obtain data download requirements from any real-time data downloader; Matching the corresponding data storage location and data storage volume whose data quality is higher than the hot storage data quality threshold based on the data downloading demand; The corresponding data storage location and the data storage amount are recommended to any real-time data downloading party.

Images