Batch data verification method and device, electronic equipment and storage medium
Through K-mean clustering algorithm and parallel task queue parallel verification, combined with incremental verification and real-time monitoring, the problem of low batch data verification efficiency in the existing technology is solved, and efficient and reliable data verification is achieved to adapt to large-scale Scale data processing.
Patent Information
- Application Number
- CN202411065381.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, batch data verification efficiency is low, especially when the data volume is huge, single-thread processing leads to a lot of time.
The target batch data is divided into K sub-data sets using K mean clustering algorithm, and these sub-data sets are allocated to N parallel task queues. N parallel task queues perform verification tasks in parallel, combining incremental checksum real-time Monitoring and feedback mechanism, dynamically adjust the verification strategy.
It significantly improves the efficiency and reliability of data verification, adapts to the needs of large-scale data processing, reduces manual intervention, and has good scalability and automation capabilities.
Smart Images

Figure CN119025532A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a batch data verification method, device, electronic device and storage medium. Background Art
[0002] The current mainstream way to process batch data is mostly to use list loop processing. Since this method uses a single thread, the data verification efficiency is very low, and it may take several minutes to process thousands of data. When the amount of data is very large, it will take even longer. Summary of the invention
[0003] In view of this, embodiments of the present application provide a batch data verification method, device, electronic device and storage medium to solve the problem of low efficiency of batch data verification in the prior art.
[0004] According to a first aspect of an embodiment of the present application, a batch data verification method is provided, the method being executed by a terminal device, comprising:
[0005] Get target batch data;
[0006] In response to determining that the target batch data is the first batch of data to be verified, obtaining a preset number K of data subset divisions, where K is a positive integer greater than 1;
[0007] Use K-means clustering algorithm to divide the target batch data into K sub-datasets;
[0008] The K sub-data sets are distributed to N parallel task queues, and the N parallel task queues perform verification of the target batch data in parallel, where N is a positive integer.
[0009] A second aspect of the embodiment of the present application provides a batch data verification device, including:
[0010] An acquisition module, configured to acquire target batch data;
[0011] The acquisition module is further configured to, in response to determining that the target batch data is the first batch of data to be verified, acquire a preset data subset division number K, where K is a positive integer greater than 1;
[0012] A partitioning module is configured to partition the target batch data into K sub-datasets using a K-means clustering algorithm;
[0013] The verification module is configured to distribute the K sub-data sets to N parallel task queues, and the N parallel task queues execute the verification of the target batch data in parallel, where N is a positive integer.
[0014] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0015] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain target batch data, and when the target batch data is the first batch of data to be verified, obtain a preset K value and use a K-means clustering algorithm to divide the target batch data into K sub-data sets, and then allocate the K sub-data sets to N parallel task queues, which execute the verification tasks of each sub-data set in parallel. The efficiency and reliability of data verification are improved by performing intelligent batch optimization and parallel verification on batch data, and the requirements of large-scale data processing can be met.
[0017] Furthermore, the technical solution of the embodiment of the present application also adopts an incremental verification method when verifying data, which further improves the consistency and accuracy of data verification. At the same time, based on real-time monitoring and feedback mechanisms, the verification strategy can be dynamically adjusted to adapt to different data scales and system resource conditions. The solution reduces manual intervention, has good scalability, and realizes data processing automation. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 It is a scenario diagram of an application scenario of an embodiment of the present application.
[0020] Figure 2 It is a flow chart of a batch data verification method provided in an embodiment of the present application.
[0021] Figure 3 It is a flow chart of the method for determining the initial K value provided in an embodiment of the present application.
[0022] Figure 4 It is a flowchart of a method for allocating K sub-data sets to N parallel task queues provided in an embodiment of the present application.
[0023] Figure 5 It is a flowchart of a method for a terminal device to verify data in a sub-data set provided in an embodiment of the present application.
[0024] Figure 6 It is a flow chart of another batch data verification method provided in an embodiment of the present application.
[0025] Figure 7 It is a flow chart of another batch data verification method provided in an embodiment of the present application.
[0026] Figure 8 It is a flow chart of another batch data verification method provided in an embodiment of the present application.
[0027] Fig. 9 It is a schematic diagram of a batch data verification device provided in an embodiment of the present application.
[0028] Fig.10 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0029] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0030] A batch data verification method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0031] Figure 1 1 is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario may include terminal devices 1 and 2 and a network 3.
[0032] Terminal device 1 may be a field device for collecting data, or a transfer device that receives data from a field device and sends the data to other devices for processing and storage. Terminal device 2 may be a device that receives batch data, processes and stores the batch data. Terminal devices 1 and 2 may communicate via network 3.
[0033] The network 5 can be a wired network connected by coaxial cable, twisted pair and optical fiber, or it can be a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present application.
[0034] The user can collect or receive batch data through the terminal device 1, and send the batch data to the terminal device 2 via the network 3. Further, the terminal device 2 pre-processes and verifies the received batch data, and stores the batch data after the verification is successful.
[0035] It should be noted that the specific types, quantities and combinations of the terminal devices 1 and 2 and the network 3 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present application do not limit this.
[0036] As mentioned above, the current mainstream way to process batch data is mostly to use list loop processing. Since this method uses a single thread, the data verification efficiency is very low, and it may take several minutes to process thousands of data. When the amount of data is very large, it will take even longer.
[0037] In view of this, an embodiment of the present application provides a batch data verification method, which obtains target batch data. When the target batch data is the first batch of data to be verified, a preset K value is obtained and a K-means clustering algorithm is used to divide the target batch data into K sub-data sets. The K sub-data sets are then assigned to N parallel task queues, and the N parallel task queues execute the verification tasks of each sub-data set in parallel. The efficiency and reliability of data verification are improved by performing intelligent batch optimization and parallel verification on batch data, and it can adapt to large-scale data processing needs.
[0038] Figure 2 It is a flow chart of a batch data verification method provided in an embodiment of the present application. Figure 2 The batch data verification method can be Figure 1 The terminal device 2 executes. Figure 2 As shown, the method comprises the following steps:
[0039] In step S201, target batch data is acquired.
[0040] In step S202, in response to determining that the target batch data is the first batch of data to be verified, a preset number K of data subset divisions is obtained.
[0041] Wherein, K is a positive integer greater than 1.
[0042] In step S203, the target batch data is divided into K sub-data sets using a K-means clustering algorithm.
[0043] In step S204, the K sub-data sets are allocated to N parallel task queues, and the N parallel task queues perform verification of the target batch data in parallel.
[0044] Wherein, N is a positive integer.
[0045] In an embodiment of the present application, the batch data verification method can be executed by a terminal device, wherein the terminal device can be a computer or a server, or can be other terminal devices with certain data processing capabilities.
[0046] In the embodiment of the present application, during batch data verification, the target batch data may be first acquired, and it may be determined whether the target batch data is the first batch of data to be verified.
[0047] In an embodiment of the present application, when it is determined that the target batch data is the first batch of data to be verified this time, the preset number of data subset divisions K can be obtained, and then the obtained K value is used to divide the target batch data into K sub-data sets using a K-means Clustering algorithm, where K is a positive integer greater than 1.
[0048] In one example, when dividing the first batch of data into sub-data sets, the K value can be set by the user according to the data characteristics and data similarity of the first batch of data.
[0049] Furthermore, using the K-means Clustering algorithm to divide the target batch data into K sub-datasets can be, first, determining K data as the centroids in the target batch data, and dividing the remaining data into a sub-dataset where the centroid is located; then updating the centroid in the divided sub-dataset to the actual centroid of the sub-dataset; next, recalculating the distance from each data to each centroid, and re-dividing the sub-dataset according to the calculation result; iteratively executing the steps of re-determining the centroid of the updated sub-dataset, calculating the distance from each data to the updated centroid, and re-dividing the sub-dataset according to the calculation result, until the re-divided sub-dataset is the same as the sub-dataset divided last time, or the number of iterations exceeds a preset threshold, and obtaining the divided K sub-datasets.
[0050] In the embodiment of the present application, the K sub-data sets obtained by the division can be allocated to N parallel task queues, and the N parallel task queues perform verification of the target batch data in parallel, where N is a positive integer.
[0051] In the embodiment of the present application, N parallel task queues can execute the target batch data verification in parallel using a parallel processing framework, for example, calling Java's Fork framework or Join framework to manage and schedule verification tasks.
[0052] According to the technical solution provided in the embodiment of the present application, by acquiring the target batch data, when the target batch data is the first batch of data for this verification, the preset K value is obtained and the K-means clustering algorithm is used to divide the target batch data into K sub-data sets, and then the K sub-data sets are allocated to N parallel task queues, and the N parallel task queues execute the verification tasks of each sub-data set in parallel. The efficiency and reliability of data verification are improved by performing intelligent batch optimization and parallel verification of batch data, which can adapt to large-scale data processing needs.
[0053] Figure 3 : is a flow chart of the method for determining the initial K value provided in the embodiment of the present application. Figure 3 As shown, the method comprises the following steps:
[0054] In step S301 , the number of data types in the target batch data is obtained.
[0055] In step S302, the data similarity distribution in the target batch data is obtained.
[0056] In step S303, the K value is determined based on the number of data types and the data similarity distribution.
[0057] As mentioned above, when dividing the first batch of data into sub-data sets, the K value can be set by the user according to the data characteristics and data similarity of the first batch of data. Among them, although the user experience can be used for reference in the specific setting, some principles still need to be followed.
[0058] That is to say, when determining the initial K value, the number of data types in the target batch data can be first obtained, where the data types include, for example, numeric types, string types, etc.; further, the data similarity distribution in the target batch data can be obtained, and then the K value can be determined based on the number of data types and the data similarity distribution. Among them, the initial K value can be proportional to the number of data types, that is, the more data types contained in the target batch data, the more sub-data sets can be divided. On the other hand, the greater the similarity of the data in the target batch data, the fewer sub-data sets can be divided, that is, the initial K value can be inversely proportional to the data similarity.
[0059] Furthermore, when determining the initial K value, you can also obtain the current load rate of the terminal device and parameters such as the system resource utilization rate and the network resource utilization rate, and determine the initial K value in combination with the load rate and resource utilization rate, so that the determined K value adapts to the real-time load and resource utilization of the terminal device, thereby optimizing the overall verification efficiency.
[0060] In the embodiment of the present application, each parallel task queue can be processed by an independent verification thread pool. That is, after the target batch data is split into small batch data sets, each small batch data set is assigned to a parallel task queue, and each task queue is processed by an independent verification thread pool.
[0061] Figure 4 is a flow chart of a method for allocating K sub-data sets to N parallel task queues provided in an embodiment of the present application. Figure 4 As shown, the method comprises the following steps:
[0062] In step S401, the current load rate of the terminal device and the system resource usage rate of the first terminal device are obtained.
[0063] Among them, the system resource utilization rate of the first terminal device is the current system resource utilization rate of the terminal device.
[0064] In step S402, the number N of thread pools is determined.
[0065] The number of threads N is inversely proportional to the load rate of the terminal device and inversely proportional to the resource usage rate of the terminal device.
[0066] In step S403, N verification thread pools are opened, and a task queue is maintained in each verification thread pool.
[0067] In step S404, the K sub-data sets are allocated to task queues maintained by N verification thread pools.
[0068] In the embodiment of the present application, the current load rate of the terminal device and the system resource utilization rate of the first terminal device can be obtained first, and the system resource utilization rate of the first terminal device is the current system resource utilization rate of the terminal device.
[0069] Then, the number N of thread pools may be determined, and the number N of threads is inversely proportional to the load rate of the terminal device and inversely proportional to the resource usage rate of the terminal device.
[0070] Next, N verification thread pools can be started, and a task queue can be maintained in each verification thread pool. Finally, the K sub-data sets are assigned to the task queues maintained by the N verification thread pools.
[0071] In the embodiment of the present application, after the K sub-data sets are allocated to the task queues maintained by the N verification thread pools, the priorities of the task queues can also be adjusted according to the system resource conditions of the terminal device to improve resource utilization.
[0072] In one example, the task queue may include sub-datasets in the data of the current batch verification, and may also include sub-datasets in the data of the previous batch verification. When system resources allow the task queue to execute verification tasks, the sub-datasets in the data of the previous batch verification may be processed first according to actual needs, or the sub-datasets in the data of the current batch verification may be processed in a limited manner.
[0073] In another example, the task queue may include a sub-data set consisting of real-time data and a sub-data set consisting of offline data in the target batch verification data. In this case, the task queue may give priority to processing the sub-data set consisting of real-time data.
[0074] That is to say, by adopting the technical solution of the embodiment of the present application, the size of the thread pool and the priority of the task queue can be adjusted according to the system resource situation of the terminal device, thereby improving resource utilization.
[0075] Figure 5 FIG. 1 is a flow chart of a method for verifying data in a sub-data set by a terminal device according to an embodiment of the present application. Figure 5 As shown, the method comprises the following steps:
[0076] In step S501, a hash value of current data in a target sub-data set is calculated, where the target sub-data set is any sub-data set in each sub-data set.
[0077] The current data hash value is the hash value of the full data in the target sub-dataset, or the hash value of the incremental data in the target sub-dataset.
[0078] In step S502, the hash value of the previous version of the data in the target sub-data set is obtained.
[0079] The previous version of the data hash value is the full data hash value or incremental data hash value obtained by the last calculation of the data in the target sub-data set.
[0080] In step S503, in response to determining that the current data hash value is the same as the previous version of the data hash value, it is determined that the data in the target sub-data set is successfully verified.
[0081] In the embodiment of the present application, when verifying the data in each sub-dataset, the current data hash value in the target sub-dataset can be calculated first, the target sub-dataset is any sub-dataset in each sub-dataset, and the current data hash value is the hash value of the full data in the target sub-dataset, or the hash value of the incremental data in the target sub-dataset. Furthermore, the previous version of the data hash value of the data in the target sub-dataset can also be obtained, and the previous version of the data hash value is the full data hash value or the incremental data hash value obtained by the last calculation of the data in the target sub-dataset.
[0082] Then, the current data hash value is compared with the previous version of the data hash value. If the two are determined to be the same, the data verification in the target sub-dataset is determined to be successful. Otherwise, the data verification in the target sub-dataset is determined to have failed.
[0083] That is to say, when performing data verification, it can be achieved by calculating the full hash value and comparing the current hash value with the previous version of the hash value, or by introducing an incremental checksum algorithm to perform differential verification on the changed part of the data, reduce repeated verification calculations, and improve overall processing performance. Among them, the incremental checksum algorithm can improve verification efficiency by comparing the hash value of the previous version of the data and only verifying the changed data part.
[0084] Figure 6 : is a flow chart of another batch data verification method provided by an embodiment of the present application. Figure 6 Steps S601 to S604 in the embodiment shown are Figure 2 Steps S201 to S204 in the illustrated embodiment are substantially the same and will not be described in detail herein. Figure 6 As shown, the method also includes the following steps:
[0085] In step S605 , in response to determining that the target batch data is not the first batch of data for this batch verification, a historical verification report is obtained.
[0086] The historical verification report at least includes the system resource usage rate of the second terminal device when verifying the previous batch of data of the target batch data.
[0087] In step S606, the current load rate of the terminal device is obtained.
[0088] In step S607, an updated K value is determined based on the current load rate of the terminal device and the system resource usage rate of the second terminal device.
[0089] The updated K value is inversely proportional to the load rate of the terminal device and inversely proportional to the resource utilization rate of the terminal device.
[0090] In step S608 , the target batch data is divided into K sub-data sets using a K-means clustering algorithm based on the updated K value.
[0091] In step S609 , the K sub-data sets are allocated to N parallel task queues, and the N parallel task queues perform verification of the target batch data in parallel.
[0092] In an embodiment of the present application, if the target batch data is not the first batch of data for this batch verification, a historical verification report can be obtained, which at least includes the system resource usage rate of the second terminal device when verifying the previous batch of data of the target batch data.
[0093] For example, if the target batch data is the Mth batch data of this batch verification, a verification report including verification of the M-1th batch data can be obtained. Furthermore, verification reports of more batch data such as the M-2th batch and the M-3th batch can be obtained as needed. The verification report can include the system resource usage rate when verifying these batch data.
[0094] In the embodiment of the present application, the current load rate of the terminal device can also be obtained, and then the updated K value is determined based on the current load rate of the terminal device and the system resource utilization rate of the second terminal device. The updated K value is inversely proportional to the load rate of the terminal device and inversely proportional to the resource utilization rate of the terminal device.
[0095] In the embodiment of the present application, the target batch data can be divided into K sub-data sets using the K-means clustering algorithm based on the updated K value. Finally, the K sub-data sets are assigned to N parallel task queues, and the N parallel task queues perform verification of the target batch data in parallel.
[0096] That is, the system resources can be monitored in real time and a feedback mechanism can be introduced to dynamically adjust the batch size and partitioning strategy for the next batch of data to cope with changes in the data processing process.
[0097] In an embodiment of the present application, the target batch data is divided into K sub-data sets using the K-means clustering algorithm based on the updated K value, and the K sub-data sets can be allocated to N parallel task queues in the following manner: determine the updated number of thread pools N based on the current load rate of the terminal device and the system resource utilization rate of the second terminal device; adjust the number of verification thread pools using the updated N value; and allocate the updated K sub-data sets to the task queues maintained by the updated N verification thread pools.
[0098] That is to say, the terminal device load rate and resource usage obtained from the real-time monitoring and feedback mechanism can be used not only to update the K value, but also to dynamically adjust the N value in real time. When the load rate is high or the resource usage is high, the number of thread pools N is reduced, and vice versa, the number of thread pools N is increased to cope with changes in the data processing process.
[0099] Figure 7 : is a flow chart of another batch data verification method provided by the embodiment of the present application. Figure 7 Steps S701 to S704 in the embodiment shown are similar to Figure 2Steps S201 to S204 in the illustrated embodiment are substantially the same and will not be described in detail herein. Figure 7 As shown, the method also includes the following steps:
[0100] In step S705, verification results of the K sub-data sets are obtained respectively.
[0101] In step S706, the verification results of the K sub-data sets are processed using consistent hash values to generate a data verification report for the target batch data.
[0102] In the embodiment of the present application, after the verification of the target batch data is performed in parallel using N parallel task queues, the verification results of the K sub-data sets can be obtained respectively. Then, the verification results of the K sub-data sets are processed using consistent hashing values to generate a data verification report for the target batch data.
[0103] In other words, the verification results returned by each thread can be summarized and analyzed, and the consistent hash value can be used to summarize the verification results of each subset to generate an overall data verification report.
[0104] Figure 8 FIG. 1 is a flow chart of another batch data verification method provided in an embodiment of the present application. Figure 8 As shown, the batch data to be verified can first be input into the intelligent batch optimization module, where data preprocessing, subset division and dynamic adjustment of the next batch are performed.
[0105] Among them, data preprocessing refers to classifying data according to data type, that is, the intelligent batch optimization module receives the data set to be saved and preprocesses the data according to the data volume and data type.
[0106] Subsetting refers to querying the load status through the system load module and using the K-means algorithm to separate the subsets. In other words, when dividing the subsets, the K-means clustering algorithm in the machine learning algorithm can be used to divide the data set into multiple subsets based on data characteristics and similarities. According to this algorithm, the large data set is split into multiple optimized verification batches, and the data size of each batch is dynamically adjusted according to the current system load and network bandwidth to adapt it to the current system resource load to optimize the overall verification efficiency.
[0107] Dynamic adjustment of the next batch refers to feedback on the use of system resources and dynamic adjustment of the next batch. That is, the system monitoring module can monitor system resources in real time and introduce a feedback mechanism to dynamically adjust the batch size and partitioning strategy for the next batch of data to cope with changes in the data processing process.
[0108] The data processed by the intelligent batch optimization module is input into the parallel verification module, which is used to perform task allocation, parallel processing and data verification on the received data.
[0109] Among them, task allocation refers to allocating subset data to sub-threads for processing, and the thread pool size of each sub-thread and the priority of the queue maintained by each thread pool can be adjusted dynamically. That is, task allocation can be to allocate the split small batch data sets to parallel task queues, and each task queue is processed by an independent verification thread pool. Adjust the thread pool size and task queue priority according to the system resource situation to improve resource utilization.
[0110] Parallel processing refers to using frameworks such as Java Fork to manage and schedule tasks, and verifying the hash value of the subset after data processing is completed. That is, you can use a parallel processing framework, such as Java's Fork framework or Join framework, to manage and schedule verification tasks. Furthermore, each verification task calculates the hash value of the data subset and performs data consistency verification.
[0111] Data verification can be done by using an incremental verification algorithm to compare differences. In other words, an incremental verification algorithm can be introduced to perform differential verification on the changed parts of the data, reduce repeated verification calculations, and improve overall processing performance. Furthermore, the incremental verification algorithm compares the data hash value of the previous version and only verifies the changed data part, thereby improving verification efficiency.
[0112] After parallel processing of the data, the verification results of each subset can be obtained. Subsequently, the verification results of each subset can be aggregated and feedback optimized using the result integration and optimization feedback module.
[0113] Result aggregation refers to summarizing the data returned by the child threads and using the consistent hash value to verify the merged results. In other words, when performing result aggregation, the verification results returned by each thread can be summarized and analyzed, and then the consistent hash value can be used to summarize the verification results of each subset to generate an overall data verification report.
[0114] Feedback optimization refers to the use of feedback performance data for data preprocessing. That is, the strategy of preprocessing and parallel verification algorithms can be adjusted through the reporting result feedback mechanism, thereby dynamically optimizing the verification process and improving the accuracy and efficiency of data verification. Finally, the processed data is saved in the database.
[0115] By adopting the technical solution provided in the embodiment of the present application, when saving and verifying large quantities of data, the intelligent batch optimization algorithm is used to greatly improve the efficiency of data verification. The consistency and accuracy of data verification are improved through incremental verification. Based on real-time monitoring and feedback mechanisms, the verification strategy can be dynamically adjusted to adapt to different data scales and system resource conditions. It reduces manual intervention, has good scalability, realizes data processing automation, and adapts to large-scale data processing needs.
[0116] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0117] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0118] Fig. 9 Schematic diagram of a batch data verification device provided by an embodiment of the present application. Fig. 9 As shown, the device comprises:
[0119] The acquisition module 901 is configured to acquire target batch data.
[0120] The acquisition module 901 is further configured to acquire a preset data subset division number K in response to determining that the target batch data is the first batch of data to be verified, where K is a positive integer greater than 1.
[0121] The partitioning module 902 is configured to partition the target batch data into K sub-data sets using a K-means clustering algorithm.
[0122] The verification module 903 is configured to distribute the K sub-data sets to N parallel task queues, and the N parallel task queues perform verification of the target batch data in parallel, where N is a positive integer.
[0123] According to the technical solution provided in the embodiment of the present application, by acquiring the target batch data, when the target batch data is the first batch of data for this verification, the preset K value is obtained and the K-means clustering algorithm is used to divide the target batch data into K sub-data sets, and then the K sub-data sets are allocated to N parallel task queues, and the N parallel task queues execute the verification tasks of each sub-data set in parallel. The efficiency and reliability of data verification are improved by performing intelligent batch optimization and parallel verification of batch data, which can adapt to large-scale data processing needs.
[0124] In the embodiment of the present application, the K value is determined in the following manner: obtaining the number of data types in the target batch data; obtaining the data similarity distribution in the target batch data; and determining the K value based on the number of data types and the data similarity distribution.
[0125] In an embodiment of the present application, each parallel task queue is processed by an independent verification thread pool; K sub-data sets are allocated to N parallel task queues, including: obtaining the current load rate of the terminal device and the system resource utilization rate of the first terminal device, the system resource utilization rate of the first terminal device is the current system resource utilization rate of the terminal device; determining the number of thread pools N, the number of threads N is inversely proportional to the load rate of the terminal device, and inversely proportional to the resource utilization rate of the terminal device; starting N verification thread pools, and maintaining a task queue in each verification thread pool; allocating K sub-data sets to the task queues maintained by N verification thread pools.
[0126] In an embodiment of the present application, the terminal device verifies the data in each sub-data set in the following manner: calculate the current data hash value in the target sub-data set, the target sub-data set is any sub-data set in each sub-data set, the current data hash value is the hash value of the full data in the target sub-data set, or the hash value of the incremental data in the target sub-data set; obtain the previous version of the data hash value of the data in the target sub-data set, the previous version of the data hash value is the full data hash value or the incremental data hash value obtained by the last calculation of the data in the target sub-data set; in response to determining that the current data hash value is the same as the previous version of the data hash value, determine that the data in the target sub-data set has been successfully verified.
[0127] In an embodiment of the present application, it also includes: in response to determining that the target batch data is not the first batch of data for this batch verification, obtaining a historical verification report, the historical verification report at least including the system resource usage rate of the second terminal device when the previous batch of data of the target batch data was verified; obtaining the current load rate of the terminal device; based on the current load rate of the terminal device and the system resource usage rate of the second terminal device, determining an updated K value, the updated K value is inversely proportional to the load rate of the terminal device and inversely proportional to the resource usage rate of the terminal device; based on the updated K value, using a K-means clustering algorithm to divide the target batch data into K sub-data sets.
[0128] In an embodiment of the present application, it also includes: determining the updated number of thread pools N based on the current load rate of the terminal device and the system resource utilization rate of the second terminal device; adjusting the number of verification thread pools using the updated N value; and allocating the updated K sub-data sets to the task queues maintained by the updated N verification thread pools.
[0129] In an embodiment of the present application, it also includes: obtaining verification results of K sub-data sets respectively; using consistent hash values to process the verification results of the K sub-data sets to generate a data verification report for the target batch data.
[0130] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0131] Fig.10 Schematic diagram of an electronic device provided in an embodiment of the present application. Fig.10 As shown, the electronic device 10 of this embodiment includes: a processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the computer program 1003, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1001 executes the computer program 1003, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0132] The electronic device 10 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 10 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art will appreciate that Fig.10 The electronic device 10 is merely an example and does not limit the electronic device 10 . The electronic device 10 may include more or fewer components than shown in the figure, or different components.
[0133] Processor 1001 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0134] The memory 1002 may be an internal storage unit of the electronic device 10, for example, a hard disk or memory of the electronic device 10. The memory 1002 may also be an external storage device of the electronic device 10, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 10. The memory 1002 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 1002 is used to store computer programs and other programs and data required by the electronic device.
[0135] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0136] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0137] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A batch data verification method, characterized in that: The method is performed by a terminal device, and the method includes: Get target batch data; In response to determining that the target batch data is the first batch of data to be verified, obtaining a preset number K of data subset divisions, where K is a positive integer greater than 1; Using a K-means clustering algorithm to divide the target batch data into K sub-datasets; The K sub-data sets are distributed to N parallel task queues, and the N parallel task queues perform verification of the target batch data in parallel, where N is a positive integer.
2. The method according to claim 1, characterized in that: The K value is determined in the following manner: Obtaining the number of data types in the target batch data; Obtaining data similarity distribution in the target batch data; The K value is determined based on the number of data types and data similarity distribution.
3. The method according to claim 1, characterized in that Each parallel task queue is processed by an independent verification thread pool; The allocating the K sub-data sets to N parallel task queues includes: Acquire the current load rate of the terminal device and the system resource usage rate of the first terminal device, where the system resource usage rate of the first terminal device is the current system resource usage rate of the terminal device; Determine the number N of thread pools, wherein the number N of threads is inversely proportional to the load rate of the terminal device and inversely proportional to the resource utilization rate of the terminal device; Open N verification thread pools and maintain a task queue in each verification thread pool; The K sub-data sets are allocated to the task queues maintained by the N verification thread pools.
4. The method according to claim 1, characterized in that: The terminal device verifies the data in each sub-data set in the following manner: Calculate a hash value of current data in a target sub-dataset, where the target sub-dataset is any sub-dataset in each sub-dataset, and the hash value of current data is a hash value of all data in the target sub-dataset, or a hash value of incremental data in the target sub-dataset; Obtaining a hash value of the last version of the data in the target sub-data set, where the hash value of the last version of the data is a hash value of the full data or a hash value of the incremental data obtained by the last calculation of the data in the target sub-data set; In response to determining that the current data hash value is the same as the previous version of the data hash value, it is determined that the data in the target sub-data set is successfully verified.
5. The method according to claim 1, characterized in that The method further comprises: In response to determining that the target batch data is not the first batch of data for this batch verification, obtaining a historical verification report, the historical verification report at least including a system resource usage rate of the second terminal device when verifying a previous batch of data of the target batch data; Get the current load rate of the terminal device; Determine an updated K value based on the current load rate of the terminal device and the system resource utilization rate of the second terminal device, wherein the updated K value is inversely proportional to the load rate of the terminal device and inversely proportional to the resource utilization rate of the terminal device; The target batch data is divided into K sub-data sets using a K-means clustering algorithm based on the updated K value.
6. The method according to claim 5, characterized in that The method further comprises: Determine the updated number N of thread pools based on the current load rate of the terminal device and the system resource usage rate of the second terminal device; Use the updated N value to adjust the number of verification thread pools; The updated K sub-datasets are distributed to the task queues maintained by the updated N verification thread pools.
7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtain verification results of the K sub-datasets respectively; The verification results of the K sub-data sets are processed using consistent hash values to generate a data verification report for the target batch data.
8. A batch data verification device, characterized in that: include: An acquisition module, configured to acquire target batch data; The acquisition module is further configured to, in response to determining that the target batch of data is the first batch of data to be verified, acquire a preset data subset division number K, where K is a positive integer greater than 1; A partitioning module is configured to partition the target batch data into K sub-data sets using a K-means clustering algorithm; The verification module is configured to distribute the K sub-data sets to N parallel task queues, and the N parallel task queues perform verification of the target batch data in parallel, where N is a positive integer.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
