A multi-thread-based data verification method, device, equipment and medium

Abstract

The embodiment of the specification discloses a kind of data verification method, device, equipment and medium based on multithreading, method includes: data center receives the new data verification rule uploaded by user end, and synchronizes the new data verification rule to real-time verification channel and historical verification channel;In real-time verification channel, the data verification rule corresponding to the current uploaded acquisition data is acquired, to verify acquisition data based on parallel thread and corresponding data verification rule, obtain the verification result of acquisition data;In historical verification channel, the corresponding historical data is matched based on new data verification rule, to determine the historical data verification information corresponding to historical data;According to the number of server CPU core, determine the number of supportable thread task, to divide time based on historical verification information and the number of supportable thread task to historical data, obtain multiple verification tasks;Multiple verification tasks are processed by multiple threads concurrently, to update the verification result of historical data.

Description

Technical Field

[0001] This specification relates to the field of data verification technology, and in particular to a data verification method, apparatus, device and medium based on multithreading. Background Technology

[0002] Data verification refers to the process of checking and verifying the accuracy, completeness, and consistency of data. Through data verification, it is ensured that data meets expected quality standards, reducing the impact of errors and inconsistencies on business and decision-making. Currently, with the continuous development of data platforms, the number of data sources is constantly increasing. However, data collected from different sources inevitably contains some quality issues. These include human error, incorrect data, outdated data, or a lack of data literacy skills within the organization. Failure to correct these data problems in a timely manner can pose serious risks and harm to data users. Therefore, timely verification of data within the data platform is a necessary step to ensure the normal functioning of the data platform.

[0003] Existing technologies typically employ hash calculations on business data and consistency comparisons of the calculated hash values ​​to verify the integrity and security of received data. Other routine data verification methods can be processed in real-time during data acquisition, generally meeting users' routine data processing needs. However, when business stakeholders propose new data verification rules, and given large volumes of historical data, the technology still suffers from slow verification speeds. Furthermore, verifying large volumes of historical data can easily lead to resource conflicts with real-time verification. Therefore, a method that can improve the efficiency and quality of data verification is needed. Summary of the Invention

[0004] To address the aforementioned technical problems, this specification provides one or more embodiments of a multi-threaded data verification method, apparatus, device, and medium.

[0005] One or more embodiments of this specification employ the following technical solutions:

[0006] This specification provides one or more embodiments of a multi-threaded data verification method, the method including:

[0007] The data platform receives new data verification rules uploaded by users and synchronizes the new data verification rules to the real-time verification channel and the historical verification channel.

[0008] In the real-time verification channel, the data verification rules corresponding to the currently uploaded data are obtained, and the data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the data.

[0009] In the historical verification channel, the corresponding historical data is matched based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data;

[0010] The number of thread tasks that can be supported is determined based on the number of CPU cores on the server. The historical data is then divided into time segments based on the historical verification information and the number of thread tasks that can be supported, to obtain multiple verification tasks.

[0011] The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data.

[0012] Optionally, in one or more embodiments of this specification, the data verification rules corresponding to the currently uploaded collected data are obtained in the real-time verification channel, and the collected data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the collected data, specifically including:

[0013] The data source and basic data information corresponding to the collected data are obtained in the real-time verification channel; wherein, the basic data information includes: execution stage and execution business type;

[0014] Based on the data source and the basic data information, determine the data verification rules corresponding to each of the collected data;

[0015] Based on the upload time of each collected data and the number of thread tasks that can be supported, the collected data is sliced ​​and grouped so that each group of collected data can be verified in parallel according to the corresponding data verification rules to obtain the verification result of the collected data.

[0016] Optionally, in one or more embodiments of this specification, the historical data verification information corresponding to the historical data is determined by matching the corresponding historical data based on the newly added data verification rules in the historical verification channel, specifically including:

[0017] Determine the scope of business impact corresponding to the newly added data verification rule in the historical verification channel;

[0018] Obtain historical data within the scope of the affected business that matches the new data verification rule, and determine the total number of historical data entries;

[0019] The total time distribution of historical data that matches the verification rules of the newly added data is determined based on the verification time of each historical data, so that the total number of historical data entries and the total time distribution of the historical data are used as the historical data verification information.

[0020] Optionally, in one or more embodiments of this specification, the historical data is divided into time segments based on the historical verification information and the number of supportable thread tasks to obtain multiple verification tasks, specifically including:

[0021] Based on the total time distribution of the historical verification information and the total number of records, determine the initial slice time range of the historical data and the number of data records corresponding to each initial slice time range;

[0022] The average number of data entries in the historical data is determined based on the number of initial slice time ranges and the total number of entries. The number of data entries corresponding to each initial slice time range is compared with the average number of data entries to determine whether to adjust the initial slice time range.

[0023] Within the limit of the number of supported thread tasks, the initial slice time range is iteratively adjusted to obtain a slice time range. Based on the slice time range, the historical data is divided into time segments to obtain multiple verification tasks corresponding to the historical data.

[0024] Optionally, in one or more embodiments of this specification, comparing the number of data entries corresponding to each initial slice time range with the average number of data entries to determine whether to adjust the initial slice time range specifically includes:

[0025] Obtain the difference between the number of data entries corresponding to each of the initial slice time ranges and the average number of data entries to obtain the difference sequence corresponding to the initial slice time range;

[0026] If it is determined that there is a sudden change in adjacent data in the difference sequence, then the initial slice time range is adjusted accordingly.

[0027] If it is determined that there are no abrupt changes in adjacent data in the difference sequence, then the initial slice time range will not be adjusted.

[0028] Optionally, in one or more embodiments of this specification, the multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data, specifically including:

[0029] Based on the concurrent processing of the multiple verification tasks by the multiple threads, the current verification result of each of the historical data is obtained;

[0030] The verification results of the historical data are retrieved to update the verification results of the historical data based on the current verification results.

[0031] Optionally, in one or more embodiments of this specification, the data platform receives newly added data verification rules uploaded by the user terminal, specifically including:

[0032] The data platform receives custom rules uploaded by users and calls existing data verification rules of the data platform.

[0033] Compare the custom rules with the existing data verification rules to determine the comparison result;

[0034] If, based on the comparison results, it is determined that the custom rule is inconsistent with the existing data verification rule, then the custom rule will be used as a new data verification rule.

[0035] This specification provides one or more embodiments of a multi-threaded data verification device, the device comprising:

[0036] The rule update unit is used to accept new data verification rules uploaded by the user terminal in the data middle platform, and to synchronize the new data verification rules to the real-time verification channel and the historical verification channel.

[0037] The first verification unit is used to obtain the data verification rules corresponding to the currently uploaded collected data in the real-time verification channel, and to verify the collected data based on the parallel thread and the corresponding data verification rules to obtain the verification result of the collected data.

[0038] The determining unit is used to match the corresponding historical data in the historical verification channel based on the newly added data verification rules, and determine the historical data verification information corresponding to the historical data.

[0039] A partitioning unit is used to determine the number of thread tasks that can be supported based on the number of server CPU cores, and to divide the historical data into time based on the historical verification information and the number of thread tasks that can be supported, thereby obtaining multiple verification tasks.

[0040] The second verification unit is used to process the multiple verification tasks concurrently through multiple threads in order to update the verification results of the historical data.

[0041] This specification provides one or more embodiments of a multi-threaded data verification device, the device comprising:

[0042] At least one processor; and,

[0043] A memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0045] The data platform accepts new data verification rules uploaded by users and synchronizes the new data verification rules to the real-time verification channel and the historical verification channel.

[0046] In the real-time verification channel, the data verification rules corresponding to the currently uploaded data are obtained, and the data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the data.

[0047] In the historical verification channel, the corresponding historical data is matched based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data;

[0048] The number of thread tasks that can be supported is determined based on the number of CPU cores on the server. The historical data is then divided into time segments based on the historical verification information and the number of thread tasks that can be supported, to obtain multiple verification tasks.

[0049] The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data.

[0050] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0051] The data platform accepts new data verification rules uploaded by users and synchronizes the new data verification rules to the real-time verification channel and the historical verification channel.

[0052] In the real-time verification channel, the data verification rules corresponding to the currently uploaded data are obtained, and the data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the data.

[0053] In the historical verification channel, the corresponding historical data is matched based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data;

[0054] The number of thread tasks that can be supported is determined based on the number of CPU cores on the server. The historical data is then divided into time segments based on the historical verification information and the number of thread tasks that can be supported, to obtain multiple verification tasks.

[0055] The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data.

[0056] The above-described at least one technical solution used in the embodiments of this specification can achieve the following beneficial effects:

[0057] By synchronizing new data verification rules to both the real-time and historical verification channels, data verification updates for both real-time and historical data are achieved, avoiding potential conflicts between the real-time and historical data verification processes. After determining the number of supportable thread tasks based on the server's CPU core count, historical data is divided into multiple verification tasks according to historical verification information and the available thread count. This process effectively balances the workload of each thread, improving the stability of the data verification process. Furthermore, using multiple threads to verify historical data solves the problem of slow data verification speed. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0059] Figure 1 This is a schematic flowchart of a multi-threaded data verification method provided in the embodiments of this specification;

[0060] Figure 2 This specification provides a schematic diagram of the internal structure of a multi-threaded data verification device as an embodiment of the present invention.

[0061] Figure 3 A schematic diagram of the internal structure of a multi-threaded data verification device provided in this specification embodiment;

[0062] Figure 4 This is a schematic diagram of the internal structure of a non-volatile storage medium provided in the embodiments of this specification. Detailed Implementation

[0063] This specification provides a data verification method, apparatus, device, and medium based on multithreading.

[0064] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0065] like Figure 1As shown in the diagram, this specification provides a flowchart of a multi-threaded data verification method. Figure 1 As can be seen, in one or more embodiments of this specification, a data verification method based on multithreading includes the following steps:

[0066] S101: The data platform receives the newly added data verification rules uploaded by the user terminal and synchronizes the newly added data verification rules to the real-time verification channel and the historical verification channel.

[0067] To efficiently and accurately perform real-time verification of multi-source data and update verification of historical data when business parties propose new data verification rules, especially when the historical data volume is large, the data platform in this embodiment synchronizes the new data verification to both the real-time verification channel and the historical verification channel after receiving the new data verification rule uploaded by the user. This facilitates reliable verification of real-time and historical data. Specifically, in one or more embodiments of this specification, the data platform receives the new data verification rule uploaded by the user, which includes the following process: First, when a business party proposes a new data verification rule, the data platform receives the custom rule uploaded by the user. To avoid the problem of duplicate definition of data verification rules, it calls existing data verification rules in the data platform, comparing the custom rule with the existing rules to determine the comparison result. If the comparison result indicates that the custom rule and the existing rules are inconsistent, then the custom rule is adopted as the new data verification rule.

[0068] S102: Obtain the data verification rules corresponding to the currently uploaded data in the real-time verification channel, and verify the data based on the parallel thread and the corresponding data verification rules to obtain the verification result of the data.

[0069] After the data verification rules are updated, the data verification rules corresponding to the currently uploaded collected data are obtained in the verification channel. The collected data is then verified based on parallel threads and the corresponding data verification rules to obtain the verification result. Specifically, in one or more embodiments of this specification, obtaining the data verification rules corresponding to the currently uploaded collected data in the real-time verification channel, and then verifying the collected data based on parallel threads and the corresponding data verification rules to obtain the verification result, specifically includes the following process:

[0070] First, the data source and basic data information corresponding to the collected data are obtained from the real-time verification channel. It should be noted that the basic data information includes the execution stage and the type of business being executed. Then, based on the data source and basic data information, the data verification rules corresponding to each collected data are determined. According to the upload time and the number of thread tasks that can be supported for each collected data, the collected data is sliced ​​and grouped so that each group of collected data can be verified in parallel based on the corresponding data verification rules, obtaining the verification results of the collected data.

[0071] S103: In the historical verification channel, match the corresponding historical data based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data.

[0072] As data verification rules are improved, to avoid the problem of historical data becoming outdated and causing serious risks and damages to users due to the difficulty in timely verification and updating of historical data based on updated data verification rules, this specification's embodiments match corresponding historical data in the historical verification channel according to newly added data verification rules, thereby determining the historical data verification information corresponding to the historical data. Specifically, in one or more embodiments of this specification, determining the historical data verification information corresponding to the historical data by matching corresponding historical data based on newly added data verification rules in the historical verification channel includes the following steps:

[0073] First, after the new data verification rules are determined, the scope of business affected by the new data verification rules is identified in the historical verification channel. Then, historical data matching the new data verification rules within this scope is retrieved, and the total number of matching historical data entries is determined. Next, the total time distribution of the historical data matching the new data verification rules is determined based on the verification time of each historical data entry. The total number of historical data entries and the total time distribution of the historical data are then used as historical data verification information.

[0074] S104: Determine the number of thread tasks that can be supported based on the number of server CPU cores, and divide the historical data into time segments based on the historical verification information and the number of thread tasks that can be supported to obtain multiple verification tasks.

[0075] To ensure that parallel threads remain within the specified range of the server's CPU core count, thereby guaranteeing data verification efficiency and CPU service performance, this specification's embodiments first determine the number of supportable thread tasks based on the server's CPU core count. Then, based on historical verification information and the number of supportable thread tasks, historical data is divided into time periods to obtain multiple verification tasks. Specifically, in one or more embodiments of this specification, dividing historical data into time periods based on historical verification information and the number of supportable thread tasks to obtain multiple verification tasks includes the following process:

[0076] First, based on the total time distribution and total number of historical verification information, the initial slice time range and the number of data entries corresponding to each initial slice time range are determined. For example, the initial slice time range is divided by week, ten-day period, month, quarter, year, etc. After division, the number of data entries in each time range is counted and combined as the number of data entries corresponding to each initial slice time range. To ensure that the number of data entries verified by each thread task is not significantly different, that is, the number of data entries corresponding to each initial slice time range is not significantly different, thereby achieving full utilization of threads, the average number of data entries in historical data is determined based on the number of initial slice time ranges and the total number of data entries. Then, the number of data entries corresponding to each initial slice time range is compared with the average number of data entries to determine whether to adjust the initial slice time range. Within the limit of the number of thread tasks that can be supported, the initial slice time range is iteratively adjusted to obtain the slice time range. Then, based on the slice time range, the historical data is divided into time periods to obtain multiple verification tasks corresponding to the historical data.

[0077] Furthermore, in one or more embodiments of this specification, comparing the number of data entries corresponding to each initial slice time range with the average number of data entries to determine whether to adjust the initial slice time range specifically includes:

[0078] First, the difference between the number of data entries for each initial slice time range and the average number of data entries is obtained, thus generating a difference sequence for each initial slice time range. If it is determined that there are abrupt changes in adjacent data within the difference sequence, then the data entry deviation for the thread task is too large when verifying based on the initial slice time range, and the initial slice time range needs to be adjusted. Conversely, if it is determined that there are no abrupt changes in adjacent data within the difference sequence, then the initial slice time range does not need to be adjusted.

[0079] S105: The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data.

[0080] After determining the verification tasks based on step S104 above, multiple threads concurrently process these tasks, thereby updating the verification results of historical data and avoiding the adverse effects of outdated or erroneous historical data on subsequent use. Specifically, in one or more embodiments of this specification, multiple threads concurrently process multiple verification tasks to update the verification results of historical data, which includes the following process: First, multiple threads concurrently process multiple verification tasks to obtain the current verification result of each historical data. Then, the verification result of the historical data is updated based on the current verification result by calling the verification result of that historical data.

[0081] like Figure 2 As shown in the diagram, this specification provides a schematic diagram of the internal structure of a multi-threaded data verification device. Figure 2 As can be seen, in one or more embodiments of this specification, a data verification device based on multithreading includes:

[0082] Rule update unit 201 is used for the data middle platform to accept new data verification rules uploaded by the user terminal and synchronize the new data verification rules to the real-time verification channel and the historical verification channel;

[0083] The first verification unit 202 is used to obtain the data verification rules corresponding to the currently uploaded collected data in the real-time verification channel, and to verify the collected data based on the parallel thread and the corresponding data verification rules to obtain the verification result of the collected data.

[0084] The determining unit 203 is used to match the corresponding historical data in the historical verification channel based on the newly added data verification rules, and determine the historical data verification information corresponding to the historical data.

[0085] The partitioning unit 204 is used to determine the number of thread tasks that can be supported based on the number of server CPU cores, and to divide the historical data into time based on the historical verification information and the number of thread tasks that can be supported, so as to obtain multiple verification tasks.

[0086] The second verification unit 205 is used to process the multiple verification tasks concurrently through multiple threads in order to update the verification results of the historical data.

[0087] like Figure 3 As shown in the figure, this specification provides a multi-threaded data verification device, which includes:

[0088] At least one processor; and,

[0089] A memory communicatively connected to the at least one processor; wherein,

[0090] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0091] The data platform accepts new data verification rules uploaded by users and synchronizes the new data verification rules to the real-time verification channel and the historical verification channel.

[0092] In the real-time verification channel, the data verification rules corresponding to the currently uploaded data are obtained, and the data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the data.

[0093] In the historical verification channel, the corresponding historical data is matched based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data;

[0094] The number of thread tasks that can be supported is determined based on the number of CPU cores on the server. The historical data is then divided into time segments based on the historical verification information and the number of thread tasks that can be supported, to obtain multiple verification tasks.

[0095] The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data.

[0096] like Figure 4 As shown in the figure, this specification provides a schematic diagram of the internal structure of a non-volatile storage medium. Figure 4 As can be seen, in one or more embodiments of this specification, a non-volatile storage medium stores computer-executable instructions 401, which are capable of:

[0097] The data platform accepts new data verification rules uploaded by users and synchronizes the new data verification rules to the real-time verification channel and the historical verification channel.

[0098] In the real-time verification channel, the data verification rules corresponding to the currently uploaded data are obtained, and the data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the data.

[0099] In the historical verification channel, the corresponding historical data is matched based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data;

[0100] The number of thread tasks that can be supported is determined based on the number of CPU cores on the server. The historical data is then divided into time segments based on the historical verification information and the number of thread tasks that can be supported, to obtain multiple verification tasks.

[0101] The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data.

[0102] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0103] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0104] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A data verification method based on multithreading, characterized in that, The method includes: The data platform receives new data verification rules uploaded by users and synchronizes the new data verification rules to the real-time verification channel and the historical verification channel. In the real-time verification channel, the data verification rules corresponding to the currently uploaded data are obtained, and the data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the data. In the historical verification channel, the corresponding historical data is matched based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data; The number of thread tasks that can be supported is determined based on the number of server CPU cores. The historical data is then divided into time segments based on the historical data verification information and the number of thread tasks that can be supported, to obtain multiple verification tasks. The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data; Based on the historical data verification information and the number of supported thread tasks, the historical data is divided into time periods to obtain multiple verification tasks, specifically including: The total number of historical data entries and the total time distribution of the historical data are used as the historical data verification information; Based on the total time distribution of the historical data verification information and the total number of records, determine the initial slice time range of the historical data and the number of data records corresponding to each initial slice time range; The average number of data entries in the historical data is determined based on the number of initial slice time ranges and the total number of entries. The number of data entries corresponding to each initial slice time range is compared with the average number of data entries to determine whether to adjust the initial slice time range. Within the limit of the number of supported thread tasks, the initial slice time range is iteratively adjusted to obtain a slice time range. Based on the slice time range, the historical data is divided into time segments to obtain multiple verification tasks corresponding to the historical data.

2. The data verification method based on multi-threading according to claim 1, characterized in that, In the real-time verification channel, the data verification rules corresponding to the currently uploaded collected data are obtained. The collected data is then verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the collected data. Specifically, this includes: The data source and basic data information corresponding to the collected data are obtained in the real-time verification channel; wherein, the basic data information includes: execution stage and execution business type; Based on the data source and the basic data information, determine the data verification rules corresponding to each of the collected data; Based on the upload time of each collected data and the number of thread tasks that can be supported, the collected data is sliced ​​and grouped so that each group of collected data can be verified in parallel according to the corresponding data verification rules to obtain the verification result of the collected data.

3. The data verification method based on multi-threading according to claim 1, characterized in that, In the historical verification channel, based on the newly added data verification rules, the corresponding historical data is matched to determine the historical data verification information corresponding to the historical data, specifically including: Determine the scope of business impact corresponding to the newly added data verification rule in the historical verification channel; Obtain historical data within the scope of the affected business that matches the new data verification rule, and determine the total number of historical data entries; The total time distribution of historical data that matches the verification rules of the newly added data is determined based on the verification time of each historical data, so that the total number of historical data entries and the total time distribution of the historical data are used as the historical data verification information.

4. The data verification method based on multi-threading according to claim 1, characterized in that, Compare the number of data entries corresponding to each of the initial slice time ranges with the average number of data entries to determine whether to adjust the initial slice time range, specifically including: Obtain the difference between the number of data entries corresponding to each of the initial slice time ranges and the average number of data entries to obtain the difference sequence corresponding to the initial slice time range; If it is determined that there is a sudden change in adjacent data in the difference sequence, then the initial slice time range is adjusted accordingly. If it is determined that there are no abrupt changes in adjacent data in the difference sequence, then the initial slice time range will not be adjusted.

5. The data verification method based on multithreading according to claim 1, characterized in that, The method of processing the multiple verification tasks concurrently through multiple threads to update the verification results of the historical data specifically includes: Based on the concurrent processing of the multiple verification tasks by the multiple threads, the current verification result of each of the historical data is obtained; The verification results of the historical data are retrieved to update the verification results of the historical data based on the current verification results.

6. The data verification method based on multi-threading according to claim 1, characterized in that, The data platform receives new data verification rules uploaded by user terminals, specifically including: The data platform receives custom rules uploaded by users and calls existing data verification rules of the data platform. Compare the custom rules with the existing data verification rules to determine the comparison result; If, based on the comparison results, it is determined that the custom rule is inconsistent with the existing data verification rule, then the custom rule will be used as a new data verification rule.

7. A data verification device based on multithreading, characterized in that, The device includes: The rule update unit is used to accept new data verification rules uploaded by the user terminal in the data middle platform, and to synchronize the new data verification rules to the real-time verification channel and the historical verification channel. The first verification unit is used to obtain the data verification rules corresponding to the currently uploaded collected data in the real-time verification channel, and to verify the collected data based on the parallel thread and the corresponding data verification rules to obtain the verification result of the collected data. The determining unit is used to match the corresponding historical data in the historical verification channel based on the newly added data verification rules, and determine the historical data verification information corresponding to the historical data. A partitioning unit is used to determine the number of thread tasks that can be supported based on the number of server CPU cores, and to divide the historical data into time based on the historical data verification information and the number of thread tasks that can be supported, thereby obtaining multiple verification tasks. The second verification unit is used to process the multiple verification tasks concurrently through multiple threads in order to update the verification results of the historical data. Based on the historical data verification information and the number of supported thread tasks, the historical data is divided into time periods to obtain multiple verification tasks, specifically including: The total number of historical data entries and the total time distribution of the historical data are used as the historical data verification information; Based on the total time distribution of the historical data verification information and the total number of records, determine the initial slice time range of the historical data and the number of data records corresponding to each initial slice time range; The average number of data entries in the historical data is determined based on the number of initial slice time ranges and the total number of entries. The number of data entries corresponding to each initial slice time range is compared with the average number of data entries to determine whether to adjust the initial slice time range. Within the limit of the number of supported thread tasks, the initial slice time range is iteratively adjusted to obtain a slice time range. Based on the slice time range, the historical data is divided into time segments to obtain multiple verification tasks corresponding to the historical data.

8. A data verification device based on multithreading, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: The data platform accepts new data verification rules uploaded by users and synchronizes the new data verification rules to the real-time verification channel and the historical verification channel. In the real-time verification channel, the data verification rules corresponding to the currently uploaded data are obtained, and the data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the data. In the historical verification channel, the corresponding historical data is matched based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data; The number of thread tasks that can be supported is determined based on the number of server CPU cores. The historical data is then divided into time segments based on the historical data verification information and the number of thread tasks that can be supported, to obtain multiple verification tasks. The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data; Based on the historical data verification information and the number of supported thread tasks, the historical data is divided into time periods to obtain multiple verification tasks, specifically including: The total number of historical data entries and the total time distribution of the historical data are used as the historical data verification information; Based on the total time distribution of the historical data verification information and the total number of records, determine the initial slice time range of the historical data and the number of data records corresponding to each initial slice time range; The average number of data entries in the historical data is determined based on the number of initial slice time ranges and the total number of entries. The number of data entries corresponding to each initial slice time range is compared with the average number of data entries to determine whether to adjust the initial slice time range. Within the limit of the number of supported thread tasks, the initial slice time range is iteratively adjusted to obtain a slice time range. Based on the slice time range, the historical data is divided into time segments to obtain multiple verification tasks corresponding to the historical data.

9. A non-volatile storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of: The data platform accepts new data verification rules uploaded by users and synchronizes the new data verification rules to the real-time verification channel and the historical verification channel. In the real-time verification channel, the data verification rules corresponding to the currently uploaded data are obtained, and the data is verified based on parallel threads and the corresponding data verification rules to obtain the verification result of the data. In the historical verification channel, the corresponding historical data is matched based on the newly added data verification rules to determine the historical data verification information corresponding to the historical data; The number of thread tasks that can be supported is determined based on the number of server CPU cores. The historical data is then divided into time segments based on the historical data verification information and the number of thread tasks that can be supported, to obtain multiple verification tasks. The multiple verification tasks are processed concurrently by multiple threads to update the verification results of the historical data; Based on the historical data verification information and the number of supported thread tasks, the historical data is divided into time periods to obtain multiple verification tasks, specifically including: The total number of historical data entries and the total time distribution of the historical data are used as the historical data verification information; Based on the total time distribution of the historical data verification information and the total number of records, determine the initial slice time range of the historical data and the number of data records corresponding to each initial slice time range; The average number of data entries in the historical data is determined based on the number of initial slice time ranges and the total number of entries. The number of data entries corresponding to each initial slice time range is compared with the average number of data entries to determine whether to adjust the initial slice time range. Within the limit of the number of supported thread tasks, the initial slice time range is iteratively adjusted to obtain a slice time range. Based on the slice time range, the historical data is divided into time segments to obtain multiple verification tasks corresponding to the historical data.

Images