Bloom filter-based data duplication checking method, device and equipment

Abstract

This application provides a data duplication verification method, apparatus, and device based on a Bloom filter, applicable to the big data field. The method is applied to a multi-core central processing unit and includes: responding to a data verification request, determining at least one mapping function; determining a first binary array corresponding to the data to be compared based on the mapping function; the first binary array representing the data to be compared; the data to be compared being all the remaining data in the verification set except for the data to be verified; determining an idle thread group in a preset thread pool, and based on a first thread in the idle thread group, determining a second binary array corresponding to the data to be verified according to the mapping function; and based on a second thread in the idle thread group, determining the verification result based on the first and second binary arrays. The method provided in this application can improve the efficiency and accuracy of data duplication verification.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to a data repeatability verification method, apparatus and equipment based on Bloom filters. Background Technology

[0002] As the scale of data continues to expand, it is usually necessary to perform repeatability checks on test data during the software testing process to avoid performing unnecessary tests on the same data multiple times.

[0003] In related technologies, manual sampling is typically used to verify data repeatability. However, this manual sampling method is prone to inaccurate results and is time-consuming. Summary of the Invention

[0004] This application provides a data repeatability verification method, apparatus, and device based on a Bloom filter to solve the problem of long repeatability verification time in related technologies.

[0005] In a first aspect, this application provides a data repeatability verification method based on a Bloom filter, applied to a multi-core central processing unit, the method comprising:

[0006] In response to a data verification request, at least one mapping function is determined, wherein the data verification request is used to request the determination of whether there is duplicate data in the set to be verified; the set to be verified contains multiple pieces of data to be verified; and the mapping function is a function used to perform data transformation processing on the data to be verified.

[0007] According to the mapping function, data mapping processing is performed on the data to be compared to determine the first binary array corresponding to the data to be compared; the first binary array represents the data to be compared; the data to be compared is all the remaining data to be verified in the set to be verified except for the data to be verified.

[0008] Idle thread groups in a preset thread pool are identified, and based on the first thread in the idle thread group, the data to be verified is mapped according to the mapping function to obtain a second binary array; the preset thread pool includes multiple thread groups, and each thread group includes two threads;

[0009] Based on the second thread in the idle thread group, a verification result is determined according to the first binary array and the second binary array. The verification result is used to characterize whether the data to be verified exists in the data to be compared.

[0010] Secondly, this application provides a data repeatability verification device based on a Bloom filter, applied to a multi-core central processing unit, the device comprising:

[0011] The first determining unit is configured to determine at least one mapping function in response to a data verification request, wherein the data verification request is used to request the determination of whether there is duplicate data in the set to be verified; the set to be verified contains multiple pieces of data to be verified; and the mapping function is a function used to perform data transformation processing on the data to be verified.

[0012] The second determining unit is used to perform data mapping processing on the data to be compared according to the mapping function, and determine the first binary array corresponding to the data to be compared; the first binary array represents the data to be compared; the data to be compared is all the remaining data to be verified in the set to be verified except for the data to be verified.

[0013] The third determining unit is used to determine the idle thread groups in the preset thread pool;

[0014] The processing unit is used to perform data mapping processing on the data to be verified according to the mapping function based on the first thread in the idle thread group to obtain a second binary array; the preset thread pool includes multiple thread groups, and each thread group includes two threads;

[0015] The fourth determining unit is used to determine a verification result based on the second thread in the idle thread group, according to the first binary array and the second binary array, wherein the verification result is used to characterize whether the data to be verified exists in the data to be compared.

[0016] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0017] The memory stores computer-executed instructions;

[0018] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any one of the first aspects.

[0020] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in any one of the first aspects.

[0021] The data duplication verification method, apparatus, and device based on Bloom filters provided in this application are applied to multi-core central processing units. By performing parallel process data duplication verification through multiple thread groups in a preset thread pool, the efficiency of data duplication verification can be improved. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 A flowchart illustrating a data repeatability verification method based on a Bloom filter, provided for an embodiment of this application;

[0024] Figure 2 A flowchart illustrating another data repeatability verification method based on a Bloom filter provided in this application embodiment;

[0025] Figure 3 A schematic diagram of an interface provided for an embodiment of this application;

[0026] Figure 4 A schematic diagram of a data repeatability verification device based on a Bloom filter provided in this application embodiment;

[0027] Figure 5 A schematic diagram of another data repeatability verification device based on a Bloom filter provided in this application embodiment;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] It should be noted that the data repeatability verification method, apparatus and equipment based on Bloom filters of this application can be used in the field of big data, or in any field other than big data. The application field of the data repeatability verification method, apparatus and equipment based on Bloom filters of this application is not limited.

[0032] As data volumes continue to expand, data redundancy verification is often required in various application scenarios. For example, when conducting software testing, redundancy verification of multiple datasets to be tested can reduce unnecessary testing time.

[0033] In related technologies, manual sampling is usually used to check the repeatability of data. However, this method is time-consuming and cannot guarantee the accuracy of the verification results.

[0034] In this application, by combining the Bloom filter provided in related technologies, the mapping of the data to be verified and the data to be compared are realized. Then, the mapped results are compared, which helps improve the accuracy of data comparison. Furthermore, when comparing the mapped results, multi-threading technology in a multi-core processor is used to execute multiple comparison processes in parallel, thereby improving data detection efficiency and ensuring the accuracy and efficiency of subsequent software testing.

[0035] The data repeatability verification method, apparatus, and equipment based on Bloom filters provided in this application aim to solve the above-mentioned technical problems in the prior art.

[0036] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Figure 1 A flowchart illustrating a data repeatability verification method based on a Bloom filter, as provided in this application embodiment, is shown below. Figure 1 As shown, the method includes the following steps:

[0038] S101. In response to a data verification request, determine at least one mapping function, wherein the data verification request is used to request the determination of whether there is duplicate data in the set to be verified; there are multiple pieces of data to be verified in the set to be verified; and the mapping function is a function used to perform data transformation processing on the data to be verified.

[0039] For example, it should be noted that a Bloom filter is a tool used in related technologies to perform duplicate verification on data. A Bloom filter includes at least one mapping function and an initial binary array (i.e., the first binary array in this embodiment). By converting the data to be verified into a binary array and converting the data to be compared with the data to be verified into a binary array (i.e., the first binary array), the presence of the data to be verified in the compared data is determined by comparing the two two-dimensional arrays obtained after the transformation.

[0040] The method provided in this embodiment is executed by a multi-core processor. When a data verification request is received, representing a data duplication check for multiple pieces of data contained in the set to be verified, at least one mapping function is determined.

[0041] In one example, when determining the mapping function, a preset default mapping function can be used as the mapping function corresponding to this data verification request. Alternatively, the mapping function corresponding to the mapping function identifier carried in the data verification request can be used as the mapping function corresponding to this data verification request. In practical applications, the mapping function can be various types of hash functions, such as additive hash functions, bitwise hash functions, multiplicative hash functions, etc.

[0042] In one example, when a multi-core processor obtains the set of data to be verified indicated by a data verification request, the set of data to be verified may include multiple data entries stored in multiple files. If it is determined that the data contained in the set of data to be verified have different formats, then the data formats of the multiple data entries can be converted to obtain a set of data to be verified with a unified data format. In this way, the accuracy of data duplication verification can be ensured through the above format conversion process.

[0043] It should be noted that this embodiment does not impose specific restrictions on the number or type of mapping functions.

[0044] S102. According to the mapping function, perform data mapping processing on the data to be compared to determine the first binary array corresponding to the data to be verified; the first binary array is used to indicate the data to be compared; the data to be compared is all the data to be verified in the set to be verified except for the data to be verified.

[0045] For example, after obtaining the set to be verified indicated by the data verification request, in order to determine whether there is duplicate data among the multiple data to be verified in the set, for each data to be verified, all other data to be verified in the set except for that data can be used as the data to be compared with that data.

[0046] Next, based on the mapping function, data mapping processing is performed on each piece of data to be verified contained in the comparison data, thereby obtaining the first binary array corresponding to the comparison data. It should be noted that the first binary array consists of multiple binary digits. When determining the first binary array according to the mapping function, each piece of data to be verified can be converted according to the mapping function, and the corresponding binary digit in the first binary array can be set to 1 according to the converted value. The specific principle of determining the first binary array according to the mapping function and the comparison data can be found in the relevant description of Bloom filters, which will not be repeated here.

[0047] S103. Determine the idle thread groups in the preset thread pool, and based on the first thread in the idle thread group, perform data mapping processing on the data to be verified according to the mapping function to obtain the second binary array; the preset thread pool includes multiple thread groups, and each thread group includes two threads.

[0048] For example, in this embodiment, a preset thread pool for performing data deduplication verification is provided in the multi-core central processing unit. The thread pool includes multiple thread groups, and each thread group consists of two threads.

[0049] After obtaining the first binary array of the data to be compared corresponding to the data to be verified, further, an idle thread group can be found in the preset thread pool, and based on the idle thread group, the duplication verification of the data to be verified can be performed according to the data to be verified and the first binary vector corresponding to the data to be verified.

[0050] Specifically, in this embodiment, an idle thread group represents a thread group consisting of two threads that are both in an idle state. It should be noted that in this embodiment, the division of thread groups in the preset thread pool is not fixed. That is, as long as there are two idle threads that are not currently assigned to any thread group, these two threads can form a thread group.

[0051] Once the idle thread group is identified, the data to be verified can be mapped and transformed based on the first thread contained in the idle thread group, according to the previously determined mapping function, thereby obtaining the second binary array corresponding to the data to be verified. Similarly, the second binary array also consists of multiple binary numbers, and the method for obtaining the second binary array is the same as that for obtaining the first binary array, which will not be repeated here.

[0052] S104. Based on the second thread in the idle thread group, determine the verification result according to the first binary array and the second binary array. The verification result is used to characterize whether the data to be verified exists in the data to be compared.

[0053] For example, after the second binary array corresponding to the data to be verified is determined based on the first thread in the idle thread group, the second thread in the idle thread group can compare the second binary array corresponding to the data to be verified with the first binary array of the data to be compared to the data to be verified, so as to determine whether the data to be verified is duplicated with the data contained in the data to be compared.

[0054] Specifically, when comparing the first and second binary arrays, the consistency can be determined by checking if the number of bits set to 1 in both data sets is the same. First, the target number of bits set to 1 in the second binary array is determined. If the target number of bits is also set to 1 in the first binary array, then the data to be verified and compared is considered a duplicate. Otherwise, it indicates that the data to be verified and compared are not duplicates.

[0055] Understandably, in this embodiment, during data duplication verification, multiple thread groups within a preset thread pool in a multi-core CPU are used to perform duplication verification on multiple pieces of data in parallel, thereby improving the efficiency of data duplication verification. Furthermore, in this embodiment, multiple threads are divided at the granularity of thread groups, and the generation of the second binary array and the comparison of the two arrays are performed on a thread group basis. This ensures the efficiency of obtaining the execution result corresponding to a single piece of data to be verified in a timely manner, and the verified (i.e., non-duplicated) data can be directly used in the software testing process, improving the overall testing efficiency and avoiding the phenomenon where multiple threads are executing the second binary array generation step, requiring waiting for all second binary arrays to be generated before obtaining the verification result of a single piece of data. In addition, when performing data comparison, converting the data into a binary array for comparison helps ensure data security.

[0056] Figure 2 A flowchart illustrating another data repeatability verification method based on a Bloom filter provided in this application embodiment is shown below. Figure 2 As shown, the method includes the following steps:

[0057] S201. In response to the data verification request, determine the number of mapping functions based on the verification error rate and the required verification duration; wherein, the data verification request is used to request the determination of whether there is duplicate data in the set to be verified; there are multiple pieces of data to be verified in the set to be verified; the mapping function is a function used to perform data transformation processing on the data to be verified; the data verification request includes: the required verification duration and the verification error rate; the required verification duration is the duration required to perform duplicate verification on the set to be verified; the verification error rate represents the probability of acceptable inaccurate verification results.

[0058] For example, the method provided in this embodiment is applied to a multi-core central processing unit. Furthermore, the data verification request received by the multi-core central processing unit in this embodiment carries the required verification duration and the acceptable verification error rate. The verification error rate can be understood as the ratio of the number of inaccurate verification results to the total number of verification results, i.e., the probability of an inaccurate verification result.

[0059] After obtaining the aforementioned verification error rate and verification time requirement, the number of mapping functions to be selected can be determined based on the correspondence between the verification error rate, verification time requirement, and mapping functions. It is understandable that the more mapping functions there are, the longer the corresponding verification time, but the lower the actual error rate of the obtained verification result. In one example, the above correspondence can be obtained through multiple repetitive verification experiments.

[0060] S202. Select a mapping function from the preset function set based on the number of mapping functions. The preset function set includes multiple mapping functions.

[0061] For example, after determining the number of mapping functions, a specified number of mapping functions can be selected from a pre-set set of preset functions. It should be noted that the preset function set can contain multiple mapping functions of different types, and the number of mapping functions of the same type can be multiple. This embodiment does not impose a specific limitation on the preset function set. In practical applications, to avoid the problem of complex and time-consuming calculations when converting data to binary arrays based on mapping functions, the function types of the mapping functions in the preset function set can all be additive hash functions, with different parameters carried by different additive hash functions.

[0062] It is understood that in this embodiment, the number of mapping functions can be determined by combining the required verification duration and the verification error rate, so as to avoid the phenomenon that the required verification duration cannot be met when there are too many mapping functions, and the problem that the inaccuracy rate of the verification result is high when there are too few mapping functions.

[0063] S203. Determine the mapping order based on the time information of each piece of data to be verified; the mapping order indicates the order in which the first binary array of the comparison data corresponding to each piece of data to be verified is generated, and the data to be verified has time information; the time information represents the time during which the data to be verified is used as test data in the software testing process.

[0064] For example, in this embodiment, each piece of data to be verified has its own time information, wherein the time information is used to characterize the time at which the data to be verified is used as test data in the software testing process. For example, if the data to be verified needs to be used as test data to execute the test process at a first moment, then the first moment can be used as the time information of the data to be verified.

[0065] Then, based on the time information, the mapping order used to characterize the order of the first binary array corresponding to the data to be verified can be determined. For example, the smaller the time information of the data to be verified, the earlier the order of the first binary array corresponding to that data will be in the mapping order.

[0066] S204. Based on the mapping order and mapping function, determine the first binary array of the data to be compared corresponding to each data to be verified in sequence. The first binary array is used to indicate the data to be compared; the data to be compared is all the data to be verified in the set to be verified except for the data to be verified.

[0067] For example, once the mapping order is determined, the first binary data of the data to be compared corresponding to each data to be verified can be determined sequentially according to the order indicated by the mapping order.

[0068] It is understood that in this embodiment, the first binary array corresponding to each piece of data to be verified can be determined sequentially using the time information corresponding to each piece of data to be verified. This method ensures that the binary array corresponding to the data to be verified with smaller time information can be determined as quickly as possible, thus allowing the verification result of the data to be verified to be determined earlier and improving subsequent testing efficiency.

[0069] S205. In response to the data verification request, establish a preset thread pool based on the verification requirement duration, the amount of data in the set to be verified, the maximum number of concurrent threads in the multi-core CPU, and the historical verification record information included in the data verification request; wherein, the historical verification record information includes the amount of data in the historical verification set corresponding to the repetitive verification in the historical period, the historical verification time information, and the number of threads in the thread pool used.

[0070] For example, in this embodiment, the multi-core CPU only requests a preset thread pool when it receives a data verification request. When determining the number of threads in the preset thread pool, this can be done based on the verification requirement duration carried in the data verification request (i.e., the verification duration required for the set to be verified this time), the maximum number of concurrent threads corresponding to the multi-core CPU, and the verification records corresponding to data duplication verification at historical times (i.e., historical verification record information).

[0071] Specifically, the historical verification record information in this embodiment includes the amount of data in the historical verification set that needs to be verified each time the historical verification process is performed, the historical verification time information corresponding to the historical verification set (i.e., how long it took to complete the repetitive verification of the historical verification set), and the number of threads in the thread pool used to complete the verification of the above-mentioned historical verification set.

[0072] When determining the number of threads in the preset thread pool requested this time, it is possible to compare the amount of data in the set to be verified at the current moment, the required verification duration, and historical verification records. This involves finding the time consumed and the number of threads at historical moments with the same amount of data, and then setting the corresponding number of threads in the thread pool at the current moment to ensure that the verification is completed within the required duration. It should be noted that the execution order of step S205 is not specifically restricted; for example, it can be executed simultaneously with step S201 or before step S201.

[0073] It is understood that in this embodiment, a preset thread pool can be constructed by combining historical verification record information, the verification requirement duration corresponding to this verification, and the data volume of the set to be verified in this verification. The number of threads in the preset thread pool can be reasonably determined to ensure that this repetitive verification can be completed within the verification requirement duration, avoid waste of thread pool resources, and help improve the resource utilization of multi-core central processing units.

[0074] S206. Determine the idle thread groups in the preset thread pool, and based on the first thread in the idle thread group, perform data mapping processing on the data to be verified according to the mapping function to obtain the second binary array; the preset thread pool includes multiple thread groups, and each thread group includes two threads.

[0075] S207. Based on the second thread in the idle thread group, determine the verification result according to the first binary array and the second binary array. The verification result is used to characterize whether the data to be verified exists in the data to be compared.

[0076] For example, the specific principles of steps S206 and S207 can be found in steps S103 and S104, and will not be repeated here.

[0077] S208. If the verification result indicates that the data to be verified exists in the data to be compared, then the location information of the data to be verified is determined. The location information is used to characterize the storage location of the data to be verified.

[0078] For example, in this embodiment, if the verification result indicates that the data to be verified and the data to be compared are duplicates, that is, the data to be verified exists in the data to be compared, then the storage location corresponding to the data to be verified can be determined. For example, the storage location of the data to be verified can be specifically represented by the file name of the document containing the data to be verified and the line number in that document, so that the data to be verified can be quickly located subsequently.

[0079] S209. Send a first prompt message to the user. The first prompt message includes the location information of the data to be verified. The first prompt message is used to indicate that the data at the location information is duplicate data.

[0080] For example, after determining the location information corresponding to the data to be verified, a first prompt message carrying the location information of the data to be verified can be sent to the user so that the user can know the location of the duplicate data to be verified in a timely manner.

[0081] Furthermore, when it is determined that the data to be verified is not among the data to be compared, that is, the data to be verified and the data to be compared are not duplicates, a second prompt message can be sent to the user to inform the user that the data to be verified will not be duplicated with any other data to be verified.

[0082] It should be noted that in this embodiment, after identifying duplicate data to be verified, the location information corresponding to the data to be verified can be returned to the user so that the user can promptly determine the location of the duplicate data to be verified, facilitating the user to review or delete the duplicate data.

[0083] In one example, after step S208, the following steps are also included:

[0084] First step: Control the display device connected to the multi-core central processing unit to display the location information of the data to be verified and the first component corresponding to the location information in the preset display interface.

[0085] The second step: In response to the control operation on the first component, delete the data to be verified corresponding to the location information.

[0086] For example, in this embodiment, after determining that there is a duplicate between the data to be verified and the corresponding data to be compared, and after determining the location information of the data to be verified, the location information corresponding to each duplicate data to be verified can be displayed on the display interface of the display device connected to the multi-core central processing unit, and each location information corresponds to a first component. For example, Figure 3 This is a schematic diagram of an interface provided for an embodiment of this application. For example... Figure 3As shown, multiple location information items are arranged in a list format, and each location information item has a corresponding first component (represented by a circular icon in the figure) before its starting character. Control operations can be performed on the first component to control the location information corresponding to that component. When a control operation is detected on the first component, the data to be verified at that location is deleted. It should be noted that in this embodiment, the control operation method for the first component is not specifically limited. Furthermore, in one possible implementation, the location information in the display interface is in the format of hyperlink text. When a control operation such as clicking on the location information is detected, or when the mouse pointer lands on the area of ​​the location information, the data to be verified at that location can be displayed. For example, the data to be verified can be displayed in the top layer of the current display interface, or the display interface can be switched to another display interface, or the document containing the data to be verified can be directly navigated to and the data to be verified can be highlighted in the document.

[0087] It is understood that in this embodiment, after determining the location information of the duplicate data to be verified, the determined location information can be displayed on the display interface, and the first component corresponding to each location information can be displayed, so that the deletion of the data to be verified corresponding to the location information can be achieved by performing relevant control operations on the first component.

[0088] In one example, after step S207, the following steps are also included:

[0089] Step 1: If it is determined that the verification results of each data to be verified contained in the set to be verified have been obtained, then the duplication rate information of the set to be verified is determined based on each verification result; wherein, the duplication rate information represents the ratio of the first quantity and the second quantity; the first quantity is the number of verification results of the data to be verified existing in the data to be compared; the second quantity is the total number of data to be verified contained in the set to be verified.

[0090] The second step involves controlling the display device connected to the multi-core central processing unit to display the repetition rate information and a prompt box in a preset display interface. The prompt box includes a second component, a third component, and a fourth component. The second component indicates that all duplicate data to be verified should be deleted directly. The third component indicates that duplicate data to be verified should be displayed. The fourth component indicates that the data to be verified does not need to be deleted, and the software testing process continues based on the set of data to be verified.

[0091] For example, in this embodiment, after obtaining the verification results corresponding to each piece of data to be verified, the number of repeated data to be verified contained in the set to be verified can be determined based on the verification results, i.e., the first number. The total number of all data to be verified contained in the set to be verified is then determined as the second number. Afterwards, the ratio between the first and second numbers (i.e., the repetition rate information) can be fed back to the user. Specifically, when feeding back the repetition rate information, it can be displayed on the display interface of the display device associated with the multi-core central processing unit, along with a preset prompt box. The prompt box includes a second component, a third component, and a fourth component.

[0092] After detecting a control operation on the second component, the multi-core CPU will delete all duplicate data to be verified from the set to be verified. That is, among multiple identical data to be verified, only one data to be verified will be retained to ensure that the data to be verified contained in the set to be verified will not be duplicated after the above operation is performed.

[0093] When a control operation on the third component is detected, the multi-core CPU controls the relevant display device to display the various duplicate data to be verified on the display interface in order to determine whether the data needs to be further deleted. Here, the display method of the data to be verified can refer to the display method in the example above, that is, the first component, the location information, and the location information are displayed in a hyperlink text format.

[0094] Once a control operation targeting the fourth component is detected, the multi-core CPU can execute the software testing process based on the data to be verified in the data set to be verified, that is, use the aforementioned data set to be verified as test data in the software testing process and execute relevant test operations.

[0095] It should be noted that this embodiment does not impose specific restrictions on the display position of the repetition rate information. The display area where the repetition rate information is located and the display area of ​​the prompt box can be different display areas in the display interface, or they can be located in the same display area. For example, the repetition rate information can be displayed in the area where the prompt box is located.

[0096] It is understood that in this embodiment, after determining the verification result corresponding to each piece of data to be verified, the repetition rate information obtained based on the verification result and a prompt box can be displayed. Then, based on the magnitude of the repetition rate information value, or by combining the repetition rate information with other factors (e.g., the testing time and accuracy requirements for subsequent software testing of the data to be verified), the selection of which component in the prompt box can be determined, thereby executing different processes and improving user satisfaction when using the above method for repetition verification.

[0097] In this embodiment, the number of mapping functions can be determined by combining the required verification duration and the verification error rate. This avoids the problem of insufficient verification duration when the number of mapping functions is too large, and the problem of high inaccuracy of verification results when the number of mapping functions is too small. Furthermore, the first binary array corresponding to each piece of data to be verified can be determined sequentially using the time information corresponding to each piece of data. This ensures that the binary array corresponding to the data with smaller time information can be determined as quickly as possible, allowing for earlier determination of the verification result and improving subsequent testing efficiency. Moreover, when determining the preset thread pool, the preset thread pool can be constructed by combining historical verification record information, the required verification duration for this verification, and the data volume of the set of data to be verified in this verification. The number of threads in the preset thread pool can be reasonably determined to ensure that this repetitive verification can be completed within the required verification duration, avoiding waste of thread pool resources and improving the resource utilization of multi-core CPUs.

[0098] Figure 4 This application provides a schematic diagram of a data repeatability verification device based on a Bloom filter, which is applied to a multi-core central processing unit, such as... Figure 4 As shown, the device includes:

[0099] The first determining unit 401 is used to determine at least one mapping function in response to a data verification request, wherein the data verification request is used to request the determination of whether there is duplicate data in the set to be verified; there are multiple data to be verified in the set to be verified; and the mapping function is a function used to perform data transformation processing on the data to be verified.

[0100] The second determining unit 402 is used to perform data mapping processing on the data to be compared according to the mapping function, and determine the first binary array corresponding to the data to be compared; the first binary array represents the data to be compared; the data to be compared is all the data to be verified in the set to be verified except for the data to be verified.

[0101] The third determining unit 403 is used to determine the idle thread groups in the preset thread pool.

[0102] Processing unit 404 is used to perform data mapping processing on the data to be verified according to the mapping function based on the first thread in the idle thread group to obtain the second binary array; the preset thread pool includes multiple thread groups, and each thread group includes two threads.

[0103] The fourth determining unit 405 is used to determine the verification result based on the second thread in the idle thread group, according to the first binary array and the second binary array. The verification result is used to characterize whether the data to be verified exists in the data to be compared.

[0104] The apparatus provided in this embodiment is used to implement the technical solution provided by the above method. Its implementation principle and technical effect are similar, and will not be described again.

[0105] Figure 5 This application provides a schematic diagram of another data repeatability verification device based on a Bloom filter, which is shown in the following embodiments. Figure 4 Based on the device structure shown, in this embodiment, the data to be verified has time information; the time information represents the time during which the data to be verified is used as test data in the software testing process; the second determining unit 402 includes:

[0106] The first determining module 4021 is used to determine the mapping order based on the time information of each data to be verified; the mapping order indicates the order in which the first binary array of the data to be compared corresponding to each data to be verified is generated.

[0107] The second determining module 4022 is used to determine the first binary array of the data to be compared corresponding to each data to be verified in sequence according to the mapping order and the mapping function.

[0108] In one example, the data verification request includes: verification requirement duration and verification error rate; the verification requirement duration is the duration required for repeatability verification of the set to be verified; the verification error rate represents the acceptable probability of inaccurate verification results; the first determining unit 401 includes:

[0109] The third determining module 4011 is used to determine the number of mapping functions in response to a data verification request, based on the verification error rate and the required verification duration.

[0110] The selection module 4012 is used to select a mapping function from a preset function set based on the number of mapping functions. The preset function set includes multiple mapping functions.

[0111] In one example, the device also includes:

[0112] The fifth determining unit 406 is used to determine the location information of the data to be verified if the verification result indicates that the data to be verified exists in the data to be compared. The location information is used to indicate the storage location of the data to be verified.

[0113] The sending unit 407 is used to send a first prompt message to the user. The first prompt message includes the location information of the data to be verified. The first prompt message is used to indicate that the data at the location information is duplicate data.

[0114] In one example, the device also includes:

[0115] The first control unit 408 is used to control the display device connected to the multi-core central processing unit to display the location information of the data to be verified and the first component corresponding to the location information in a preset display interface after the fifth determining unit 406 determines the location information of the data to be verified if the verification result indicates that the data to be verified exists in the data to be compared.

[0116] The deletion unit 409 is used to delete the data to be verified corresponding to the location information in response to a control operation on the first component.

[0117] In one example, the device also includes:

[0118] The sixth determining unit 410 is used to determine the repetition rate information of the data set to be verified based on each verification result if it is determined that the verification results of each data set to be verified contained in the data set to be verified have been obtained; wherein, the repetition rate information represents the ratio of the first quantity and the second quantity; the first quantity is the number of verification results of the data to be verified existing in the data to be compared; the second quantity is the total number of data to be verified contained in the data set to be verified.

[0119] The second control unit 411 is used to control the display device connected to the multi-core central processing unit to display repetition rate information and prompt boxes in a preset display interface. The prompt boxes include a second component, a third component, and a fourth component. The second component indicates that all duplicate data to be verified should be deleted directly. The third component indicates that duplicate data to be verified should be displayed. The fourth component indicates that the data to be verified does not need to be deleted and the software testing process continues based on the set of data to be verified.

[0120] In one example, the data verification request includes: the verification requirement duration; the device also includes:

[0121] Establishment unit 412 is used to respond to data verification requests and establish a preset thread pool based on the verification requirement duration, the amount of data in the set to be verified, the maximum number of concurrent threads in the multi-core CPU, and historical verification record information. The historical verification record information includes the amount of data in the historical verification set corresponding to the repeated verification in the historical period, the historical verification time information, and the number of threads in the thread pool used.

[0122] The apparatus provided in this embodiment is used to implement the technical solution provided by the above method. Its implementation principle and technical effect are similar, and will not be described again.

[0123] This application provides an electronic device, including: a memory and a processor;

[0124] Memory; memory used to store processor-executable instructions;

[0125] The processor is used to execute methods according to executable instructions.

[0126] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the electronic device includes:

[0127] The electronic device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can invoke logical instructions stored in the memory 292 to execute the methods of the above embodiments.

[0128] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0129] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, thereby implementing the methods in the above-described method embodiments.

[0130] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0131] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods.

[0132] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements any one of the methods.

[0133] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0134] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for data duplication checking based on a Bloom filter, characterized in that, Applied to a multi-core central processing unit, the method includes: In response to a data verification request, at least one mapping function is determined, wherein the data verification request is used to request the determination of whether there is duplicate data in the set to be verified; the set to be verified contains multiple pieces of data to be verified; and the mapping function is a function used to perform data transformation processing on the data to be verified. According to the mapping function, data mapping processing is performed on the data to be compared to determine the first binary array corresponding to the data to be compared; the first binary array represents the data to be compared; the data to be compared is all the remaining data to be verified in the set to be verified except for the data to be verified. Idle thread groups in a preset thread pool are identified, and based on the first thread in the idle thread group, the data to be verified is mapped according to the mapping function to obtain a second binary array; the preset thread pool includes multiple thread groups, and each thread group includes two threads; Based on the second thread in the idle thread group, a verification result is determined according to the first binary array and the second binary array. The verification result is used to characterize whether the data to be verified exists in the data to be compared. The data verification request includes: verification requirement duration and verification error rate; the verification requirement duration is the duration required to perform repeatability verification on the set to be verified; the verification error rate represents the probability of acceptable inaccurate verification results. Determine at least one mapping function, including: The number of mapping functions is determined based on the verification error rate and the required verification duration. Based on the number of mapping functions, a mapping function is selected from a preset function set, which includes multiple mapping functions.

2. The method of claim 1, wherein, The data to be verified has time information; the time information indicates the time during which the data to be verified is used as test data in the software testing process. According to the mapping function, data mapping processing is performed on the data to be compared to determine the first binary array corresponding to the data to be compared, including: The mapping order is determined based on the time information of each of the data to be verified; the mapping order indicates the order in which the first binary array of the data to be compared corresponding to each of the data to be verified is generated. Based on the mapping order and the mapping function, the first binary array of the data to be compared corresponding to each of the data to be verified is determined sequentially.

3. The method according to any one of claims 1-2, characterized in that, The method further includes: If the verification result indicates that the data to be verified exists in the data to be compared, then the location information of the data to be verified is determined, and the location information is used to characterize the storage location of the data to be verified. A first prompt message is sent to the user, the first prompt message including the location information of the data to be verified; the first prompt message is used to indicate that the data at the location information is duplicate data.

4. The method of claim 3, wherein, After determining the location information of the data to be verified, the method further includes: The display device connected to the multi-core central processing unit is controlled to display the location information of the data to be verified and the first component corresponding to the location information in a preset display interface; In response to a control operation on the first component, the data to be verified corresponding to the location information is deleted.

5. The method of claim 3, wherein, The method further includes: If it is determined that the verification results of each piece of data to be verified contained in the set to be verified have been obtained, then the duplication rate information of the set of data to be verified is determined based on each of the verification results; wherein, the duplication rate information represents the ratio of a first quantity and a second quantity; the first quantity is the number of verification results of the data to be verified existing in the data to be compared; the second quantity is the total number of data to be verified contained in the set to be verified. The display device connected to the multi-core central processing unit is controlled to display the repetition rate information and a prompt box in a preset display interface. The prompt box includes a second component, a third component, and a fourth component. The second component indicates that all duplicate data to be verified should be deleted directly. The third component indicates that duplicate data to be verified should be displayed. The fourth component indicates that the data to be verified does not need to be deleted, and the software testing process continues based on the set of data to be verified.

6. The method of claim 3, wherein, The data verification request includes: a verification duration; the method further includes: In response to the data verification request, a preset thread pool is established based on the verification requirement duration, the data volume of the set to be verified, the maximum number of concurrent threads of the multi-core CPU, and historical verification record information; wherein, the historical verification record information includes the data volume of the historical verification set corresponding to the repetitive verification in the historical period, the historical verification time information, and the number of threads in the thread pool used.

7. A Bloom filter based data duplication checking apparatus, characterized by, The device, applied to a multi-core central processing unit, includes: The first determining unit is configured to determine at least one mapping function in response to a data verification request, wherein the data verification request is used to request the determination of whether there is duplicate data in the set to be verified; the set to be verified contains multiple pieces of data to be verified; and the mapping function is a function used to perform data transformation processing on the data to be verified. The second determining unit is used to perform data mapping processing on the data to be compared according to the mapping function, and determine the first binary array corresponding to the data to be compared; the first binary array represents the data to be compared; the data to be compared is all the remaining data to be verified in the set to be verified except for the data to be verified. The third determining unit is used to determine the idle thread groups in the preset thread pool; The processing unit is used to perform data mapping processing on the data to be verified according to the mapping function based on the first thread in the idle thread group to obtain a second binary array; the preset thread pool includes multiple thread groups, and each thread group includes two threads; The fourth determining unit is used to determine a verification result based on the second thread in the idle thread group, according to the first binary array and the second binary array, wherein the verification result is used to characterize whether the data to be verified exists in the data to be compared; The data verification request includes: verification requirement duration and verification error rate; the verification requirement duration is the duration required to perform repeatability verification on the set to be verified; the verification error rate represents the probability of acceptable inaccurate verification results. The first determining unit includes a third determining module and a selection module; The third determining module is used to determine the number of mapping functions based on the verification error rate and the verification requirement duration; The selection module is used to select a mapping function from a preset function set based on the number of mapping functions, wherein the preset function set includes multiple mapping functions.

8. An electronic device, comprising: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

Images