Data processing method and apparatus, electronic device, and storage medium

Abstract

This application discloses a data processing method, apparatus, electronic device, and storage medium. The method includes: performing a bucketing operation on the dataset to be processed based on attribute information to obtain at least one target bucket; determining a target type sorting result for each element in each target bucket, and performing verification processing on the element based on the target type sorting result to obtain a target verification result for the element; wherein each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket; and merging the target verification results of each element in each target bucket to obtain a processing result for the dataset to be processed.

Description

Technical Field

[0001] The embodiments of this application relate to, but are not limited to, the field of computer technology, and in particular to a data processing method and apparatus, electronic device and storage medium. Background Technology

[0002] Data validation is a verification operation performed to ensure the correctness and integrity of data. In related technologies, the process of validating batch data requires defining a validation rule for each type of data. This approach suffers from problems such as long validation time, high time complexity, and low accuracy. Furthermore, due to the large amount of data being validated, issues such as memory anomalies and service unavailability can easily arise. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data processing method and apparatus, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0006] Based on the attribute information of the dataset to be processed, the dataset to be processed is divided into buckets to obtain at least one target bucket; wherein, the dataset to be processed includes at least one element of at least one type, and each target bucket includes at least one element of at least one type;

[0007] For each element in each target bucket, the target type sorting result of the target bucket is determined, and the element is verified based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket;

[0008] The target verification results of each element in each target bucket are merged to obtain the processing result of the dataset to be processed.

[0009] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0010] The bucketing module is used to perform bucketing operations on the dataset to be processed based on the attribute information of the dataset to be processed, to obtain at least one target bucket; wherein the dataset to be processed includes at least one element of at least one type, and each target bucket includes at least one element of at least one type;

[0011] The verification module is used to determine the target type sorting result of each element in each target bucket, and to perform verification processing on the element based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket;

[0012] The merging module is used to merge the verification results of each element in each of the target buckets to obtain the processing result of the dataset to be processed.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps in the above-described method.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0015] Fifthly, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, the computer program implementing the above-described method when read and executed by a computer.

[0016] In this embodiment, the dataset to be processed is bucketed based on its attribute information to obtain at least one target bucket. For each element in each target bucket, a target type sorting result is determined, and the element is validated based on the target type sorting result to obtain a target validation result. Each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket. The target validation results of each element in each target bucket are merged to obtain the processing result of the dataset to be processed. In this way, firstly, by binning the dataset to be processed, the possibility of system memory anomalies or unavailability due to excessively large data can be reduced; secondly, by using the target type sorting results of the buckets, each element in the bucket is automatically validated, which shortens the validation time and improves the validation accuracy, thereby reducing the complexity and cost of validation time; finally, by using the already classified elements in the buckets to determine the target type sorting results in real time, the accuracy of the target type sorting results can be improved, thereby reducing the number of subsequent validations of each element, which can further shorten the validation time and reduce the complexity of validation time. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method provided in an embodiment of this application;

[0018] Figure 2 A schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application;

[0019] Figure 3 A schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application;

[0020] Figure 4 A schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application;

[0021] Figure 5A This is a schematic diagram of the composition structure of a data processing system provided in an embodiment of this application;

[0022] Figure 5B This is a schematic diagram illustrating the implementation process of validating elements in a bucket, as provided in an embodiment of this application.

[0023] Figure 5C A schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0030] Data validation is a verification operation performed to ensure the correctness and integrity of data. To facilitate rapid data uploads by teams, common systems (e.g., operational systems, office systems) provide data import functions. However, these systems often directly pass unvalidated data downstream, forcing downstream systems to perform validation based on simple data validation rules. This leads to resource waste and poor availability. Furthermore, data validation rules in related technologies rely on regular expression algorithms within each system, using simple conditional regular expressions for validation, such as checking the quantity of data. This method requires custom validation rules and code customization for each data type, lacking universality. Alternatively, data can be sorted by time and compared with standard data to check for missing data; if no missing data is found, the validation is considered correct. This method suffers from low accuracy, high validation time, and high time complexity. Additionally, the large volume of data being validated can easily lead to memory errors and service unavailability.

[0031] This application provides a data processing method. First, by binning the dataset to be processed, the possibility of system memory anomalies or unavailability due to excessively large data volumes can be reduced. Second, by utilizing the target type sorting results of the bins, each element in the bin is automatically validated, shortening the validation time and improving validation accuracy, thereby reducing the complexity and cost of validation time. Finally, by determining the target type sorting results in real time using the categorized elements in the bins, the accuracy of the target type sorting results can be improved, thereby reducing the number of subsequent validations of each element, further shortening the validation time and reducing validation time complexity. The data processing method provided in this application can be executed by an electronic device, which can be a laptop, tablet, desktop computer, set-top box, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), or other types of terminals, or can be implemented as a server. A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0032] Figure 1 This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes steps S11 to S13, wherein:

[0033] Step S11: Based on the attribute information of the dataset to be processed, perform a bucketing operation on the dataset to be processed to obtain at least one target bucket.

[0034] Here, the dataset to be processed includes at least one element of at least one type, and each target bucket includes at least one element of at least one type. Elements can be elements from any suitable scenario. For example, a phone number, a file containing several ID card numbers, an information field, etc. The type of an element can include, but is not limited to, basic types, extended types, and unknown types. Basic types can be some pre-defined commonly used types; in implementation, these basic types can include, but are not limited to, phone numbers, ID card numbers, addresses, email addresses, dates of birth, ethnicities, genders, bank card numbers, employee numbers, ages, social media accounts, etc. Extended types can be types defined by the user based on actual needs, on the basis of basic types. For example, serial numbers, bill numbers, etc. Unknown types indicate that the type of the element is unknown; in implementation, the types of elements in this dataset can include, but are not limited to, at least one of the basic types and at least one of the extended types.

[0035] In some implementations, depending on business requirements, the type of each element in the dataset can be directly specified, or the type of each element can be left unspecified. In practice, each element in the dataset can be of the same type or different types. For example, every element in the dataset might be an ID card. Or, for instance, some elements in the dataset might be phone numbers, and some might be email addresses, etc.

[0036] The attributes of a dataset may include, but are not limited to, the total number of elements, the total size of the dataset, and extrema information. The total number of elements represents the total number of elements in the dataset. The total size of the dataset represents the size of each element in the dataset. Extrema information may include, but is not limited to, the first extremum, the second extremum, etc. The first extremum represents the size of the largest element in the dataset, and the second extremum represents the size of the smallest element in the dataset. For example, a dataset may contain 1000 elements, with a total size of 9.6 MB (megabytes), a minimum element size of 10 KB (kilobytes), and a maximum element size of 30 KB.

[0037] The binning method can include, but is not limited to, the default method, the random method, the uniform method, and the custom method. The custom method can include, but is not limited to, the previous method, the most frequently used method, and the user's preferred method. In implementation, those skilled in the art can choose the binning method according to actual needs; this application's embodiments do not impose limitations.

[0038] In some implementations, the binning method can be determined based on preset rules, and the dataset can be binned using this method. These preset rules may include, but are not limited to, default, random, uniform, custom, user preference, usage frequency, and user operation information. In implementation, those skilled in the art can independently set preset rules according to actual needs; this application's embodiments do not impose such limitations.

[0039] For example, electronic devices may provide configuration options through which users can customize the binning method. Alternatively, the binning method may be determined based on the attribute information of the user's operation, which may include, but is not limited to, the type of operation, operation distance, speed, location, duration, and number of times. In implementation, those skilled in the art can independently set the correspondence between the operation's attribute information and the binning method according to actual needs; this application's embodiments do not impose such limitations.

[0040] In some implementations, the binning method can be determined based on the type of operation; that is, different types correspond to different binning methods. For example, when the operation gesture is a swipe operation, the binning method can be the default method; when the operation gesture is a multiple-click operation, the binning method can be the uniform method.

[0041] In some implementations, the binning method can be determined in real time based on the operation distance; that is, different operation distances correspond to different binning methods. For example, when the sliding distance is within a first distance range, the binning method is uniform; when the sliding distance is within a second distance range, the binning method is random.

[0042] In some implementations, the dataset can be bucketed based on the total number of elements, the total size of the dataset, and maximum / minimum value information.

[0043] For example, based on the total number of elements and a preset threshold, the first target number of buckets is determined. The dataset is then divided into buckets evenly according to this first target number, resulting in each target bucket having a roughly uniform number of elements. The threshold represents the maximum number of elements that each bucket can process in the current device. For instance, if the dataset contains 16,000 elements, and each bucket can process a maximum of 1,000 elements, the first target number of buckets could be 16 (16,000 / 1,000). In this case, evenly dividing the dataset, each target bucket would contain 1,000 elements. Similarly, if the dataset contains 14,900 elements, and each bucket can process a maximum of 1,000 elements, the first target number of buckets could be 16. Where ROUND(x) represents rounding up x, then according to the uniform binning operation, the number of elements in the 12 target bins is 931 (14900 / 16), and the number of elements in the remaining 4 target bins is 932 (14900 / 16+1).

[0044] For example, based on the total size of the dataset and the target memory required to process it, the second target number of buckets is determined. Then, based on the second target number of buckets, the target memory, and the maximum / minimum values, the dataset is randomly divided into buckets to obtain each target bucket. At this point, the number of elements in each target bucket is within a set range, which is determined based on the first maximum / minimum value, the target memory, and the second maximum / minimum value. For instance, if the target memory required to process the dataset is 0.5MB, the total size of the dataset is 9.6MB, the first maximum / minimum value is 30KB, and the second maximum / minimum value is 10KB, then the second target number of buckets is 32. At this point, each bucket has an actual memory of 0.3MB. The elements are divided into buckets randomly, and the number of elements contained in the 16 target buckets is... Between 0.01MB and 0.03MB.

[0045] For example, based on the first and second target quantities, a third target quantity is determined. Then, according to the third target quantity of the bucket, the total number of elements, target memory, and extreme value information, the dataset is divided into buckets using uniform or random methods to obtain each target bucket. The method for determining the third target quantity may include, but is not limited to, the first target quantity, the second target quantity, the mean / variance / mean squared error between the first and second target quantities, or the mean / variance / mean squared error after weighting the first and second target quantities. In implementation, those skilled in the art can choose the method for determining the third target quantity according to actual needs; this application does not limit this method.

[0046] Step S12: For each element in each target bucket, determine the target type sorting result of the target bucket, and perform verification processing on the element based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket.

[0047] Here, the target type sorting result may include, but is not limited to, the initial type sorting result, the updated previous type sorting result, etc. The initial type sorting result may include, but is not limited to, user-defined type sorting results, type sorting results based on historical data, etc. In implementation, the target type sorting result for each target bucket may be the same or different.

[0048] For example, the initial type sorting result can be the user-preset type sorting result, that is: if the preset type sorting result is {phone number, ID card number, address, email, ethnicity}, then the initial type sorting result is {phone number, ID card number, address, email, ethnicity}.

[0049] For example, the initial type sorting result can be a type sorting result obtained based on historical data. For instance, the type sorting result with the most occurrences can be selected from multiple type sorting results in the historical data as the initial type sorting result.

[0050] For example, based on the target validation result of the previous element, the sorting result of the previous target type is updated, and the updated sorting result of the previous target type is used as the sorting result of the target type for the corresponding bucket. For instance, if the sorting result of the previous target type is: {phone number, ID card number, address, email, ethnicity}, and the target validation result of the previous element indicates that the data is an ID card number, then the sorting result of this target type would be: {ID card number, phone number, address, email, ethnicity}.

[0051] The verification process represents the validation of elements according to a preset verification method. Different types of elements correspond to different verification methods. For example, for an ID card number, any suitable verification algorithm can be used, such as the MOD11-2 algorithm or regular expression verification. In implementation, those skilled in the art can choose the verification algorithm according to actual needs; this application's embodiments do not impose limitations.

[0052] The target validation result may include, but is not limited to, a pass for one type of validation and a failure for another. In implementation, this target validation result characterizes the type of the element; if a certain type of validation passes, that type is adopted as the element's type. For example, if an element meets the validation criteria for an ID card, then the element is an ID card. Similarly, if an element meets the validation criteria for a phone number, then the element is a phone number.

[0053] In some implementations, each type is defined using tags, and after the element's verification process is complete, the target verification result for each element is stored as key-value pairs. The tag value for the element is stored on the key, and the element itself is stored on the value. For example, the tag value for an ID card number is 01, the tag value for a phone number is 02, the tag value for an email address is 03, the tag value for a birth date is 04, the tag value for an address is 05, the tag value for an ethnicity is 06, the tag value for other categories is -001, and the tag value for invalid categories is -002, etc.

[0054] Step S13: Merge the target verification results of each element in each target bucket to obtain the processing result of the dataset to be processed.

[0055] Here, the merging process involves classifying each element in each target bucket, specifically by type, to determine the number of elements in each type. For example, this dataset might contain 8000 phone numbers and 2000 ID card numbers. In practice, the merging process includes at least one merge operation.

[0056] The processing results of the dataset include, but are not limited to, the classification results of the dataset, the number of types in the dataset, the number of elements in each type, and the proportion of elements in each type. For example, if the dataset has a total of 10,000 elements and 16 target buckets, after the first merge of these 16 target buckets, 8 target buckets remain; after the second merge of these 8 target buckets, 4 target buckets remain; after the third merge of these 4 target buckets, 2 target buckets remain; and after the fourth merge of these 2 target buckets, 1 target bucket remains, then the time complexity of the merge is 4(log216).

[0057] In this embodiment, the dataset to be processed is bucketed based on its attribute information to obtain at least one target bucket. For each element in each target bucket, a target type sorting result is determined, and the element is validated based on the target type sorting result to obtain a target validation result. Each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket. The target validation results of each element in each target bucket are merged to obtain the processing result of the dataset to be processed. In this way, firstly, by binning the dataset to be processed, the possibility of system memory anomalies or unavailability due to excessively large data can be reduced; secondly, by using the target type sorting results of the buckets, each element in the bucket is automatically validated, which shortens the validation time and improves the validation accuracy, thereby reducing the complexity and cost of validation time; finally, by using the already classified elements in the buckets to determine the target type sorting results in real time, the accuracy of the target type sorting results can be improved, thereby reducing the number of subsequent validations of each element, which can further shorten the validation time and reduce the complexity of validation time.

[0058] In some implementations, "determining the target type sorting result of the target bucket" in step S12 includes steps S121 and / or S122, wherein:

[0059] Step S121: If the element satisfies the first preset condition, determine the initial type sorting result of the target bucket and use the initial type sorting result as the target type sorting result.

[0060] Here, the first preset condition may include, but is not limited to, the first element, other elements at specified positions, etc. During implementation, the initial type sorting result may include, but is not limited to, user-defined type sorting results, type sorting results based on historical data, etc. Among these, the type sorting result based on historical data may include, but is not limited to, the type sorting result with the largest number of elements, the type sorting result with a number close to a preset value, and the type sorting result at a specified position (e.g., the first, the last).

[0061] In some implementations, the method for determining the initial type sorting result may include, but is not limited to, custom, random, default, or specified order. In implementation, those skilled in the art can independently set the method for determining the initial type sorting result according to actual needs, and the embodiments of this application do not impose limitations. For example, the initial type sorting result can be determined according to a specified order. For instance, if a custom type sorting result exists, it is used as the initial type sorting result; if no custom type sorting result exists, the type sorting result obtained based on historical data is used as the initial type sorting result. Another example is to determine the initial type sorting result randomly. That is, a type sorting result is randomly selected from the custom type sorting result and the type sorting result obtained based on historical data as the initial type sorting result.

[0062] In some implementations, "determining the initial type sorting result of the target bucket" in step S121 includes steps S1211 and / or S1212, wherein:

[0063] Step S1211: Use the preset type sorting result as the initial type sorting result.

[0064] Here, users can pre-set the sorting results by type. In some implementations, this sorting result can be related to the purpose of the dataset. For example, if a user sets the sorting result to {phone number, other}, then the sorting result includes two types. In implementation, those skilled in the art can set the sorting result according to actual needs, and this application embodiment does not impose any limitations.

[0065] Step S1212: Based on the dataset to be processed, determine at least one historical dataset, and based on the type sorting result corresponding to each element in each historical dataset, determine the initial type sorting result of the target bucket.

[0066] Here, the methods for determining historical datasets may include, but are not limited to, their purpose and sender identifier. In implementation, those skilled in the art can choose the method for determining historical datasets according to actual needs; this application embodiment does not impose any limitations. For example, a dataset whose purpose is substantially consistent with the dataset to be processed can be selected from multiple historical datasets as the historical dataset.

[0067] In some implementations, the type sorting result corresponding to each element in each historical dataset can be saved.

[0068] The methods for determining the initial type sorting result may include, but are not limited to, the type sorting result with the largest number of types, the type sorting result with a number close to a preset value, or the type sorting result at a specified position (e.g., the first or the last). In implementation, those skilled in the art can choose the method for determining the initial type sorting result according to actual needs, and the embodiments of this application do not limit it.

[0069] In some implementations, the initial type sorting result S can be determined using the following formula (1-1). n ,Right now:

[0070]

[0071] in, Characterizing the sorting result S of this type m The number of times used to validate each element in each historical dataset. This represents the number of times each element in each historical dataset is validated. express With All Sm The proportion between them, Max(A) Sm The representation is based on the S corresponding to the highest selection percentage. m As the S n .

[0072] Step S122: If the element does not meet the first preset condition, update the previous target type sorting result based on the target verification result of the previous element to obtain the target type sorting result.

[0073] Here, the update methods can include, but are not limited to, replacement, changing the position of the type, etc. For example, if the previous target type sorting result is: {phone number, ID card number, address, other}, and the target validation result of the previous element indicates that the data is an ID card number, then the target type sorting result will be: {ID card number, phone number, address, other}.

[0074] In this embodiment, when an element meets a first preset condition, an initial type sorting result of the target bucket is determined, and this initial type sorting result is used as the target type sorting result; and / or, when an element does not meet the first preset condition, the previous target type sorting result is updated based on the classification result of the element preceding it, thus obtaining the target type sorting result. In this way, on the one hand, by setting a first preset condition to select different target type sorting results, the accuracy of the target type sorting result can be improved, thereby reducing the number of subsequent checks on each element, and thus shortening the check time and reducing the complexity of the check time; on the other hand, by updating the target type sorting result in real time using the target check result of the previous element, the accuracy of the target type sorting result can be further improved, thereby further reducing the number of subsequent checks on each element, and thus further shortening the check time and reducing the complexity of the check time.

[0075] In some embodiments, step S12 includes step S131, wherein:

[0076] Step S131: For each target bucket, determine the target number of threads and the target type sorting result of the target bucket, schedule the target number of threads in parallel, and perform verification processing on each element based on the target type sorting result to obtain the target verification result of each element.

[0077] Here, the target number of threads can be at least one, and at least one thread can be scheduled in parallel to perform verification processing on at least one element simultaneously.

[0078] The target number of threads can be determined in ways including, but not limited to, user-defined methods, random methods, and device load. In implementation, those skilled in the art can choose the method for determining the target number of threads according to actual needs, and this application does not impose any limitations. For example, the target number of threads may be specified as 4.

[0079] In this embodiment, for each target bucket, a target number of threads and a target type sorting result for the target bucket are determined. The target number of threads are then scheduled in parallel. Based on the target type sorting result, each element is validated to obtain a target validation result for each element. This parallel processing of multiple elements by a target number of threads shortens the validation time and improves the device's response efficiency.

[0080] In some implementations, "determining the target number of threads" in step S131 includes steps S1311 to S1313, wherein:

[0081] Step S1311: Determine the CPU time consumption and the target utilization rate of the CPU.

[0082] Here, CPU time represents the time taken for the CPU to perform one verification process. Methods for determining CPU time may include, but are not limited to, the longest time taken, the average time taken for multiple types, or the time taken for a specific type. In practice, those skilled in the art can choose the method for determining CPU time based on actual needs; this application does not impose such limitations.

[0083] The target CPU utilization rate represents the CPU utilization rate used by the device to perform verification processing on elements. Methods for determining the target CPU utilization rate may include, but are not limited to, the highest CPU utilization rate, the lowest CPU utilization rate, or the average utilization rate of multiple CPU utilization rates. In implementation, those skilled in the art can choose the method for determining the target CPU utilization rate according to actual needs; this application does not impose such limitations.

[0084] In some implementations, "determining CPU time consumption" in step S1311 includes steps S141 and / or S142, wherein:

[0085] Step S141: If the target type sorting result meets the third preset condition, determine the CPU time consumption based on the waiting time corresponding to each type in the target type sorting result.

[0086] Here, the third preset condition may include, but is not limited to, the target type sorting result being based on historical data, or the target type sorting result being custom. During implementation, each type in the target type sorting result has a corresponding waiting time, which may be the same or different.

[0087] Methods for determining CPU time consumption may include, but are not limited to, a single wait time, the mean / variance / mean squared error of multiple wait times, or the mean / variance / mean squared error of multiple wait times after weighting them. In implementation, those skilled in the art can choose the method for determining CPU time consumption according to actual needs; this application does not limit this method. For example, if the target type sorting result is {phone number, ID card number, email, date of birth, address, ethnicity}, and the wait time for a phone number is 1 millisecond (ms), the wait time for an ID card number is 2 ms, the wait time for an email is 1 ms, the wait time for a date of birth is 1 ms, the wait time for an address is 2 ms, and the wait time for an ethnicity is 2 ms, then the average of the multiple wait times can be used as the CPU time consumption, i.e., 1.5 ms.

[0088] In some implementations, the CPU time CM can be determined using the following formula (1-2):

[0089]

[0090] Among them, SM i The value represents the waiting time corresponding to the i-th type, and n represents the number of types included in the target type sorting result.

[0091] Step S142: If the target type sorting result does not meet the third preset condition, determine the target waiting time from the waiting time corresponding to each type in the target type sorting result, and use the target waiting time as the CPU time consumption.

[0092] Here, the method for determining the target waiting time may include, but is not limited to, the waiting time corresponding to the first type, the waiting time corresponding to the last type, and the waiting time corresponding to types in other positions. In implementation, those skilled in the art can choose the method for determining the target waiting time according to actual needs; this application embodiment does not limit this. For example, the waiting time corresponding to the first type in the target type sorting result can be used as the target waiting time. For instance, if the target type sorting result is {phone number, ID card number, other}, then the waiting time of 1ms corresponding to the phone number can be used as the target waiting time.

[0093] In some implementations, "determining the target utilization rate of the CPU" in step S1311 includes steps S151 to S152, wherein:

[0094] Step S151: Determine the remaining CPU utilization rate based on the current remaining CPU utilization rate and the preset CPU ratio.

[0095] Here, the preset CPU ratio represents the remaining CPU utilization after the dataset verification process is completed. In some implementations, the current remaining CPU utilization can be obtained in any suitable way, such as through system functions or specified files. In practice, those skilled in the art can choose the method for obtaining the current remaining CPU utilization according to actual needs; this application does not limit this method. For example, in Java, the current remaining CPU utilization can be obtained through the system functions getSystemLoadAverage() and getAvailableProcessors(). getSystemLoadAverage() is used to obtain the system average load, and getAvailableProcessors() is used to obtain the number of available CPU cores. For example, the ratio between the system average load and the number of available CPU cores can be used as the current remaining CPU utilization LC.

[0096] The methods for determining the remaining CPU utilization rate may include, but are not limited to, the current remaining CPU utilization rate, a preset CPU ratio, the product of the current remaining CPU utilization rate and a first ratio, or the product of weighted averages of the current remaining CPU utilization rate and the first ratio. The first ratio may include, but is not limited to, a preset CPU ratio, or the difference between a preset value and a preset CPU ratio. In implementation, those skilled in the art can choose the method for determining the remaining CPU utilization rate according to actual needs; this application does not limit such methods.

[0097] In some implementations, the remaining CPU utilization UM can be determined using the following formulas (1-3):

[0098] UM = CB * (1 - LC)(1 - 3);

[0099] Wherein, CB represents the current remaining CPU utilization rate, and LC represents the preset CPU ratio.

[0100] Step S152: Determine the target utilization rate of the CPU based on the preset threshold utilization rate and the utilization rate of the remaining CPU.

[0101] Here, the threshold utilization rate represents the utilization rate of the highest-used CPU. Methods for determining the target CPU utilization rate may include, but are not limited to, the threshold utilization rate, the remaining CPU utilization rate, the difference / mean / variance / mean squared error between the threshold utilization rate and the remaining CPU utilization rate, or the difference / mean / variance / mean squared error after weighting the threshold utilization rate and the remaining CPU utilization rate. In implementation, those skilled in the art can choose the method for determining the target CPU utilization rate according to actual needs; this application does not limit this method. For example, the minimum value between the threshold utilization rate and the remaining CPU utilization rate can be taken as the target CPU utilization rate. For instance, if the threshold utilization rate is 10%, the current remaining CPU utilization rate is 60%, and the preset CPU ratio is 30%, then the remaining CPU utilization rate is 0.3*(1-0.6), which is 0.12. Since 0.1 is less than 0.12, the threshold utilization rate is taken as the target CPU utilization rate.

[0102] In some implementations, the target CPU utilization C can be determined using the following formulas (1-4):

[0103] C = min(UC:MC)(1-4);

[0104] Where UC represents the threshold utilization rate and MC represents the remaining CPU utilization rate.

[0105] Step S1312: Determine the fourth ratio between the threshold waiting time and the CPU time consumption.

[0106] Here, the threshold waiting time represents the sum of the waiting times corresponding to each type in the target type sorting result. The methods for determining the fourth ratio may include, but are not limited to, the ratio between the threshold waiting time and the CPU time, the ratio between the CPU time and the threshold waiting time, the ratio after weighting the threshold waiting time and the CPU time separately, and so on. In implementation, those skilled in the art can choose the method for determining the fourth ratio according to actual needs; this application embodiment does not limit this.

[0107] In some implementations, the fourth ratio PT can be determined using the following formulas (1-5):

[0108]

[0109] Among them, SM i The value represents the waiting time for the i-th type, n represents the number of types included in the target type sorting result, and CM represents the CPU time.

[0110] Step S1313: Based on the number of CPU cores, the fourth ratio, and the target utilization rate of the CPU, the target number of threads is obtained.

[0111] Here, the number of CPU cores can be obtained in any suitable way, such as through system functions, specified files, or custom methods. In implementation, those skilled in the art can choose the method for obtaining the number of CPU cores according to actual needs; this application does not limit this method. For example, for Java, the number of CPU cores can be obtained through the system function `Runtime.getRuntime().availableProcessors()`.

[0112] The methods for determining the target number of threads may include, but are not limited to, the product of the number of CPU cores, a preset value, and the target CPU utilization, or a weighted product of the number of CPU cores, the preset value, and the target CPU utilization. The preset value is based on a fourth ratio, such as the fourth ratio itself, or the sum / difference between the fourth ratio and a certain constant. In implementation, those skilled in the art can choose the method for determining the target number of threads according to actual needs; this application does not limit this method.

[0113] In some implementations, the target number T of threads can be determined using the following formulas (1-6):

[0114] T = CN * C * (1 + PT)(1 - 6);

[0115] Where CN represents the number of CPU cores, PT represents the fourth ratio, and C represents the target utilization rate of the CPU.

[0116] For example, if the CPU has 10 cores, the fourth ratio is 9, and the target CPU utilization is 0.1, then according to the above formula (1-6), the target number of threads is 10*0.1*(1+9), which is 10.

[0117] In this embodiment, the CPU time consumption and target utilization rate are determined; a fourth ratio between a preset threshold waiting time and the CPU time consumption is determined; and the target number of threads is obtained based on the number of CPU cores, the fourth ratio, and the target utilization rate. Thus, determining the number of threads by using the number of CPU cores, time consumption, and utilization rate not only improves the accuracy of the thread count but also increases system utilization while ensuring normal system operation, thereby improving the efficiency of verification processing and system response efficiency.

[0118] Figure 2 This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application, such as... Figure 2 As shown, the method includes steps S21 to S24, wherein:

[0119] Step S21: Determine the first target number of buckets based on the total number of elements in the dataset to be processed and the preset quantity threshold.

[0120] Here, the total number of elements represents the total number of elements in the dataset, and the number threshold represents the maximum number of elements that can be processed in each bucket of the current device.

[0121] The methods for determining the first target quantity may include, but are not limited to, a fifth ratio between the total number of elements and a quantity threshold, rounding the fifth ratio, taking the logarithm, rounding, and exponentiation of the rounded fifth ratio, and a sixth ratio obtained by weighting the total number of elements and the quantity threshold, rounding the sixth ratio, and taking the logarithm, rounding, and exponentiation of the sixth ratio, etc. In implementation, those skilled in the art can choose the method for determining the first target quantity according to actual needs; the embodiments in this application do not impose limitations.

[0122] In some implementations, the first target quantity BNum1 can be determined by the following formula (2-1):

[0123]

[0124] Where AllNum represents the total number of elements, f represents the quantity threshold, and ROUND() represents rounding up.

[0125] In some embodiments, step S21 includes steps S212 to S212, wherein:

[0126] Step S211: Determine the first initial number of buckets based on the total number of elements and the quantity threshold.

[0127] Here, the first initial quantity is used to characterize the number of average buckets. The method for determining the first initial quantity may include, but is not limited to, a fifth ratio between the total number of elements and a quantity threshold, rounding the fifth ratio, a sixth ratio obtained by weighting the total number of elements and the quantity threshold, and rounding the sixth ratio, etc. In implementation, those skilled in the art can choose the method for determining the first initial quantity according to actual needs; this application embodiment does not impose any limitations.

[0128] In some implementations, the first initial quantity b1 can be determined by the following formula (2-2):

[0129]

[0130] Where AllNum represents the total number of elements, f represents the quantity threshold, and ROUND() represents rounding up.

[0131] Step S212: Determine the first target quantity of the buckets based on the first initial quantity of the buckets.

[0132] Here, the methods for determining the first target quantity may include, but are not limited to, pairing, rounding, and exponencing the first initial quantity, or pairing, rounding, and exponencing the first initial quantity after weighting. In implementation, those skilled in the art can choose the method for determining the first target quantity according to actual needs; this application does not limit such methods.

[0133] In some implementations, the first target quantity BNum1 can be determined by the following formula (2-3):

[0134]

[0135] Where b1 represents the first initial quantity, and ROUND() represents rounding up.

[0136] Step S22: Based on the first target number of buckets and the total number of elements, the dataset to be processed is divided into buckets to obtain at least one target bucket.

[0137] Here, the bucketing method can include, but is not limited to, the default method, the random method, the uniform method, and the custom method. The custom method can include, but is not limited to, the previous method, the most frequently used method, or the user's preferred method. In implementation, those skilled in the art can choose the bucketing method according to actual needs; this application embodiment does not limit this choice. For example, the dataset to be processed is bucketed using a uniform method to obtain each target bucket. In this case, the number of elements in each target bucket is basically the same, which is the ratio between the total number of elements and the first target number.

[0138] In some embodiments, step S22 includes steps S221 to S222, wherein:

[0139] Step S221: Determine the first ratio between the total number of elements and the first target number of the buckets.

[0140] Here, the first ratio is a positive integer. The method for determining the first ratio may include, but is not limited to, the ratio between the total number of elements and the first target quantity, or the ratio obtained by weighting the total number of elements and the first target quantity separately. In implementation, those skilled in the art can choose the method for determining the first ratio according to actual needs; this application does not limit this method.

[0141] In some implementations, the first ratio M1 can be determined by the following formula (2-4):

[0142]

[0143] Where AllNum represents the total number of elements, and BNum represents the number of the first target.

[0144] In some implementations, the remainder between the total number of elements and the first target quantity can be 0 or not.

[0145] Step S222: According to the first ratio, perform the bucketing process on the dataset to be processed to obtain each target bucket.

[0146] Here, the number of elements in each target bucket is not less than the first ratio. During implementation, if the remainder between the total number of elements and the first target number is 0, then the number of elements in each target bucket is the first ratio; if the remainder between the total number of elements and the first target number is not 0, then the number of elements in some target buckets is the first ratio, and the number of elements in the remaining target buckets is the first ratio + 1.

[0147] For example, if the dataset contains 16,000 elements, and each bucket can handle a maximum of 1,000 elements, then according to formula (2-1) above, the first target number of buckets is 16. Therefore, by performing the bucketing operation in a uniform manner, each target bucket will contain 1,000 elements. indivual.

[0148] For example, if the dataset contains 14,900 elements, and each bucket can process a maximum of 1,000 elements, then according to formula (2-1) above, the first target number of buckets is 16. If the bucketing operation is performed evenly, and since the remainder between 14,900 and 16 is 4, the 12 target buckets will contain 931 elements. The remaining four target buckets contain a total of 932 elements.

[0149] In this embodiment, a first ratio is determined between the total number of elements and the first target number of buckets; the dataset to be processed is then binned according to the first ratio to obtain each target bucket. This approach improves the accuracy of the first ratio by determining the total number of elements and the first target number of buckets, thereby enhancing the accuracy and uniformity of subsequent binning. Furthermore, using the first ratio for binning improves the uniformity of the number of elements among the buckets, allowing for precise control of the verification time for each bucket and reducing the complexity of the verification process.

[0150] Step S23: For each element in each target bucket, determine the target type sorting result of the target bucket, and perform verification processing on the element based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket.

[0151] Step S24: Merge the target verification results of each element in each target bucket to obtain the processing result of the dataset to be processed.

[0152] Here, steps S23 and S24 correspond to steps S12 to S13 mentioned above, respectively. In implementation, the specific implementation methods of steps S12 to S13 mentioned above can be referred to.

[0153] In this embodiment, a first target number of buckets is determined based on the total number of elements in the dataset to be processed and a preset quantity threshold. Based on the first target number of buckets and the total number of elements, the dataset to be processed is bucketed to obtain at least one target bucket. For each element in each target bucket, a target type sorting result is determined, and the element is validated based on the target type sorting result to obtain a target validation result. Each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket. The target validation results of each element in each target bucket are merged to obtain the processing result of the dataset to be processed. Thus, on the one hand, determining the first target number of buckets by using the total number of elements and a quantity threshold improves the accuracy of the first target number, thereby improving the accuracy of subsequent bucketing operations. On the other hand, bucketing operations based on the total number of elements and the first target number of buckets ensure that the number of elements in each bucket is essentially the same, which not only improves the uniformity of the number of elements in each bucket, but also allows for precise control of the validation time of each bucket and reduces the complexity of the validation time.

[0154] Figure 3 This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application, such as... Figure 3 As shown, the method includes steps S31 to S36, wherein:

[0155] Step S31: Determine the target memory required to process the dataset to be processed.

[0156] Here, the target memory represents the memory required to process the dataset. Methods for determining the target memory may include, but are not limited to, a memory threshold, remaining memory, the difference / mean / variance / mean squared error between the memory threshold and remaining memory, or the difference / mean / variance / mean squared error after weighting the memory threshold and remaining memory. The memory threshold represents the maximum execution memory of the system, and the remaining memory is obtained based on the current remaining memory and a preset memory ratio. The preset memory ratio represents the percentage of memory remaining after completing the verification processing of the dataset. In implementation, those skilled in the art can choose the method for determining the target memory according to actual needs; this application does not limit this method.

[0157] In some embodiments, step S31 includes steps S311 to S312, wherein:

[0158] Step S311: Determine the remaining memory based on the current remaining memory and the preset memory ratio.

[0159] Here, the preset memory ratio represents the percentage of memory remaining after the validation processing of the dataset is completed.

[0160] In some implementations, the remaining memory can be obtained in any suitable way, such as through system functions or specified files. In practice, those skilled in the art can choose the appropriate method to obtain the remaining memory based on actual needs; this application does not impose any limitations.

[0161] The methods for determining the remaining memory may include, but are not limited to, the current remaining memory, the product of the current remaining memory and the memory percentage, or the product of weighted averages of the current remaining memory and the memory percentage. In implementation, those skilled in the art can choose the method for determining the remaining memory according to actual needs; this application does not limit such methods.

[0162] In some implementations, the remaining memory gl can be determined using the following formula (3-1):

[0163] gl = gb * gn(3-1);

[0164] Where gb represents the preset memory ratio, and gn represents the current remaining memory.

[0165] Step S312: Based on a preset memory threshold and the remaining memory, determine the target memory required to process the dataset to be processed.

[0166] Here, the memory threshold represents the maximum execution memory of the system. Methods for determining the target memory may include, but are not limited to, a memory threshold, remaining memory, the difference / mean / variance / mean squared error between the memory threshold and remaining memory, or the difference / mean / variance / mean squared error after weighting the memory threshold and remaining memory. In implementation, those skilled in the art can choose the method for determining the target memory according to actual needs; this application does not limit this method.

[0167] In some implementations, the target memory dg can be determined using the following formula (3-2):

[0168] dg = min(g:gl)(3-2);

[0169] Where g represents the memory threshold and gl represents the remaining memory.

[0170] In this embodiment, the remaining memory to be processed is determined based on the current remaining memory and a preset memory ratio; the target memory required to process the dataset to be processed is determined based on a preset memory threshold and the remaining memory to be processed. Thus, determining the target memory using the current remaining memory, the preset memory ratio, and the preset memory threshold improves the accuracy of the target memory, thereby improving the precision of subsequent bucketing operations.

[0171] Step S32: Based on the total size of the dataset to be processed and the target memory, determine the second target number of buckets.

[0172] Here, the methods for determining the second target quantity may include, but are not limited to, the seventh ratio between the total size and the target memory, rounding the seventh ratio, taking the logarithm, rounding, and exponentiation of the rounded seventh ratio, and then weighting the total size and the target memory to obtain an eighth ratio, rounding the eighth ratio, and then taking the logarithm, rounding, and exponentiation of the eighth ratio, etc. In implementation, those skilled in the art can choose the method for determining the second target quantity according to actual needs; this application embodiment does not impose limitations.

[0173] In some implementations, the second target quantity BNum2 can be determined by the following formula (3-3):

[0174]

[0175] Where AllG represents the total size, dg represents the target memory, and ROUND() represents rounding up.

[0176] In some embodiments, step S32 includes steps S321 to S322, wherein:

[0177] Step S321: Determine the second initial number of buckets based on the total size of the dataset to be processed and the target memory.

[0178] Here, the second initial quantity is used to characterize the number of average buckets. The methods for determining the second initial quantity may include, but are not limited to, a seventh ratio between the total size and the target memory, rounding the seventh ratio, an eighth ratio obtained by weighting the total size and the target memory, and rounding the eighth ratio. In implementation, those skilled in the art can choose the method for determining the second initial quantity according to actual needs; this application embodiment does not impose limitations.

[0179] In some implementations, the second initial quantity b2 can be determined by the following formula (3-4):

[0180]

[0181] Where AllG represents the total size, dg represents the target memory, and ROUND() represents rounding up.

[0182] Step S322: Determine the second target quantity of the buckets based on the second initial quantity of the buckets.

[0183] Here, the methods for determining the second target quantity may include, but are not limited to, pairing, rounding, and exponencing the second initial quantity, or pairing, rounding, and exponencing the second initial quantity after weighting. In implementation, those skilled in the art can choose the method for determining the second target quantity according to actual needs; this application does not limit such methods.

[0184] In some implementations, the second target quantity BNum2 can be determined using the following formula (3-5):

[0185]

[0186] Where b2 represents the second initial quantity, and ROUND() represents rounding up.

[0187] Step S33: Determine the memory information of the buckets based on the second target number of the buckets and the total size of the dataset to be processed.

[0188] Here, the bucket's memory information represents the bucket's actual memory size. Methods for determining the memory information may include, but are not limited to, the ratio between the total size and the second target quantity, or the ratio after weighting the total size and the second target quantity. In implementation, those skilled in the art can choose the method for determining the memory information according to actual needs; this application's embodiments do not impose limitations.

[0189] In some implementations, the memory information MG can be determined using the following formulas (3-6):

[0190]

[0191] Where AllG represents the total size, and BNum2 represents the number of second targets.

[0192] Step S34: Based on the memory information of the bucket and the maximum / minimum value information of the dataset to be processed, perform the bucketing process on the dataset to be processed to obtain at least one target bucket.

[0193] Here, the bucketing method can include, but is not limited to, the default method, the random method, the uniform method, and the custom method. The custom method can include, but is not limited to, the previous method, the most frequently used method, or the user's preferred method. In implementation, those skilled in the art can choose the bucketing method according to actual needs; this application embodiment does not impose any limitations. For example, the dataset to be processed is bucketed randomly to obtain each target bucket. In this case, the number of elements in each target bucket is within a set range, which is determined based on maximum / minimum value information and memory information.

[0194] In some implementations, the extremum information includes a first extremum and a second extremum; step S34 includes steps S341 to S343, wherein:

[0195] Step S341: Determine the second ratio between the memory information of the bucket and the first maximum value.

[0196] Here, the first maximum / minimum value represents the size of the largest element in the dataset. For example, if the largest element in the dataset is 30KB, then the first maximum / minimum value is 30KB. The methods for determining the second ratio may include, but are not limited to, the ratio between memory information and the first maximum / minimum value, or the ratio after weighting the memory information and the first maximum / minimum value separately. In implementation, those skilled in the art can choose the method for determining the second ratio according to actual needs; this application does not limit this approach.

[0197] In some implementations, the second ratio X1 can be determined by the following formula (3-7):

[0198]

[0199] Where MG represents memory information and AG represents the first maximum value.

[0200] Step S342: Determine the third ratio between the memory information of the bucket and the second maximum value.

[0201] Here, the second extreme value represents the size of the smallest element in the dataset. For example, if the smallest element in the dataset is 10KB, then the second extreme value is 10KB. The methods for determining the third ratio may include, but are not limited to, the ratio between memory information and the second extreme value, or the ratio after weighting the memory information and the second extreme value. In implementation, those skilled in the art can choose the method for determining the third ratio according to actual needs; this application does not limit this approach.

[0202] In some implementations, the third ratio X2 can be determined by the following formula (3-8):

[0203]

[0204] Where MG represents memory information and NG represents the second maximum value.

[0205] Step S343: Perform the bucketing process on the dataset to be processed to obtain each target bucket.

[0206] Here, the number of elements in each target bucket is between the second and third ratios. In practice, the number of elements in each target bucket can be the same or different. For example, if the target memory required to process this dataset is 0.5MB, the total size of the dataset is 9.6MB, the first maximum value is 30KB, and the second maximum value is 10KB, then according to the above formula (3-3), the second target number is 32. At this time, according to the above formula (3-6), the memory information is 0.3MB. According to the random bucketing operation, according to the above formula (3-7), the second ratio is 0.01MB, and according to the above formula (3-8), the third ratio is 0.03MB. Correspondingly, the number of elements contained in the 16 target buckets is between 0.01MB and 0.03MB.

[0207] In this embodiment, a second ratio between the memory information of the bucket and the first maximum / minimum value is determined; a third ratio between the memory information of the bucket and the second maximum / minimum value is determined; and the dataset to be processed is subjected to the bucketing process to obtain each target bucket. Thus, on the one hand, determining the second and third ratios using memory information and maximum / minimum value information can improve the accuracy of the ratios, thereby improving the accuracy and uniformity of subsequent bucketing; on the other hand, performing bucketing operations using the second and third ratios can precisely control the memory resources of each bucket, ensuring normal system operation while improving system utilization, thereby improving the efficiency of verification processing and system response efficiency.

[0208] Step S35: For each element in each target bucket, determine the target type sorting result of the target bucket, and perform verification processing on the element based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket.

[0209] Step S36: Merge the target verification results of each element in each target bucket to obtain the processing result of the dataset to be processed.

[0210] Here, steps S35 and S36 correspond to steps S12 to S13 above, respectively. In implementation, the specific implementation methods of steps S12 to S13 above can be referred to.

[0211] In this embodiment, the following steps are taken: First, the target memory required to process the dataset to be processed is determined. Second, a second target number of buckets is determined based on the total size of the dataset to be processed and the target memory. Third, the memory information of the buckets is determined based on the second target number of buckets and the total size of the dataset to be processed. Fourth, the dataset to be processed is bucketed based on the memory information of the buckets and the maximum / minimum value information of the dataset to be processed, resulting in at least one target bucket. Fifth, for each element in each target bucket, a target type sorting result is determined, and the element is validated based on the target type sorting result to obtain a target validation result for the element. Each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket. Sixth, the target validation results of each element in each target bucket are merged to obtain the processing result of the dataset to be processed. In this way, on the one hand, by determining the second target number of buckets based on the total size of the dataset to be processed and the target memory, the accuracy of the second target number can be improved, thereby improving the accuracy of subsequent bucketing operations; on the other hand, by performing bucketing operations based on the memory information of the buckets and the maximum and minimum value information of the dataset to be processed, the memory resources of each bucket can be precisely controlled, which not only ensures the normal use of the system but also improves the system utilization rate, thereby improving the efficiency of verification processing and the system response efficiency.

[0212] Figure 4 This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application, such as... Figure 4 As shown, the method includes steps S41 to S43, wherein:

[0213] Step S41: Based on the attribute information of the dataset to be processed, perform a bucketing operation on the dataset to be processed to obtain at least one target bucket.

[0214] Here, step S41 corresponds to step S11 mentioned above. In implementation, the specific implementation method of step S11 mentioned above can be referred to.

[0215] Step S42: For each element in each target bucket, determine the target type sorting result of the target bucket, determine an initial type from at least one type in the target type sorting result, perform the verification process on the element using the verification method corresponding to the initial type, obtain an initial verification result, and obtain the target verification result of the element based on the initial verification result; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket.

[0216] Here, the initial type is one of the types in the target type sorting result. In implementation, the method for determining the initial type can include, but is not limited to, randomness, specifying a position (first, last, etc.), or customization. Those skilled in the art can choose the method for determining the initial type according to actual needs, and this application embodiment does not impose any limitations. For example, if the target type sorting result represents a sorting of types from highest to lowest probability, the first type can be used as the initial type.

[0217] The initial validation result may include, but is not limited to, the initial type validation passing or failing. In implementation, if the initial type validation passes, the initial validation result is used as the target validation result, and the initial type is used as the element's type. If the initial type validation fails, the next initial type is determined from at least one type in the target type sorting result. The element is then validated using the validation method corresponding to the next initial type to obtain the next initial validation result. Based on this next initial validation result, the target validation result for the element is obtained.

[0218] The number of validation processes does not exceed the number of types in the target type sorting result.

[0219] In some implementations, the step S42, "obtaining the target verification result of the element based on the initial verification result," includes steps S421 and / or S422, wherein:

[0220] Step S421: If the initial verification result meets the second preset condition, the initial verification result is taken as the target verification result.

[0221] Here, the second preset condition indicates that the initial verification result is that the initial type verification has passed.

[0222] Step S422: If the initial verification result does not meet the second preset condition, determine the next initial type from at least one type in the target type sorting result, use the verification method corresponding to the next initial type to perform the verification processing on the element, obtain the next initial verification result, and obtain the target verification result based on the next initial verification result.

[0223] Here, the next initial type differs from the initial type. In implementation, the method for determining the next initial type is the same as the method for determining the initial type. For example, if the target type sorting result includes {phone number, ID card number, other}, and the initial type is phone number, then the next initial type could be ID card number.

[0224] In this embodiment, if the initial verification result meets the second preset condition, the initial verification result is used as the target verification result; and / or, if the initial verification result does not meet the second preset condition, a next initial type is determined from at least one type in the target type sorting results, and the element is subjected to the verification processing using the verification method corresponding to the next initial type to obtain a next initial verification result, and the target verification result is obtained based on the next initial verification result. Thus, by obtaining the target verification result of the element through at least one verification process, the accuracy of the target verification result can be improved, thereby improving the accuracy of the dataset processing results.

[0225] Step S43: Merge the target verification results of each element in each target bucket to obtain the processing result of the dataset to be processed.

[0226] Here, step S43 corresponds to step S13 mentioned above. In implementation, the specific implementation method of step S13 mentioned above can be referred to.

[0227] In this embodiment, the dataset to be processed is binned based on attribute information to obtain at least one target bucket. For each element in each target bucket, a target type sorting result is determined. An initial type is determined from at least one type in the target type sorting result. The element is then validated using the validation method corresponding to the initial type to obtain an initial validation result. Based on the initial validation result, a target validation result is obtained for the element. Each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket. The target validation results of each element in each target bucket are merged to obtain the processing result of the dataset to be processed. This approach, on the one hand, by determining an initial type from multiple types and performing validation processing according to the initial type, shortens the validation time and improves validation accuracy, thereby reducing the complexity of the validation time. On the other hand, obtaining the target validation result of an element through at least one validation process improves the accuracy of the target validation result, thus improving the accuracy of the dataset processing result.

[0228] Figure 5A This is a schematic diagram of the composition structure of a data processing system provided in an embodiment of this application, such as... Figure 5A As shown, the system includes an input component 51, a processing component 52, and an output component 53, wherein:

[0229] The input component 51 is used to receive the dataset to be processed;

[0230] The processing component 52 is configured to perform bucketing on the dataset to be processed based on the attribute information of the dataset to be processed, to obtain at least one target bucket; for each element in each target bucket, determine the target type sorting result of the target bucket, and perform verification processing on the element based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket; and merge the target verification results of each element in each target bucket to obtain the processing result of the dataset to be processed.

[0231] The output component 53 is used to output the processing result.

[0232] Here, the input component 51 can be any suitable component capable of receiving functions, such as a receiver or a mouse. In implementation, those skilled in the art can independently determine the implementation method of this input component according to actual needs; this application embodiment does not impose any limitations.

[0233] The processing component 52 can be any suitable component capable of performing processing functions, such as a processor. In implementation, those skilled in the art can independently determine the implementation method of this processing component according to actual needs; the embodiments of this application do not impose limitations.

[0234] The output component 53 can be any suitable component capable of performing output functions, such as a display or a display page. In implementation, those skilled in the art can independently determine the implementation method of the output component according to actual needs; this application embodiment does not impose any limitations.

[0235] The number of datasets to be processed can be at least one. The datasets to be processed can be selected by the user through the input module of an electronic device, or they can be sent by other electronic devices. The input module can include, but is not limited to, a keyboard, mouse, touchscreen, touchpad, audio input device, etc.

[0236] In some implementations, the attribute information of the dataset to be processed includes the total number of elements; the processing component 52 is further configured to: determine a first target number of buckets based on the total number of elements and a preset quantity threshold; determine a first ratio between the total number of elements and the first target number of buckets; and perform the bucketing process on the dataset to be processed according to the first ratio to obtain each target bucket.

[0237] Here, the number of elements in each target bucket is not less than the first ratio. In implementation, using the total number of elements in the dataset for the bucketing algorithm ensures that the dataset is evenly distributed across the buckets, with the number of elements between buckets having an error of less than 1. This reduces the algorithm's time complexity, and the uniform data volume in each bucket allows for precise control over the processing time for each bucket.

[0238] In some implementations, the attribute information of the dataset to be processed includes the total size and maximum / minimum value information of the dataset to be processed; the processing component 52 is further configured to: determine the remaining memory based on the current remaining memory and a preset memory ratio; determine the target memory required to process the dataset to be processed based on a preset memory threshold and the remaining memory; determine a second target number of buckets based on the total size of the dataset to be processed and the target memory; determine the memory information of the buckets based on the second target number of buckets and the total size of the dataset to be processed; determine a second ratio between the memory information of the buckets and a first maximum / minimum value in the maximum / minimum value information; determine a third ratio between the memory information of the buckets and a second maximum / minimum value in the maximum / minimum value information; and perform the bucketing process on the dataset to be processed to obtain each target bucket.

[0239] Here, the number of elements in each target bucket falls between the second and third ratios. In implementation, the bucketing algorithm is performed using the device's memory, controlling memory resource size to ensure that element verification processing does not consume excessive system resources, thus preventing other system functions from becoming unavailable. This precise control of memory resources avoids impacting other system functions, thereby improving system availability.

[0240] In some implementations, when the dataset to be processed is of a specified type, the target type sorting result only includes the specified type; the processing component 52 is further configured to: perform verification processing on each element in each target bucket according to the verification method corresponding to the specified type, and obtain the target verification result of the element. The specified type may include, but is not limited to, basic types, extended types, etc.

[0241] Here, each type corresponds to a different verification method. If the target verification result of the element indicates that the specified type verification passes, the element is stored as a key-value pair, where the key is the specified type and the value is the element itself. If the target verification result of the element indicates that the specified type verification fails, the element is stored as a key-value pair, where the key is the invalid type and the value is the element itself. In some implementations, each type can be defined using tags. For example, the tag value for an ID card number is 01, the tag value for a phone number is 02, the tag value for an email address is 03, the tag value for a birth date is 04, the tag value for an address is 05, the tag value for an ethnicity is 06, the tag value for other types is -001, and the tag value for invalid types is -002, etc.

[0242] In some implementations, when the type of the dataset to be processed is unspecified, the target type sorting result includes the initial type sorting result or the updated previous type sorting result; the processing component 52 is further configured to: determine the initial type sorting result of the target bucket when the element meets the first preset condition, and use the initial type sorting result as the target type sorting result; when the element does not meet the first preset condition, update the previous target type sorting result based on the target verification result of the previous element to obtain the target type sorting result.

[0243] Here, the initial type sorting result can be a preset type sorting result, or it can be determined based on the type sorting result corresponding to each element in at least one historical dataset.

[0244] In some embodiments, the processing component 52 is further configured to: determine an initial type from at least one type in the target type sorting results; perform the verification processing on the element using the verification method corresponding to the initial type to obtain an initial verification result; if the initial verification result satisfies a second preset condition, use the initial verification result as the target verification result; if the initial verification result does not satisfy the second preset condition, determine a next initial type from at least one type in the target type sorting results, perform the verification processing on the element using the verification method corresponding to the next initial type to obtain a next initial verification result, and obtain the target verification result based on the next initial verification result.

[0245] Here, the initial type can be the first or last type in the target type sorting result. In implementation, this initial type represents the most likely type of the current element.

[0246] Figure 5B This application provides a schematic diagram of an implementation process for validating elements in a bucket, as illustrated in the embodiments of this application. Figure 5B As shown, the method includes steps S511 to S518, wherein:

[0247] Step S511: For the first element in the bucket, determine the initial type from at least one type in the determined initial type sorting results, and use the verification method corresponding to the initial type to perform verification processing on the first element to obtain the initial verification result of the first element.

[0248] Step S512: If the initial verification result of the first element passes (corresponding to the aforementioned second preset condition), proceed to step S514; otherwise, proceed to step S511.

[0249] Step S513: Use the initial verification result as the target verification result of the first element, update the initial type sorting result according to the target verification result of the first element, and use the updated initial type sorting result as the target type sorting result.

[0250] Step S514: For the second element, determine the initial type from at least one type in the target type sorting results, and use the verification method corresponding to the initial type to perform verification processing on the second element to obtain the initial verification result of the second element.

[0251] Step S515: Determine whether the initial verification result of the second element passes, then proceed to step S516; otherwise, proceed to step S514.

[0252] Step S516: Determine whether the second element is the last element. If not, proceed to step S517; otherwise, proceed to step S518.

[0253] Step S517: Take the initial verification result of the second element as the target verification result of the second element, update the target type sorting result according to the target verification result of the second element, take the updated target type sorting result as the new target type sorting result, and take the next second element as the second element, and proceed to step S514.

[0254] Step S518: Output the percentage of each type of element.

[0255] In some implementations, the processing component 52 is further configured to: for each target bucket, determine the target number of threads and the target type sorting result of the target bucket, schedule the target number of threads in parallel, and perform verification processing on each element based on the target type sorting result to obtain the target verification result of each element.

[0256] Figure 5C This is a schematic diagram illustrating the implementation flow of a data processing method provided in an embodiment of this application, such as... Figure 5C As shown, the method includes steps S521 to S524, wherein:

[0257] Step S521: Receive the dataset to be processed;

[0258] Step S522: Based on the attribute information of the dataset to be processed, perform a bucketing operation on the dataset to be processed to obtain at least one target bucket;

[0259] Here, you can perform uniform bucketing based on the total number of elements in the dataset, or random bucketing based on the total size and maximum / minimum value information of the dataset.

[0260] Step S523: For each target bucket, schedule threads of the target number in parallel, perform verification processing on each element based on the target type sorting result, and obtain the target verification result for each element.

[0261] Step S524: Merge the data in pairs to obtain the processing result of the dataset.

[0262] Here, the processing results include the type of each element and the percentage of each type.

[0263] In some embodiments, the processing component 52 is further configured to: determine the CPU time consumption based on the waiting time corresponding to each type in the target type sorting result when the target type sorting result meets a third preset condition; or, when the target type sorting result does not meet the third preset condition, determine a target waiting time from the waiting time corresponding to each type in the target type sorting result and use the target waiting time as the CPU time consumption; determine the utilization rate of the remaining CPU based on the current remaining CPU utilization rate and a preset CPU ratio; determine the target utilization rate of the CPU based on a preset threshold utilization rate and the remaining CPU utilization rate; determine a fourth ratio between the threshold waiting time and the CPU time consumption; and obtain the target number of threads based on the number of CPU cores, the fourth ratio, and the target utilization rate of the CPU.

[0264] For example, if each bucket contains M elements, and there are n element types (e.g., ID card number, phone number, email address, date of birth, address, ethnicity, etc.), and each type has a corresponding regular expression for validation, then during the validation process for each element in the bucket:

[0265] (1) The existing verification method requires n verifications for each element. Therefore, the number of verifications performed in each bucket is M*n, and the time complexity of the conventional algorithm is M*n.

[0266] (2) The verification method provided in this application first determines the target type sorting result, then determines the initial type from at least one type in the target type sorting, and verifies the element according to the verification method of the initial type. The initial type represents the element as most likely to be of that initial type; that is, only one verification is needed to obtain the target verification result of the element. Therefore, the minimum number of verifications performed in each bucket is M. The time complexity W can then be expressed by the following formula (5-1):

[0267]

[0268] Among them, S0 represents the position information of the most numerous type in the initial type sorting result, P i S represents the proportion of the i-th type. i The order of the i-th type is represented by T, where T is the target number of threads.

[0269] For P n *M*S n In terms of P n *M represents the quantity of type n in this bucket, P n *M*S nThis represents the total number of executions for type n. If the total number of elements M in the bucket is 10000, the target type sorting result of the bucket is {phone number, ID number, email, date of birth, address, ethnicity}, and the ethnicity percentage is 1%, since the ethnicity is in the order of 5 in the target type sorting result, it takes 6 executions to verify an ethnicity element from phone number to ethnicity. Therefore, the number of ethnicities in the bucket is 100, and the number of times the ethnicity type elements in the bucket need to be verified is 600.

[0270] For example, if the elements in the bucket are all phone numbers, and the target number of threads is 10,

[0271] When the target type sorting result is a custom type sorting result {phone number, others}, the element type n is 2, the order of phone numbers is 0 and the percentage is 100%, and the order of others is 1 and the percentage is 0, that is: S0 = 0, S1 = 1, P1 = 0. According to the above formula (5-1), the time complexity can be obtained as:

[0272] If the target type sorting result is the sorting result based on the historical dataset {phone number, ID card number, email, date of birth, address, ethnicity}, then the element type n is 6. The order of phone numbers is 0, with a percentage of 100%; the order of ID card numbers is 1, with a percentage of 0%; the order of email numbers is 2, with a percentage of 0%; the order of date of birth is 3, with a percentage of 0%; the order of addresses is 4, with a percentage of 0%; and the order of ethnicity is 5, with a percentage of 0%. That is: S0=0, S1=1, P1=0, S2=2, P2=0, S3=3, P3=0, S4=4, P4=0, S5=5, P5=0. According to the above formula (5-1), the time complexity can be obtained as:

[0273]

[0274] For example, if most of the elements in the bucket are phone numbers, and the target number of threads is 10,

[0275] Given that the target type sorting result is based on a historical dataset, resulting in {phone number, ID card number, email, date of birth, address, ethnicity}, where phone numbers account for 60%, ID card numbers for 30%, emails for 8%, and addresses for 2%, then the element type n is 6. The order of phone numbers is 0 (60%), ID card numbers are 1 (30%), emails are 2 (8%), and addresses are 4 (2%). Therefore, S0 = 0, S1 = 1, P1 = 30%, S2 = 2, P2 = 8%, S3 = 4, and P3 = 2%. According to formula (5-1), the time complexity is:

[0276] Given that the target type sorting result is based on a historical dataset and is sorted as {ID number, phone number, email, date of birth, address, ethnicity}, where phone numbers account for 60%, ID numbers for 30%, emails for 8%, and addresses for 2%, then the element type n is 6. The order of ID numbers is 0 (30%), the order of phone numbers is 1 (60%), the order of emails is 2 (8%), and the order of addresses is 4 (2%). Therefore, S0 = 1, S1 = 1, P1 = 60%, S2 = 2, P2 = 8%, S3 = 4, and P3 = 2%. According to formula (5-1), the time complexity is:

[0277] Therefore, based on the time complexity above, we can see that the closer the types of the elements in the bucket are to the same type, the lower the required time complexity; the fewer types in the target type sorting result, the lower the complexity; and the more the target type sorting result matches the proportion of each type in the current bucket, the lower the complexity.

[0278] In this embodiment, firstly, by binning the dataset to be processed, the possibility of system memory anomalies or unavailability due to excessively large data can be reduced; secondly, by using the target type sorting results of the bins, each element in the bin is automatically verified without manual intervention, which shortens the verification time and improves the verification accuracy, thereby reducing the complexity of verification time and labor costs; finally, by using the classified elements in the bins to determine the target type sorting results in real time, the accuracy of the target type sorting results can be improved, thereby reducing the number of times each element is verified subsequently, which can further shorten the verification time and reduce the complexity of verification time.

[0279] Based on the foregoing embodiments, this application provides a data processing device, which includes the included units and the modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0280] Figure 6 This is a schematic diagram of the composition structure of a data processing device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device 60 includes a binning module 61, a verification module 62, and a merging module 63, wherein:

[0281] The bucketing module 61 is used to perform bucketing operations on the dataset to be processed based on the attribute information of the dataset to be processed, to obtain at least one target bucket; wherein the dataset to be processed includes at least one element of at least one type, and each target bucket includes at least one element of at least one type;

[0282] The verification module 62 is used to determine the target type sorting result of each element in each target bucket, and to perform verification processing on the element based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket;

[0283] The merging module 63 is used to merge the verification results of each element in each target bucket to obtain the processing result of the dataset to be processed.

[0284] In some implementations, the attribute information of the dataset to be processed includes the total number of elements; the bucketing module 61 is further configured to: determine a first target number of buckets based on the total number of elements and a preset quantity threshold; and perform the bucketing process on the dataset to be processed based on the first target number of buckets and the total number of elements to obtain each target bucket.

[0285] In some embodiments, the bucketing module 61 is further configured to: determine a first ratio between the total number of elements and the first target number of buckets; perform the bucketing process on the dataset to be processed according to the first ratio to obtain each target bucket; wherein the number of elements in each target bucket is not less than the first ratio.

[0286] In some implementations, the attribute information of the dataset to be processed includes the total size and maximum / minimum value information of the dataset to be processed; the bucketing module 61 is further configured to: determine the target memory required to process the dataset to be processed; determine a second target number of buckets based on the total size of the dataset to be processed and the target memory; determine the memory information of the buckets based on the second target number of buckets and the total size of the dataset to be processed; and perform the bucketing process on the dataset to be processed based on the memory information of the buckets and the maximum / minimum value information to obtain each target bucket.

[0287] In some implementations, the extremum information includes a first extremum and a second extremum; the bucketing module 61 is further configured to: determine a second ratio between the memory information of the bucket and the first extremum; determine a third ratio between the memory information of the bucket and the second extremum; perform the bucketing process on the dataset to be processed to obtain each target bucket, wherein the number of elements in each target bucket is between the second ratio and the third ratio.

[0288] In some implementations, the bucketing module 61 is further configured to: determine the remaining memory based on the current remaining memory and a preset memory ratio; and determine the target memory required to process the dataset to be processed based on a preset memory threshold and the remaining memory.

[0289] In some embodiments, the verification module 62 is further configured to: determine the initial type sorting result of the target bucket when the element satisfies the first preset condition, and use the initial type sorting result as the target type sorting result; and / or, when the element does not satisfy the first preset condition, update the previous target type sorting result based on the target verification result of the previous element to obtain the target type sorting result.

[0290] In some implementations, the verification module 62 is further configured to: use a preset type sorting result as the initial type sorting result; determine at least one historical dataset based on the dataset to be processed, and determine the initial type sorting result of the target bucket based on the type sorting result corresponding to each element in each historical dataset.

[0291] In some embodiments, the verification module 62 is further configured to: determine an initial type from at least one type in the target type sorting results; perform the verification processing on the element using the verification method corresponding to the initial type to obtain an initial verification result; and obtain the target verification result based on the initial verification result.

[0292] In some embodiments, the verification module 62 is further configured to: if the initial verification result satisfies the second preset condition, use the initial verification result as the target verification result; and / or, if the initial verification result does not satisfy the second preset condition, determine the next initial type from at least one type in the target type sorting result, perform the verification processing on the element using the verification method corresponding to the next initial type, obtain the next initial verification result, and obtain the target verification result based on the next initial verification result.

[0293] In some implementations, the verification module 62 is further configured to: for each target bucket, determine the target number of threads and the target type sorting result of the target bucket, schedule the target number of threads in parallel, and perform verification processing on each element based on the target type sorting result to obtain the target verification result of each element.

[0294] In some implementations, the verification module 62 is further configured to: determine the CPU's latency and the target utilization rate of the CPU; determine a fourth ratio between a preset threshold waiting time and the CPU's latency; and obtain the target number of threads based on the number of CPU cores, the fourth ratio, and the target utilization rate of the CPU.

[0295] In some embodiments, the verification module 62 is further configured to: determine the CPU time consumption based on the waiting time corresponding to each type in the target type sorting result when the target type sorting result meets a third preset condition; and / or, determine a target waiting time from the waiting time corresponding to each type in the target type sorting result when the target type sorting result does not meet the third preset condition, and use the target waiting time as the CPU time consumption.

[0296] In some implementations, the verification module 62 is further configured to: determine the utilization rate of the remaining CPU based on the current utilization rate of the remaining CPU and a preset CPU ratio; and determine the target utilization rate of the CPU based on a preset threshold utilization rate and the utilization rate of the remaining CPU.

[0297] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0298] It should be noted that, in the embodiments of this application, if the above-described spray control method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0299] Correspondingly, embodiments of this application provide an electronic device including a memory and a processor. The memory stores a computer program that can run on the processor. The processor executes the computer program to implement the steps of the methods provided in the above embodiments.

[0300] Correspondingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0301] Correspondingly, embodiments of this application provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium; in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0302] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of an electronic device in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the electronic device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0303] The processor 701 typically controls the overall operation of the electronic device 700.

[0304] Communication interface 702 enables electronic devices to communicate with other terminals or servers via a network.

[0305] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the electronic device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0306] It should be noted that the descriptions of the above-described storage media, computer program products, and device embodiments are similar to the descriptions of the above-described method embodiments, and have similar beneficial effects. For technical details not disclosed in the storage media and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0307] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0308] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0309] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0310] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this application may all be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in hardware or in a combination of hardware and software functional units.

[0311] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0312] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0313] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Based on the attribute information of the dataset to be processed, the dataset to be processed is divided into buckets to obtain at least one target bucket; wherein, the dataset to be processed includes at least one element of at least one type, and each target bucket includes at least one element of at least one type; For each element in each target bucket, the target type sorting result of the target bucket is determined, and the element is verified based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket; The target verification results of each element in each target bucket are merged to obtain the processing result of the dataset to be processed; The attribute information of the dataset to be processed includes the total size and maximum / minimum value information of the dataset to be processed; The process of bucketing the dataset to be processed based on its attribute information to obtain at least one target bucket includes: Determine the target memory required to process the dataset to be processed; Based on the total size of the dataset to be processed and the target memory, determine the second target number of buckets; The memory information of the buckets is determined based on the second target number of the buckets and the total size of the dataset to be processed; Based on the memory information and the maximum / minimum value information of the buckets, the bucketing operation is performed on the dataset to be processed to obtain each target bucket.

2. The method according to claim 1, characterized in that, The attribute information of the dataset to be processed includes the total number of elements; The process of bucketing the dataset to be processed based on its attribute information to obtain at least one target bucket includes: Based on the total number of elements and the preset quantity threshold, determine the first target number of buckets; Based on the first target number of buckets and the total number of elements, the dataset to be processed is subjected to the bucketing operation to obtain each target bucket.

3. The method according to claim 2, characterized in that, The step of performing the bucketing operation on the dataset to be processed based on the first target number of buckets and the total number of elements to obtain each target bucket includes: Determine a first ratio between the total number of elements and the first target number of the buckets; According to the first ratio, the dataset to be processed is subjected to the bucketing operation to obtain each target bucket; wherein the number of elements in each target bucket is not less than the first ratio.

4. The method according to claim 1, characterized in that, The extreme value information includes a first extreme value and a second extreme value; The step of performing the bucketing operation on the dataset to be processed based on the memory information and the maximum / minimum value information of the buckets to obtain each target bucket includes: Determine a second ratio between the memory information of the bucket and the first maximum / minimum value; Determine a third ratio between the memory information of the bucket and the second maximum / minimum value; The dataset to be processed is subjected to the bucketing operation to obtain each target bucket, wherein the number of elements in each target bucket is between the second ratio and the third ratio.

5. The method according to claim 1 or 4, characterized in that, Determining the target memory required to process the dataset to be processed includes: Based on the current remaining memory and the preset memory ratio, determine the remaining memory to be used; Based on a preset memory threshold and the remaining memory, the target memory required to process the dataset to be processed is determined.

6. The method according to any one of claims 1 to 4, characterized in that, The determination of the target type sorting result of the target bucket includes at least one of the following: If the element satisfies the first preset condition, determine the initial type sorting result of the target bucket, and use the initial type sorting result as the target type sorting result; If the element does not meet the first preset condition, the target type sorting result is updated based on the target verification result of the previous element to obtain the target type sorting result.

7. The method according to claim 6, characterized in that, The determination of the initial type sorting result of the target bucket includes one of the following: Use the preset type sorting result as the initial type sorting result; Based on the dataset to be processed, at least one historical dataset is determined, and based on the type sorting result corresponding to each element in each historical dataset, the initial type sorting result of the target bucket is determined.

8. The method according to any one of claims 1 to 4, characterized in that, The step of performing validation processing on the element based on the sorting result of the target type to obtain the target validation result of the element includes: Determine the initial type from at least one type in the sorting results of the target type; The element is subjected to the verification process using the verification method corresponding to the initial type to obtain the initial verification result; Based on the initial verification result, the target verification result is obtained.

9. The method according to claim 8, characterized in that, The process of obtaining the target verification result based on the initial verification result includes at least one of the following: If the initial verification result meets the second preset condition, the initial verification result shall be used as the target verification result; If the initial verification result does not meet the second preset condition, the next initial type is determined from at least one type in the target type sorting result, and the verification process is performed on the element using the verification method corresponding to the next initial type to obtain the next initial verification result. Based on the next initial verification result, the target verification result is obtained.

10. The method according to any one of claims 1 to 4, characterized in that, For each element in each target bucket, the process of determining the target type sorting result of the target bucket and performing verification processing on the element based on the target type sorting result to obtain the target verification result of the element includes: For each target bucket, determine the target number of threads and the target type sorting result of the target bucket, schedule the target number of threads in parallel, and perform verification processing on each element based on the target type sorting result to obtain the target verification result of each element.

11. The method according to claim 10, characterized in that, Determining the target number of threads includes: Determine the CPU time consumption and the target utilization rate of the CPU; Determine a fourth ratio between the threshold waiting time and the CPU's latency; The target number of threads is obtained based on the number of CPU cores, the fourth ratio, and the target utilization rate of the CPU.

12. The method according to claim 11, characterized in that, The determination of CPU time consumption includes at least one of the following: If the target type sorting result meets the third preset condition, the CPU time consumption is determined based on the waiting time corresponding to each type in the target type sorting result; If the target type sorting result does not meet the third preset condition, a target waiting time is determined from the waiting time corresponding to each type in the target type sorting result, and the target waiting time is used as the CPU time consumption.

13. The method according to claim 11 or 12, characterized in that, Determining the target utilization rate of the CPU includes: Based on the current remaining CPU utilization rate and the preset CPU ratio, determine the remaining CPU utilization rate; The target utilization rate of the CPU is determined based on the preset threshold utilization rate and the remaining CPU utilization rate.

14. A data processing apparatus, characterized in that, The device includes: The bucketing module is used to perform bucketing operations on the dataset to be processed based on the attribute information of the dataset to be processed, to obtain at least one target bucket; wherein the dataset to be processed includes at least one element of at least one type, and each target bucket includes at least one element of at least one type; The verification module is used to determine the target type sorting result of each element in each target bucket, and to perform verification processing on the element based on the target type sorting result to obtain the target verification result of the element; wherein, each target type sorting result represents the result of sorting at least one type, and each target type sorting result is obtained based on the classified elements in the corresponding target bucket; The merging module is used to merge the verification results of each element in each of the target buckets to obtain the processing result of the dataset to be processed. The attribute information of the dataset to be processed includes the total size and extreme value information of the dataset to be processed; the processing component is used to: determine the remaining memory based on the current remaining memory and a preset memory ratio; determine the target memory required to process the dataset to be processed based on a preset memory threshold and the remaining memory; determine a second target number of buckets based on the total size of the dataset to be processed and the target memory; determine the memory information of the buckets based on the second target number of buckets and the total size of the dataset to be processed; determine a second ratio between the memory information of the buckets and a first extreme value in the extreme value information; determine a third ratio between the memory information of the buckets and a second extreme value in the extreme value information; and perform the bucketing operation on the dataset to be processed to obtain each target bucket.

15. An electronic device comprising a processor and a memory, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 13.

Images