Method, apparatus, device and storage medium for processing data files

Abstract

An embodiment of the present application provides a method for processing data files and related products, which relates to the field of big data. The method includes: if a first rule set is not configured for the data file to be processed, verifying the sharding rules in the rule pool, and determining a second rule set from multiple experience sets according to at least one matching rule obtained through verification; performing sharding processing on the data file to be processed through the second rule set, and sending the obtained sharded files to the server, where the first rule set is a rule set adapted to the server. If there is an adapted first rule set, sharding processing can be performed through the first rule set. If not, the second rule set can be determined from past experience according to the verification operation, and sharding processing can be performed through the second rule set. Therefore, for sharding operations in different scenarios, the inference efficiency can be significantly improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology. Specifically, the present application relates to a method, apparatus, electronic device, and computer-readable storage medium for processing data files. Background Art

[0002] With the advent of the era of artificial intelligence, various industries have begun to focus on mining potential value from massive amounts of data. For example, operators, financial institutions, governments, etc. perform inferences based on a large amount of data generated in various scenarios by applying AI models to obtain the value hidden in the data. Due to the limitation of server resources, before performing inferences, the big data to be inferred is generally segmented, and inferences are performed based on the segmented data.

[0003] Currently, there are two inference methods. Among them, the first method, after obtaining a big data file, only performs basic sharding on the big data file and then performs inferences based on the sharded files. This method consumes a huge amount of server resources (mainly referring to CPU and memory), and the inference efficiency is relatively low. The second method only makes slight improvements to the basic sharding. Its inference efficiency has been improved for a small number of scenarios, but the improvement effect is not obvious for most scenarios. Summary of the Invention

[0004] The solution shown in the embodiments of the present application aims to solve one of the above technical problems.

[0005] According to one aspect of the embodiments of the present application, there is provided a method for processing a data file, the method including:

[0006] If a first rule set is not configured for the data file to be processed, verify the sharding rules in the rule pool, and determine a second rule set from multiple experience sets according to at least one matching rule obtained by the verification;

[0007] Perform sharding processing on the data file to be processed through the second rule set, and send the obtained sharded files to the server, where the first rule set is a rule set adapted to the server.

[0008] In a possible implementation manner, each sharding rule is configured with a priority level; verifying the sharding rules in the rule pool may specifically include:

[0009] Verify each sharding rule in the rule pool in order from the highest priority level to the lowest priority level to determine the first matching degree of each sharding rule with the server resources; determine the sharding rules corresponding to the first matching degrees greater than the preset threshold as the matching rules.

[0010] In another possible implementation manner, determining a second rule set from multiple experience sets according to at least one matching rule obtained by the verification may specifically include:

[0011] Determine the second matching degree between each experience set and at least one matching rule, where the experience set includes at least one sharding rule in the rule pool; determine the second rule set according to the experience set corresponding to the maximum second matching degree.

[0012] In another possible implementation, each sharding rule in the rule pool is verified to determine the first matching degree between each sharding rule and the server resources, including:

[0013] Determine the first resource of each sharding rule, where the first resource is the server resources required for the corresponding sharding rule to process the data file to be processed; compare the matching degree between the second resource and each first resource to determine the first matching degree between the corresponding sharding rule and the second resource, where the second resource is the currently provided server resources.

[0014] In another possible implementation, if the second rule set includes a feature protection sharding rule; perform sharding processing on the data file to be processed through the second rule set, specifically including:

[0015] Perform sharding on the data file to be processed according to other sharding rules in the second rule set to obtain at least two sharded files; adjust the data that meets the conditions in each sharded file among the at least two sharded files.

[0016] In another possible implementation, if the data file to be processed is configured with a first rule set, the method includes:

[0017] Perform sharding processing on the data file to be processed through the first rule set, and send the obtained sharded files to the server;

[0018] Wherein, the first rule set is a rule set adapted to the server, including: the first rule set is adapted to the inference model configured in the server.

[0019] In another possible implementation, if the data file to be processed is not configured with a first rule set, the method further includes:

[0020] If the total memory occupied by the data file to be processed is not greater than the first threshold, and the total number of entries of the data file to be processed is not greater than the second threshold, send the data file to be processed to a single server.

[0021] According to another aspect of the embodiments of the present application, there is provided a processing device for data files, the device includes:

[0022] A verification module, configured to verify the sharding rules in the rule pool if the data file to be processed is not configured with a first rule set, and determine a second rule set from multiple experience sets according to at least one matching rule obtained from the verification;

[0023] A sharding module, configured to shard a data file to be processed according to a second rule set and send the obtained sharded files to a server, where the first rule set is a rule set adapted to the server.

[0024] According to another aspect of the embodiments of the present application, there is provided an electronic device, which includes:

[0025] A memory, a processor, and a computer program stored on the memory, where the processor executes the computer program to implement the steps of a method for a data file as shown in the present application.

[0026] According to still another aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a method for a data file as shown in the present application are implemented.

[0027] The beneficial effects brought by the technical solutions provided in the embodiments of the present application are:

[0028] The embodiments of the present application provide a method for processing a data file, including: if a first rule set is not configured for the data file to be processed, verifying the sharding rules in the rule pool, and determining a second rule set from multiple experience sets according to at least one matching rule obtained by the verification; sharding the data file to be processed according to the second rule set and sending the obtained sharded files to the server; where the first rule set is a rule set adapted to the server. In the implementation of the present application, if there is a first rule set adapted to the server, the data file to be processed can be sharded according to the first rule set. If not, the second rule set can be determined from past experience according to the verification operation, and the data file to be processed can be sharded according to the second rule set. Therefore, for sharding operations in different scenarios, the inference efficiency can be significantly improved. Description of the Drawings

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application.

[0030] Figure 1a It is a schematic flowchart of a traditional offline inference method provided by an embodiment of the present application;

[0031] Figure 1b It is a schematic flowchart of an offline inference method based on basic sharding provided by an embodiment of the present application;

[0032] Figure 2 It is a schematic flowchart of a method for processing a data file provided by an embodiment of the present application;

[0033] Figure 3Schematic diagram of an application scenario for processing data files provided by an embodiment of the present application;

[0034] Figure 4 Schematic diagram of the structure of a data file processing device provided by an embodiment of the present application;

[0035] Figure 5 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0036] The accompanying drawings in the application depict the embodiments of the present application. It should be understood that the implementation manners described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and do not constitute limitations on the technical solutions of the embodiments of the present application.

[0037] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the terms "comprising" and "including" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation of other features, information, data, steps, operations, elements, components and / or their combinations supported by the art of the present technology. It should be understood that when we say an element is "connected" or "coupled" to another element, this element can be directly connected or coupled to the other element, or it can mean that this element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein can include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by this term. For example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0038] To make the objectives, technical solutions and advantages of the present application clearer, the following will further describe the implementation manners of the present application in detail in conjunction with the accompanying drawings.

[0039] First, introduce and explain several terms related to the present application:

[0040] Continuing from the background art, Figure 1a and Figure 1b are respectively the specific implementation flowcharts of the first method and the second method.

[0041] Among them, Figure 1aShows a schematic flow diagram of a traditional offline inference method. After receiving a large data file, the large data file is subjected to basic sharding, and the obtained sharded files are sent to the inference nodes of the server for inference. However, this method consumes a huge amount of server resources (mainly referring to CPU and memory), and the inference efficiency is relatively low.

[0042] Among them, Figure 1b Shows a schematic flow diagram of an offline inference method based on basic sharding. After receiving a large data file, any sharding rule is selected from the rule pool to shard the large data file. Although the sharding operation in this method helps to improve the inference efficiency in some scenarios, the improvement effect is not obvious for most scenarios. For example, if the data volume of the sharded files processed by the inference model changes greatly and data integrity is required, the sharded files obtained by a single sharding rule cannot meet this requirement.

[0043] A data file processing method, device, electronic device, and computer-readable storage medium provided by the present application aim to solve the above technical problems in the prior art.

[0044] The technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application will be described below through the description of several exemplary embodiments. It should be noted that the following embodiments can refer to, draw on, or combine with each other. For the same terms, similar features, and similar implementation steps in different embodiments, they will not be described repeatedly.

[0045] See Figure 2 , a data file processing method is provided in an embodiment of the present application. This method is applied to a terminal, and the terminal can be an electronic device such as a computer. The method includes steps S210 to S220:

[0046] S210, if the first rule set is not configured for the data file to be processed, verify the sharding rules in the rule pool, and determine a second rule set from multiple experience sets according to at least one matching rule obtained by the verification.

[0047] Among them, the data file to be processed can be an offline large data file. The characteristic of an offline large data file is that the data volume or the occupied memory is large. The large data file can be a file with a single record of data, or a folder that includes multiple sub-files with record data. For a folder, the total number of entries of the data file to be processed is the sum of the total number of entries of each sub-file, and the total occupied memory is the memory occupied by the folder.

[0048] Specifically, all the effective sharding rules are recorded in the rule pool. Each sharding rule can be configured with a unique identifier, interface information, and other information. Among them, the first rule set or the second rule set can include one or more sharding rules.

[0049] S220, perform sharding processing on the data file to be processed through the second rule set, and send the obtained sharded files to the server. The first rule set is a rule set adapted to the server.

[0050] Among them, before preparing to process offline big data, an inference model can be configured for it in the server. After configuring the inference model, a customized rule set adapted to the inference model can also be configured for it. The sharded files obtained by processing the big data file through the customized rule set can meet the requirements of the inference model for the sharded files. Among them, the customized rule set is the first rule set.

[0051] The embodiment of the present application provides a method for processing data files, including: if the first rule set is not configured for the data file to be processed, verify the sharding rules in the rule pool, and determine the second rule set from multiple experience sets according to at least one matching rule obtained by the verification; perform sharding processing on the data file to be processed through the second rule set, and send the obtained sharded files to the server; where the first rule set is a rule set adapted to the server. In the implementation of the present application, if there is a first rule set adapted to the server, the data file to be processed can be sharded through the first rule set. If not, the second rule set can be determined from past experience according to the verification operation, and the data file to be processed can be sharded through the second rule set. Therefore, for sharding operations in different scenarios, the inference efficiency can be significantly improved.

[0052] The embodiment of the present application also provides a possible implementation manner. Each sharding rule is configured with a priority level; verifying the sharding rules in the rule pool includes steps Sa1-Sa2 (not shown in the figure).

[0053] Sa1, verify each sharding rule in the rule pool in order from high to low according to the priority level to determine the first matching degree of each sharding rule with the server resources.

[0054] Optionally, the rule pool includes but is not limited to the following sharding rules: data volume sharding rule, inference node efficiency sharding rule, number of threads sharding rule, file size sharding rule, feature protection sharding rule.

[0055] In one example, the priority levels can be set in the following ways, including but not limited to: setting the priority levels from high to low as the first level, the second level, and the third level. For example, setting the priority levels of the data volume sharding rule and the file size sharding rule as the first level, setting the priority levels of the inference node efficiency sharding rule and the number of threads sharding rule as the second level, and setting the feature protection sharding rule as the third level. Specifically, based on this way of setting the priority levels, the verification process can include: verifying the sharding rules of the corresponding levels in sequence from the first level, the second level, and the third level.

[0056] In one example, a high priority level can be used to set sharding rules with a higher usage frequency. The sharding rules of this level can meet the requirements of most inference models for sharded files, such as the data volume sharding rule; a low priority level can be used to set sharding rules with a lower usage frequency, such as the feature protection sharding rule.

[0057] In one example, each sharding rule can be understood as follows. Data volume sharding rule: Shard the data file to be processed according to the second threshold. For example, if the total number of entries in the data file to be processed is 1 million and the second threshold is 300,000, then 4 sharded files can be obtained, and the data volume of each sharded file is: 300,000, 300,000, 300,000, 100,000; where the second threshold can be input by the user or be a default value. File size sharding rule: Shard the data file to be processed according to the first threshold. For example, if the memory occupied by the data file to be processed is 8G and the first threshold is 2G, then 4 sharded files can be obtained, and the size of each sharded file is 2G. Inference node efficiency sharding rule: Shard by referring to the servers with remaining memory greater than zero, and the inference model is configured in these servers. Number of threads sharding rule: Shard according to the total number of threads for processing the sharded files. For example, if the total number of threads is 3, the data file to be processed is divided into 3 sharded files with the same occupied memory; where the total number of threads can be input by the user or be a default value. Feature protection sharding rule, during the sharding process, is generally used as the last sharding rule for verifying the integrity of the data of a single sharded file.

[0058] In one possible implementation, the method further includes:

[0059] In response to an add operation carrying a new sharding rule, add the new sharding rule to the rule pool and set a priority level for the new sharding rule.

[0060] In one possible implementation, the method further includes:

[0061] In response to an update operation for a target sharding rule, update the priority level of the target sharding rule.

[0062] With the progress and development of the inference model, the old sharding rules may not be able to adapt to the changes and become less frequently used or unused. The priority level of the old sharding rules can be reduced; or when none of the existing sharding rules are applicable, new sharding rules can also be extended to enrich the rule pool.

[0063] An embodiment of this application also provides a possible implementation. If the second rule set includes a feature protection sharding rule, the data file to be processed is sharded through the second rule set, which may specifically include:

[0064] The data file to be processed is sharded according to other sharding rules in the second rule set to obtain at least two sharded files; and the data that meets the conditions in each sharded file among the at least two sharded files is adjusted based on the feature protection sharding rule.

[0065] Specifically, when the data file to be processed is sharded through the second rule set, each sharding rule in the second rule set needs to be executed. It should be noted that when using the sharding rule for processing, there is no need to refer to the priority level of the sharding rule.

[0066] Specifically, the following processing is performed on each sharded file among the at least two sharded files: retrieving the target features of each piece of data in each sharded file, sorting each piece of data in each sharded file, matching the first piece of data and the last piece of data in each sharded file, moving the data that meets the conditions in each sharded file, and merging the data that meets the conditions.

[0067] In one example, after obtaining sharded file A, sharded file B, and sharded file C, each sharded file describes the production details data for each quarter in a certain year. Among them, sharded file A: the production details data for the 1st quarter of 2019, the 2nd quarter of 2019, the 3rd quarter of 2019, and the 4th quarter of 2019; sharded file B includes: the production details data for the 1st quarter of 2020, the 2nd quarter of 2020, the 3rd quarter of 2020, the 4th quarter of 2020, and the 1st quarter of 2021; sharded file C: the production details data for the 2nd quarter of 2021, the 3rd quarter of 2021, and the 4th quarter of 2021. After performing the above processing on sharded file A, sharded file B, and sharded file C, it is found that the data for "the 1st quarter of 2021" in sharded file B needs to be adjusted, so the data for "the 1st quarter of 2021" is moved from sharded file B to sharded file C.

[0068] It should be noted that this example is only used to illustrate a usage method of the feature protection sharding rule and cannot be used as a limitation to it. There may be other ways for the feature protection sharding rule, and this application does not limit it.

[0069] The embodiment of the present application also provides a possible implementation manner, which verifies each sharding rule in the rule pool to determine the first matching degree between each sharding rule and the server resources. Specifically, it may include:

[0070] Determine the first resources of each sharding rule. The first resources are the server resources required for the corresponding sharding rule to process the data file to be processed; compare the matching degree between the second resources and each first resource to determine the first matching degree between the corresponding sharding rule and the second resources. The second resources are the server resources currently provided.

[0071] Among them, the second resources may include the remaining memory resources of each provided server. The first resources may include the server memory resources required for each sharding file.

[0072] Specifically, obtain the total number of entries of the data file to be processed and the total occupied memory, and perform the following operations for any sharding rule: Obtain the simulated sharding of the data file to be processed by the sharding rule to obtain the memory occupied by each sharding file. Compare the memory occupied by each sharding file with the remaining memory resources of each server in the second resources in an orderly manner to obtain the first matching degree between the first resources and the second resources.

[0073] Optionally, the orderly comparison process may include starting the comparison from the sharding file with the highest occupied memory in the sharding files in sequence. Multiple metrics may be used to determine the first matching degree during the comparison process, such as the first metric and the second metric. Among them, the first metric includes: whether all sharding files can be matched to a server (for example, the remaining memory provided by the server can satisfy the sharding file). If matched, the value of the first metric is the full score value; if not matched, the value of the first metric is 0. The second metric includes: after determining the server matched to each sharding file, determine the utilization rate of each server, and determine the value of the second metric through all utilization rates.

[0074] By performing the above verification operations on each sharding rule during the verification phase, it can be ensured that the provided server resources can be adapted to process the obtained sharding files.

[0075] Sa2, determine the sharding rules corresponding to the first matching degree greater than the preset threshold as the matching rules.

[0076] Among them, the determined matching rules may be one or more than one.

[0077] The embodiment of the present application also provides a possible implementation manner, which determines the second rule set from multiple experience sets according to at least one matching rule. Specifically, it may include:

[0078] Determine the second matching degree between each set of experiences and at least one matching rule, where the set of experiences includes at least one sharding rule in the rule pool; determine the second rule set according to the set of experiences corresponding to the maximum second matching degree.

[0079] Optionally, filter out multiple inference processes with relatively high inference efficiency from the historical inference process, and determine the set of sharding rules corresponding to the multiple inference processes as the set of experiences. Alternatively, receive the set of sharding rules input by the user, and determine the input set of sharding rules as the set of experiences. Wherein, each set of experiences may include one sharding rule or more than one sharding rule.

[0080] Optionally, the process of determining the second matching degree includes: if there is a single matching rule, determine the second matching degree by calculating the proportion of the matching rule in the set of experiences, and determine the set of experiences with the highest second matching degree as the second rule set. If there are more than one matching rules, filter out the eligible sets of experiences on the condition that the set of experiences includes at least one matching rule, and the second matching degree of other ineligible sets of experiences is zero; then, calculate the second matching degree between each eligible set of experiences and the corresponding matching rule, and perform the following operations for each eligible set of experiences: determine the sharding rules that overlap between the set of experiences and the set composed of all matching rules, and sequentially determine the first proportion of the overlapping sharding rules in all matching rules and the second proportion of the overlapping sharding rules in the set of experiences, and determine the second matching degree according to the first proportion and the second proportion.

[0081] Optionally, if the set of experiences corresponding to the maximum second matching degree includes one set of experiences, determine the single set of experiences as the second rule set; if the set of experiences corresponding to the maximum second matching degree includes more than one set of experiences, filter out the second rule set from the more than one set of experiences according to the priority level of the sharding rules.

[0082] In one example, filtering out the second rule set from more than one set of experiences according to the priority level of the sharding rules may specifically include: counting the first total number of sharding rules belonging to the first level in each set of experiences, and determining the set of experiences with the largest first total number as the second rule set; if there are more than one same first total number when counting the sharding rules of the first level, count the second total number of sharding rules belonging to the second level in each set of experiences, and determine the set of experiences with the largest second total number as the second rule set. And so on until the second rule set is determined.

[0083] An embodiment of the present application also provides a possible implementation manner. If the data file to be processed is configured with a first rule set, the method includes:

[0084] Perform sharding processing on the data file to be processed through the first rule set, and send the obtained sharded file to the server.

[0085] Among them, the first rule set is a rule set adapted to the server, including: the first rule set is adapted to the inference model configured in the server.

[0086] In one example, inference models A, B, and C are configured in the server, and the amount of data processed by each inference model is different. The amount of data in the sharded files processed by inference model A is moderate and the business is simple, and the data volume sharding rule can be specified as the first rule set of inference model A; the amount of data in the sharded files processed by inference model B varies greatly and the business is simple, and the data volume sharding rule and the inference node efficiency sharding rule can be specified; the amount of data in the sharded files processed by inference model C is moderate. Since inference model C is a time-series model and requires data integrity, the data volume sharding rule and the feature protection sharding rule can be specified as the first rule set of inference model C.

[0087] In the actual processing process, the memory sizes of the large data files to be analyzed vary. Some large data files are relatively large, and some large data files are relatively small. There is no need to waste time sharding.

[0088] The embodiment of the present application also provides a possible implementation. If the first rule set is not configured for the data file to be processed, the method further includes:

[0089] If the total memory occupied by the data file to be processed is not greater than the first threshold and the total number of entries in the data file to be processed is not greater than the second threshold, the data file to be processed is sent to a single server.

[0090] The data file processing method shown in the embodiment of the present application can be applied to various sharding scenarios of large data files. To more clearly describe the data file processing method, the embodiment of the present application also provides a flow schematic diagram of an offline large data file processing solution to illustrate the method, as Figure 3 shown. This solution includes steps S1001 - step S1006.

[0091] S1001, the system receives a large data file A.

[0092] S1002, the system determines whether the file A is configured with a customized rule set (corresponding to the first rule set).

[0093] Among them, if the customized rule set is configured, S1005 is executed. If the customized rule set is not configured, S1003 is executed.

[0094] S1003, the system starts the verification module to verify the file A.

[0095] Among them, obtain the data volume and occupied memory of File A. If the data volume of File A is not greater than the data volume threshold (corresponding to the second threshold), and the memory occupied by File A is not greater than the memory threshold (corresponding to the first threshold), then directly execute S1006; otherwise, execute S1004.

[0096] S1004, the system checks all the sharding rules in the File A verification rule pool and filters out the sharding rule set (corresponding to the second rule set) that adapts to File A.

[0097] Among them, the rule pool includes the following sharding rules: data volume rule, inference node efficiency sharding rule, number of threads sharding rule, file size sharding rule, and feature protection sharding rule. Additionally, new sharding rules can be added to the rule pool.

[0098] Among them, each sharding rule in the rule pool has a set priority level, and referring to each sharding rule's priority level is included in the process of determining the sharding rule set.

[0099] S1005, the system performs sharding processing on File A according to the customized rule set or the sharding rule set.

[0100] Specifically, after the sharding processing, at least one sharded file is obtained. For example, Sharded File 1, Sharded File 2, Sharded File 3, and Sharded File 4.

[0101] S1006, the system sends Sharded File 1, Sharded File 2, Sharded File 3, and Sharded File 4 to the server cluster for processing.

[0102] Among them, the server cluster includes multiple servers, each server is an inference node, and each inference node is configured with an inference model. After the server cluster receives the above sharded files, they are assigned to the corresponding server nodes for processing.

[0103] See Figure 4 , an embodiment of the present application provides a processing device for data files. The device 400 may include: a verification module 410 and a sharding module 420.

[0104] The verification module 410 is used to, if the first rule set is not configured for the data file to be processed, verify the sharding rules in the rule pool and determine the second rule set from multiple experience sets according to at least one matching rule obtained from the verification.

[0105] The sharding module 420 is used to perform sharding processing on the data file to be processed through the second rule set and send the obtained sharded files to the server. The first rule set is a rule set adapted to the server.

[0106] In a possible implementation, each sharding rule is configured with a priority level; when the verification module 410 verifies the sharding rules in the rule pool and obtains at least one matching rule, it is specifically used for:

[0107] Verify each sharding rule in the rule pool in order from the highest priority level to the lowest priority level to determine the first matching degree between each sharding rule and the server resources; determine the sharding rules corresponding to the first matching degrees greater than the preset threshold as the matching rules.

[0108] In a possible implementation, when the verification module 410 determines the second rule set from multiple experience sets according to at least one matching rule, it is specifically used for:

[0109] Determine the second matching degree between each experience set and at least one matching rule, where the experience set includes at least one sharding rule in the rule pool; determine the second rule set according to the experience set corresponding to the maximum second matching degree.

[0110] In a possible implementation, when the verification module 410 verifies each sharding rule in the rule pool and determines the first matching degree between each sharding rule and the server resources, it is specifically used for:

[0111] Determine the first resources of each sharding rule, where the first resources are the server resources required for the corresponding sharding rule to process the data file to be processed; compare the matching degree between the second resources and each first resource to determine the first matching degree between the corresponding sharding rule and the second resources, and the second resources are the currently provided server resources.

[0112] In a possible implementation, if the second rule set includes a feature protection sharding rule, when the sharding module 420 performs sharding processing on the data file to be processed through the second rule set to obtain sharded files, it is specifically used for:

[0113] Perform sharding on the data file to be processed according to other sharding rules in the second rule set to obtain at least two sharded files;

[0114] Adjust the data that meets the conditions in each of the at least two sharded files.

[0115] In a possible implementation, if the data file to be processed is configured with a first rule set, the sharding module 420 is further used for:

[0116] Process the data file to be processed through the first rule set and send the obtained sharded files to the server.

[0117] Wherein, the first rule set is a rule set adapted to the server, including: the first rule set is adapted to the inference model configured in the server.

[0118] In a possible implementation, if the first rule set is not configured for the data file to be processed, the verification module 410 can also be used to:

[0119] If the total memory occupied by the data file to be processed is not greater than the first threshold, and the total number of entries in the data file to be processed is not greater than the second threshold, send the data file to be processed to a single server.

[0120] The device according to the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principle is similar. The actions performed by each module in the device according to the embodiments of the present application correspond to the steps in the method according to the embodiments of the present application. For the detailed function descriptions of the modules of the device, reference can be specifically made to the descriptions in the corresponding methods shown above, and details are not described herein again.

[0121] An electronic device is provided in the embodiments of the present application, including a memory, a processor, and a computer program stored on the memory. The processor executes the above computer program to implement the steps of a method for processing a data file. Compared with the related art, it can achieve:

[0122] In an optional embodiment, an electronic device is provided, as Figure 5 shown, Figure 5 The electronic device 5000 shown includes: a processor 5001 and a memory 5003. Among them, the processor 5001 and the memory 5003 are connected, such as through a bus 5002. Optionally, the electronic device 5000 may further include a transceiver 5004, and the transceiver 5004 can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data, etc. It should be noted that in practical applications, the transceiver 5004 is not limited to one, and the structure of the electronic device 5000 does not constitute a limitation to the embodiments of the present application.

[0123] The processor 5001 may be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of the present application. The processor 5001 may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0124] The bus 5002 may include a path for transmitting information between the above components. The bus 5002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 5002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 5 it is only represented by a thick line in Figure 5 , but it does not mean that there is only one bus or one type of bus.

[0125] The memory 5003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, which is not limited herein.

[0126] The memory 5003 is used to store the computer program for implementing the embodiments of the present application and is controlled by the processor 5001 to execute. The processor 5001 is used to execute the computer program stored in the memory 5003 to implement the steps shown in the foregoing method embodiments.

[0127] Among them, the electronic device includes but is not limited to: computer devices.

[0128] The embodiments of the present application provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0129] The embodiments of the present application also provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0130] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the description, claims and the above-mentioned drawings of the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in words.

[0131] It should be understood that although the flowcharts of the embodiments of the present application indicate the respective operation steps by arrows, the execution order of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated in this document, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart can be executed in other orders according to requirements. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage among these sub-steps or stages can also be executed at different times respectively. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and the embodiments of the present application do not limit this.

[0132] The above are only optional implementation manners of some implementation scenarios of the present application. It should be noted that for those of ordinary skill in the art, without departing from the technical concept of the solution of the present application, using other similar implementation means based on the technical idea of the present application also belongs to the protection scope of the embodiments of the present application.

Claims

1. A method for processing data files, characterized in that, the method includes: If the first rule set is not configured for the data file to be processed, verify the sharding rules in the rule pool, and determine the second rule set from multiple experience sets according to at least one matching rule obtained by the verification; Perform sharding processing on the data file to be processed through the second rule set, and send the obtained sharded files to the server, where the first rule set is a rule set adapted to the server; wherein, each sharding rule is configured with a priority level; verifying the sharding rules in the rule pool includes: Verify each sharding rule in the rule pool in order from high to low priority to determine the first matching degree of each sharding rule with the server resources; Determine the sharding rule corresponding to the first matching degree greater than the preset threshold as the matching rule.

2. The method according to claim 1, characterized in that, determining the second rule set from multiple experience sets according to at least one matching rule obtained by the verification includes: Determine the second matching degree of each experience set with the at least one matching rule, where the experience set includes at least one sharding rule in the rule pool; Determine the second rule set according to the experience set corresponding to the largest second matching degree.

3. The method according to claim 1 or 2, characterized in that, verifying each sharding rule in the rule pool to determine the first matching degree of each sharding rule with the server resources includes: Determine the first resource of each sharding rule, where the first resource is the server resource required for the corresponding sharding rule to process the data file to be processed; Compare the matching degree of the second resource with each first resource to determine the first matching degree of the corresponding sharding rule with the second resource, where the second resource is the currently provided server resource.

4. The method according to claim 1, characterized in that, If the second rule set includes a feature protection sharding rule; performing sharding processing on the data file to be processed through the second rule set includes: Perform sharding on the data file to be processed according to other sharding rules in the second rule set to obtain at least two sharded files; Adjust the data that meets the conditions in each of the at least two sharded files.

5. The method according to claim 1, characterized in that, If the first rule set is configured for the data file to be processed, the method includes: Perform sharding processing on the data file to be processed through the first rule set, and send the obtained sharded files to the server; wherein, the first rule set is a rule set adapted to the server, including: The first rule set is adapted to the inference model configured in the server.

6. The method according to claim 1, characterized in that, If the first rule set is not configured for the data file to be processed, the method further includes: If the total memory occupied by the data file to be processed is not greater than the first threshold, and the total number of entries of the data file to be processed is not greater than the second threshold, send the data file to be processed to a single server.

7. A data file processing device, characterized in that, The device includes: A verification module, configured to verify the sharding rules in the rule pool if the data file to be processed is not configured with a first rule set, and determine a second rule set from multiple experience sets according to at least one matching rule obtained by the verification; A sharding module, configured to perform sharding processing on the data file to be processed through the second rule set, and send the obtained sharded files to a server, where the first rule set is a rule set adapted to the server; Wherein, the verification module is specifically configured to: Verify each sharding rule in the rule pool in order from highest to lowest priority to determine a first matching degree between each sharding rule and server resources; Determine the sharding rule corresponding to the first matching degree greater than a preset threshold as the matching rule.

8. An electronic device, including a memory, a processor, and a computer program stored on the memory, Characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, on which a computer program is stored, Characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1-6 are implemented.

Images