Data normalization processing method and device

Abstract

The application discloses a data normalization processing method and device, and applies to a server of a data normalization processing system, and comprises the following steps: determining a starting identifier chain of a processing flow of each sub-data in at least one sub-data of original data; acquiring a waiting time of each starting identifier chain, a standard execution time of each starting identifier, and a data processing amount of each starting identifier sub-chain; updating each execution component sequence corresponding to each starting identifier chain according to the waiting time, the standard execution time and the data processing amount, so that the execution time length of the total processing flow is less than the execution time length before the updating; and processing the original data according to the updated each execution component sequence until each execution component sequence is executed completely, so that target data is obtained. According to the application, the execution order of the execution component is adjusted according to the waiting time of the starting identifier chain, the standard execution time of the starting identifier and the data processing amount of the starting identifier sub-chain, the memory pressure is reduced, and the processing time length is shortened.

Description

Technical Field

[0001] This application relates to the field of big data technology, specifically to data normalization processing methods and apparatus. Background Technology

[0002] When acquiring data from multiple data sources, the formats and naming of data items often differ. For example, when banks acquire tax data from various tax bureaus, the number and naming of data items vary from bureau to bureau. This makes it difficult for banks to extract the required information from the raw data provided by multiple tax bureaus, or the extracted information may contain numerous errors. Therefore, data from multiple data sources needs to be standardized. Current technical solutions require creating numerous data tables in the database or customizing different processing subsystems for different data sources, resulting in high code duplication, excessive memory consumption, and long processing times, failing to meet usage requirements. Therefore, how to reduce memory pressure and shorten the processing time during data standardization becomes a further technical problem that needs to be solved. Summary of the Invention

[0003] This application proposes a data normalization processing method and apparatus to solve the problems of excessive memory resource consumption and long processing time, reduce memory pressure, and shorten the data normalization processing time.

[0004] In a first aspect, embodiments of this application provide a data normalization processing method, applied to a server of a data normalization processing system, the method comprising:

[0005] A chain of start identifiers is defined to determine the processing flow of each sub-data in at least one sub-data of the original data. Multiple start identifiers in each chain of start identifiers correspond one-to-one with multiple execution components in the execution component sequence. Multiple sub-chains of start identifiers in the chain of start identifiers correspond one-to-one with multiple subsets of execution components in the execution component sequence. The overall processing flow of the original data includes the processing flow of each sub-data in the at least one sub-data.

[0006] The waiting time of each startup identifier chain, the standard execution time of each startup identifier in the plurality of startup identifiers, and the data processing volume of each startup identifier subchain in each startup identifier chain are obtained, and the startup identifier in each startup identifier subchain corresponds one-to-one with the execution component in the corresponding execution component subset;

[0007] Based on the waiting time, the standard execution time, and the data processing volume, update each execution component sequence corresponding to each startup identifier chain so that the total execution time of the processing flow is less than the execution time before the update;

[0008] The original data is processed according to the updated sequence of each execution component until each sequence of execution components is completed, and the target data is obtained.

[0009] Secondly, embodiments of this application provide a data normalization processing apparatus, applied to a server of a data normalization processing system, the apparatus comprising:

[0010] The first processing unit is used to determine the start identifier chain of the processing flow of each sub-data in at least one sub-data of the original data. Multiple start identifiers in each start identifier chain correspond one-to-one with multiple execution components in the execution component sequence. Multiple start identifier sub-chains in the start identifier chain correspond one-to-one with multiple subsets of execution components in the execution component sequence. The overall processing flow of the original data includes the processing flow of each sub-data in the at least one sub-data.

[0011] The first receiving unit is used to obtain the waiting time of each startup identifier chain, the standard execution time of each startup identifier among the plurality of startup identifiers, and the data processing volume of each startup identifier sub-chain in each startup identifier chain, wherein each startup identifier in each startup identifier sub-chain corresponds one-to-one with the execution components in the corresponding execution component subset;

[0012] The second processing unit is configured to update each execution component sequence corresponding to each start identifier chain according to the waiting time, the standard execution time, and the data processing volume so that the total execution time of the processing flow is less than the execution time before the update; and to process the original data according to the updated execution component sequence until each execution component sequence is completed to obtain the target data.

[0013] Thirdly, embodiments of this application provide a server including a processor, a memory, and one or more programs stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps of the method as described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program / instructions stored thereon, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the method described in any one of the first aspects.

[0015] As can be seen, in this application, the server first determines the start identifier chain of the processing flow for each sub-data in at least one sub-data of the original data. Multiple start identifiers in each start identifier chain correspond one-to-one with multiple execution components in the execution component sequence, and multiple sub-chains of start identifiers in the start identifier chain correspond one-to-one with multiple subsets of execution components in the execution component sequence. The total processing flow of the original data includes the processing flow for each sub-data in at least one sub-data. Next, the server obtains the waiting time for each start identifier chain, the standard execution time for each start identifier among the multiple start identifiers, and the data processing volume for each start identifier sub-chain in each start identifier chain. Each start identifier in each sub-chain of start identifiers corresponds one-to-one with the execution components in the corresponding subset of execution components. Then, based on the waiting time, the standard execution time, and the data processing volume, the server updates each execution component sequence corresponding to each start identifier chain so that the total processing flow execution time is less than the execution time before the update. Finally, the server processes the original data according to each updated execution component sequence until each execution component sequence is completed, obtaining the target data. Since the execution components are invoked by the start identifier, when new data processing scenarios are needed, the execution components can be extended and the start identifier chain can be adjusted and configured. There is no need to create a large number of data tables and processing subsystems, which reduces memory pressure. Furthermore, adjusting the execution order of the execution components shortens the time for data normalization processing. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of a data normalization processing system provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram of the server structure in a data normalization processing system provided in an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of the server structure in another data normalization processing system provided in this application embodiment;

[0020] Figure 4 This is a flowchart illustrating a data normalization processing method provided in an embodiment of this application;

[0021] Figure 5This is a schematic diagram of the structure of a startup identifier chain provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of another startup identifier chain structure provided in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram illustrating a data normalization processing method provided in an embodiment of this application.

[0024] Figure 8 This is a schematic diagram illustrating another data normalization processing method provided in an embodiment of this application.

[0025] Figure 9 This is a timing diagram of a data normalization processing method provided in an embodiment of this application;

[0026] Figure 10a This is a functional unit block diagram of a data normalization processing device provided in an embodiment of this application;

[0027] Figure 10b This is a functional unit block diagram of another data normalization processing device provided in the embodiments of this application;

[0028] Figure 11 This is a structural block diagram of a server provided in an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0030] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] In the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone; A and B exist simultaneously; B exists alone. Among them, A and B can be singular or plural.

[0033] In this embodiment, the symbol " / " can indicate that the preceding and following objects are in an "or" relationship. Alternatively, the symbol " / " can also represent a division sign, i.e., performing a division operation. For example, A / B can mean A divided by B.

[0034] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0035] In the embodiments of this application, "equal to" can be used with "greater than" and is applicable to technical solutions used when "greater than" is used; it can also be used with "less than" and is applicable to technical solutions used when "less than" is used. When "equal to" is used with "greater than", it is not used with "less than"; when "equal to" is used with "less than", it is not used with "greater than".

[0036] To better understand the solutions of the embodiments of this application, the terminal devices, related concepts and background that may be involved in the embodiments of this application will be introduced below.

[0037] (1) Data standardization: Extract, transform, sort, and filter raw data from different data sources with different carriers, inconsistent formats, and different business fields to form standardized, structured data with unified semantics and format.

[0038] (2) Startup Identifier: A unique identifier keyword bound to the execution component. Each startup identifier corresponds one-to-one with an execution component. Upon receiving the startup identifier, the executor in the server can trigger the associated execution component to perform related operations. Common execution components include field-mapping, sorting, grouping, and filtering. The identifier keyword in parentheses in the examples above is the startup identifier.

[0039] (3) Startup identifier chain: refers to a chain structure that organizes a group of startup identifiers in a standard format. Usually, the startup identifiers contained in the startup identifier chain are the key values ​​contained in the nodes of the chain structure.

[0040] (4) Startup Identifier Parameter: Together with the startup identifier, it forms the key and value in the chain node, where the key is the startup identifier and the value is the startup identifier parameter. For example, if-Null is an identifier that adds a default value when the field is empty. Its startup identifier and startup identifier parameter are "if-Null": {"nsrlx": "small-scale taxpayer"}, and the value of if-Null, {"nsrlx": "small-scale taxpayer"}, is the startup identifier parameter.

[0041] (5) Execution Component: A reusable program capable of processing data. Execution components are unique, atomic, and correspond one-to-one with the startup identifier. The execution component receives calls from the executor in the server and, based on the startup identifier and its parameters in the startup identifier chain, performs the corresponding specific operation on the input data. Examples include string clipping, merging multiple data lists (add-stream) / joining, etc.

[0042] (6) MLCATBP algorithm: Multi-Level Capacity And Time Balanced Priority, a multi-level capacity and time balanced priority algorithm used to evaluate the dynamic execution priority of basic execution components and to perform unified scheduling and execution of execution components.

[0043] Current technical solutions require creating numerous data tables in the database to store data or customizing different processing subsystems based on different data sources when performing data standardization. This results in high code duplication, excessive memory consumption, and excessively long processing times, failing to meet usage requirements.

[0044] To address the aforementioned issues, embodiments of this application provide a data normalization processing method and apparatus. This method is applied to a server in a data normalization processing system. The server can invoke execution components via a startup identifier. When new data processing scenarios arise, the execution components can be expanded, and the startup identifier chain adjusted and configured. This eliminates the need to create numerous data tables and processing subsystems, reducing memory pressure. Furthermore, adjusting the execution order of the execution components shortens the data normalization processing time.

[0045] Please see Figure 1 , Figure 1 This is a schematic diagram of the data normalization processing system provided in an embodiment of this application. For example... Figure 1 As shown, the data normalization processing system 100 includes a terminal device 110 and a server 120. The terminal device 110 is communicatively connected to the server 120. The terminal device can be a mobile phone, tablet computer, or laptop computer, etc., and the server 120 can be a single server, a server cluster consisting of several servers, or a cloud computing service center, etc.

[0046] In the current data normalization process, server 120 needs to normalize data from multiple data sources. Current technical solutions require creating numerous tables in the database to store data or customizing different processing subsystems for different data sources. This results in high code duplication, excessive memory consumption, and excessively long processing times, failing to meet usage requirements.

[0047] In the daily use of the data normalization processing system 100, the server 120 determines the start identifier chain for the processing flow of each sub-data in at least one sub-data of the original data. Multiple start identifiers in each start identifier chain correspond one-to-one with multiple execution components in the execution component sequence. Multiple sub-chains of start identifiers in the start identifier chain correspond one-to-one with multiple subsets of execution components in the execution component sequence. The overall processing flow of the original data includes the waiting time for each start identifier chain to be obtained, the standard execution time for each start identifier among the multiple start identifiers, and the data processing volume for each sub-chain of start identifiers in each start identifier chain. Each start identifier in a sub-chain of start identifiers corresponds one-to-one with the execution components in the corresponding subset of execution components. Based on the waiting time, standard execution time, and data processing volume, each execution component sequence corresponding to each start identifier chain is updated so that the execution time of the overall processing flow is less than the execution time before the update. The original data is processed according to each updated execution component sequence until each execution component sequence is completed, resulting in the target data.

[0048] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of server 1 in the data normalization processing system provided in this application embodiment. For example... Figure 2 As shown, server 200 includes processor 210 and memory 220, with processor 210 communicatively connected to memory 220. Memory 220 stores one or more programs, which are configured to be executed by processor 210. The function of the one or more programs is to determine the start identifier chain for the processing flow of each sub-data in at least one sub-data of the original data. Multiple start identifiers in each start identifier chain correspond one-to-one with multiple execution components in the execution component sequence. Multiple sub-chains of start identifiers in each start identifier chain correspond one-to-one with multiple subsets of execution components in the execution component sequence. The overall processing flow of the original data includes the waiting time for each start identifier chain to be obtained, the standard execution time for each start identifier among the multiple start identifiers, and the data processing volume for each sub-chain of start identifiers. Each start identifier in each sub-chain of start identifiers corresponds one-to-one with the execution components in the corresponding subset of execution components. Based on the waiting time, standard execution time, and data processing volume, each execution component sequence corresponding to each start identifier chain is updated so that the execution time of the overall processing flow is less than the execution time before the update. The original data is processed according to each updated execution component sequence until each execution component sequence is completed, resulting in the target data.

[0049] Please see Figure 3 , Figure 3 This is a schematic diagram of the server structure in another data normalization processing system provided in this application embodiment. For example... Figure 3 As shown, server 300 includes configuration cache 310, execution engine 320, verifier 330, scheduler 340, parser 350, executor 360, database 370 and intermediate data cache pool 380; configuration cache 310, verifier 330, scheduler 340, parser 350 and database 370 are respectively connected to execution engine 320, scheduler 340 and intermediate data cache pool 380 are respectively connected to executor 360.

[0050] The configuration cache 310 is used to cache the startup identifier chain that has passed verification.

[0051] The execution engine 320 is used to load and parse the raw data to form a unified data stream, verify the startup identifier chain and write it into the configuration cache 310, and pass the startup identifier chain and data stream to the scheduler 340.

[0052] The verifier 330 is used to verify whether the input data of the startup identifier chain and the execution component conforms to the specifications.

[0053] The scheduler 340 is used to calculate the execution priority of the start identifier chain and its secondary nodes based on the MLCATBP algorithm, and send the relevant start identifier to the executor 360 to perform data normalization processing according to the priority.

[0054] The parser 350 is used to parse raw data in formats such as XML / JSON / PDF / EXCEL and convert it into a data stream with the same data structure.

[0055] The executor 360 is used to encapsulate the startup identifier and startup identifier parameters, and to call the execution components of the relevant category.

[0056] The database 370 is used for data normalization processing of streaming data, starting identifier chains, processing flow configuration files, and data after data normalization processing.

[0057] The intermediate data cache pool 380 is used to cache intermediate data generated during the processing of the startup identifier or startup identifier subchain, so that the same startup identifier or startup identifier subchain can be reused in the future.

[0058] The data normalization processing method provided in the embodiments of this application is described below.

[0059] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data normalization processing method provided in an embodiment of this application, applicable to, for example... Figure 1 The data normalization processing system 100 shown includes a server 120, which comprises a terminal device 110 and a server 120. The terminal device 110 is communicatively connected to the server 120. As shown in the figure, the method includes the following steps:

[0060] Step S401: Determine the start identifier chain of the processing flow of each sub-data in at least one sub-data of the original data. Multiple start identifiers in each start identifier chain correspond one-to-one with multiple execution components in the execution component sequence. Multiple start identifier sub-chains in the start identifier chain correspond one-to-one with multiple subsets of execution components in the execution component sequence. The overall processing flow of the original data includes the processing flow of each sub-data in the at least one sub-data.

[0061] The execution component sequence is a sequence obtained by sorting the multiple execution components under the joint constraints of a first timing condition and a second timing condition. The first timing condition refers to the first timing constraint relationship of the multiple execution component subsets, and the multiple execution component subsets correspond one-to-one with the multiple data processing flow nodes included in the processing flow. The first timing constraint relationship is used to characterize the execution order of the multiple data processing flow nodes. The second timing condition refers to the second timing constraint relationship of at least two data processing operations in the target data processing flow node corresponding to the target execution component subset, and the execution components in the target execution component subset correspond one-to-one with the data processing operations in the target data processing flow node. The second timing constraint relationship is used to characterize the execution order of the at least two data processing operations.

[0062] The original data can be a collection of data from different data sources, or a collection of data in different formats or versions from the same data source.

[0063] The step of determining the start identifier chain of the processing flow for each sub-data in at least one sub-data of the original data includes: the execution engine determining the start identifier chain of the processing flow for the original data.

[0064] The step of determining the start identifier chain for the processing flow of each sub-data in at least one sub-data of the original data includes: converting the original data into a data stream of a preset data structure, wherein the preset type includes List<Map<k, v>> type.

[0065] The step of determining the start identifier chain for the processing flow of each sub-data in at least one sub-data of the original data includes: the parser converts the original data into a data stream of a preset data structure, wherein the preset type includes List<Map<k, v>> type.

[0066] The step of determining the start identifier chain for the processing flow of each sub-data in at least one sub-data of the original data includes: the execution engine sending the start identifier chain and the data stream to the scheduler, and the scheduler receiving the start identifier chain and the data stream.

[0067] The startup identifier chain is stored in the configuration cache or database of the server in the form of a processing flow configuration file. The processing flow configuration file includes the number of the processing flow, the data source number corresponding to the processing flow, the name of the processing flow, the identifier (id) of the startup identifier chain, the file name of the original data, the data normalization processing serial number of the original data, the file name of the target data, and the startup identifier chain.

[0068] The step of determining the start identifier chain of the processing flow for each sub-data in at least one sub-data of the original data includes: the execution engine determining the start identifier chain of the processing flow for the original data.

[0069] Please refer to Figure 5 , Figure 5 This is a schematic diagram of a startup identifier chain provided in an embodiment of this application. For example... Figure 5 As shown, the startup identifier chain includes multiple startup identifier sub-chains. Each startup identifier sub-chain includes at least one startup identifier and a startup identifier parameter corresponding to the at least one startup identifier. Multiple startup identifier sub-chains are called to process the corresponding raw data rawDateListA, rawDateListB, rawDateListAN, and dateListMap to obtain the processed data.

[0070] Please refer to Figure 6 , Figure 6 This is a schematic diagram of another startup identifier chain structure provided in an embodiment of this application. For example... Figure 6 As shown, the startup identifier chain includes multiple startup identifier sub-chains.

[0071] In one possible embodiment, determining the start identifier chain for the processing flow of each sub-data in at least one sub-data of the original data includes: obtaining the data source number of the original data; determining guidance information for the start identifier chain based on the data source number; and determining the start identifier chain based on the guidance information.

[0072] The step of determining the start identifier chain for the processing flow of each sub-data in at least one sub-data of the original data includes: the execution engine obtaining the data source number of the original data; the execution engine determining the guidance information of the start identifier chain based on the data source number; and the execution engine determining the start identifier chain based on the guidance information.

[0073] The guidance information includes at least one of the following: the number of the processing flow, the number of the processing flow, the data source number corresponding to the processing flow, the name of the processing flow, the identifier (id) of the start identifier chain, the file name of the original data, the data normalization processing serial number of the original data, and the file name of the target data.

[0074] The execution engine's determination of the startup identifier chain based on the guidance information includes: determining whether a processing flow configuration file containing the guidance information is stored in the configuration cache; if yes, determining the startup identifier chain in the processing flow configuration file in the configuration cache as the startup identifier chain; if no, determining whether the processing flow configuration file is stored in the database; if yes, determining the startup identifier chain in the processing flow configuration file in the database as the startup identifier chain; if no, stopping data normalization processing and displaying an exception.

[0075] As can be seen in this example, determining the start identifier chain through guidance information can quickly and easily identify the start identifier chain for data normalization, thereby shortening the time required for data normalization.

[0076] In one possible embodiment, determining the startup identifier chain based on the guidance information includes: determining whether the startup identifier chain is correct; if not, stopping data normalization processing and displaying an error; if yes, storing the startup identifier chain in a configuration cache, the configuration cache being used to non-permanently store the startup identifier chain for reading and rereading.

[0077] The step of determining the startup identifier chain based on the guidance information includes: a verifier determining whether the startup identifier chain is correct; if not, stopping the data normalization process and displaying an error; if yes, storing the startup identifier chain in a configuration cache, wherein the configuration cache is used to non-permanently store the startup identifier chain for reading and rereading.

[0078] Before the verifier determines whether the startup identifier chain is correct, the process further includes: the execution engine sending the startup identifier chain to the verifier.

[0079] As can be seen, in this example, validating the startup identifier can ensure the accuracy of component calls, reduce call errors, and thus shorten the time for data normalization processing.

[0080] In one possible embodiment, after determining whether the startup identifier chain is correct, the method further includes: determining whether the startup identifier chain exists in the configuration cache; if not, obtaining the startup identifier chain from the database and storing the startup identifier chain in the configuration cache, wherein the database is used to permanently store the startup identifier chain for reading and rereading; if yes, obtaining the startup identifier chain from the configuration cache.

[0081] The method further includes, after determining whether the startup identifier chain is correct, the execution engine determines whether the startup identifier chain exists in the configuration cache; if not, it retrieves the startup identifier chain from the database and stores the startup identifier chain in the configuration cache, wherein the database is used to permanently store the startup identifier chain for reading and re-reading; if yes, it retrieves the startup identifier chain from the configuration cache.

[0082] If the startup identifier chain does not exist in the database, the data normalization process will stop and an error will be displayed.

[0083] As can be seen in this example, the startup identifier chain is retrieved from the configuration cache first, thereby shortening the time of data normalization processing; if the startup identifier chain does not exist in the configuration cache, the startup identifier chain is retrieved from the database and stored in the configuration cache, so that the database does not need to be accessed again when the startup identifier chain is called next time, thereby shortening the time of data normalization processing.

[0084] Step S402: Obtain the waiting time of each startup identifier chain, the standard execution time of each startup identifier among the plurality of startup identifiers, and the data processing volume of each startup identifier sub-chain in each startup identifier chain, wherein each startup identifier in each startup identifier sub-chain corresponds one-to-one with the execution components in the corresponding execution component subset.

[0085] The step of obtaining the waiting time of each start identifier chain, the standard execution time of each start identifier among the plurality of start identifiers, and the data processing volume of each start identifier sub-chain in each start identifier chain, wherein each start identifier in each start identifier sub-chain corresponds one-to-one with the execution components in the corresponding execution component subset, includes: the scheduler obtaining the waiting time of each start identifier chain, the standard execution time of each start identifier among the plurality of start identifiers, and the data processing volume of each start identifier sub-chain in each start identifier chain, wherein each start identifier in each start identifier sub-chain corresponds one-to-one with the execution components in the corresponding execution component subset.

[0086] The waiting time of the startup identifier chain refers to the time that the entire sequence of execution components corresponding to the startup identifier chain waits in a queue to be executed.

[0087] The waiting time of the startup identifier chain refers to the time that the entire sequence of execution components corresponding to the startup identifier chain waits in the executor to be executed.

[0088] The standard execution time of the startup identifier refers to the execution time of the execution component corresponding to the startup identifier when processing standard data, and the standard data is data with a unit data volume.

[0089] The data processing volume refers to the amount of data actually processed by the subset of execution components corresponding to the start identifier subchain.

[0090] Step S403: Based on the waiting time, the standard execution time, and the data processing volume, update each execution component sequence corresponding to each start identifier chain so that the total execution time of the processing flow is less than the execution time before the update.

[0091] The step of updating each execution component sequence corresponding to each start identifier chain according to the waiting time, the standard execution time, and the data processing volume so that the total execution time of the processing flow is less than the execution time before the update includes: the scheduler updating each execution component sequence corresponding to each start identifier chain according to the waiting time, the standard execution time, and the data processing volume so that the total execution time of the processing flow is less than the execution time before the update.

[0092] In the execution component sequence, the plurality of execution component subsets are arranged according to the first timing constraint relationship, and at least two execution components in each of the plurality of execution component subsets are arranged according to the second timing constraint relationship.

[0093] The step of updating each execution component sequence corresponding to each startup identifier chain based on the waiting time, the standard execution time, and the data processing volume so that the total execution time of the processing flow is less than the execution time before the update includes: the execution time of some of the startup identifier chains corresponding to each execution component sequence after the update is less than the execution time before the update, the execution time of some of the startup identifier chains corresponding to each execution component sequence remains unchanged, and the execution time of some of the startup identifier chains corresponding to each execution component sequence is greater than the execution time before the update, and the total execution time of the process is less than the execution time before the update; the updated part of the startup The execution time of each execution component sequence corresponding to the identifier chain is less than the execution time before the update; the execution time of each execution component sequence corresponding to the remaining startup identifier chain is greater than the execution time before the update, and the total execution time is less than the execution time before the update; the execution time of each execution component sequence corresponding to the updated startup identifier chain is less than the execution time before the update; the execution time of each execution component sequence corresponding to the remaining startup identifier chain remains unchanged, and the total execution time is less than the execution time before the update; the execution time of each execution component sequence corresponding to all updated startup identifier chains is less than the execution time before the update.

[0094] In one possible embodiment, the at least one may be multiple, and the step of updating each execution component sequence corresponding to each start identifier chain according to the waiting time, the standard execution time, and the data processing volume so that the total processing time is less than the execution time before the update includes: determining a first execution priority for each start identifier chain and a second execution priority for each start identifier sub-chain in each start identifier chain, wherein each start identifier sub-chain corresponds one-to-one with a subset of execution components, the first execution priority is used to indicate the probability that the start identifier chain is executed first, and the second execution priority is used to indicate the probability that the start identifier sub-chain is executed first; and updating each execution component sequence corresponding to each start identifier chain according to the first execution priority and the second execution priority so that the total processing time is less than the execution time before the update.

[0095] Wherein, at least one can be multiple, and the step of updating each execution component sequence corresponding to each start identifier chain according to the waiting time, the standard execution time, and the data processing volume so that the total processing time is less than the execution time before the update includes: the scheduler determining a first execution priority for each start identifier chain and a second execution priority for each start identifier sub-chain in each start identifier chain, wherein each start identifier sub-chain corresponds one-to-one with the subset of execution components, the first execution priority is used to indicate the probability that the start identifier chain is executed first, and the second execution priority is used to indicate the probability that the start identifier sub-chain is executed first; and updating each execution component sequence corresponding to each start identifier chain according to the first execution priority and the second execution priority so that the total processing time is less than the execution time before the update.

[0096] The method for determining the first execution priority and the second execution priority includes: calculating the first execution priority and the second execution priority using the MLCATBP algorithm.

[0097] The step of using the MLCATBP algorithm to calculate the first execution priority and the second execution priority includes: obtaining the priority setting and multi-level setting of the MLCATBP algorithm. The priority setting is used to indicate the influence relationship between the multi-level setting and the execution priority. The priority setting includes a priority that is directly proportional to the waiting time, inversely proportional to the standard execution time, and inversely proportional to the data processing volume. The multi-level setting includes the fourth and fifth levels of the start identifier subchain level, as well as the first, second, and third levels of the start identifier chain level.

[0098] The priority setting and multi-level setting can be changed according to the data type being processed.

[0099] The step of updating each execution component sequence corresponding to each startup identifier chain according to the first execution priority and the second execution priority so that the execution time of the total processing flow is less than the execution time before the update includes: sorting each startup identifier chain according to the size of the first execution priority, and executing the startup identifier chain with the larger first execution priority first; in each startup identifier chain, sorting each startup identifier chain subset according to the size of the second execution priority, and executing the startup identifier sub-chain with the larger second priority first.

[0100] As can be seen, in this example, the MLCATBP algorithm is used to bind a priority to the starter identifier chain and a priority to each starter identifier sub-chain in the starter identifier chain. This comprehensively considers the waiting time for data normalization processing, the standard execution time, and the amount of data to be processed. The execution plan can be flexibly adjusted to improve the performance of data normalization processing and shorten the processing time.

[0101] In one possible embodiment, determining the first execution priority of each startup identifier chain and the second execution priority of each startup identifier sub-chain within each startup identifier chain, wherein each startup identifier sub-chain corresponds one-to-one with a subset of execution components, the first execution priority indicating the probability that the startup identifier chain is executed first, and the second execution priority indicating the probability that the startup identifier sub-chain is executed first, includes: determining a first-level priority of each startup identifier chain based on the standard execution time, the first-level priority indicating the probability that the startup identifier chain is executed first at the standard execution time level; determining a second-level priority of each startup identifier chain based on the data processing volume, the second-level priority indicating the probability that the startup identifier chain is executed first at the data processing volume level; determining a third-level priority of each startup identifier chain based on the waiting time, the third-level priority indicating the probability that the startup identifier chain is executed first at the waiting time level; obtaining a preset first performance impact coefficient; and determining the first execution priority of each startup identifier chain based on the first-level priority, the second-level priority, the third-level priority, and the first performance impact coefficient.

[0102] The process of determining a first execution priority for each start identifier chain and a second execution priority for each start identifier sub-chain within that chain, wherein each start identifier sub-chain corresponds one-to-one with a subset of execution components, and the first execution priority indicates the probability that the start identifier chain will be executed first, and the second execution priority indicates the probability that the start identifier sub-chain will be executed first, includes: the scheduler determining a first-level priority for each start identifier chain based on the standard execution time, the first-level priority indicating the probability that the start identifier chain will be executed first at the standard execution time level; determining a second-level priority for each start identifier chain based on the data processing volume, the second-level priority indicating the probability that the start identifier chain will be executed first at the data processing volume level; determining a third-level priority for each start identifier chain based on the waiting time, the third-level priority indicating the probability that the start identifier chain will be executed first at the waiting time level; obtaining a preset first performance impact coefficient; and determining a first execution priority for each start identifier chain based on the first-level priority, the second-level priority, the third-level priority, and the first performance impact coefficient.

[0103] The step of determining the first-level priority of each startup identifier chain based on the standard execution time, where the first-level priority indicates the probability that the startup identifier chain will be executed first at the standard execution time level, includes: calculating the standard execution time of each startup identifier chain based on the standard execution time; determining a first-level calculation formula based on a preset priority setting; obtaining the original first-level priority of each startup identifier chain based on the first-level calculation formula and the standard execution time of each startup identifier chain; and performing first-level normalization processing on the original first-level priority of each startup identifier chain according to a preset first-level normalization formula to obtain the first-level priority of each startup identifier chain, wherein the first-level normalization processing refers to mapping the original first-level priority to a certain interval, where the certain interval includes the interval 0 to 1.

[0104] The formula for calculating the standard execution time of each startup identifier chain based on the standard execution time can be:

[0105]

[0106] STm is the standard execution time of the startup identifier chain m;

[0107] The The sum of the standard execution times of all startup identifiers in the i-th startup identifier subset of the startup identifier chain m, i.e., the standard execution time of the i-th startup identifier subchain;

[0108] The The sum of the standard execution times of all startup identifier subchains in the startup identifier chain m is the standard execution time of the startup identifier chain m.

[0109] The calculation formula for the first level can be:

[0110]

[0111] The P STm 'This refers to the original first-level priority;

[0112] The The sum of the standard execution times for all startup identifier chains m;

[0113] The The percentage of the standard execution time of the startup identifier chain m in the sum of the standard execution times of all startup identifier chains;

[0114] Because the preset priority is set to be inversely proportional to the standard execution time, the original first-level priority is obtained through the following formula:

[0115]

[0116] The first-level normalization formula can be:

[0117]

[0118] The P STm Priority is set at the first level.

[0119] The It is the sum of the original first-level priorities of all start identifier chains m.

[0120] The step of determining the second-level priority of each startup identifier chain based on the data processing volume, wherein the second-level priority is used to indicate the probability that the startup identifier chain will be executed first at the data processing volume level, includes: calculating the data processing volume of each startup identifier chain based on the data processing volume; determining a second-level calculation formula based on a preset priority setting; obtaining the original second-level priority of each startup identifier chain based on the second-level calculation formula and the data processing volume of each startup identifier chain; and performing second-level normalization processing on the original second-level priority of each startup identifier chain according to a preset second-level normalization formula to obtain the second-level priority of each startup identifier chain, wherein the second-level normalization processing refers to mapping the original second-level priority to a certain interval, the certain interval including the interval 0 to 1.

[0121] The formula for calculating the data processing volume of the startup identifier chain based on the data processing volume can be:

[0122]

[0123] Dm is the data processing volume of the start identifier chain m;

[0124] Dmi is the data processing volume of the i-th sub-chain of the start identifier in the start identifier chain m;

[0125] The The sum of the data processing volume of all start identifier subchains in the start identifier chain m is the data processing volume of the start identifier chain m.

[0126] The calculation formula for the second level can be:

[0127]

[0128] The P Dm 'This refers to the original second-level priority;

[0129] The This is the sum of the data processing volume of all start identifier chains m, which is the total data processing volume of the total processing flow.

[0130] The The percentage of the data processing volume of the start identifier chain m in the total data processing volume;

[0131] Because the preset priority is set to be inversely proportional to the amount of data processed, the original second-level priority is obtained through the following formula:

[0132]

[0133] The second-level normalization formula can be:

[0134]

[0135] The P Dm It is the second level priority;

[0136] The It is the sum of the original second-level priorities of all starter identifier chains m.

[0137] The step of determining the third-level priority of each startup identifier chain based on the waiting time of each startup identifier chain, wherein the third-level priority is used to indicate the probability that the startup identifier chain will be executed first at the waiting time level, includes: determining a third-level calculation formula based on a preset priority setting; and obtaining the third-level priority of each startup identifier chain based on the third-level calculation formula and the waiting time.

[0138] The calculation formula for the third level can be:

[0139]

[0140] The P WTm This is the priority of the third level;

[0141] The WTm is the waiting time of the startup identifier chain m;

[0142] The This is the sum of the wait times for all starter identifier chains;

[0143] The preset priority is set so that the priority is proportional to the waiting time, that is, the third-level priority is obtained by the following formula:

[0144]

[0145] The first performance influence coefficient can be a first computing power influence factor λ, a first memory space influence factor μ, or both the first computing power influence factor λ and the first memory space influence factor μ.

[0146] The formula for determining the first execution priority of each startup identifier chain based on the first-level priority, the second-level priority, the third-level priority, and the first performance impact coefficient can be:

[0147] P m =λ*(P STm +P WTm )+μ*PDm

[0148] The P m This is the first execution priority.

[0149] As can be seen, in this example, by comprehensively considering the waiting time, the amount of data processed, and the standard processing time to determine the overall execution priority of the start identifier chain, the execution plan can be flexibly adjusted to improve the performance of data normalization processing and shorten the time of data normalization processing.

[0150] In one possible embodiment, determining the first execution priority of each startup identifier chain and the second execution priority of each startup identifier sub-chain within each startup identifier chain, wherein each startup identifier sub-chain corresponds one-to-one with a subset of execution components, the first execution priority indicating the probability that a startup identifier chain is executed first, and the second execution priority indicating the probability that a startup identifier sub-chain is executed first, includes: determining a fourth-level priority of each startup identifier sub-chain based on the standard execution time, the fourth-level priority indicating the probability that a startup identifier sub-chain is executed first at the standard execution time level; determining a fifth-level priority of each startup identifier sub-chain based on the data processing volume, the fifth-level priority indicating the probability that a startup identifier sub-chain is executed first at the data processing volume level; obtaining a preset second performance impact coefficient; and determining the second execution priority of each startup identifier sub-chain within each startup identifier chain based on the fourth-level priority, the fifth-level priority, and the second performance impact coefficient.

[0151] The step of determining the first execution priority of each startup identifier chain and the second execution priority of each startup identifier sub-chain within each startup identifier chain, wherein each startup identifier sub-chain corresponds one-to-one with a subset of execution components, the first execution priority indicating the probability that the startup identifier chain is executed first, and the second execution priority indicating the probability that the startup identifier sub-chain is executed first, includes: determining a fourth-level priority of each startup identifier sub-chain based on the standard execution time, the fourth-level priority indicating the probability that the startup identifier sub-chain is executed first at the standard execution time level; determining a fifth-level priority of each startup identifier sub-chain based on the data processing volume, the fifth-level priority indicating the probability that the startup identifier sub-chain is executed first at the data processing volume level; obtaining a preset second performance impact coefficient; and determining the second execution priority of each startup identifier sub-chain within each startup identifier chain based on the fourth-level priority, the fifth-level priority, and the second performance impact coefficient.

[0152] The step of determining the fourth-level priority of each startup identifier subchain based on the standard execution time, wherein the fourth-level priority is used to indicate the probability that the startup identifier subchain will be executed first at the standard execution time level, includes: calculating the standard execution time of each startup identifier subchain based on the standard execution time of each startup identifier; determining the fourth-level calculation formula based on a preset priority setting; obtaining the original fourth-level priority of each startup identifier subchain based on the fourth-level calculation formula and the standard execution time of each startup identifier subchain; and performing fourth-level normalization processing on the original fourth-level priority of each startup identifier subchain according to a preset fourth-level normalization formula to obtain the fourth-level priority of each startup identifier subchain, wherein the fourth-level normalization processing refers to mapping the original fourth-level priority to a certain interval, wherein the certain interval includes the interval 0 to 1.

[0153] The formula for calculating the standard execution time of each startup identifier subchain based on the standard execution time can be:

[0154]

[0155] STmi is the standard execution time of the startup identifier subchain;

[0156] Tmi is the standard execution time of the i-th startup identifier in the startup identifier subchain;

[0157] The The sum of the standard execution times of all startup identifiers in the startup identifier subchain is the standard execution time of the startup identifier subchain.

[0158] The calculation formula for the fourth level can be:

[0159]

[0160] The P STmi 'This refers to the original fourth-level priority;

[0161] The The sum of the standard execution times of all startup identifier subchains in the startup identifier chain, i.e., the standard execution time of the startup identifier chain;

[0162] The The percentage of the standard execution time of the startup identifier subchain in the standard execution time of the startup identifier chain;

[0163] Because the preset priority is set to be inversely proportional to the standard execution time, the original fourth-level priority is obtained through the following formula:

[0164]

[0165] The fourth-level normalization formula can be:

[0166]

[0167] The P STmi Priority for the fourth level:

[0168] The The sum of the original fourth-level priorities of all starter identifier subchains in the shown starter identifier chain.

[0169] The step of determining the fifth-level priority of each startup identifier subchain based on the data processing volume, wherein the fifth-level priority is used to indicate the probability that the startup identifier subchain will be executed first at the data processing volume level, includes: determining a fifth-level calculation formula based on a preset priority setting; obtaining the original fifth-level priority of each startup identifier subchain based on the fifth-level calculation formula and the data processing volume; and performing fifth-level normalization processing on the original fifth-level priority of each startup identifier subchain according to a preset fifth-level normalization formula to obtain the fifth-level priority of each startup identifier subchain, wherein the fifth-level normalization processing refers to mapping the original fifth-level priority to a certain interval, wherein the certain interval includes the interval 0 to 1.

[0170] The calculation formula for the fifth level can be:

[0171]

[0172] The P Dmi 'This refers to the original fifth-level priority;

[0173] The The sum of the data processing volume of all the subchains of the start identifier in the start identifier chain, i.e., the data processing volume of the start identifier chain;

[0174] The The percentage of the data processing volume of the startup identifier subchain in the total data processing volume of the startup identifier chain;

[0175] Because the preset priority is set to be inversely proportional to the amount of data processed, the original fifth-level priority is obtained through the following formula:

[0176]

[0177] The fourth-level normalization formula can be:

[0178]

[0179] The P Dmi This is the fifth level priority;

[0180] The It is the sum of the fifth-level priorities of all start identifier subchains.

[0181] The step of obtaining a preset second performance impact coefficient can be a second computing power impact factor α, a second memory space impact factor β, or both the second computing power impact factor α and the second memory space impact factor β.

[0182] The formula for determining the second execution priority based on the fourth-level priority, the fifth-level priority, and the second performance impact coefficient can be:

[0183] P mi =α*P STmi +β*P Dmi ;

[0184] The P mi This is the second execution priority.

[0185] For example, when the startup identifier chain includes startup identifier chain 5 and startup identifier chain 6, the steps of determining the first execution priority, determining the second execution priority, and updating each execution component sequence corresponding to each startup identifier chain according to the first execution priority and the second execution priority so that the execution time of the total processing flow is less than the execution time before the update are as follows:

[0186] As shown in Table 1, the startup identifier chain 5 includes:

[0187]

[0188]

[0189] Table 1

[0190] As shown in Table 2, the startup identifier chain 6 includes:

[0191]

[0192] Table 2

[0193] As shown in Figure 3 below, Rawdata1, Rawdata2, and Rawdata3 in startup identifier chain 5 and Rawdata4, Rawdata5, and Rawdata6 in startup identifier chain 6 are determined by data normalization processing requirements. These requirements are determined by the source of the original data and the fields of the original data. There are two scenarios: First, the fields required for the target data come from different original data sources, and the fields in these different original data sources are different. These fields are then combined to form the target data, corresponding to startup identifier chain 5. Second, the fields required for the target data come from different original data sources, and the fields in these different original data sources are the same. These fields are then superimposed to form the target data, corresponding to startup identifier chain 6.

[0194]

[0195] Table 3

[0196] All fields of the target data are shown in Table 4:

[0197] Target data Fields zx_nsrjcxx nsrsbh, nsrmc, frxm, fddbrsjhm, scjydz, djrq, zczb, hyrs, hydm, xydj, pjsj zxzcfzbxx qmye, ncye, bsrq, mc, xm, skssqq, skssqz

[0198] Table 4

[0199] The standard execution time is shown in Table 5:

[0200]

[0201]

[0202] Table 5

[0203] The startup identifier chain wait time is shown in Table 6:

[0204] Startup Identifier Chain Waiting time Startup Identifier Chain 5 200ms Startup Identifier Chain 6 100ms

[0205] Table 6

[0206] The first performance parameters are shown in Table 7:

[0207] First performance parameters value λ 2 μ 1

[0208] Table 7

[0209] The second performance parameters are shown in Table 8:

[0210] First performance parameters value α 2 β 1

[0211] Table 8

[0212] Based on the data above, the first execution priority calculation process is as follows:

[0213] Calculate the original first-level priority:

[0214] The STm of the start identifier chain 5 is (10+20+40+40)+(20+30+30)+(10+30+10)=240ms;

[0215] The STm of the start identifier chain 6 is (10+40+80+500)+(20+40+30+50+60)+(100+40+500+80)=1550ms;

[0216] P of the starting identifier chain 5 STm = 1 / [240 / (240+1550)] = 1 / (24 / 179) = 7.458;

[0217] P of the starting identifier chain 6 STm = 1 / [1550 / (240+1550)] = 1 / (155 / 179) = 1.155;

[0218] Perform first-level normalization to obtain the first-level priority:

[0219] P of the starting identifier chain 5 STm =7.458 / (7.458+1.155)=0.866;

[0220] P of the starting identifier chain 6 STm =1.155 / (7.458+1.155)=0.134;

[0221] That is, the first-level priority of the start identifier chain 5 is 0.866; the first-level priority of the start identifier chain 6 is 0.134.

[0222] Calculate the original second-level priority:

[0223] The starting identifier chain 5's Dm = 10000 + 1000 + 100 = 11100;

[0224] The starting identifier chain 6's Dm = 5 + 5000 + 800 = 5805;

[0225] P of the starting identifier chain 5 Dm = 1 / [11100 / (11100+5805)] = 1 / (11100 / 16905) = 1.523;

[0226] P of the starting identifier chain 6 Dm = 1 / [5805 / (11100+5805)] = 1 / (5805 / 16905) = 2.912;

[0227] Perform a second-level normalization process to obtain the second-level priority:

[0228] P of the starting identifier chain 5 Dm =1.523 / (1.523+2.912)=0.343;

[0229] P of the starting identifier chain 6 Dm =2.912 / (1.523+2.912)=0.657;

[0230] That is, the second-level priority of the starter identifier chain 5 is 0.343; the second-level priority of the starter identifier chain 6 is 0.657.

[0231] Calculate the third-level priority:

[0232] Set the WTm of identifier chain 5 to 200;

[0233] Set the WTm of identifier chain 6 to 100;

[0234] P of the starting identifier chain 5 WTm =200 / (200+100) = 2 / 3 = 0.667;

[0235] P of the starting identifier chain 5 WTm =100 / (200+100)=1 / 3;

[0236] That is, the third-level priority of the starter identifier chain 5 is 0.667; the third-level priority of the starter identifier chain 6 is 0.333.

[0237] Then, the first execution priority P of the start identifier chain 5 is... m =2*(0.866+0.667)+1*0.343=3.409;

[0238] First execution priority P of start identifier chain 6 m =2*(0.134+0.333)+1*0.657=1.591;

[0239] Therefore, startup identifier chain 5 is executed before startup identifier chain 6;

[0240] Based on the data above, the calculation process for the second execution priority is as follows:

[0241] Calculate the original fourth-level priority:

[0242] P of the starting identifier subchain Rawdata1 (10000 entries) STmi =1 / {(10+20+40+40) / [(10+20+40+40)+(20+30+30)+(10+30+10)]}=1 / (11 / 24)=2.18;

[0243] P starts the identifier subchain Rawdata2 (1000 entries) STmi =1 / {(20+30+30) / [(10+20+40+40)+(20+30+30)+(10+30+10)]}=1 / (8 / 24)=0.333=3;

[0244] P of the starting identifier subchain Rawdata3 (100 entries) STmi =1 / {(10+30+10) / [(10+20+40+40)+(20+30+30)+(10+30+10)]}=1 / (5 / 24)=4.8;

[0245] P of the starting identifier subchain Rawdata4 (5 entries) STmi =1 / {(10+40+80+500) / [(10+40+80+500)+(20+40+30+50+60)+(100+40+500+80)]}=1 / (63 / 155)=2.46;

[0246] P starts the identifier subchain Rawdata5 (5000 entries) STmi =1 / {(20+40+30+50+60) / [(10+40+80+500)+(20+40+30+50+60)+(100+40+500+80)]}=1 / (20 / 155)=7.75;

[0247] P of the starting identifier subchain Rawdata6 (800 entries) STmi =1 / {(100+40+500+80) / [(10+40+80+500)+(20+40+30+50+60)+100+40+500+80)]}=1 / (72 / 155)=2.153;

[0248] Perform fourth-level normalization to obtain the fourth-level priority:

[0249] P of the starting identifier subchain Rawdata1 (10000 entries) STmi =2.18 / (2.18+3+4.8)=0.218;

[0250] P starts the identifier subchain Rawdata2 (1000 entries) STmi =3 / (2.18+3+4.8)=0.301;

[0251] P of the starting identifier subchain Rawdata3 (100 entries) STmi=4.8 / (2.18+3+4.8) = 0.481;

[0252] P of the starting identifier subchain Rawdata4 (5 entries) STmi =2.46 / (2.46+7.75+2.153)=0.199;

[0253] P starts the identifier subchain Rawdata5 (5000 entries) STmi =7.75 / (2.46+7.75+2.153)=0.627;

[0254] P of the starting identifier subchain Rawdata6 (800 entries) STmi =2.153 / (2.46+7.75+2.153)=0.174;

[0255] Calculate the original fifth-level priority:

[0256] P of the starting identifier subchain Rawdata1 (10000 entries) Dmi = 1 / [10000 / (10000+1000+100)] = 1 / (100 / 111) = 1.11;

[0257] P starts the identifier subchain Rawdata2 (1000 entries) Dmi =1000 / (10000+1000+100) =1(10 / 111) =11.1;

[0258] P of the starting identifier subchain Rawdata3 (100 entries) Dmi =100 / (10000+1000+100) = 1(1 / 111) = 111;

[0259] P of the starting identifier subchain Rawdata4 (5 entries) Dmi =1 / [5 / (5+5000+800)] =1 / (5 / 5805) =1161;

[0260] P starts the identifier subchain Rawdata5 (5000 entries) Dmi = 1 / [5000 / (5+5000+800)] = 1 / (5000 / 5805) = 1.161;

[0261] P of the starting identifier subchain Rawdata6 (800 entries) Dmi = 1 / [800 / (5+5000+800)] = 1 / (800 / 5805) = 7.256;

[0262] Perform fifth-level normalization to obtain the fifth-level priority:

[0263] P of the starting identifier subchain Rawdata1 (10000 entries) Dmi =1.11 / (1.11+11.1+111)=0.009;

[0264] P starts the identifier subchain Rawdata2 (1000 entries) Dmi =11.1 / (1.11+11.1+111)=0.09;

[0265] P of the starting identifier subchain Rawdata3 (100 entries) Dmi =111 / (1.11+11.1+111)=0.901;

[0266] P of the starting identifier subchain Rawdata4 (5 entries) Dmi =1161 / (1161+1.161+7.256)=0.993;

[0267] P starts the identifier subchain Rawdata5 (5000 entries) Dmi =1.161 / (1161+1.161+7.256)=0.00099;

[0268] P of the starting identifier subchain Rawdata6 (800 entries) Dmi =7.256 / (1161+1.161+7.256)=0.00601;

[0269] Then, the second execution priority P of the identifier subchain Rawdata1 (10000 entries) is started. mi =2 * 0.218 + 1 * 0.009 = 0.445;

[0270] The second execution priority P of the start identifier subchain Rawdata2 (1000 entries) mi =2 * 0.301 + 1 * 0.09 = 0.692;

[0271] The second execution priority P of the start identifier subchain Rawdata3 (100 entries) mi =2 * 0.481 + 1 * 0.901 = 1.863;

[0272] Therefore, the execution order of the startup identifier subchain in startup identifier chain 5 is: Rawdata3 (100 entries) → Rawdata2 (1000 entries) → Rawdata1 (10000 entries);

[0273] The second execution priority P of the start identifier subchain Rawdata4 (5 entries) mi =2 * 0.199 + 1 * 0.993 = 1.391;

[0274] The second execution priority P of the start identifier subchain Rawdata5 (5000 entries) mi =2 * 0.627 + 1 * 0.00099 = 1.255;

[0275] The second execution priority P of the start identifier subchain Rawdata6 (800 entries) mi =2 * 0.174 + 1 * 0.00601 = 0.354;

[0276] Therefore, the execution order of the startup identifier subchain in startup identifier chain 6 is: Rawdata4 (5 entries) → Rawdata5 (5000 entries) → Rawdata6 (800 entries).

[0277] As can be seen, in this example, by comprehensively considering the waiting time, the amount of data processed, and the standard processing time to determine the execution priority of the start identifier subchain, the execution plan can be flexibly adjusted to improve the performance of data normalization processing and shorten the time of data normalization processing.

[0278] Step S404: Process the original data according to the updated sequence of each execution component until each sequence of execution components is completed, and obtain the target data.

[0279] The step of processing the original data according to the updated sequence of each execution component until each sequence of execution components is completed to obtain the target data includes: the executor processing the original data according to the updated sequence of each execution component until each sequence of execution components is completed to obtain the target data.

[0280] The scheduler sends the updated sequence of each execution component to the executor and the data stream, and the executor receives the updated sequence of each execution component and the data stream.

[0281] The process involves processing the original data according to the updated sequence of each execution component until each execution component sequence has been completed, resulting in the target data. Prior to this, the process includes: determining whether the data stream is empty; if so, stopping the data normalization process and displaying an error; if not, proceeding to the next step.

[0282] The step of processing the original data according to the updated sequence of each execution component until the sequence of each execution component is completed and the target data is obtained further includes: the executor sending the data stream to the verifier so that the verifier can determine whether the data stream is empty; if so, the data normalization process is stopped and an anomaly is displayed; if not, the verifier sends the data stream to the executor to continue to the next step.

[0283] In one possible embodiment, processing the original data according to the updated sequence of each execution component until each execution component sequence is completed to obtain the target data includes: obtaining the largest identical starter identifier sub-chain of each starter identifier chain, wherein the largest identical starter identifier sub-chain is at least two starter identifier sub-chains that have the same order and the same starter identifier sub-chain parameters, wherein the starter identifier sub-chain parameters are the values ​​pointed to by the starter identifier sub-chains; and storing the intermediate data generated when processing the original data according to the largest identical starter identifier sub-chain in an intermediate data cache pool, wherein the intermediate data cache pool is used to store and read intermediate data for reading and rereading.

[0284] The step of processing the original data according to the updated sequence of each execution component until the sequence of each execution component is completed to obtain the target data includes: the execution engine obtaining the largest identical sub-chain of each start identifier chain, wherein the largest identical sub-chain of start identifiers is at least two start identifier sub-chains that have the same order and the same start identifier sub-chain parameters, wherein the start identifier sub-chain parameters are the values ​​pointed to by the start identifier sub-chains; and the executor storing the intermediate data generated when processing the original data according to the largest identical sub-chain of start identifiers in an intermediate data cache pool, wherein the intermediate data cache pool is used to store and read intermediate data for reading and rereading.

[0285] The step of obtaining the maximum identical starter identifier subchain for each starter identifier chain, wherein the maximum identical starter identifier subchain is at least two starter identifier subchains in each starter identifier chain that have the same arrangement order and the same corresponding starter identifier subchain parameters, and the starter identifier subchain parameters are the values ​​pointed to by the starter identifier subchains, includes: determining whether there are at least two starter identifier subchains in each starter identifier chain that have the same arrangement order and the same corresponding starter identifier subchain parameters; if so, then determining the at least two starter identifier subchains that have the same arrangement order, the same corresponding starter identifier subchain parameters, and the largest number of starter identifier subchains as the maximum identical starter identifier subchain.

[0286] The step of determining whether at least two starter identifier chains in each starter identifier chain have at least two starter identifier subchains with the same order and the same starter identifier subchain parameters includes the following steps: First, take out the first starter identifier chain, and then iterate through the second to nth starter identifier chains in turn. Assuming the current chain is the i-th chain, determine whether there is a common starter identifier subchain in the i-th chain. If not, continue to determine the next starter identifier chain, i.e., the (i+1)-th starter identifier chain. If there is a common starter identifier subchain in the i-th chain and the starter identifier subchain parameters are the same, then compare the starter identifier and starter identifier parameters in the starter identifier subchains in turn. If they are not the same, continue to determine the next starter identifier chain, i.e., the (i+1)-th starter identifier chain. If they are the same, continue to determine whether the remaining starter identifier subchains and starter identifier subchain parameters in the i-th chain are the same, until the first different starter identifier subchain appears.

[0287] The startup identifier subchain parameters are the key and value in the chain node that together with the startup identifier subchain, where the key is the startup identifier subchain and the value is the startup identifier parameter.

[0288] Please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating a data normalization processing method provided in an embodiment of this application. For example... Figure 7 As shown, the starter identifier subchains with the same starter identifier subchains and the same starter identifier subchain parameters in starter identifier chain 1 and starter identifier chain 2 are: rawData1, field-mapping, clipper. Then the largest common starter identifier subchain between starter identifier chain 1 and starter identifier chain 2 is {rawData1, field-mapping, clipper}.

[0289] Please refer to Figure 8 , Figure 8 This is a schematic diagram of another data normalization processing method provided in the embodiments of this application. The start identifier subchains with the same start identifier subchains and the same start identifier subchain parameters in start identifier chain 3 and start identifier chain 4 are: rawData1, field-mapping. Then the largest common start identifier subchain between start identifier chain 3 and start identifier chain 4 is {rawData1, field-mapping}.

[0290] As can be seen, in this example, storing the intermediate data when executing the largest identical start identifier subchain in the intermediate data cache pool can reduce the generation of duplicate intermediate data during execution and shorten the time for reading intermediate data, thereby improving the performance of data normalization processing and shortening the duration of data normalization processing.

[0291] This reduces the time required for data normalization, thereby improving the performance of data normalization and shortening the processing time.

[0292] In one possible embodiment, after storing the intermediate data generated when processing the original data according to the largest identical start identifier subchain in an intermediate data cache pool, the method further includes: when it is detected that the start identifier chain includes the largest identical start identifier subchain, calling the corresponding intermediate data in the intermediate data cache pool and continuing subsequent processing.

[0293] The method further includes, after storing the intermediate data generated when processing the original data according to the largest identical start identifier subchain in an intermediate data cache pool, the method further includes: when the executor detects that the start identifier chain includes the largest identical start identifier subchain, calling the corresponding intermediate data in the intermediate data cache pool and continuing subsequent processing.

[0294] Please refer to Figure 9 , Figure 9 This is a timing diagram of a data normalization processing method provided in an embodiment of this application. For example... Figure 9 As shown, applied to, for example Figure 1 The data normalization processing system 100 shown includes a server 120, which comprises a terminal device 110 and a server 120. The terminal device 110 is communicatively connected to the server 120. The data normalization processing method includes:

[0295] Parser 350 converts the raw data into a data stream;

[0296] Secondly, the parser 350 sends the data stream to the execution engine 320;

[0297] Secondly, the execution engine 320 determines whether a startup identifier chain exists in the configuration cache 310;

[0298] Secondly, the execution engine 320 retrieves the startup identifier chain from the configuration cache 310;

[0299] Secondly, the execution engine 320 sends the startup identifier chain to the verifier 330 so that the verifier 330 can verify whether the startup identifier is correct;

[0300] Secondly, the verifier 330 sends the verification result of the start identifier chain to the execution engine 320;

[0301] Secondly, the execution engine 320 stores the startup identifier chain in the configuration cache 310 based on the startup identifier chain verification result;

[0302] Secondly, execution engine 320 obtains the largest subchain of identical startup identifiers;

[0303] Secondly, the execution engine 320 sends the data stream and the start identifier chain to the scheduler 340;

[0304] Next, scheduler 340 calculates the first execution priority of the start identifier chain and the second execution priority of the start identifier subchain;

[0305] Secondly, scheduler 340 sends the data stream, the start identifier chain, the first execution priority, and the second execution priority to executor 360;

[0306] Secondly, the executor 360 sends the data stream to the verifier 330 so that the verifier 330 can verify whether the data stream is empty;

[0307] Secondly, the executor 360 obtains the data stream verification result from the verifier 330;

[0308] Secondly, executor 360 calls the execution components according to the sequence of execution components corresponding to the first and second execution priorities;

[0309] Secondly, the executor 360 will send the intermediate data when executing the largest identical start identifier subchain to the intermediate data cache pool 380 or retrieve the intermediate data in the intermediate data cache pool 380, and send the execution component sequence execution completion information to the scheduler 340.

[0310] As can be seen, in this example, reading intermediate data from the intermediate data cache pool can shorten the time for reading intermediate data. This not only reduces the generation of duplicate intermediate data during execution, but also improves the performance of data normalization processing and shortens the processing time.

[0311] As can be seen, in this embodiment, the server first determines the start identifier chain of the processing flow of each sub-data in at least one sub-data of the original data. Multiple start identifiers in each start identifier chain correspond one-to-one with multiple execution components in the execution component sequence, and multiple sub-chains of start identifiers in the start identifier chain correspond one-to-one with multiple subsets of execution components in the execution component sequence. The total processing flow of the original data includes the processing flow of each sub-data in at least one sub-data. Next, the server obtains the waiting time of each start identifier chain, the standard execution time of each start identifier among multiple start identifiers, and the data processing volume of each start identifier sub-chain in each start identifier chain. Each start identifier in each sub-chain of start identifiers corresponds one-to-one with the execution components in the corresponding subset of execution components. Then, based on the waiting time, standard execution time, and data processing volume, the server updates each execution component sequence corresponding to each start identifier chain so that the execution time of the total processing flow is less than the execution time before the update. Finally, the server processes the original data according to each updated execution component sequence until each execution component sequence is completed, obtaining the target data. Since the execution components are invoked by the start identifier, when new data processing scenarios are needed, the execution components can be extended and the start identifier chain can be adjusted and configured. There is no need to create a large number of data tables and processing subsystems, which reduces memory pressure. Furthermore, adjusting the execution order of the execution components shortens the time for data normalization processing.

[0312] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the server includes the corresponding hardware structure and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0313] For embodiments consistent with those shown above, please refer to... Figure 10a , Figure 10a This is a functional unit block diagram of a data normalization processing device provided in an embodiment of this application, such as... Figure 10aAs shown, the data normalization processing device 1000 includes: a first processing unit 1001, configured to determine a chain of start identifiers for the processing flow of each sub-data in at least one sub-data of the original data, wherein multiple start identifiers in each start identifier chain correspond one-to-one with multiple execution components in the execution component sequence, and multiple sub-chains of start identifiers in the start identifier chain correspond one-to-one with multiple subsets of execution components in the execution component sequence, and the overall processing flow of the original data includes the processing flow of each sub-data in the at least one sub-data; and a first receiving unit 1002, configured to acquire the waiting time of each start identifier chain, and the waiting time of each of the multiple start identifiers. The standard execution time of the start identifier and the data processing volume of each start identifier sub-chain in each start identifier chain, wherein each start identifier in each start identifier sub-chain corresponds one-to-one with the execution components in the corresponding execution component subset; the second processing unit 1003 is used to update each execution component sequence corresponding to each start identifier chain according to the waiting time, the standard execution time, and the data processing volume so that the execution time of the total processing flow is less than the execution time before the update; and to process the original data according to the updated execution component sequence until each execution component sequence is completed to obtain the target data.

[0314] In one possible embodiment, regarding the initiation identifier chain of the processing flow for each sub-data in determining at least one sub-data of the original data, the first processing unit 1001 is specifically configured to: obtain the data source number of the original data; determine guidance information for the initiation identifier chain based on the data source number; and determine the initiation identifier chain based on the guidance information.

[0315] In one possible embodiment, in determining the startup identifier chain according to the guidance information, the first processing unit 1001 is specifically configured to: determine whether the startup identifier chain is correct; if not, stop data normalization processing and display an error; if yes, store the startup identifier chain in a configuration cache, the configuration cache being used to non-permanently store the startup identifier chain for reading and rereading.

[0316] In one possible embodiment, after determining whether the startup identifier chain is correct, the data normalization processing device 1000 is further configured to: determine whether the startup identifier chain exists in the configuration cache; if not, obtain the startup identifier chain from the database and store the startup identifier chain in the configuration cache, wherein the database is used to permanently store the startup identifier chain for reading and rereading; if yes, obtain the startup identifier chain from the configuration cache.

[0317] In one possible embodiment, where at least one or more, the second processing unit 1003 is specifically configured to: determine a first execution priority for each startup identifier chain and a second execution priority for each startup identifier subchain within each startup identifier chain, wherein the startup identifier subchain corresponds one-to-one with the subset of execution components; the first execution priority indicates the probability that the startup identifier chain will be executed first, and the second execution priority indicates the probability that the startup identifier subchain will be executed first; and update each execution component sequence corresponding to each startup identifier chain according to the first execution priority and the second execution priority so that the total processing time is less than the execution time before the update.

[0318] In one possible embodiment, regarding determining a first execution priority for each startup identifier chain and a second execution priority for each startup identifier sub-chain within each startup identifier chain, wherein each startup identifier sub-chain corresponds one-to-one with a subset of execution components, the first execution priority indicates the probability that the startup identifier chain will be executed first, and the second execution priority indicates the probability that the startup identifier sub-chain will be executed first, the second processing unit 1003 is specifically configured to: determine a first-level priority for each startup identifier chain based on the standard execution time, the first-level priority indicating the probability that the startup identifier chain will be executed first at the standard execution time level; determine a second-level priority for each startup identifier chain based on the data processing volume, the second-level priority indicating the probability that the startup identifier chain will be executed first at the data processing volume level; determine a third-level priority for each startup identifier chain based on the waiting time, the third-level priority indicating the probability that the startup identifier chain will be executed first at the waiting time level; obtain a preset first performance impact coefficient; and determine a first execution priority for each startup identifier chain based on the first-level priority, the second-level priority, the third-level priority, and the first performance impact coefficient.

[0319] In one possible embodiment, regarding the determination of a first execution priority for each start identifier chain and a second execution priority for each start identifier subchain within each start identifier chain, wherein each start identifier subchain corresponds one-to-one with a subset of execution components, the first execution priority indicates the probability that the start identifier chain will be executed first, and the second execution priority indicates the probability that the start identifier subchain will be executed first. Specifically, the second processing unit 1003 is configured to: determine a fourth-level priority for each start identifier subchain based on the standard execution time, the fourth-level priority indicating the probability that the start identifier subchain will be executed first at the standard execution time level; determine a fifth-level priority for each start identifier subchain based on the data processing volume, the fifth-level priority indicating the probability that the start identifier subchain will be executed first at the data processing volume level; obtain a preset second performance impact coefficient; and determine the second execution priority for each start identifier subchain within each start identifier chain based on the fourth-level priority, the fifth-level priority, and the second performance impact coefficient.

[0320] In one possible embodiment, regarding the processing of the original data according to the updated sequence of each execution component until the sequence of each execution component is completed to obtain the target data, the second processing unit 1003 is specifically configured to: obtain the largest identical start identifier sub-chain of each start identifier chain, wherein the largest identical start identifier sub-chain is at least two start identifier sub-chains that have the same order and the same start identifier sub-chain parameters, wherein the start identifier sub-chain parameters are the values ​​pointed to by the start identifier sub-chains; and store the intermediate data generated when processing the original data according to the largest identical start identifier sub-chain in an intermediate data cache pool, wherein the intermediate data cache pool is used to store and read intermediate data for reading and rereading.

[0321] In one possible embodiment, after storing the intermediate data generated when processing the original data according to the maximum identical start identifier subchain in an intermediate data cache pool, the data normalization processing device 1000 is further configured to: when it is detected that the start identifier chain includes the maximum identical start identifier subchain, call the corresponding intermediate data in the intermediate data cache pool and continue subsequent processing.

[0322] It is understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the method embodiment section in this application should be adapted to the device embodiment section in a synchronous manner, and will not be repeated here.

[0323] When using integrated units, such as Figure 10b As shown, Figure 10b This is a functional unit block diagram of another data normalization processing device provided in an embodiment of this application. Figure 10b In this document, the data normalization processing apparatus 1010 includes a processing module 1012 and a communication module 1011. The processing module 1012 controls and manages the operation of the data normalization processing apparatus, for example, executing the steps of the first processing unit 1001, the first receiving unit 1002, and the second processing unit 1003, and / or performing other processes of the technology described herein. The communication module 1011 supports interaction between the data normalization processing apparatus and other devices. Figure 10b As shown, the data normalization processing device may further include a storage module 1013, which is used to store the program code and data of the data normalization processing device.

[0324] The processing module 1012 can be a processor or server, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 1011 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 1013 can be a memory.

[0325] All relevant content in each scenario involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. The above data normalization processing device 1010 can execute the above... Figure 4 The data normalization process shown is as follows.

[0326] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0327] Figure 11 This is a structural block diagram of a server provided in an embodiment of this application. For example... Figure 11 As shown, server 1100 may include one or more of the following components: processor 1101, memory 1102 coupled to processor 1101, wherein memory 1102 may store one or more computer programs, which may be configured to implement the methods described in the above embodiments when executed by one or more processors 1101. Server 1100 may be server 120 in the above embodiments.

[0328] Processor 1101 may include one or more processing cores. Processor 1101 connects to various parts of server 1100 via various interfaces and lines, and performs various functions and processes data of server 1100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 1102, and by calling data stored in memory 1102. Optionally, processor 1101 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 1101 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 1101 and may be implemented separately through a communication chip.

[0329] The memory 1102 may include random access memory (RAM) or read-only memory (ROM). The memory 1102 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1102 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the server 1100 during use.

[0330] It is understood that server 1100 may include more or fewer structural elements than those shown in the above block diagram, and this is not limited thereto. Embodiments of this application provide a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the method described in any possible embodiment.

[0331] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0332] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0333] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0334] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0335] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., which are various media capable of storing program code.

[0336] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A data normalization processing method, characterized in that, A server applied to a data normalization processing system, the method comprising: A chain of start identifiers is defined to determine the processing flow of each sub-data in at least one sub-data of the original data. Multiple start identifiers in each chain of start identifiers correspond one-to-one with multiple execution components in the execution component sequence. Multiple sub-chains of start identifiers in the chain of start identifiers correspond one-to-one with multiple subsets of execution components in the execution component sequence. The overall processing flow of the original data includes the processing flow of each sub-data in the at least one sub-data. The waiting time of each startup identifier chain, the standard execution time of each startup identifier in the plurality of startup identifiers, and the data processing volume of each startup identifier subchain in each startup identifier chain are obtained, and the startup identifier in each startup identifier subchain corresponds one-to-one with the execution component in the corresponding execution component subset; Based on the waiting time, the standard execution time, and the data processing volume, update each execution component sequence corresponding to each startup identifier chain so that the total execution time of the processing flow is less than the execution time before the update; The original data is processed according to the updated sequence of each execution component until each sequence of execution components is completed, and the target data is obtained. Wherein, at least one can be multiple, and the step of updating each execution component sequence corresponding to each startup identifier chain according to the waiting time, the standard execution time, and the data processing volume so that the total execution time of the processing flow is less than the execution time before the update includes: A first execution priority and a second execution priority of each start identifier chain and each start identifier sub-chain in each start identifier chain are determined, wherein each start identifier sub-chain corresponds one-to-one with a subset of execution components. The first execution priority indicates the probability that the start identifier chain is executed first, and the second execution priority indicates the probability that the start identifier sub-chain is executed first. The method for determining the first execution priority and the second execution priority includes: calculating the first execution priority and the second execution priority using the MLCATBP algorithm, including: obtaining the priority setting and multi-level setting of the MLCATBP algorithm, wherein the priority setting indicates the influence relationship between the multi-level setting and the execution priority, and the priority setting includes that the priority is directly proportional to the waiting time, inversely proportional to the standard execution time, and inversely proportional to the data processing volume. Based on the first execution priority and the second execution priority, update each execution component sequence corresponding to each start identifier chain so that the total execution time of the process is less than the execution time before the update.

2. The method according to claim 1, characterized in that, The step of determining a first execution priority for each start identifier chain and a second execution priority for each start identifier sub-chain within each start identifier chain, wherein each start identifier sub-chain corresponds one-to-one with a subset of execution components, the first execution priority indicating the probability that a start identifier chain will be executed first, and the second execution priority indicating the probability that a start identifier sub-chain will be executed first, includes: The first-level priority of each startup identifier chain is determined based on the standard execution time, and the first-level priority is used to indicate the probability that the startup identifier chain is executed first at the standard execution time level; The second-level priority of each initiation identifier chain is determined based on the data processing volume, and the second-level priority is used to indicate the probability that the initiation identifier chain is executed first at the data processing volume level; The third-level priority of each initiation identifier chain is determined based on the waiting time, and the third-level priority is used to indicate the probability that the initiation identifier chain is executed first at the waiting time level; Obtain the preset first performance impact coefficient; The first execution priority of each startup identifier chain is determined based on the first level priority, the second level priority, the third level priority, and the first performance impact coefficient.

3. The method according to claim 1, characterized in that, The step of determining a first execution priority for each start identifier chain and a second execution priority for each start identifier sub-chain within each start identifier chain, wherein each start identifier sub-chain corresponds one-to-one with a subset of execution components, the first execution priority indicating the probability that a start identifier chain will be executed first, and the second execution priority indicating the probability that a start identifier sub-chain will be executed first, includes: The fourth-level priority of each initiation identifier subchain is determined based on the standard execution time, and the fourth-level priority is used to indicate the probability that the initiation identifier subchain is executed first at the standard execution time level; The fifth-level priority of each initiation identifier subchain is determined based on the data processing volume, and the fifth-level priority is used to indicate the probability that the initiation identifier subchain is executed first at the data processing volume level. Obtain the preset second performance impact coefficient; The second execution priority of each startup identifier subchain in each startup identifier chain is determined based on the fourth-level priority, the fifth-level priority, and the second performance impact coefficient.

4. The method according to any one of claims 1-3, characterized in that, The step of processing the original data according to the updated sequence of each execution component until each sequence of execution components is completed, to obtain the target data, includes: Obtain the largest identical start identifier subchain for each start identifier chain. The largest identical start identifier subchain is at least two start identifier subchains that have the same arrangement order and the same start identifier subchain parameters. The start identifier subchain parameters are the values ​​pointed to by the start identifier subchains. Intermediate data generated when processing the original data according to the largest identical start identifier subchain is stored in an intermediate data cache pool, which is used to store and retrieve intermediate data for reading and re-reading.

5. The method according to claim 4, characterized in that, After storing the intermediate data generated during the processing of the original data according to the largest identical start identifier subchain in the intermediate data cache pool, the method further includes: When it is detected that the startup identifier chain includes the largest identical startup identifier sub-chain, the corresponding intermediate data in the intermediate data cache pool is called, and subsequent processing continues.

6. The method according to claim 1, characterized in that, The chain of start identifiers for determining the processing flow of each sub-data in at least one sub-data of the original data includes: Obtain the data source number of the original data; The guidance information for determining the startup identifier chain is determined based on the data source number; The launch identifier chain is determined based on the guidance information.

7. The method according to claim 6, characterized in that, Determining the startup identifier chain based on the guidance information includes: Determine whether the startup identifier chain is correct; If not, stop data normalization and display an anomaly; If so, the startup identifier chain is stored in the configuration cache, which is used to non-permanently store the startup identifier chain for reading and rereading.

8. The method according to claim 7, characterized in that, After determining whether the startup identifier chain is correct, the method further includes: Determine whether the startup identifier chain exists in the configuration cache; If not, the startup identifier chain is retrieved from the database and stored in the configuration cache. The database is used to permanently store the startup identifier chain for reading and re-reading. If so, then retrieve the startup identifier chain from the configuration cache.

9. A data normalization processing device, characterized in that, A server used in a data normalization processing system, the apparatus being configured to perform the method as described in any one of claims 1-8, the apparatus comprising: The first processing unit is used to determine the start identifier chain of the processing flow of each sub-data in at least one sub-data of the original data. Multiple start identifiers in each start identifier chain correspond one-to-one with multiple execution components in the execution component sequence. Multiple start identifier sub-chains in the start identifier chain correspond one-to-one with multiple subsets of execution components in the execution component sequence. The overall processing flow of the original data includes the processing flow of each sub-data in the at least one sub-data. The first receiving unit is used to obtain the waiting time of each startup identifier chain, the standard execution time of each startup identifier among the plurality of startup identifiers, and the data processing volume of each startup identifier sub-chain in each startup identifier chain, wherein each startup identifier in each startup identifier sub-chain corresponds one-to-one with the execution components in the corresponding execution component subset; The second processing unit is configured to update each execution component sequence corresponding to each start identifier chain according to the waiting time, the standard execution time, and the data processing volume so that the total execution time of the processing flow is less than the execution time before the update; and to process the original data according to the updated execution component sequence until each execution component sequence is completed to obtain the target data.

Images