A method, device, equipment and storage medium for data processing
By verifying the configuration files of data processing, the problem of frequent errors in code processing is solved, the development cost is reduced, and the accuracy and efficiency of data processing is improved.
Patent Information
- Application Number
- CN202011025727.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-25
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-09-25
AI Technical Summary
In the prior art, errors are prone to occur when processing data through code, resulting in high development costs, high development difficulty, low accuracy of data processing, and affecting data processing efficiency.
By obtaining the target input data, determine the associated target configuration file, and verify the configuration file before performing data processing to ensure that it meets the preset requirements before performing data processing.
It reduces labor development costs, avoids errors in the data processing process, and improves the correctness of configuration files and the efficiency of data processing.
Smart Images

Figure CN112181519B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to big data technology, and more particularly to a method, apparatus, device and storage medium for data processing. Background Art
[0002] The application of big data is mainly concentrated in distributed caching, storage, indexing and computing. In banking business, there are often tasks such as large-scale customer asset analysis, customer behavior data and product evaluation analysis. Big data is also increasingly used in these data processing tasks.
[0003] In the prior art, staff write the code for the data processing flow by analyzing a large number of data analysis and computing tasks in actual system applications. Each task type corresponds to different codes. When the parameters change, the code needs to be updated, which makes the learning cost for developers too high and the development difficulty increased. In addition, errors are prone to occur during the code writing process, and the accuracy of data processing is low, which affects the data processing efficiency. Summary of the invention
[0004] The embodiments of the present invention provide a method, device, equipment and storage medium for data processing to improve the efficiency of data processing.
[0005] In a first aspect, an embodiment of the present invention provides a method for data processing, the method comprising:
[0006] Responding to the data processing instruction, acquiring target input data;
[0007] Determining a target configuration file associated with the target input data according to an association relationship between the input data and the configuration file;
[0008] Verifying the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements;
[0009] If it meets the requirements, the target input data is processed according to the target configuration file to obtain the target output data.
[0010] In a second aspect, an embodiment of the present invention further provides a data processing device, the device comprising:
[0011] An input data acquisition module, used to acquire target input data in response to a data processing instruction;
[0012] A target file determination module determines a target configuration file associated with the target input data according to an association relationship between the input data and the configuration file;
[0013] A target file verification module is used to verify the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements;
[0014] The output data acquisition module is used to process the target input data according to the target configuration file to obtain target output data if it meets the requirements.
[0015] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method as described in any embodiment of the present invention is implemented.
[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the data processing method as described in any embodiment of the present invention.
[0017] The embodiment of the present invention obtains target input data, determines a target configuration file associated with the target input data, and verifies the target configuration file before processing data according to the target configuration file to determine whether the target configuration file meets the preset requirements. If so, data processing is performed. The problem that the code is prone to errors when data processing is performed through code in the prior art is solved, the manpower development cost is reduced, and errors in the data processing flow are avoided by verifying the configuration file, the correctness of the configuration file is improved, and the efficiency of data processing is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a flowchart of a data processing method in Embodiment 1 of the present invention;
[0019] Figure 2 is a flow chart of a data processing method in Embodiment 2 of the present invention;
[0020] Figure 3 is a structural block diagram of a data processing device in Embodiment 3 of the present invention;
[0021] Figure 4 It is a structural diagram of a computer device in Embodiment 4 of the present invention. DETAILED DESCRIPTION
[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0023] Embodiment 1
[0024] Figure 1 This is a flow chart of a data processing method provided by the first embodiment of the present invention. This embodiment is applicable to the case of processing data. The method can be executed by a data processing device. Figure 1 As shown, the method specifically comprises the following steps:
[0025] Step 110: Responding to the data processing instruction, obtaining target input data.
[0026] The user issues a data processing instruction, and the server responds to the data processing instruction and obtains the target input data from the system that issues the data processing instruction. The target input data may include the user's unique identifier, date and time, and product details. According to the data processing instruction, the required target input data is determined. For example, if the user's data processing instruction is a profit and loss analysis, the target input data is the user data and product data associated with calculating the profit and loss analysis.
[0027] In this embodiment, optionally, before obtaining the target input data in response to the data processing instruction, it also includes: determining the target task type according to the data processing instruction; and determining the target task ID of the target task type according to the association between the task type and the task ID.
[0028] Specifically, the data processing instruction may include a data source for obtaining the target input data, and may also include a target task type. The task type may be a type for calculating the product period rate of return, annualized rate of return, volatility, etc., or may be an analysis of user position profit and loss, etc. According to the data processing instruction, the target task type may be determined. Different task types are associated with a unique task ID. According to the target task type, the target task ID of the target task type may be obtained. After obtaining the target task ID, the target task ID may be added to the target input data. The beneficial effect of such a setting is that the target task type and the target task ID may be clearly defined, which is conducive to the subsequent determination of the target configuration file according to the target task ID, avoiding configuration file confirmation errors, and improving the accuracy and efficiency of data processing.
[0029] Step 120: Determine a target configuration file associated with the target input data according to the association relationship between the input data and the configuration file.
[0030] Among them, the association relationship between different input data and configuration files can be preset, and the input data can include information such as task ID, data source number, data name and data field, and the information in the input data can be associated with the configuration file. The data source can be associated with the configuration file, and the name of the input data can also be associated with the configuration file. For example, there are configuration files 1 and 2 in the configuration file, and configuration file 1 is associated with the product revenue amount. When the input data is the product revenue amount, configuration file 1 is used for processing. After determining the target input data, the target configuration file is determined according to the information in the target input data.
[0031] In this embodiment, optionally, determining a target configuration file associated with the target input data according to an association relationship between the input data and the configuration file includes: determining a target configuration file associated with the target task ID according to an association relationship between the task ID and the configuration file.
[0032] Specifically, the association relationship between the task ID and the configuration file can be pre-built. After the target input data is determined, the target task ID is obtained from the target input data, and the target configuration file associated with the target task ID is determined according to the association relationship between the task ID and the configuration file. The beneficial effect of such a setting is that by searching for the target configuration file according to the task ID, errors in searching for the target configuration file can be avoided, thereby improving data processing efficiency.
[0033] Step 130 : Verify the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements.
[0034] Among them, after determining the target configuration file, the target configuration file is first verified, and data processing is performed after determining that the target configuration file meets the preset requirements. Each configuration file contains the data processing flow of the task type corresponding to the configuration file. The data can be processed according to the data processing flow. For example, the data can be calculated according to the preset formula to obtain the final result. The preset requirement is that there can be no loops in the data processing flow in the configuration file. In the configuration file of each task type, the data processing flow is a directed acyclic flow, that is, the data processing flow is a non-cyclic flow. If there is a loop in the target processing flow of the target configuration file, the target processing flow cannot obtain the final output data, the target configuration file does not meet the preset requirements, and there is an error in the target configuration file; if there is no loop in the target processing flow, the target configuration file meets the preset requirements and data processing can continue.
[0035] In this embodiment, optionally, the target processing flow in the target configuration file is verified to determine whether the target configuration file meets the preset requirements, including: parsing the target processing flow in the target configuration file to obtain intermediate input data and intermediate output data of each step in the target processing flow; and determining whether the target configuration file meets the preset requirements based on the data dependency relationship between the intermediate input data and the intermediate output data between different steps.
[0036] Specifically, the target processing flow in the target configuration file is obtained, and the target processing flow is parsed to obtain each data processing step in the target processing flow. The source of the intermediate input data and the destination of the intermediate output data required for each data processing step are determined. The intermediate input data is the input data of the intermediate step in the target processing flow, including the input data of the first step, and the intermediate output data is the output data of the intermediate step in the target processing flow, including the output data of the last step. The data dependency between each two steps is determined. For example, there are three steps, namely step one, step two and step three. The data dependency between step one and step two, the data dependency between step one and step three, and the data dependency between step two and step three are determined. The data dependency refers to the relationship between the input and output data between different steps. If the output data of one step is the input data of another step, there is a data dependency between the two steps. According to the data dependency between different steps, it can be determined whether the target configuration file meets the preset requirements. For example, it can be determined whether the input data of the previous step contains the output data of the subsequent step to determine whether the target configuration file meets the preset requirements. The beneficial effect of such a setting is that by checking the target configuration file, errors in the data processing process can be avoided, the time for data processing can be saved, problems can be discovered before data processing, and the efficiency of data processing can be improved.
[0037] In this embodiment, optionally, determining whether the target configuration file meets the preset requirements is performed based on the data dependency relationship between the intermediate input data and the intermediate output data between different steps, including: determining whether the acquisition of the intermediate input data of the first step in the target processing flow depends on the intermediate output data of the second step; if not, determining that the target configuration file meets the preset requirements.
[0038] Specifically, judging the data dependency relationship between the intermediate input data and the intermediate output data between different steps can be to judge whether the intermediate input data of a data processing step is derived from the output data of other data processing steps, that is, to determine whether the acquisition of the intermediate input data of the first step in the target processing flow depends on the intermediate output data of the second step. Among them, the first step is located before the second step in the target processing flow, and the order of each step is preset. The first step and the second step can be connected or not. If the intermediate input data of the first step depends on the intermediate output data of the second step, it means that after the intermediate output data of the second step is obtained, there is a loop process to reach the first step, and the target processing flow has a loop, which does not meet the preset requirements of the target configuration file. If the intermediate input data of the first step does not depend on the intermediate output data of the second step, it is determined that the target configuration file meets the preset requirements. For example, the intermediate input data of the first step can be the intermediate output data of any step before the first step, or it can be the target input data of the data source. The beneficial effect of such a setting is that the dependency relationship of the intermediate data of each step is verified, the omission of steps is avoided, the accuracy of verification is improved, and the efficiency of data processing is improved.
[0039] In this embodiment, optionally, the target processing flow in the target configuration file is verified to determine whether the target configuration file meets the preset requirements, and also includes: running the target processing flow in the target configuration file according to preset input data to determine whether preset output data is obtained; if the preset output data is obtained, it is determined that the target configuration file meets the preset requirements.
[0040] Specifically, preset input data for verifying the target configuration file can be pre-set. When the target configuration file needs to be verified, the preset input data is input, and the target processing flow is run. If the target processing flow is a non-cyclic flow, the final preset output data can be obtained. If the preset output data cannot be obtained, it means that the target processing flow is a cyclic flow, which does not meet the preset requirements of the target configuration file. After obtaining the preset output data, the output preset output data can be compared with the corresponding standard output data. The standard output data is the correct output data corresponding to the preset input data calculated in advance. If the output preset output data is consistent with the standard output data, it means that there is no problem with the calculation formula in the target configuration file, and the correct result can be obtained. Therefore, data processing can be performed through the target configuration file. The beneficial effect of such a setting is that by inputting the preset input data, the verification result can be directly obtained, and there is no need to check the data dependency of each step, avoiding the omission of step inspection, effectively saving time, and improving the efficiency and accuracy of data verification.
[0041] Step 140: If it is in compliance, the target input data is processed according to the target configuration file to obtain the target output data.
[0042] Among them, if the target configuration file meets the preset requirements, the target input data is processed according to the target configuration file and calculated according to the preset target processing flow. For example, the target input data can be added, averaged or calculated for yield to obtain the target output data, that is, the result of data processing.
[0043] In this embodiment, optionally, after processing the target input data according to the target configuration file to obtain the target output data, the method further includes: storing the target output data according to the target output data persistence strategy in the target configuration file.
[0044] Specifically, the configuration file may include a data processing flow, which is used to calculate the input data and obtain the output data. The configuration file may also include an output data persistence strategy, which is used to persist the output data. After obtaining the target output data, the target output data persistence strategy in the target configuration file is obtained to determine the storage method, storage type and storage location of the target output data. According to the target output data persistence strategy, the target output data is stored. The beneficial effect of such a setting is that after obtaining the output data, persistent storage can be automatically performed without manually searching for storage information. The configuration of the output data is completed through the configuration file, thereby improving the efficiency of data processing, saving manpower and time, and reducing human errors.
[0045] The technical solution of this embodiment obtains target input data, determines the target configuration file associated with the target input data, and verifies the target configuration file before processing data according to the target configuration file to determine whether the target configuration file meets the preset requirements. If it does, data processing is performed. This solves the problem in the prior art that when data is processed through code, the code is prone to errors, reduces human development costs, and avoids errors in the data processing process by verifying the configuration file, improves the correctness of the configuration file, and improves the efficiency of data processing.
[0046] Embodiment 2
[0047] Figure 2 This is a flow chart of a data processing method provided by the second embodiment of the present invention. This embodiment is further optimized based on the above embodiment. The method can be executed by a data processing device. Figure 2 As shown, the method specifically comprises the following steps:
[0048] Step 210: Generate configuration files corresponding to different task types according to the data processing flow of different task types; wherein the configuration files include data source configuration, user-defined function configuration and data flow configuration.
[0049] Among them, multiple configuration files can be determined in advance according to the data processing flow of different task types, so that the configuration files correspond to the task types. Different task types have different data processing flows, and the data processing flow refers to the specific processing process of the input data. The staff loads the data processing flow of different task types into the server, and the server packages the data processing flow and generates a configuration file. The configuration file may include data source configuration, custom function configuration and data flow configuration. The data source configuration refers to the configuration of data information in the data processing flow, and the data information at least includes the data type, data access method and data storage format. The data information may be input data information, intermediate data information in the processing process and output data information. For example, the data type of the input data can be determined according to the data source configuration, and the storage location of the output data can also be determined. The data source configuration can indicate the fields and formats of the required input data, so that after obtaining the target input data, the target input data can be selected and extracted to obtain the input data that is really needed, and the target input data can be converted into a format type that can be processed by the configuration file. The target input data can be HDFS (Hadoop Distributed File System) files, HBase (Hadoop database, an open source non-relational distributed database) tables or Hive (a data warehouse tool based on Hadoop) tables and other data. According to the data source configuration, various types of heterogeneous data can be extracted and converted. The configuration file uses datasource as the source node, and the data source configuration can be one of the data. The data source configuration can configure data description information, storage type, data access information, field name, field type and field alias, etc. Effectively reduce the development cost of staff, reduce the difficulty of development, reduce the writing of code, avoid code errors, and improve data processing efficiency.
[0050] The custom function configuration in the configuration file can be configured as UDF (User-Defined Function) and UDAF (User-Defined Aggregation Function). The custom function configuration is stored in the path of the JAVA class, and the path of the JAVA class is configured in the configuration file. In other words, by configuring the JAVA class path in the configuration file, you can dynamically load the custom function and use it to process and calculate data. For example, the path of Hive's UDF can be org.apache.hadoop.hive.ql.exec.UDF. By configuring custom functions, the flexibility and ease of data processing are increased, which is conducive to the expansion and update of the configuration file.
[0051] Data flow configuration refers to the step configuration of the data processing flow, which includes at least data reading configuration, data processing configuration and data persistence configuration. The actual process of data processing is parsed to obtain each step. Data reading configuration refers to the requirements for configuring data when reading, which is convenient for determining the field content and data format of data reading. Data processing configuration is to splice each step to generate a complete data processing flow, so as to process the read data and obtain output data. Custom functions can be used in data processing configuration to calculate data. Data persistence configuration is the configuration of output data, which is used to persist output data and store output data in a specified location. Through data flow configuration, the complete process of data processing can be determined, and the staff does not need to encode. The relevant processes of big data processing are abstracted, which has high flexibility and scalability, reduces errors caused by code errors, and improves the accuracy and efficiency of data processing.
[0052] Step 220: In response to the data processing instruction, obtain target input data.
[0053] Step 230: Determine a target configuration file associated with the target input data according to the association relationship between the input data and the configuration file.
[0054] Step 240: Verify the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements.
[0055] Step 250: If it is in compliance, the target input data is processed according to the target configuration file to obtain the target output data.
[0056] Among them, if the target configuration file is successfully verified and meets the preset requirements, the target configuration file can be used for data processing.
[0057] In this embodiment, optionally, the target input data is processed according to the target configuration file to obtain the target output data, including: converting the target input data according to the data source configuration in the target configuration file to obtain the data to be processed; obtaining the path of the target JAVA class in the target configuration file; dynamically loading the target custom function according to the path of the target JAVA class; and processing the data to be processed based on the target custom function according to the data flow configuration to obtain the target output data.
[0058] Specifically, after obtaining the target input data, the format of the target input data may not be processed by the target configuration file, and the target input data needs to be converted into a format that can be recognized by the target configuration file. The data source configuration in the target configuration file can be used to convert the format of the target input data to obtain the data to be processed. For example, the data to be processed can be hexadecimal or binary. According to the data flow configuration, the processing flow of the data to be processed is determined. In the data processing process, a custom function is required. The custom function is configured in the path of the JAVA class. The path of the JAVA class is configured in the configuration file. When the custom function needs to be read, the custom function can be dynamically loaded. According to the path of the target JAVA class in the target configuration file, the target custom function can be dynamically loaded. After obtaining the target custom function, the data to be processed is processed according to the data flow configuration in the target configuration file to obtain the target output data. The beneficial effect of this setting is that the input data format is converted through the data source configuration, the custom function configuration is used to load the custom function, and the data flow configuration is used to determine the data processing flow. This simplifies the code writing, clearly divides the steps of data processing, avoids code confusion or loss, and completes the creation and operation of data processing tasks in a configurable manner, thereby improving the efficiency and accuracy of data processing.
[0059] The embodiment of the present invention configures configuration files of different task types, saves the process of writing codes, obtains target input data, determines the target configuration file associated with the target input data, and verifies the target configuration file before processing data according to the target configuration file to determine whether the target configuration file meets the preset requirements. If it meets the requirements, data processing is performed. The problem that the code is prone to errors when data processing is performed through code in the prior art is solved, the manpower development cost is reduced, and errors in the data processing flow are avoided by verifying the configuration file, the correctness of the configuration file is improved, and the efficiency of data processing is improved.
[0060] Embodiment 3
[0061] Figure 3This is a structural block diagram of a data processing device provided in the third embodiment of the present invention, which can execute a data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 3 As shown, the device specifically includes:
[0062] An input data acquisition module 301 is used to acquire target input data in response to a data processing instruction;
[0063] The target file determination module 302 determines the target configuration file associated with the target input data according to the association relationship between the input data and the configuration file;
[0064] The target file verification module 303 is used to verify the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements;
[0065] The output data acquisition module 304 is used to process the target input data according to the target configuration file to obtain the target output data if it meets the requirements.
[0066] Optionally, the device further comprises:
[0067] A target type determination module, used to determine the target task type according to the data processing instruction before obtaining the target input data in response to the data processing instruction;
[0068] The target ID determination module is used to determine the target task ID of the target task type according to the association relationship between the task type and the task ID.
[0069] Optionally, the target file determination module 302 is specifically configured to:
[0070] According to the association relationship between the task ID and the configuration file, a target configuration file associated with the target task ID is determined.
[0071] Optionally, the target file verification module 303 includes:
[0072] A processing flow parsing unit is used to parse the target processing flow in the target configuration file to obtain intermediate input data and intermediate output data of each step in the target processing flow;
[0073] The data relationship determination unit is used to determine whether the target configuration file meets the preset requirements according to the data dependency relationship between the intermediate input data and the intermediate output data between different steps.
[0074] Optionally, the data relationship determination unit is specifically used to:
[0075] Determining whether acquisition of intermediate input data of a first step in a target processing flow depends on intermediate output data of a second step;
[0076] If not, it is determined that the target profile meets the preset requirements.
[0077] Optionally, the first step is located before the second step in the target processing flow.
[0078] Optionally, the target file verification module 303 is further specifically used for:
[0079] According to the preset input data, the target processing flow in the target configuration file is run to determine whether the preset output data is obtained;
[0080] If the preset output data is obtained, it is determined that the target configuration file meets the preset requirements.
[0081] Optionally, the device further comprises:
[0082] The data persistence module is used to process the target input data according to the target configuration file to obtain the target output data, and then store the target output data according to the target output data persistence strategy in the target configuration file.
[0083] Optionally, the device further comprises:
[0084] The configuration file generation module is used to generate configuration files corresponding to different task types according to the data processing flow of different task types before obtaining the target input data in response to the data processing instruction; wherein the configuration file includes data source configuration, custom function configuration and data flow configuration.
[0085] Optionally, data source configuration refers to the configuration of data information in the data processing flow; the data information at least includes data type, data access method and data storage format.
[0086] Optionally, the custom function configuration is stored in the JAVA class path, and the JAVA class path is configured in the configuration file.
[0087] Optionally, data flow configuration refers to step configuration of the data processing flow, and the step configuration at least includes data reading configuration, data processing configuration and data persistence configuration.
[0088] Optionally, the output data acquisition module 304 is specifically used for:
[0089] According to the data source configuration in the target configuration file, the target input data is converted to obtain the data to be processed;
[0090] Get the path of the target JAVA class in the target configuration file;
[0091] Dynamically load the target custom function according to the path of the target JAVA class;
[0092] According to the data flow configuration and based on the target custom function, the data to be processed is processed to obtain the target output data.
[0093] The embodiment of the present invention obtains target input data, determines a target configuration file associated with the target input data, and verifies the target configuration file before processing data according to the target configuration file to determine whether the target configuration file meets the preset requirements. If so, data processing is performed. The problem that the code is prone to errors when data processing is performed through code in the prior art is solved, the manpower development cost is reduced, and errors in the data processing flow are avoided by verifying the configuration file, the correctness of the configuration file is improved, and the efficiency of data processing is improved.
[0094] Embodiment 4
[0095] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary computer device 400 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The computer device 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0096] like Figure 4 As shown, the computer device 400 is in the form of a general computing device. The components of the computer device 400 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).
[0097] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.
[0098] Computer device 400 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 400, including volatile and non-volatile media, removable and non-removable media.
[0099] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Computer device 400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data medium interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.
[0100] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.
[0101] The computer device 400 may also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 400, and / or communicate with any device that enables the computer device 400 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via input / output (I / O) interface 411. Furthermore, the computer device 400 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via network adapter 412. As shown, the network adapter 412 communicates with other modules of the computer device 400 via bus 403. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0102] The processing unit 401 executes various functional applications and data processing by running the program stored in the system memory 402, for example, implementing a data processing method provided by an embodiment of the present invention, including:
[0103] Responding to the data processing instruction, acquiring target input data;
[0104] Determine a target configuration file associated with the target input data according to the association relationship between the input data and the configuration file;
[0105] Verify the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements;
[0106] If it meets the requirements, the target input data is processed according to the target configuration file to obtain the target output data.
[0107] Embodiment 5
[0108] Embodiment 5 of the present invention further provides a storage medium containing computer executable instructions, on which a computer program is stored. When the program is executed by a processor, a data processing method provided in the embodiment of the present invention is implemented, including:
[0109] Responding to the data processing instruction, acquiring target input data;
[0110] Determine a target configuration file associated with the target input data according to the association relationship between the input data and the configuration file;
[0111] Verify the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements;
[0112] If it meets the requirements, the target input data is processed according to the target configuration file to obtain the target output data.
[0113] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0114] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0115] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0116] Computer program code for performing the operation of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0117] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A method for data processing, characterized in that: include: In response to the data processing instruction, target input data is obtained, wherein the target input data includes a user unique identifier, a date and time, and product details, and the data processing instruction is a user demand instruction, and the data processing instruction includes a data source for obtaining the target input data and a target task type; Determining a target configuration file associated with the target input data according to an association relationship between the input data and the configuration file; Verifying the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements; If they meet the requirements, the target input data is processed according to the target configuration file to obtain target output data, wherein the processing of the target input data refers to summing, averaging or calculating the rate of return of the target input data; The step of verifying the target processing flow in the target configuration file to determine whether the target configuration file meets preset requirements includes: Parse the target processing flow in the target configuration file to obtain intermediate input data and intermediate output data of each step in the target processing flow; Determining whether the target configuration file meets preset requirements according to the data dependency relationship between the intermediate input data and the intermediate output data between different steps; The step of determining whether the target configuration file meets preset requirements according to the data dependency relationship between the intermediate input data and the intermediate output data between different steps includes: Determine whether the acquisition of the intermediate input data of the first step in the target processing flow depends on the intermediate output data of the second step, wherein determining the data dependency relationship between the intermediate input data and the intermediate output data of different steps refers to determining whether the intermediate input data of one data processing step is derived from the intermediate output data of other data processing steps; If not, determining that the target configuration file meets the preset requirements; Wherein, the first step is located before the second step in the target processing flow; The step of verifying the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements further includes: According to the preset input data, the target processing flow in the target configuration file is run to determine whether the preset output data is obtained, wherein the target configuration file also includes a calculation formula corresponding to the target processing flow; If the preset output data is obtained, it is determined that the target configuration file meets the preset requirements; Wherein, after processing the target input data according to the target configuration file to obtain the target output data, the method further includes: The target output data is stored according to the target output data persistence strategy in the target configuration file.
2. The method according to claim 1, characterized in that Before obtaining target input data in response to the data processing instruction, the method further includes: Determining a target task type according to the data processing instruction; According to the association relationship between the task type and the task ID, the target task ID of the target task type is determined.
3. The method according to claim 2, characterized in that Determining a target configuration file associated with the target input data according to the association relationship between the input data and the configuration file includes: According to the association relationship between the task ID and the configuration file, a target configuration file associated with the target task ID is determined.
4. The method according to claim 1, characterized in that: Before obtaining target input data in response to the data processing instruction, the method further includes: According to the data processing flow of different task types, configuration files corresponding to different task types are generated; wherein the configuration files include data source configuration, user-defined function configuration and data flow configuration.
5. The method according to claim 4, characterized in that The data source configuration refers to the configuration of data information in the data processing flow; the data information at least includes data type, data access method and data storage format.
6. The method according to claim 4, characterized in that The custom function configuration is stored in the path of the JAVA class, and the path of the JAVA class is configured in the configuration file.
7. The method according to claim 4, characterized in that The data flow configuration refers to the step configuration of the data processing flow, and the step configuration at least includes data reading configuration, data processing configuration and data persistence configuration.
8. The method according to claim 4, characterized in that Processing the target input data according to the target configuration file to obtain target output data includes: According to the data source configuration in the target configuration file, the target input data is converted to obtain the data to be processed; Get the path of the target JAVA class in the target configuration file; Dynamically load the target custom function according to the path of the target JAVA class; According to the data flow configuration and based on the target custom function, the data to be processed is processed to obtain the target output data.
9. A data processing device, characterized in that: include: An input data acquisition module, used to acquire target input data in response to a data processing instruction, wherein the target input data includes a user unique identifier, a date and time, and product details, and the data processing instruction is a user demand instruction, and the data processing instruction includes a data source for acquiring the target input data and a target task type; A target file determination module determines a target configuration file associated with the target input data according to an association relationship between the input data and the configuration file; A target file verification module is used to verify the target processing flow in the target configuration file to determine whether the target configuration file meets the preset requirements; an output data acquisition module, for processing the target input data according to the target configuration file to obtain target output data if the target input data meets the requirements, wherein processing the target input data refers to summing, averaging or calculating the rate of return of the target input data; The target file verification module includes: A processing flow parsing unit is used to parse the target processing flow in the target configuration file to obtain intermediate input data and intermediate output data of each step in the target processing flow; A data relationship determination unit, configured to determine whether the target configuration file meets preset requirements according to the data dependency relationship between the intermediate input data and the intermediate output data between different steps; The data relationship determination unit is configured to: Determine whether the acquisition of the intermediate input data of the first step in the target processing flow depends on the intermediate output data of the second step, wherein determining the data dependency relationship between the intermediate input data and the intermediate output data of different steps refers to determining whether the intermediate input data of one data processing step is derived from the intermediate output data of other data processing steps; If not, determining that the target configuration file meets the preset requirements; Wherein, the first step is located before the second step in the target processing flow; The target file verification module is further configured to: According to the preset input data, the target processing flow in the target configuration file is run to determine whether the preset output data is obtained, wherein the target configuration file also includes a calculation formula corresponding to the target processing flow; If the preset output data is obtained, it is determined that the target configuration file meets the preset requirements; The data persistence module is used to store the target output data according to the target output data persistence strategy in the target configuration file after processing the target input data according to the target configuration file to obtain the target output data.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the data processing method as described in any one of claims 1-8 is implemented.
11. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the data processing method as described in any one of claims 1 to 8 when executed by a computer processor.
