Data processing method, device, computer equipment and storage medium

Abstract

The present application is applicable to the field of big data technology, and provides a data processing method, device, computer equipment and storage medium, wherein a data processing method is provided, wherein the first script file is used to describe the logical process of a first data processing tool performing data processing on the data to be processed by obtaining the data to be processed and a first script file; according to the preset search condition of the regular expression, the first target field that meets the regular expression in the first script file is searched; according to the preset replacement condition of the regular expression, the first target field is replaced with the second target field to obtain a second script file, and the second script file is used to describe the logical process of a second data processing tool performing data processing on the data to be processed; the second data processing tool is called to execute the second script file, and the data to be processed is processed to obtain a data processing result. Thus, the data to be processed is processed by the new engine, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] With the development of big data technology, the amount of data to be processed in various business scenarios is increasing, and the business scenarios are becoming more and more complex, so the performance requirements for computing engines are getting higher and higher. At present, traditional data processing tools use Java to encapsulate all data processing functions into JAR (Java Archive) packages. With the development of big data, some data processing functions in JAR packages are no longer applicable to the current data processing environment. However, due to the high degree of coupling between functions in JAR packages, data processing tools are very complex and have very poor maintainability, making it difficult to remove useless functions in JAR packages, resulting in reduced data processing performance of data processing tools. Summary of the invention

[0003] In view of this, embodiments of the present application provide a data processing method, apparatus, computer device, and computer-readable storage medium to solve the problem of low data processing performance in data processing tools in the prior art.

[0004] A first aspect of an embodiment of the present application provides a data processing method, including:

[0005] Acquire the data to be processed and a first script file, where the first script file is used to describe the logical process of data processing by the first data processing tool on the data to be processed;

[0006] According to a preset regular expression search condition, searching for a first target field in the first script file that meets the regular expression;

[0007] According to a replacement condition of a preset regular expression, the first target field is replaced with the second target field to obtain a second script file, where the second script file is used to describe a logical process of data processing performed by the second data processing tool on the data to be processed;

[0008] The second data processing tool is called to execute the second script file, and data processing is performed on the data to be processed to obtain a data processing result.

[0009] A second aspect of an embodiment of the present application provides a data processing device, including:

[0010] An acquisition module, used for acquiring data to be processed and a first script file, wherein the first script file is used for describing a logical process of data processing by a first data processing tool on the data to be processed;

[0011] A search module, used to search for a first target field in a first script file that meets the regular expression according to a preset search condition of the regular expression;

[0012] A replacement module, used for replacing the first target field with the second target field according to a replacement condition of a preset regular expression, to obtain a second script file, wherein the second script file is used for describing a logical process of data processing performed by the second data processing tool on the data to be processed;

[0013] The execution module is used to call the second data processing tool to execute the second script file, perform data processing on the data to be processed, and obtain a data processing result.

[0014] A third aspect of an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the computer device, wherein the processor implements the steps of the data processing method provided by the first solution when executing the computer program.

[0015] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the data processing method provided by the first solution are implemented.

[0016] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device implements each step of the data processing method provided by the first solution.

[0017] The data processing method, device, computer equipment and computer-readable storage medium provided by the embodiments of the present application have the following beneficial effects:

[0018] By obtaining the data to be processed and the first script file, the processing logic suitable for processing the data to be processed in the original computing engine is extracted; according to the preset regular expression search condition, the first target field that meets the regular expression in the first script file is searched, and according to the preset regular expression replacement condition, the first target field is replaced with the second target field to obtain the second script file, so as to convert the processing logic of the original engine into the processing logic of the new engine, so that useful processing logic can be extracted, thereby avoiding the influence of useless processing logic on data processing efficiency; and the new engine can be an oracle engine, and there is no high coupling between the extracted processing logics; finally, the second data processing tool is called to execute the second script file, and the data to be processed is processed to obtain the data processing result, so that the new engine can process the data to be processed and improve the data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 It is a flow chart of an implementation of a data processing method provided in an embodiment of the present application;

[0021] Figure 2 is a flow chart of an implementation of a data processing method provided by another embodiment of the present application;

[0022] Figure 3 is a flow chart of an implementation of a data processing method provided in yet another embodiment of the present application;

[0023] Figure 4 is a structural block diagram of a data processing device provided in an embodiment of the present application;

[0024] Figure 5 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0026] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0027] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0029] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0031] As described in the background technology, for example, a credit card points processing tool accumulates points for consumption records in a specified scenario, which requires the points processing tool to identify the consumption scenario corresponding to the consumption record. As society develops, consumption scenarios become more and more diverse, and the processing rules of the points processing tool also need to be updated. However, the traditional points processing tool is a Java engine. Due to the limitations of the Java engine itself, the coupling degree between the functions of the Java engine is high, making it difficult to remove useless functions in the Java engine and difficult to add new functions. Ultimately, the Java engine not only cannot achieve performance optimization, but the high coupling degree will also lead to reduced processing performance.

[0032] Therefore, an embodiment of the present application proposes a data processing method that can be executed by a computer device to achieve the migration of useful processing rules in a first data processing tool to a second data processing tool, such as migrating useful processing rules in a Java engine to an Oracle engine, so as to remove useless processing rules and decouple the processing rules, thereby improving the processing performance of the points system.

[0033] As an example and not a limitation, computer devices include mobile phones, tablet computers, wearable devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks or personal digital assistants (PDAs), cloud servers, physical servers and other computer devices. The embodiments of the present application do not impose any restrictions on the specific types of computer devices.

[0034] See also Figure 1 , Figure 1 A flow chart of an implementation of a data processing method provided in an embodiment of the present application is shown.

[0035] S101, obtaining data to be processed and a first script file, where the first script file is used to describe a logical process of a first data processing tool performing data processing on the data to be processed;

[0036] In the above S101, the above data to be processed can be obtained from the data input table of the second data processing tool, which can be credit card consumption details (such as records of purchasing goods, living expenses, etc.), phone card consumption details, membership card consumption details, etc. In this embodiment, the data processing tool for the points system scheduling is changed from the original first data processing tool to the second data processing tool, and the input parameters of the second data processing tool are changed from the original data input table to the new data input table, thereby changing the data acquisition path, so that the data to be processed is obtained from the original data input table of the first data processing tool, and is changed to the data input table of the second data processing tool to obtain the data to be processed, thereby realizing the migration of the acquisition path of the data to be processed.

[0037] The first data processing tool is the original data processing tool scheduled by the points system, such as the Java engine that encapsulates the data processing function as a JAR package in the background technology, and the first script file is a logic code file for processing the data to be processed. It should be understood that since the data processing logic of the data processing function is different, the corresponding first script file can be pre-configured according to each data processing logic, that is, there can be multiple first script files.

[0038] Optionally, after obtaining the data to be processed, one or more first script files are extracted from the JAR package of the Java engine according to a preset extraction condition, so that only the first script file applicable to the current data processing environment is obtained, so that the first script file can be used to process the data to be processed later, and other script files in the JAR package that are not applicable to the current data processing environment can be left unprocessed. Exemplarily, the extraction condition can be to extract the first script file corresponding to the preset function name based on the preset function name.

[0039] S102, searching for a first target field in a first script file that meets the regular expression according to a preset regular expression search condition;

[0040] In the above S102, since JAVA encapsulates the jar package, the jar package contains a large number of useless functions, and the coupling degree of the functions in the jar package is high, so the existing integral calculation engine is very complex and the maintainability is very poor, so the first script file currently needed is converted into a second script file based on the oracle engine, which can remove the first script file that is not needed at present, and use oracle as the calculation engine to reduce the coupling degree between rules.

[0041] The above regular expression is a regular expression for searching and replacing a target text segment in a certain text, which includes but is not limited to a search rule for finding a target text segment from a certain text or a replacement rule for replacing a target text segment with another text segment. In this embodiment, the field of the script segment in the first script file is found through the search condition of the regular expression, so that the field is subsequently replaced with a target field that can be recognized and run by the Oracle engine, thereby realizing rule conversion between different engines.

[0042] The above search conditions are search rules for finding a target text segment from a certain text. As an example but not a limitation, the search conditions can be to find a text segment containing a preset field in a certain text, to find a text segment in a preset form in a certain text, etc. For example, for ease of description, "123456789" is used as the script segment of the first script file, and according to the preset regular expression used to find a group of consecutive three-digit numbers in the first script file, the search condition "(?'test'\d)(?=(\d{3})+(?!\d))" matches the three groups of consecutive three-digit numbers "123", "456", "789" in the script segment.

[0043] S103, according to a replacement condition of a preset regular expression, the first target field is replaced with a second target field to obtain a second script file, where the second script file is used to describe a logical process of data processing performed by the second data processing tool on the data to be processed;

[0044] In the above S103, the above second data processing tool can be an oracle engine. Compared with the java engine, the oracle engine can support big data calculation more efficiently, and the internal coupling degree of the oracle engine is low. The above replacement condition is a replacement rule for replacing the target text segment with another text segment. In this embodiment, according to a preset rule table, the computer code corresponding to the first script file is converted into the computer code corresponding to the second script file, wherein the rule table contains the correspondence between the first script file and the second script file, or contains the correspondence between the first field of the program language (such as java) of the first script file and the second field of the program language (such as oracle) of the second script file.

[0045] Exemplarily, based on the correspondence between the first script file and the second script file, the computer code corresponding to the first script file can be completely replaced with the computer code corresponding to the second script file; or based on the correspondence between the java field and the oracle field, the java field of the computer code corresponding to the first script file can be replaced with the oracle field of the computer code corresponding to the second script file to obtain the second script file.

[0046] Optionally, the above-mentioned preset rule table is a rule table of an oracle engine, in which the function name to be converted corresponding to the first script file of the java engine is added as a rule conversion identification, and a conversion condition for converting the first script file into a second script file is added in the rule table. Thus, after the oracle engine extracts the first script file, it identifies whether the function name corresponding to the first script file needs to be converted. If conversion is required, it means that the first script file is a rule file applicable to the current data processing environment, and is converted to the second script file based on the above conversion condition; if conversion is not required, it means that the first script file is not a rule file applicable to the current data processing environment, and the next first script file is extracted.

[0047] Furthermore, based on actual needs, rule conversion identification conditions and conversion conditions can be added or deleted in the rule table to identify whether the first script file needs rule conversion based on the rule conversion identification conditions, and perform rule conversion on the first script file that needs rule conversion based on the conversion conditions, thereby realizing the addition or deletion of rules without affecting the use of other rules, thereby achieving decoupling between rules.

[0048] S104, calling a second data processing tool to execute a second script file, performing data processing on the data to be processed, and obtaining a data processing result.

[0049] In the above S104, the above data processing includes but is not limited to data classification, data matching and data calculation. In this embodiment, it is determined whether the data to be processed meets the preset requirements according to the second script file. If it meets the preset requirements, all the data to be processed that meet the preset requirements are calculated to obtain the calculation result.

[0050] For example, if a user uses a credit card to spend 100 yuan in a supermarket, the consumption details are written into the data input table of the Oracle engine, and a first script file containing functions for determining whether the consumption details are consumed in a specified scenario, for matching the points value of the consumption details consumed in the specified scenario, and for accumulating the points value is obtained. The first script file is converted into a second script file that can be run by the Oracle engine. Based on the second script file, it is determined whether the consumption details are consumed in the specified scenario. If so, the consumption details are classified as meeting the requirements to achieve data classification of the consumption details; point values ​​are allocated to consumption details that meet the requirements (such as 100 yuan = 1 point value) to achieve data matching of the consumption details; the point value is added to the total point value of the credit card account to obtain the point accumulation result to achieve accumulation of the point value.

[0051] From the above, it can be seen that the data processing method provided in this embodiment converts the integral calculation rules of the Java engine into the integral calculation rules processed by the Oracle engine, and adopts the integral calculation rules of the Oracle engine to perform integral calculation, thereby removing a large number of useless functions in the Java engine, making the reconstructed engine more concise and less coupled, reducing the complexity of the integral calculation engine and improving maintainability.

[0052] In one possible implementation, Figure 1 On the basis of the embodiment shown, the above S101 also includes S1011 to S1013:

[0053] S1011, obtaining business data in a data input table of a second data processing tool;

[0054] S1012, calculating the key values ​​corresponding to the plurality of preset features of the business data, and obtaining a plurality of data to be processed corresponding to the plurality of key values;

[0055] In the above S1011 and S1012, the data input table is the input parameter of the second data processing tool, and the business data can be the consumption details of multiple credit cards. Since most of the current points systems adopt distributed systems and the data is stored in multiple devices in a dispersed manner, data sharding is used to determine the distribution of data on multiple storage devices to achieve uniform distribution, load balancing, and small data migration. In this embodiment, data sharding can be achieved by calculating the distribution of the key according to the key value of the data or the hash value of the key value, where the key value is the key value.

[0056] Exemplarily, the Key values ​​corresponding to multiple preset features of the business data are calculated respectively. For example, a piece of business data is a consumption detail of purchasing Coke for 3 yuan at Walmart on May 1, 2020. The preset features may be consumption time (May 1, 2020), consumption scene (Walmart), consumption type (Coke), and consumption amount (3 yuan). According to the preset Key values ​​corresponding to the preset features, multiple Key values ​​of the business data are obtained. For example, if the preset key value of the consumption time is 1, then one of the key values ​​of the consumption details can be 1.

[0057] S1013, storing the to-be-processed data whose key value is within a preset key value range in a preset storage space corresponding to the preset key range.

[0058] In the above S1013, after calculating the Key value, the data to be processed can be stored in a preset storage space by dividing the number segments, taking the modulus, retrieving the table, and using the consistent hash value, thereby obtaining the data to be processed distributed on multiple devices, so that multiple devices can call the oracle engine to concurrently process the data to be processed, thereby improving the processing efficiency.

[0059] See also Figure 2 ,exist Figure 1 Based on the examples, Figure 2 is a flowchart of a data processing method provided by another embodiment of the present application. It should be understood that Figure 1 The same steps as those in the embodiment are not repeated here. The first script file of the data processing method provided in this embodiment includes a first sub-query script. The specific process of step S102 and step S103 is as follows: Figure 2 As shown, specifically, step S102 includes steps S201 and S202, and step S103 specifically includes steps S203 and S204. The details are as follows:

[0060] S201, identifying a first sub-query script in a first script file according to a preset script conversion identification function;

[0061] S202, searching for a plurality of first target fields that meet the regular expression in the first subquery script according to the search condition of the regular expression;

[0062] In the above S201 and S202, the subquery function is a function of the query condition of another function. In the SQL language, a SELECT-FROM-WHERE statement is called a query block. When obtaining the answer to a query requires multiple steps of operation, a query must first be created to determine the value that the user does not know but is contained in the database. The query block that is nested in the condition of the WHERE clause or HAVING phrase of another query block is called a subquery or inner query.

[0063] Since JAVA encapsulates a jar package, the jar package contains a large number of useless functions, and the coupling degree of the functions in the jar package is high, only the first script file currently needed is converted into a second script file based on the oracle engine, so that the first script file that is not currently needed can be removed, and oracle is used as the calculation engine to reduce the coupling degree between rules. This embodiment uses regular expressions to find the fields of the function script, so that the fields can be replaced with target fields that can be recognized and run by the oracle engine in the future, thereby realizing the conversion of the script file.

[0064] In some cases, there are custom functions in the business rule table of the Oracle engine that need to be obtained by linking other tables using a select statement. The select statement is stored in another parameter table. Therefore, the custom function (i.e., the first subquery function) is converted into a corresponding select statement through substitution. For example, the first subquery function of this embodiment includes a substr function, a datepart function, or a dateadd function, wherein the substr function, the datepart function, and the dateadd function are respectively used to intercept, calculate, and provide engine screening for the data to be processed in the specified scenario.

[0065] Optionally, in the rule conversion job table for converting the Java engine to the Oracle engine, the conversion rules and conversion identification of the substr function, the datepart function and the dateadd function are added, wherein the conversion rules are converted according to the function conversion process in the business rule table (such as regular expressions), and the conversion identification is to add the function name to be converted at a preset position. Finally, in actual application, based on the above conversion rules and conversion identification, the Java engine rules are converted to Oracle engine rules. Optionally, in this process, the conversion of the subquery in the first script file can also be included, and the converted subquery is saved in the preset business rule table.

[0066] S203, according to the replacement condition of the regular expression, replacing multiple first target fields that meet the regular expression in the first subquery script with second target fields that can be recognized by the second data processing tool;

[0067] S204: Assemble the multiple second target fields into a second sub-query script to obtain a second script file including the second sub-query script.

[0068] In the above S203 and S204, the Oracle engine uses regular expressions to filter data through the where condition, and a single filtering rule may contain multiple conditions (i.e., multiple first sub-query functions corresponding to sub-query conditions), so the select functions between different conditions are assembled into query SQL statements (i.e., second sub-query functions) using "and" through dynamic SQL. During the application process, the Oracle engine converts and identifies the pre-configured sub-query function, takes out the corresponding first sub-query function in the first script file, converts the first sub-query function into the second sub-query function, and then matches the data to be processed according to the second sub-query function.

[0069] See also Figure 3 ,exist Figure 1 Based on the embodiment shown, Figure 3 A flowchart of a data processing method provided by another embodiment of the present application is shown. It should be understood that Figure 1 The same steps are not repeated here. The specific process of step S104 of the data processing method provided in this embodiment is as follows Figure 3 As shown, specifically, step S104 includes S301 to S303. The details are as follows:

[0070] S301, calling a second data processing tool to execute a second script file, and determining whether the data to be processed meets the data processing conditions in the second script file;

[0071] In the above S301, based on the above second script file, the data to be processed is filtered by rules by data matching to remove the data to be processed that does not meet the conditions. For example, the points system only accumulates the consumption points for daily shopping, then the second script file of the oracle engine determines whether the data to be processed is the consumption details of daily shopping. If the data to be processed is the consumption details of Wal-Mart, it means that it meets the conditions, and if the data to be processed is the consumption details of a travel agency, it means that it does not meet the conditions.

[0072] Optionally, the data to be processed is further verified, for example, the consumption time of the data to be processed is verified to identify whether the consumption time corresponding to the data to be processed is within a specified time period. If so, it means that the condition is met, if not, it means that the condition is not met.

[0073] S302, adding the data to be processed that meets the data processing conditions to a preset intermediate table;

[0074] In the above S302, since there may be multiple data to be processed at the same time, for example, a user pays for multiple items in an online shopping cart, and different items have different merchants, that is, there are multiple payees, there may be multiple data to be processed that meet the requirements at the same time. Therefore, in order to avoid the existence of multiple data to be processed at the same time, which may cause errors in subsequent operations of the data to be processed, the data to be processed is saved in a preset intermediate table. Optionally, the data to be processed that meet the requirements can be inserted into the preset intermediate table in the form of an insert statement.

[0075] S303, performing data operation on the data to be processed in the preset intermediate table to obtain a data processing result.

[0076] In the above S303, the final score of the last statistics and all the data in the current preset intermediate table are accumulated to obtain the final score of this time. It should be understood that the preset intermediate table may include the final score of the last statistics, or may not include the final score of the last statistics and go to the database to obtain the final score of the last statistics for accumulation.

[0077] Optionally, it is also possible to verify whether there is new data to be processed in the preset intermediate table. If so, it is determined whether there are duplicate processing results in the preset intermediate table based on the correspondence between the data insertion record and the data to be processed, thereby further avoiding repeated calculation of integrals.

[0078] In a possible implementation manner, before S302, the step further includes:

[0079] S3021, perform data cleaning on the preset intermediate table according to the preset data cleaning script.

[0080] In the above S3021, since the to-be-processed data previously existing in the preset intermediate table affects the subsequent integral calculation, it is necessary to clear the to-be-processed data previously existing in the preset intermediate table. It should be understood that the preset intermediate table can be cleaned multiple times, that is, the preset intermediate table is cleaned each time before S302 is executed, or the preset intermediate table is cleaned multiple times before S302 is executed each time.

[0081] In all embodiments of the present application, the corresponding data processing results are obtained based on the second data processing tool. Specifically, the data processing results are obtained by the second data processing tool through data classification and data calculation. Uploading the data processing results to the blockchain can ensure its security and fairness and transparency to users. The user device can download the processing results from the blockchain to verify whether the processing results have been tampered with. The blockchain referred to in this example is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database. It is a string of data blocks generated by cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain may include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0082] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0083] See also Figure 4 , Figure 4 is a structural block diagram of a data processing device provided in an embodiment of the present application. In this embodiment, the mobile terminal includes various units for executing Figures 1 to 3 For details, please refer to the steps in the corresponding embodiment. Figures 1 to 3 as well as Figures 1 to 3 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 4 , the data processing device 40 comprises:

[0084] An acquisition module 401 is used to acquire data to be processed and a first script file, wherein the first script file is used to describe a logical process of a first data processing tool performing data processing on the data to be processed;

[0085] A search module 402 is used to search for a first target field in the first script file that meets the regular expression according to a preset search condition of the regular expression;

[0086] The replacement module 403 is used to replace the first target field with the second target field according to the replacement condition of the preset regular expression, and obtain a second script file, wherein the second script file is used to describe the logical process of the second data processing tool performing data processing on the data to be processed.

[0087] The execution module 404 is used to call the second data processing tool to execute the second script file, perform data processing on the data to be processed, and obtain a data processing result.

[0088] It should be understood that Figure 4 In the structural block diagram of the data processing device shown in FIG. Figures 1 to 3 The steps in the corresponding embodiments, and for Figures 1 to 3 Each step in the corresponding embodiment has been explained in detail in the above embodiment. Figures 1 to 3 as well as Figures 1 to 3 The relevant descriptions in the corresponding embodiments are not repeated here.

[0089] Figure 5 is a structural block diagram of a computer device provided by another embodiment of the present application. Figure 5 As shown, the computer device 50 of this embodiment includes: a processor 51, a memory 52, and a computer program 53 stored in the memory 52 and executable on the processor 51, such as a program of a data processing method. When the processor 51 executes the computer program 53, the steps in each embodiment of the above-mentioned data processing method are implemented, such as Figure 1 S101 to S104 shown, or Figure 2 and Figure 3 Alternatively, the processor 51 implements the above when executing the computer program 53 Figure 4 The functions of each unit / module in the corresponding embodiment are, for example, Figure 4 For details on the functions of modules 401 to 404, please refer to Figure 4 The relevant descriptions in the corresponding embodiments are not repeated here.

[0090] Exemplarily, the computer program 53 may be divided into one or more units / modules, which are stored in the memory 52 and executed by the processor 51 to complete the present application. The one or more units / modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 53 in the computer 50. For example, the computer program 53 may be divided into an acquisition module, a search module, a replacement module, and an execution module, and the specific functions of each unit / module are as described above.

[0091] The turntable device may include, but is not limited to, a processor 51 and a memory 52. ​​Those skilled in the art will appreciate that Figure 5 It is only an example of the computer device 50 and does not constitute a limitation of the computer device 50. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the turntable device may also include input and output devices, network access devices, buses, etc.

[0092] The processor 51 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0093] The memory 52 may be an internal storage unit of the computer device 50, such as a hard disk or memory of the computer device 50. The memory 52 may also be an external storage device of the computer device 50, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 50. Further, the memory 52 may also include both an internal storage unit of the computer device 50 and an external storage device. The memory 52 is used to store the computer program and other programs and data required by the turntable device. The memory 52 may also be used to temporarily store data that has been output or is to be output.

[0094] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire data to be processed and a first script file, where the first script file is used to describe a logical process of a first data processing tool performing data processing on the data to be processed; According to a preset regular expression search condition, searching for a first target field in the first script file that meets the regular expression; According to a replacement condition of a preset regular expression, the first target field is replaced with a second target field to obtain a second script file, wherein the second script file is used to describe a logical process of data processing by a second data processing tool on the data to be processed; Calling the second data processing tool to execute the second script file, performing data processing on the data to be processed, and obtaining a data processing result; The first script file includes a first sub-query script. Accordingly, searching the first target field in the first script file that meets the regular expression according to the preset regular expression search condition includes: According to a preset script conversion recognition function, identifying the first sub-query script in the first script file; According to the search condition of the regular expression, searching for a plurality of the first target fields in the first subquery script that conform to the regular expression; The replacing condition of the preset regular expression, replacing the first target field with the second target field to obtain a second script file, includes: According to the replacement condition of the regular expression, replacing the plurality of first target fields in the first subquery script that conform to the regular expression with the second target fields that can be recognized by the second data processing tool; The plurality of the second target fields are assembled into a second sub-query script to obtain the second script file including the second sub-query script.

2. The data processing method according to claim 1, characterized in that: Before obtaining the data to be processed and the first script file, the method further includes: Acquire business data in a data input table of the second data processing tool; Calculate the key values ​​corresponding to the plurality of preset features of the business data, and obtain the plurality of data to be processed corresponding to the plurality of key values; The to-be-processed data whose key value is within a preset key value range is stored in a preset storage space corresponding to the preset key range.

3. The data processing method according to claim 1, characterized in that: The calling of the second data processing tool to execute the second script file, performing data processing on the data to be processed, and obtaining a data processing result includes: Calling the second data processing tool to execute the second script file, and determining whether the data to be processed meets the data processing conditions in the second script file; Add the data to be processed that meets the data processing conditions to the preset intermediate table; Perform data operations on the data to be processed in the preset intermediate table to obtain the data processing result.

4. The data processing method according to claim 3, characterized in that: Before adding the data to be processed that meets the data processing conditions into the preset intermediate table, the method further includes: According to the preset data cleaning script, data cleaning is performed on the preset intermediate table.

5. The data processing method according to any one of claims 1 to 4, characterized in that: The data processing result is stored in a preset blockchain.

6. A data processing device, characterized in that: include: An acquisition module, used for acquiring data to be processed and a first script file, wherein the first script file is used for describing a logical process of data processing by a first data processing tool on the data to be processed; A search module, used for searching the first target field in the first script file that meets the regular expression according to a preset search condition of the regular expression; A replacement module, used for replacing the first target field with a second target field according to a replacement condition of a preset regular expression, to obtain a second script file, wherein the second script file is used for describing a logical process of data processing performed by a second data processing tool on the data to be processed; An execution module, used for calling the second data processing tool to execute the second script file, performing data processing on the data to be processed, and obtaining a data processing result; The first script file includes a first sub-query script, and the search module is specifically used for: According to a preset script conversion recognition function, identifying the first sub-query script in the first script file; According to the search condition of the regular expression, searching for a plurality of the first target fields in the first subquery script that conform to the regular expression; The replacing condition of the preset regular expression is used to replace the first target field with the second target field to obtain a second script file, including: According to the replacement condition of the regular expression, replacing the plurality of first target fields in the first subquery script that conform to the regular expression with the second target fields that can be recognized by the second data processing tool; The plurality of the second target fields are assembled into a second sub-query script to obtain the second script file including the second sub-query script.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images