A data processing logic unification method, device and computer equipment

Abstract

Embodiments of the present application disclose a data processing logic unification method and device and a computer device. Embodiments of the present application include: obtaining a script file in a database, parsing the script file to obtain attribute information corresponding to the script file; performing traceability processing on the script file based on the attribute information corresponding to the script file to obtain traceability attribute information corresponding to a field in the script file; comparing data processing logic corresponding to the attribute information and data processing logic of the traceability attribute information to obtain a comparison result; and performing unification processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information based on the comparison result to obtain unification-processed data processing logic. The data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information can be comprehensively and quickly identified to determine whether there is repeated data processing logic in the database, and the repeated data processing logic can be then modified to make the data processing logic of the same field in the database consistent and unified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a data processing logic unification method and device and computer equipment. BACKGROUND

[0002] In the era of big data, there may be thousands of processing scripts in the database. There may be multiple indicators for the same indicator. Due to the errors in the understanding of the indicators by different business personnel, the differences in the analysis and understanding of different demand analysis personnel, the errors in the scripts processing logic given by different data analysis personnel, the differences in the development of different data developers, and the differences in each link such as historical optimization of indicators, all of which may lead to inconsistent data, thereby affecting the business development process. SUMMARY

[0003] The embodiments of the present application provide a data processing logic unification method, device, computer equipment and storage medium, which can comprehensively and quickly identify whether there is repeated data processing logic in the database, and then can transform the repeated data processing logic, so that the data processing logic of the same field in the database is consistent and unified.

[0004] The embodiments of the present application provide a data processing logic unification method, comprising:

[0005] Obtaining a script file in a database, parsing the script file to obtain attribute information corresponding to the script file;

[0006] Based on the attribute information corresponding to the script file, performing traceability processing on the script file to obtain traceability attribute information corresponding to a field in the script file;

[0007] Comparing the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information to obtain a comparison result;

[0008] Based on the comparison result, performing unification processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information to obtain unification processing data processing logic.

[0009] Correspondingly, the embodiments of the present application also provide a data processing logic unification device, comprising:

[0010] A parsing unit is configured to obtain a script file in a database, parse the script file to obtain attribute information corresponding to the script file;

[0011] A traceability unit is configured to perform traceability processing on the script file based on the attribute information corresponding to the script file to obtain traceability attribute information corresponding to a field in the script file;

[0012] a comparison unit, configured to perform comparison processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceable attribute information, to obtain a comparison result;

[0013] a unification unit, configured to perform unification processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceable attribute information based on the comparison result, to obtain data processing logic after unification processing.

[0014] In an embodiment, the parsing unit can include:

[0015] a first parsing subunit, configured to parse the script file to obtain a temporary table corresponding to the script file;

[0016] a second parsing subunit, configured to parse the temporary table to obtain attribute information corresponding to the script file.

[0017] In an embodiment, the trace unit can include:

[0018] a layer trace processing subunit, configured to perform layer trace processing on the script file based on the attribute information corresponding to the script file, to obtain layer trace attribute information of a field in the script file corresponding to a layer to which the script file belongs;

[0019] a library trace processing subunit, configured to perform library trace processing on the script file based on the layer trace attribute information corresponding to the script file, to obtain traceable attribute information of the field in the script file in the database.

[0020] In an embodiment, the layer trace processing subunit can include:

[0021] a parsing module, configured to parse the attribute information corresponding to the script file to obtain field information;

[0022] a relationship determination module, configured to determine a temporary dependency relationship corresponding to the field information;

[0023] a hierarchical recursive processing module, configured to perform hierarchical recursive processing on the field information based on the temporary dependency relationship, to obtain layer trace attribute information of a field in the script file corresponding to a layer to which the script file belongs.

[0024] In an embodiment, the library trace processing subunit can include:

[0025] a traversal module, configured to traverse the layer trace attribute information to obtain a dependency relationship corresponding to the layer trace attribute information;

[0026] The recursive processing module is configured to recursively process the dependency relationship to obtain a source dependency relationship.

[0027] The acquisition module is configured to acquire traceability attribute information corresponding to the source dependency relationship.

[0028] In an embodiment, the comparison unit can include:

[0029] The first standardization processing subunit is configured to standardize the attribute information and the traceability attribute information to obtain standardized attribute information and standardized traceability attribute information.

[0030] The second standardization processing subunit is configured to standardize a data processing logic of the standardized attribute information to obtain a standardized data processing logic corresponding to the standardized attribute information, and standardize a data processing logic of the standardized traceability attribute information to obtain a standardized data processing logic corresponding to the standardized traceability attribute information.

[0031] The information determination subunit is configured to determine matching standardized attribute information and standardized traceability attribute information from the standardized attribute information and the standardized traceability attribute information.

[0032] The comparison subunit is configured to compare the standardized data processing logics corresponding to the matching standardized attribute information and the standardized traceability attribute information to obtain a comparison result.

[0033] In an embodiment, the comparison subunit can include:

[0034] The first comparison processing module is configured to compare the standardized data processing logics corresponding to the matching standardized attribute information and the standardized traceability attribute information.

[0035] The condition determination module is configured to determine a filtering condition corresponding to the standardized attribute information and a filtering condition corresponding to the standardized traceability attribute information when the standardized data processing logic corresponding to the standardized attribute information is the same as the standardized data processing logic corresponding to the standardized traceability attribute information.

[0036] The second comparison processing module is configured to compare the filtering condition corresponding to the standardized attribute information and the filtering condition corresponding to the standardized traceability attribute information to obtain the comparison result.

[0037] In an embodiment, the comparison subunit can further include:

[0038] The description information determination module is configured to determine the description information corresponding to the standardized attribute information and the description information corresponding to the standardized provenance attribute information when the standardized processing logic corresponding to the standardized attribute information is different from the standardized processing logic corresponding to the standardized provenance attribute information.

[0039] The third comparison processing module is configured to compare the description information corresponding to the standardized attribute information and the description information corresponding to the standardized provenance attribute information to obtain a comparison result.

[0040] Correspondingly, the embodiment of the present application further provides a computer device, which comprises a memory and a processor; the memory stores a computer program, and the processor is configured to run the computer program in the memory to execute the data processing logic unification method provided by any of the embodiments of the present application.

[0041] Correspondingly, the embodiment of the present application further provides a storage medium, which stores a computer program; when the computer program is executed by a processor, the data processing logic unification method provided by any of the embodiments of the present application is implemented.

[0042] The embodiment of the present application comprises the following steps: obtaining a script file in a database, parsing the script file to obtain attribute information corresponding to the script file, performing provenance processing on the script file based on the attribute information corresponding to the script file to obtain provenance attribute information corresponding to a field in the script file, comparing the data processing logic corresponding to the attribute information and the data processing logic of the provenance attribute information to obtain a comparison result, and performing unified processing on the data processing logic corresponding to the attribute information and the data processing logic of the provenance attribute information based on the comparison result to obtain the unified processing data processing logic. The repeated data processing logic in the database can be comprehensively and quickly identified, and then the repeated data processing logic can be modified to make the data processing logic of the same field in the database consistent and unified. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0044] Figure 1 is a scene schematic diagram of the data processing logic unification method provided by the embodiment of the present application;

[0045] Figure 2 is a flowchart of the data processing logic unification method provided by the embodiment of the present application;

[0046] Figure 3 is another flow diagram of a data processing logic unification method provided by an embodiment of the present application;

[0047] Figure 4 is a structural diagram of a data processing logic unification apparatus provided by an embodiment of the present application;

[0048] Figure 5 is a structural diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. However, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0050] An embodiment of the present application provides a data processing logic unification method, which can be executed by a data processing logic unification apparatus. The data processing logic unification apparatus can be integrated in a computer device having at least one data processing logic unification hardware. The computer device can include at least one of a computer device, a server, and the like. That is, the data processing logic unification method provided by an embodiment of the present application can be executed by a computer device, and can also be executed by a server, and can also be executed by a terminal and a server capable of mutual communication.

[0051] The terminal can be a smart phone, a tablet computer, a notebook computer, a personal computer (PC), a smart home, a wearable electronic device, a VR / AR device, a vehicle-mounted computer, and the like. The server can be an interworking server or a background server between multiple heterogeneous systems, and can also be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, and the like.

[0052] In an embodiment, as shown in FIG. 1, the data processing logic unification method provided by an embodiment of the present application includes the following steps. Figure 1As shown, the data processing logic unification apparatus can be integrated in the terminal 10 or the server 11 to implement the data processing logic unification method proposed in the embodiments of the present application. Specifically, the server 11 can acquire a script file in a database, parse the script file to obtain attribute information corresponding to the script file, perform provenance processing on the script file based on the attribute information corresponding to the script file to obtain provenance attribute information corresponding to a field in the script file, perform comparison processing on data processing logic corresponding to the attribute information and data processing logic of the provenance attribute information to obtain a comparison result, and perform unification processing on the data processing logic corresponding to the attribute information and the data processing logic of the provenance attribute information based on the comparison result to obtain unification-processed data processing logic.

[0053] The following will be described in detail respectively. It should be noted that the sequence of the following embodiments is not used to limit the preferred sequence of the embodiments.

[0054] The embodiments of the present application will be described from the perspective of a data processing logic unification apparatus, which can be integrated in a computer device.

[0055] As shown in the figure, Figure 2 A data processing logic unification method is provided, and the specific process includes:

[0056] 101. Acquire a script file in a database, parse the script file to obtain attribute information corresponding to the script file.

[0057] The database is a "warehouse for organizing, storing and managing data according to data structure". It is a long-term storage in a computer, an organized, shareable and unified management of a large amount of data collection.

[0058] The script file is an executable file written in a specific descriptive language according to a certain format.

[0059] In an embodiment, multiple script files in a database can be acquired. For example, script files corresponding to each job table in a data warehouse system can be read. The script file can be a processed script.

[0060] In an embodiment, the script file can be parsed to obtain attribute information corresponding to the script file. The attribute information of the script file can be used to describe the content contained in the script file. For example, the attribute information of the script file can include field name, field processing logic, where condition, where condition dependent table and field, join condition, join condition dependent table and field, etc.

[0061] In an embodiment, the temporary table is a table in the database for storing some intermediate result sets. The temporary table is only visible in the current connection, and when the connection is closed, the database automatically deletes the table and releases all space. The temporary table is often used to store information in the database, thereby saving space. Therefore, the script file can be parsed to obtain the temporary table corresponding to the script file. Then, the temporary table is parsed to obtain the attribute information corresponding to the script file.

[0062] Specifically, the step of "parsing the script file to obtain the attribute information corresponding to the script file" can include:

[0063] parsing the script file to obtain the temporary table corresponding to the script file;

[0064] parsing the temporary table to obtain the attribute information corresponding to the script file.

[0065] In an embodiment, each script file can be parsed to obtain all temporary tables of each script file. Then, each temporary table is parsed to obtain the temporary table name, the field name, the field processing logic, the where condition, the where condition dependent table and field, the join condition, and the join condition dependent table and field.

[0066] 102. Based on the attribute information corresponding to the script file, the script file is traced to obtain the trace attribute information corresponding to the field in the script file.

[0067] In an embodiment, in order to ensure that the data processing logic corresponding to the same field in the database is the same, the script file can be traced to obtain the trace attribute information corresponding to the field in the script file.

[0068] Wherein, the script file can be traced to find out that the field in the script file originates from which script file.

[0069] Wherein, the trace attribute information can refer to the final where condition, the final where condition dependent table and field, the final join condition, and the final join condition dependent table and field corresponding to the field in the script file.

[0070] In an embodiment, the step of "based on the attribute information corresponding to the script file, the script file is traced to obtain the trace attribute information corresponding to the field in the script file" can include:

[0071] Based on the attribute information corresponding to the script file, the script file is layer-traced to obtain the layer trace attribute information corresponding to the field in the script file in the layer to which the script file belongs;

[0072] The script file is subjected to library trace processing based on the layer trace attribute information corresponding to the script file, to obtain trace attribute information of the field in the script file in the database.

[0073] In an embodiment, a plurality of layers can be included in the database, and each layer can store a corresponding script file. When the script file is subjected to trace processing, the script file can be subjected to layer trace processing to obtain layer trace attribute information of the field in the script file corresponding to the layer to which the script file belongs. Then, the script file is subjected to library trace processing to obtain trace attribute information of the field in the script file in the database.

[0074] In an embodiment, the step of "subjecting the script file to layer trace processing based on the attribute information corresponding to the script file, to obtain layer trace attribute information of the field in the script file corresponding to the layer to which the script file belongs" can include:

[0075] parsing the attribute information corresponding to the script file to obtain field information;

[0076] determining temporary dependency relationships corresponding to the field information;

[0077] subjecting the field information to hierarchical recursive processing based on the temporary dependency relationships, to obtain layer trace attribute information of the field in the script file corresponding to the layer to which the script file belongs.

[0078] The temporary dependency relationship can refer to a reference relationship of the field information in the script file. For example, if a B field in an A script file references a field in a C script file, the B field and the C script file have a temporary dependency relationship.

[0079] In an embodiment, the attribute information of the script file can be parsed to obtain field information, i.e., to obtain the fields included in the script file. Then, the temporary dependency relationships corresponding to the fields in the script file can be determined. Then, the field information can be subjected to hierarchical recursive processing based on the temporary dependency relationships, to obtain layer trace attribute information of the field in the script file corresponding to the layer to which the script file belongs. For example, the script file from which the field originates can be constantly found through the temporary dependency relationships, and then the attribute information corresponding to the script file from which the field originates can be obtained to obtain the layer trace attribute information.

[0080] In one embodiment, the script file corresponding to each job table in the database can be read. Then, each script file is parsed to obtain all temporary tables for that script file. Each temporary table can then be parsed to obtain the temporary table name, field names, field processing logic, WHERE conditions, WHERE condition dependent tables and fields, JOIN conditions, and JOIN condition dependent tables and fields. Through the temporary table name, field names, field processing logic, WHERE conditions, WHERE condition dependent tables and fields, JOIN conditions, and JOIN condition dependent tables and fields, the final table to be written and all fields written to that table can be identified.

[0081] Then, based on the temporary dependencies corresponding to each field written to the table, we can recursively obtain the final processing logic for each field written to the table, the final logic dependency table and field, the final WHERE condition, the final WHERE condition dependency table and field, the final JOIN condition, and the final JOIN condition dependency table and field.

[0082] For example, by performing layer-by-layer tracing on the script file, we can find that the database name corresponding to the field "cert_type" is "opdm", the table name is "op_f_ev_sz_brn_campus_card_batch_issu_detail_list_fd", the data processing logic is "case cert_type_cd when'1'then'ID card'else cert_type_cd end ascert_type", the temporary dependency is "dwd.d_ev_fb_appsvr_open_batch_seq_detail_fd|cert_type_cd", and the join condition is "1@r@2@r@3@r@4@r@5".

[0083] Then, based on the layer-based source tracing attribute information corresponding to the script file, library source tracing processing can be performed on the script file to obtain the source tracing attribute information of the fields in the script file in the database. Specifically, the step "based on the layer-based source tracing attribute information corresponding to the script file, perform library source tracing processing on the script file to obtain the source tracing attribute information of the fields in the script file in the database" can include:

[0084] Traverse the layer-based source attribute information to obtain the dependency relationships corresponding to the layer-based source attribute information;

[0085] Recursively process the dependencies to obtain the source dependencies;

[0086] Obtain the source dependency information corresponding to the source attribute.

[0087] In one embodiment, the results of layer-by-layer tracing can be extracted. Then, the field processing logic, WHERE conditions, WHERE condition dependencies on tables and fields, JOIN conditions, and JOIN condition dependencies on tables and fields are traversed recursively, continuously replacing the corresponding logic, tables, and fields to obtain the final deep parsing results. The tracing attribute information can then be stored in a database.

[0088] By performing library tracing on the script file, we can obtain the deep field processing logic and deep logic dependencies corresponding to the fields.

[0089] For example, by performing library source tracing on the script file, we can find that the database name corresponding to the field "cert_type" is "opdm", the table name is "op_f_ev_sz_brn_campus_card_batch_issu_detail_list_fd", the data processing logic is "case cert_type_cd when'1'then'ID card'else cert_type_cd end ascert_type", and the dependency relationship is "dwd.d_ev_fb_appsvr_open_batch_seq_detail_fd|cert_type_cd". Furthermore, we can also find that the deep field processing logic corresponding to the field "cert_type" is "caseid_typewhen'1'then'ID card'elseid_typeend", and the deep dependency logic is "raw.fb_appsvr_t_acct_open_batch_entry|id_type".

[0090] 103. Compare the data processing logic corresponding to the attribute information with the data processing logic of the source attribute information to obtain the comparison results.

[0091] In one embodiment, in order to solve the problem of duplicate processing and make the data processing logic corresponding to the same attribute information consistent, the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information can be compared and processed.

[0092] In this context, the data processing logic corresponding to attribute information can refer to the caliber, which refers to the data retrieval standard. For example, in a data warehouse, the data caliber refers to the data retrieval standard for certain fields. For instance, the source or calculation method of a certain field should be consistent; this is the standard. For metrics, the caliber is the calculation logic of that metric, i.e., the calculation standard.

[0093] In an embodiment, there are multiple ways to compare the data processing logic corresponding to the attribute information and the data processing logic of the traceable attribute information.

[0094] For example, the data processing logic corresponding to the attribute information and the data processing logic of the traceable attribute information can be directly compared.

[0095] In an embodiment, in order to improve the accuracy of comparison, the attribute information and the traceable attribute information can be standardized, and the data processing logic can be standardized. Then, the standardized data is compared.

[0096] Specifically, the step of "comparing the data processing logic corresponding to the attribute information and the data processing logic of the traceable attribute information to obtain a comparison result" can include:

[0097] standardizing the attribute information and the traceable attribute information to obtain standardized attribute information and standardized traceable attribute information;

[0098] standardizing the data processing logic of the standardized attribute information to obtain standardized data processing logic corresponding to the standardized attribute information, and standardizing the data processing logic of the standardized traceable attribute information to obtain standardized data processing logic corresponding to the standardized traceable attribute information;

[0099] determining matching standardized attribute information and standardized traceable attribute information in the standardized attribute information and the standardized traceable attribute information;

[0100] comparing the standardized processing logic corresponding to the matching standardized attribute information and the standardized traceable attribute information to obtain a comparison result.

[0101] In an embodiment, the attribute information and the traceable attribute information can be standardized to obtain standardized attribute information and standardized traceable attribute information. For example, the fields in the attribute information can include prefixes or suffixes, so that the matching attribute information and the traceable attribute information become unmatched. For example, the field in the attribute information can be "cert_type.xxx", so that the prefix or suffix can be removed after standardization. For example, the source table in the attribute information and the source table of the traceable attribute information can also have inconsistent formats, so the source table in the attribute information and the format of the traceable attribute information can be standardized.

[0102] In one embodiment, the data processing logic for standardized attribute information can also be standardized to obtain standardized data processing logic corresponding to the standardized attribute information, and the data processing logic for standardized traceability attribute information can also be standardized to obtain standardized data processing logic corresponding to the standardized traceability attribute information. For example, the annotation information carried in the data processing logic can be deleted, etc.

[0103] For example, for the field "cert_type", its normalized field processing logic is "caseid_typewhen'1'then'ID card'elseid_typeend", and its normalized logic dependency can be "fb_appsvr_t_acct_open_batch_entry|id_type".

[0104] In one embodiment, matching standardized attribute information and standardized traceability attribute information can be determined from standardized attribute information and standardized traceability attribute information. For example, standardized attribute information may include multiple fields, and standardized traceability attribute information may also include multiple fields. Matching fields can be determined from the multiple fields of standardized attribute information and the multiple fields of standardized traceability attribute information, i.e., identical fields can be determined.

[0105] In one embodiment, the standardized processing logic corresponding to the matching standardized attribute information and the standardized traceability attribute information can be compared to obtain the comparison result.

[0106] Specifically, the step of "comparing the standardized processing logic corresponding to the matching standardized attribute information and the standardized traceability attribute information to obtain the comparison result" may include:

[0107] The standardized processing logic corresponding to the matching standardized attribute information and the standardized traceability attribute information is compared and processed.

[0108] When the standardization processing logic corresponding to the standardized attribute information is the same as the standardization processing logic corresponding to the standardized traceability attribute information, determine the filtering conditions corresponding to the standardized attribute information and the filtering conditions corresponding to the standardized traceability attribute information.

[0109] The filtering conditions corresponding to the standardized attribute information and the filtering conditions corresponding to the standardized traceability attribute information are compared to obtain the comparison results.

[0110] In an embodiment, the standardized processing logic corresponding to the matched standardized attribute information and the standardized provenance attribute information can be compared, i.e., whether the standardized processing logic corresponding to the matched standardized attribute information and the standardized provenance attribute information is the same. If the same, the filtering condition corresponding to the standardized attribute information and the filtering condition corresponding to the standardized provenance attribute information can be determined. The filtering condition can refer to the filtering condition included in the data processing logic.

[0111] Then, the filtering condition corresponding to the standardized attribute information and the filtering condition corresponding to the standardized provenance attribute information can be compared to obtain a comparison result. For example, whether the filtering conditions are consistent or whether the filtering conditions have a containing relationship can be determined.

[0112] In addition, in an embodiment, the embodiments of the present application can also include:

[0113] When the standardized processing logic corresponding to the standardized attribute information and the standardized processing logic corresponding to the standardized provenance attribute information are different, the description information corresponding to the standardized attribute information and the description information corresponding to the standardized provenance attribute information are determined.

[0114] The description information corresponding to the standardized attribute information and the description information corresponding to the standardized provenance attribute information are compared to obtain a comparison result.

[0115] In an embodiment, when the standardized processing logic corresponding to the standardized attribute information and the standardized processing logic corresponding to the standardized provenance attribute information are different, the description information corresponding to the standardized attribute information and the description information corresponding to the standardized provenance attribute information can be determined. For example, the description keyword in the standardized attribute information and the description keyword corresponding to the standardized provenance attribute information can be determined.

[0116] Then, the description information corresponding to the standardized attribute information and the description information corresponding to the standardized provenance attribute information can be determined. The description information corresponding to the standardized attribute information and the description information corresponding to the standardized provenance attribute information can be compared by using artificial to obtain a comparison result.

[0117] By comparing, it can be determined whether the attribute information and the provenance attribute information match.

[0118] 104. Based on the comparison result, the data processing logic corresponding to the attribute information and the data processing logic of the provenance attribute information are uniformly processed to obtain a uniformly processed data processing logic.

[0119] For example, the matched field is recorded in a preset table, and the table includes all the fields processed repeatedly in the database. Then, the fields and the fields processed repeatedly can be unified, so that the field is processed only once in the script file. Then, the repeated processing logic in other script files can be directly obtained from the table, so as to achieve consistent data processing logic and avoid repeated processing.

[0120] The embodiment of the present application proposes a data processing logic unification method, which comprises: obtaining a script file in a database, parsing the script file to obtain attribute information corresponding to the script file; performing traceability processing on the script file based on the attribute information corresponding to the script file to obtain traceability attribute information corresponding to a field in the script file; comparing the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information to obtain a comparison result; and performing unified processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information based on the comparison result to obtain the unified processing data processing logic. The repeated data processing logic in the database can be comprehensively and quickly identified, and then the repeated data processing logic can be modified to make the data processing logic of the same field in the database consistent and unified.

[0121] According to the method described in the above embodiment, the following examples will be further described in detail.

[0122] The embodiment of the present application will be described by taking the integration of the data processing logic unification method in a server as an example.

[0123] In an embodiment, as shown in Figure 3 A data processing logic unification method, the specific process is as follows:

[0124] 201. The server obtains a script file in a database, parses the script file to obtain attribute information corresponding to the script file.

[0125] In an embodiment, a plurality of script files in the database can be obtained. For example, the script file corresponding to each job table in the data warehouse system can be read. The script file can be a processed script.

[0126] In an embodiment, each script file can be parsed to obtain all temporary tables of each script file. Then, each temporary table is parsed to obtain the temporary table name, field name, field processing logic, where condition, where condition dependent table and field, join condition, and join condition dependent table and field.

[0127] 202. Based on the attribute information corresponding to the script file, the server performs source tracing processing on the script file to obtain the source attribute information corresponding to the fields in the script file.

[0128] In one embodiment, the script file corresponding to each job table in the database can be read. Then, each script file is parsed to obtain all temporary tables for that script file. Each temporary table can then be parsed to obtain the temporary table name, field names, field processing logic, WHERE conditions, WHERE condition dependent tables and fields, JOIN conditions, and JOIN condition dependent tables and fields. Through the temporary table name, field names, field processing logic, WHERE conditions, WHERE condition dependent tables and fields, JOIN conditions, and JOIN condition dependent tables and fields, the final table to be written and all fields written to that table can be identified.

[0129] Then, based on the temporary dependencies corresponding to each field written to the table, we can recursively obtain the final processing logic for each field written to the table, the final logic dependency table and field, the final WHERE condition, the final WHERE condition dependency table and field, the final JOIN condition, and the final JOIN condition dependency table and field.

[0130] For example, by performing layer-by-layer tracing on the script file, we can find that the database name corresponding to the field "cert_type" is "opdm", the table name is "op_f_ev_sz_brn_campus_card_batch_issu_detail_list_fd", the data processing logic is "case cert_type_cd when'1'then'ID card'else cert_type_cd end ascert_type", the temporary dependency is "dwd.d_ev_fb_appsvr_open_batch_seq_detail_fd|cert_type_cd", and the join condition is "1@r@2@r@3@r@4@r@5".

[0131] In one embodiment, the results of layer-by-layer tracing can be extracted. Then, the field processing logic, WHERE conditions, WHERE condition dependencies on tables and fields, JOIN conditions, and JOIN condition dependencies on tables and fields are traversed recursively, continuously replacing the corresponding logic, tables, and fields to obtain the final deep parsing results. The tracing attribute information can then be stored in a database.

[0132] By performing library tracing on the script file, we can obtain the deep field processing logic and deep logic dependencies corresponding to the fields.

[0133] For example, by performing library source tracing on the script file, we can find that the database name corresponding to the field "cert_type" is "opdm", the table name is "op_f_ev_sz_brn_campus_card_batch_issu_detail_list_fd", the data processing logic is "case cert_type_cd when'1'then'ID card'else cert_type_cd end ascert_type", and the dependency relationship is "dwd.d_ev_fb_appsvr_open_batch_seq_detail_fd|cert_type_cd". Furthermore, we can also find that the deep field processing logic corresponding to the field "cert_type" is "caseid_typewhen'1'then'ID card'elseid_typeend", and the deep dependency logic is "raw.fb_appsvr_t_acct_open_batch_entry|id_type".

[0134] 203. The data processing logic corresponding to the server's identification attribute information and the data processing logic of the traceability attribute information are compared and processed to obtain the comparison results.

[0135] In one embodiment, the caliber processing logic standardization and the caliber source table and field standardization can be performed. In order to shield the influence of table naming, the results after deep source exploration can be standardized.

[0136] Then, the standardized processing logic can be compared with the standardized source table and fields. If they are the same, the filtering conditions can be compared to determine whether the filtering conditions are consistent or contain each other.

[0137] 204. Based on the comparison results, the server performs unified processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information to obtain the unified data processing logic.

[0138] Record the successfully matched table fields into the final inconsistent record table. At this point, all the duplicate processing fields for this field in the database have been generated. Further design modifications are needed to ensure that this field is processed only once in a certain table, and then the duplicate processing logic in other tables directly retrieves the data from this table, so as to achieve consistency and avoid duplicate processing.

[0139] The embodiment of the application provides an operation method, which comprises the following steps: a server acquires a script file in a database, analyzes the script file, and obtains attribute information corresponding to the script file; the server performs traceability processing on the script file based on the attribute information corresponding to the script file, and obtains traceability attribute information corresponding to a field in the script file; the server compares data processing logic corresponding to the attribute information with data processing logic of the traceability attribute information, and obtains a comparison result; and the server performs unified processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information based on the comparison result, and obtains data processing logic after unified processing. Whether repeated data processing logic exists in the database can be comprehensively and quickly identified, and then the repeated data processing logic can be reformed, so that the data processing logic of the same field in the database is consistent and unified.

[0140] In order to better implement the data processing logic unification method provided by the embodiment of the application, in an embodiment, a data processing logic unification device is further provided, which can be integrated in a computer device. The meanings of the nouns are the same as those in the data processing logic unification method of the product, and the specific implementation details can be referred to the description in the method embodiment.

[0141] In an embodiment, a data processing logic unification device is provided, which can be integrated in a computer device, for example, a television, as shown in the figure, the data processing logic unification device comprises an analysis unit 301, a traceability unit 302, a comparison unit 303 and a unification unit 304, and the specific implementation is as follows: Figure 4 The analysis unit 301 is configured to acquire a script file in a database, analyze the script file, and obtain attribute information corresponding to the script file.

[0142] The traceability unit 302 is configured to perform traceability processing on the script file based on the attribute information corresponding to the script file, and obtain traceability attribute information corresponding to a field in the script file.

[0143] The comparison unit 303 is configured to compare data processing logic corresponding to the attribute information with data processing logic of the traceability attribute information, and obtain a comparison result.

[0144] The unification unit 304 is configured to perform unified processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information based on the comparison result, and obtain data processing logic after unified processing.

[0145] In an embodiment, the analysis unit 301 can comprise:

[0146]

[0147] ​The first parsing subunit is configured to parse the script file to obtain temporary table corresponding to the script file.

[0148] The second parsing subunit is configured to parse the temporary table to obtain attribute information corresponding to the script file.

[0149] In an embodiment, the trace unit 302 can include:

[0150] The layer trace processing subunit is configured to perform layer trace processing on the script file based on the attribute information corresponding to the script file to obtain layer trace attribute information of a field in the script file corresponding to a layer to which the script file belongs.

[0151] The library trace processing subunit is configured to perform library trace processing on the script file based on the layer trace attribute information corresponding to the script file to obtain trace attribute information of the field in the script file in the database.

[0152] In an embodiment, the layer trace processing subunit can include:

[0153] The parsing module is configured to parse the attribute information corresponding to the script file to obtain field information.

[0154] The relationship determining module is configured to determine temporary dependency relationship corresponding to the field information.

[0155] The hierarchical recursive processing module is configured to perform hierarchical recursive processing on the field information based on the temporary dependency relationship to obtain layer trace attribute information of a field in the script file corresponding to a layer to which the script file belongs.

[0156] In an embodiment, the library trace processing subunit can include:

[0157] The traversal module is configured to traverse the layer trace attribute information to obtain dependency relationship corresponding to the layer trace attribute information.

[0158] The recursive processing module is configured to perform recursive processing on the dependency relationship to obtain source dependency relationship.

[0159] The obtaining module is configured to obtain trace attribute information corresponding to the source dependency relationship.

[0160] In an embodiment, the comparison unit 303 can include:

[0161] The first standardization processing subunit is configured to perform standardization processing on the attribute information and the trace attribute information to obtain standardized attribute information and standardized trace attribute information.

[0162] The second standardization processing subunit is configured to standardize the data processing logic of the standardized attribute information to obtain the standardized data processing logic corresponding to the standardized attribute information, and standardize the data processing logic of the standardized traceability attribute information to obtain the standardized data processing logic corresponding to the standardized traceability attribute information.

[0163] The information determination subunit is configured to determine matching standardized attribute information and standardized traceability attribute information from the standardized attribute information and the standardized traceability attribute information.

[0164] The comparison subunit is configured to compare the standardized processing logic corresponding to the matching standardized attribute information and standardized traceability attribute information to obtain a comparison result.

[0165] In an embodiment, the comparison subunit can include:

[0166] The first comparison processing module is configured to compare the standardized processing logic corresponding to the matching standardized attribute information and standardized traceability attribute information.

[0167] The condition determination module is configured to determine the filtering condition corresponding to the standardized attribute information and the filtering condition corresponding to the standardized traceability attribute information when the standardized processing logic corresponding to the standardized attribute information and the standardized processing logic corresponding to the standardized traceability attribute information are the same.

[0168] The second comparison processing module is configured to compare the filtering condition corresponding to the standardized attribute information and the filtering condition corresponding to the standardized traceability attribute information to obtain the comparison result.

[0169] In an embodiment, the comparison subunit can further include:

[0170] The description information determination module is configured to determine the description information corresponding to the standardized attribute information and the description information corresponding to the standardized traceability attribute information when the standardized processing logic corresponding to the standardized attribute information and the standardized processing logic corresponding to the standardized traceability attribute information are different.

[0171] The third comparison processing module is configured to compare the description information corresponding to the standardized attribute information and the description information corresponding to the standardized traceability attribute information to obtain a comparison result.

[0172] In implementation, each of the above units can be implemented as an independent entity, or can be combined as the same or several entities, and the implementation of each of the above units can refer to the method embodiments described above, which will not be described here.

[0173] The data processing logic unification device can automatically predict the asset level of the consultation object, thereby improving the reliability of predicting the asset level of the consultation object.

[0174] The computer device can include a computer device or a server, for example, the computer device can be a data processing logic unification computer device, which can be an intelligent television or the like; for another example, the computer device can be a server, such as a data processing logic unification server or the like. Figure 5 As shown in FIG. 1, a structure schematic diagram of a computer device related to an embodiment of the present application is shown, in particular:

[0175] The computer device can include a processor 401 with one or more processing cores, a memory 402 with one or more computer readable storage media, a power supply 403, an input unit 404, and the like. Those skilled in the art can understand that the computer device structure shown in FIG. 1 does not constitute a limitation on the computer device, and can include more or less components than those shown in the figure, or combine certain components, or different component arrangements. Among them: Figure 5 The computer device structure shown in FIG. 1 does not constitute a limitation on the computer device, and can include more or less components than those shown in the figure, or combine certain components, or different component arrangements. Among them:

[0176] The processor 401 is the control center of the computer device, connects various parts of the computer device through various interfaces and lines, executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, thereby overall monitoring the computer device. Optionally, the processor 401 can include one or more processing cores; preferably, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401.

[0177] The memory 402 can be used to store software programs and modules, and the processor 401 executes various function applications and data processing logic by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide the processor 401 with access to the memory 402.

[0178] The computer device also includes a power supply 403 for powering the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as charge management, discharge management, and power consumption management through the power management system. The power supply 403 can also include one or more direct current or alternating current power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator, and the like.

[0179] The computer device can also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0180] Although not shown, the computer device can also include a display unit, etc., which will not be described here. Specifically, in the present embodiment, the processor 401 in the computer device loads one or more executable files corresponding to the processes of the application programs into the memory 402 according to the following instructions, and runs the application programs stored in the memory 402 by the processor 401, so as to realize various functions, as follows:

[0181] Obtaining a script file in a database, parsing the script file to obtain attribute information corresponding to the script file;

[0182] Based on the attribute information corresponding to the script file, performing traceability processing on the script file to obtain traceability attribute information corresponding to a field in the script file;

[0183] Comparing the data processing logic corresponding to the attribute information with the data processing logic of the traceability attribute information to obtain a comparison result;

[0184] Based on the comparison result, the data processing logic corresponding to the attribute information and the data processing logic of the traceable attribute information are uniformly processed to obtain uniformly processed data processing logic.

[0185] The specific implementation of the above operations can refer to the foregoing embodiments, which will not be described here again.

[0186] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method provided in various optional implementation manners in the above embodiments.

[0187] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a computer program, or by a computer program controlling related hardware, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0188] For this purpose, the embodiments of the present application further provide a storage medium having a computer program stored therein, which can be loaded by a processor to execute the steps in any one of the data processing logic unification methods provided by the embodiments of the present application. For example, the computer program can execute the following steps:

[0189] Obtaining a script file in a database, parsing the script file to obtain attribute information corresponding to the script file;

[0190] Based on the attribute information corresponding to the script file, performing traceable processing on the script file to obtain traceable attribute information corresponding to a field in the script file;

[0191] Performing comparison processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceable attribute information to obtain a comparison result;

[0192] Based on the comparison result, the data processing logic corresponding to the attribute information and the data processing logic of the traceable attribute information are uniformly processed to obtain uniformly processed data processing logic.

[0193] The specific implementation of the above operations can refer to the foregoing embodiments, which will not be described here again.

[0194] Since the computer program stored in the storage medium can execute the steps in any one of the data processing logic unification methods provided by the embodiments of the present application, the beneficial effects that can be achieved by any one of the data processing logic unification methods provided by the embodiments of the present application can be achieved, which is described in detail in the foregoing embodiments and will not be described here.

[0195] The foregoing has described in detail the data processing logic unification method, device and computer equipment provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples in this paper. The foregoing example description is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A unified data processing logic method, characterized in that, include: Obtain the script file from the database, parse the script file, and obtain the temporary table corresponding to the script file; The temporary table is parsed to obtain the attribute information corresponding to the script file; Based on the attribute information corresponding to the script file, layer tracing processing is performed on the script file to obtain the layer tracing attribute information of the fields in the script file corresponding to the layer to which the script file belongs; based on the layer tracing attribute information corresponding to the script file, library tracing processing is performed on the script file to obtain the tracing attribute information of the fields in the script file corresponding to the database. The data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information are compared to obtain the comparison result. Based on the comparison results, the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information are processed in a unified manner to obtain the unified data processing logic.

2. The method according to claim 1, characterized in that, Based on the attribute information corresponding to the script file, layer tracing processing is performed on the script file to obtain the layer tracing attribute information of the fields in the script file corresponding to the layer to which the script file belongs, including: The attribute information corresponding to the script file is parsed to obtain the field information; Determine the temporary dependency relationships corresponding to the field information; Based on the temporary dependency relationship, the field information is processed hierarchically and recursively to obtain the layer-based traceability attribute information of the field in the script file corresponding to the layer to which the script file belongs.

3. The method according to claim 1, characterized in that, Based on the layer tracing attribute information corresponding to the script file, the script file is subjected to library tracing processing to obtain the tracing attribute information of the fields in the script file in the database, including: The layer source tracing attribute information is traversed to obtain the dependency relationship corresponding to the layer source tracing attribute information; The dependency relationships are recursively processed to obtain the source dependency relationships; Obtain the source dependency relationship corresponding to the source attribute information.

4. The method according to claim 1, characterized in that, The data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information are compared to obtain the comparison results, including: The attribute information and the traceability attribute information are standardized to obtain standardized attribute information and standardized traceability attribute information. The data processing logic of the standardized attribute information is standardized to obtain the standardized data processing logic corresponding to the standardized attribute information, and the data processing logic of the standardized traceability attribute information is standardized to obtain the standardized data processing logic corresponding to the standardized traceability attribute information. Determine matching standardized attribute information and standardized traceability attribute information from the standardized attribute information and the standardized traceability attribute information; The standardized processing logic corresponding to the matched standardized attribute information and standardized traceability attribute information is compared to obtain the comparison result.

5. The method according to claim 4, characterized in that, The standardized processing logic corresponding to the matched standardized attribute information and standardized traceability attribute information is compared to obtain the comparison results, including: The standardized processing logic corresponding to the matched standardized attribute information and standardized traceability attribute information is compared and processed. When the standardization processing logic corresponding to the standardization attribute information is the same as the standardization processing logic corresponding to the standardization traceability attribute information, the filtering conditions corresponding to the standardization attribute information and the filtering conditions corresponding to the standardization traceability attribute information are determined. The filtering conditions corresponding to the standardized attribute information and the filtering conditions corresponding to the standardized traceability attribute information are compared to obtain the comparison result.

6. The method according to claim 5, characterized in that, The method further includes: When the standardization processing logic corresponding to the standardization attribute information is different from the standardization processing logic corresponding to the standardization traceability attribute information, the description information corresponding to the standardization attribute information and the description information corresponding to the standardization traceability attribute information are determined. The description information corresponding to the standardized attribute information and the description information corresponding to the standardized traceability attribute information are compared to obtain the comparison result.

7. A unified data processing logic device, characterized in that, include: The acquisition unit is used to acquire a script file from the database, parse the script file, and obtain a temporary table corresponding to the script file. The temporary table is parsed to obtain the attribute information corresponding to the script file; The tracing unit is used to perform layer tracing processing on the script file based on the attribute information corresponding to the script file, to obtain the layer tracing attribute information of the fields in the script file corresponding to the layer to which the script file belongs; and to perform library tracing processing on the script file based on the layer tracing attribute information corresponding to the script file, to obtain the tracing attribute information of the fields in the script file corresponding to the database. The comparison unit is used to compare the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information to obtain a comparison result. A unified unit is used to perform unified processing on the data processing logic corresponding to the attribute information and the data processing logic of the traceability attribute information based on the comparison results, so as to obtain the unified data processing logic.

8. A computer device, characterized in that, It includes a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to perform the data processing logic unification method according to any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores multiple computer programs, which are adapted for loading by a processor to execute the data processing logic unification method according to any one of claims 1 to 6.

Images