Data processing method and device, equipment and medium

Abstract

The invention relates to the field of big data, the scheme can be applied to the fields of medical treatment and finance, and the invention provides a data processing method and device, equipment and a medium, and the method comprises the steps: generating a unique execution identifier for each business process; for the business process, tracking a complete business process from initiation to termination, identifying key data nodes in the process, and determining a business source table corresponding to data generated by each key data node; designing a data processing architecture according to a preset data construction specification and index logic model; according to the data processing architecture and service analysis requirements, sequentially associating, cleaning and integrating the identified service source tables based on the execution identifiers, and outputting corresponding detail data tables; and aggregating and summarizing the detail data table according to service requirements to generate a corresponding core wide table. The data management complexity can be reduced, and the data processing efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of big data, and the solution can be applied to the medical and financial fields, particularly to a data processing method, apparatus, equipment, and medium. Background Technology

[0002] In industries such as finance, healthcare, and the internet, data monitoring is a core support for business operation optimization and iteration. The financial sector needs to integrate transaction and risk control data tables to monitor fund flows and risk thresholds; the healthcare industry relies on diagnosis and treatment records to ensure medical quality and resource allocation; and the internet industry analyzes operational effectiveness through user and order tables. The traditional approach involves: after defining monitoring indicators and definitions, organizing relevant data tables and their primary and foreign key relationships, adding new fields directly to the source data table, and then retrieving data from the source table using SQL for calculation. However, this approach has significant drawbacks: First, performance bottlenecks are prominent when multiple tables are linked. When the financial and healthcare industries involve hundreds or even tens of millions of data tables, SQL execution is slow, consuming significant resources and potentially interfering with the normal operation of core business tables. Second, the lack of hierarchical data storage leads to difficulties in data traceability and confusion regarding data lineage. Third, the adaptability of new indicators is poor; newly added fields directly intrude into source tables shared by multiple business lines, significantly reducing the tables' versatility over time.

[0003] Therefore, existing data monitoring system construction schemes suffer from low data processing efficiency and high resource consumption in large-scale data scenarios. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, device, and medium, aiming to solve the problems of low data processing efficiency and high resource consumption in existing data monitoring system construction schemes under large-scale data scenarios.

[0005] In a first aspect, embodiments of the present invention provide a data processing method, the method comprising: Generate a unique execution identifier for each business process; For the aforementioned business process, trace its complete business flow from initiation to termination, identify the key data nodes in the process, and clarify the business source table corresponding to the data generated by each key data node; Design the data processing architecture based on the pre-defined data construction specifications and indicator logic model; Based on the data processing architecture and business analysis requirements, and based on the execution identifier, the identified business source tables are sequentially associated, cleaned, and integrated to output the corresponding detailed data tables. The detailed data tables are aggregated and summarized according to business requirements to generate the corresponding core wide table.

[0006] Secondly, embodiments of the present invention also provide a data processing apparatus, the apparatus comprising: The generation unit is used to generate a unique execution identifier for each business process; The identification unit is used to track the complete business process from initiation to termination for the business process, identify the key data nodes in the process, and determine the business source table corresponding to the data generated by each key data node. The design unit is used to design the data processing architecture based on the preset data construction specifications and indicator logic model; The output unit is used to sequentially associate, clean and integrate the identified business source tables based on the execution identifier according to the data processing architecture and business analysis requirements, and output the corresponding detailed data table. The wide table generation unit is used to aggregate and summarize the detailed data table according to business needs, and generate the corresponding core wide table.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect above.

[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the method described in the first aspect.

[0009] This invention provides a data processing method, apparatus, device, and medium. The method includes: generating a unique execution identifier for each business process; tracing the complete business flow from initiation to termination for each business process, identifying key data nodes in the flow, and clarifying the business source table corresponding to the data generated by each key data node; designing a data processing architecture based on preset data construction specifications and indicator logic models; sequentially associating, cleaning, and integrating the identified business source tables based on the execution identifier according to the data processing architecture and business analysis requirements, and outputting corresponding detailed data tables; and aggregating and summarizing the detailed data tables according to business requirements to generate a corresponding core wide table. This invention uses a unique execution identifier to connect the data links of the entire business process, efficiently solving the problem of multi-source business table linkage and significantly improving the efficiency of large-scale data association; relying on data construction specifications and data processing architecture, it makes data storage and traceability logic clear, reduces data management complexity, and improves data processing efficiency; the cleaned and integrated detailed data tables and the aggregated core wide table ensure data quality and indicator calculation accuracy, significantly reduce resource consumption, and avoid affecting core business. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating the data processing method provided in an embodiment of the present invention; Figure 2 A schematic block diagram of a data processing apparatus provided in an embodiment of the present invention; Figure 3 A schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. Embodiments of this invention provide a data processing method, apparatus, device, and medium. The data processing method can be applied to a terminal device or a server. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, and wearable device; the server can be a single server or a server cluster composed of multiple servers. The invention will now be described in detail through specific embodiments.

[0016] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S150.

[0017] S110. Generate a unique execution identifier for each business process.

[0018] In this embodiment, a globally unique execution identifier is generated for each independent business process, including but not limited to a marketing campaign, a patient treatment process, or a credit approval transaction. The execution identifier is used as a key field and embedded into all relevant business system tables involved in the business process (such as marketing channel table, patient medical record table, financial transaction table, etc.), providing a unique physical association key for subsequent cross-system and enterprise-wide data connection and unified analysis.

[0019] In one embodiment, step S110 includes: generating a corresponding business level code segment based on the business level and data entity type to which the business process belongs; generating a sequence number segment for the business process based on a preset identifier generation algorithm; and connecting the business level code segment and the sequence number segment to form an execution identifier for the corresponding business process.

[0020] In this embodiment, a corresponding business level code segment is generated based on the business level and data entity type to which the business process belongs. The business level code segment consists of 13 fixed-rule numbers, which respectively represent the naming of different types of tables under various business levels such as bu (business unit), business line, user table, order table, and transaction table. A unique sequence number segment is generated for the business process according to a preset identifier generation algorithm (such as the 19-bit snowflake algorithm). The business level code segment and the sequence number segment are connected to form the execution identifier of the corresponding business process.

[0021] S120. For the business process, trace its complete business flow from initiation to termination, identify the key data nodes in the process, and clarify the business source table corresponding to the data generated by each key data node.

[0022] In this embodiment, for the business process (such as a patient treatment process or a credit approval transaction), the complete business process from initiation to termination is tracked. For example, in the treatment process, key data nodes such as registration, examination, diagnosis, and prescription are identified; in the credit approval process, key data nodes such as application, review, signing, and loan disbursement are identified; and the business source table corresponding to the data generated by each key data node is clarified, providing a foundation for subsequent data integration based on a unified identifier.

[0023] In one embodiment, step S120 includes: tracing the complete business process from initiation, intermediate transitions to termination according to the execution sequence of the business process; identifying the core links in the complete business process that directly affect the business results as key data nodes; the key data nodes include, but are not limited to, business initiation nodes, status transition nodes, user interaction nodes, and result feedback nodes; determining the business source table corresponding to each key data node based on the execution identifier and user identifier; the business source table is a multi-user batch record table under the corresponding key data node.

[0024] In this embodiment, the complete business process from initiation, intermediate transitions to termination is traced according to the execution sequence of the business process; the core links in the complete business process that directly affect the business result are identified as key data nodes; the key data nodes include, but are not limited to, business initiation nodes, state transition nodes, user interaction nodes, and result feedback nodes; based on the execution identifier and user identifier, the business source table corresponding to each key data node is determined; the business source table is a data table used to store the records generated by relevant users at this node in the business process, and its table structure includes the execution identifier field and the user identifier field; the business source table corresponds to the key data nodes respectively; wherein, the business initiation node corresponds to the drop table recording the business initiation action, the state transition node corresponds to the channel table recording the reach status, the user interaction node corresponds to the tracking point table recording the user's front-end interaction, and the result feedback node corresponds to the collection table recording the final business effect.

[0025] In marketing campaign scenarios, user identifiers can be "user IDs". Business source tables typically include: a distribution table that records the actions of issuing campaign strategies, a channel table that records the status of being reached through channels such as SMS or APP push, an event tracking table that records user interaction behaviors such as clicking and browsing on the client, and a collection table (such as an order table) that records the user's final conversion behavior.

[0026] S130. Design the data processing architecture based on the preset data construction specifications and indicator logic model.

[0027] In this embodiment, based on pre-defined data construction specifications (including principles such as strict data layering, high cohesion and low coupling) and indicator logic models (using dimensional modeling methods to define atomic and derived indicator formulas), an enterprise-level data processing architecture is designed, clarifying the technical positioning, data source, and processing responsibilities of each layer (such as the source layer, detail layer, and aggregation layer). The source layer stores raw data (business source tables) from various business systems. The detail layer, based on unified execution and user identifiers, cleans, standardizes, and integrates data from different business source tables in the source layer, forming high-quality, cross-system detailed factual data. The aggregation layer, according to the indicator logic model, performs topic-oriented pre-calculation and aggregation on the detail layer data (such as cumulative aggregation by organization, product, and time dimensions), generating a wide indicator table (core wide table) that can be directly used for analysis and reporting.

[0028] In one embodiment, step S130 includes: constructing a data construction planning model based on the data construction specifications and the indicator logic model; and designing a data processing architecture containing multiple logical layers according to the data construction planning model; wherein the data processing architecture includes at least a source layer, a detail layer, and a summary layer.

[0029] In this embodiment, a data construction planning model is constructed based on the data construction specifications (including principles such as strict data layering, high cohesion, and low coupling) and the indicator logic model. The indicator logic model employs a dimensional modeling method, defining formulas for atomic and derived indicators. Specifically, the derived indicator Y = F{@atomic indicator, @calculation cycle, @business constraint, @calculation granularity}. Based on the data construction planning model, a data processing architecture with multiple logical layers is designed. This architecture includes at least a source layer, a detail layer, and a summary layer. The source layer stores raw data from various business systems (i.e., business source tables), and the detail layer is based on a unified execution identifier and user identifier. The data processing architecture cleans, standardizes, and integrates data from different business source tables in the source layer to form high-quality, cross-system detailed fact data. The aggregation layer performs topic-oriented pre-calculation and aggregation on the detailed layer data according to the indicator logic model (such as cumulative aggregation by organization, product, and time dimensions) to generate indicator wide tables (core wide tables) that can be directly used for analysis and reporting. The data processing architecture also includes a dimension layer, an interface layer, and a tag layer. The dimension layer provides dimension association support for the detailed layer (such as time dimension, product dimension, etc.), the interface layer provides secure data service interfaces for upper-layer data applications, and the tag layer constructs user profile tags or business feature tags based on the detailed layer and aggregation layer data.

[0030] In one embodiment, after designing a data processing architecture comprising multiple logical levels based on the data construction planning model, the method further includes: in response to business changes, determining the specific logical level affected by the business changes; and adaptively adjusting the data processing logic of the specific logical level to integrate the business changes into the data processing architecture.

[0031] In this embodiment, when business changes occur, the impact of the changes is accurately isolated and quickly responded to based on the data processing architecture. For example: Financial Scenario (New Data Source): If a bank adds a new internet loan product channel, it needs to access a new data source. This change only affects the data source layer. The solution is to add a data access and format standardization task for this channel in the data source layer. The indicator calculation logic of the detail layer, summary layer and upper layer does not need to be modified.

[0032] Healthcare Scenario (Business Rule Changes): If a hospital changes the statistical scope of the "patient satisfaction" indicator from "follow-up within 3 days of discharge" to "follow-up within 7 days of discharge", this change mainly affects the summary layer. The solution is to only adjust the calculation logic of the corresponding indicator in the summary layer, while the process of accessing the original follow-up data in the source layer and integrating the patient visit data in the detail layer remains unchanged.

[0033] S140. Based on the data processing architecture and business analysis requirements, and using the execution identifier, sequentially associate, clean, and integrate the identified business source tables to output the corresponding detailed data tables.

[0034] In this embodiment, based on the data processing architecture and business analysis requirements (such as tracking a complete diagnosis and treatment process), the identified business source tables are sequentially associated, cleaned, and integrated based on the execution identifier to output a high-quality detailed data table.

[0035] In one embodiment, step S140 includes: determining data processing rules based on the business analysis requirements; the data processing rules include at least time period rules, business limitation rules, and data association logic; based on the logical positioning of the corresponding level in the data processing architecture, performing cross-table joins, quality cleaning, and structural integration on the data of the business source table in sequence according to the execution identifier and the data processing rules, to generate a corresponding detailed data table.

[0036] In this embodiment, data processing rules are determined based on the business analysis requirements. These rules include at least time period rules (defining the time window and slicing logic for data analysis), business limitation rules (clarifying the dimensions and conditions for data filtering), and data association logic (specifying the order and method of concatenating multiple business source tables based on execution identifiers). Based on the logical positioning of the corresponding level in the data processing architecture, cross-table joins, quality cleaning, and structural integration are performed on the data of the identified multiple business source tables (such as publishing, channel tables, tracking tables, and recycling tables) based on the execution identifiers and the data processing rules to generate corresponding detailed data tables.

[0037] S150. Aggregate and summarize the detailed data table according to business needs to generate the corresponding core wide table.

[0038] In this embodiment, the detailed data table is aggregated and summarized according to business needs (such as generating daily reports, real-time dashboards, or performance evaluation reports) to generate the corresponding core wide table.

[0039] In one embodiment, step S150 includes: performing grouping and aggregation operations on the detailed data table according to the analysis topic and aggregation dimension corresponding to the business requirements, to obtain the corresponding related dimension fields; performing a pre-calculation operation on the detailed data table to obtain the corresponding calculation results; wherein the pre-calculation operation is performed based on the derived indicator calculation logic defined in the indicator logic model; merging the calculation results with the related dimension fields to generate a core wide table that directly supports business queries and analysis.

[0040] In this embodiment, based on the analysis theme (such as user activity analysis) and aggregation dimensions (such as time period, user segmentation, etc.) corresponding to the business requirements, a grouping and aggregation operation is performed on the detailed data table to obtain the corresponding relevant dimension fields; a pre-calculation operation is performed on the detailed data table to obtain the corresponding calculation results (such as click-through rate, cumulative amount, etc.); wherein, the pre-calculation operation is performed based on the derived indicator calculation logic defined in the indicator logic model; the calculation results are merged with the relevant dimension fields to generate a core wide table that directly supports business queries and analysis.

[0041] In summary, this invention establishes a unified data link across the entire business process through a unique execution identifier, efficiently solving the problem of linking multiple source business tables and significantly improving the efficiency of large-scale data association. Based on data construction standards and data processing architecture, it ensures clear data storage and traceability logic, reduces data management complexity, and improves data processing efficiency. The cleaned and integrated detailed data tables and the aggregated core wide table guarantee data quality and the accuracy of indicator calculations, significantly reducing resource consumption and preventing core business operations from being affected.

[0042] Figure 2This is a schematic block diagram of a data processing apparatus provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above data processing methods, the present invention also provides a data processing apparatus, which is configured in a terminal device or a server. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, and wearable device; the server can be a single server or a server cluster composed of multiple servers. For details, please refer to... Figure 2 The data processing device 700 includes: The generation unit 701 is used to generate a unique execution identifier for each business process; The identification unit 702 is used to track the complete business process from initiation to termination for the business process, identify the key data nodes in the process, and identify the business source table corresponding to the data generated by each key data node. Design unit 703 is used to design the data processing architecture based on the preset data construction specifications and indicator logic model; The output unit 704 is used to sequentially associate, clean and integrate the identified business source tables based on the execution identifier according to the data processing architecture and business analysis requirements, and output the corresponding detailed data table. The wide table generation unit 705 is used to aggregate and summarize the detailed data table according to business requirements to generate the corresponding core wide table.

[0043] In some embodiments, when the generation unit 701 performs the step of generating a unique execution identifier for each business process, it is specifically used for: Generate a corresponding business level code segment based on the business level and data entity type of the business process; generate a sequence number segment for the business process based on a preset identifier generation algorithm; and connect the business level code segment and the sequence number segment to form the execution identifier of the corresponding business process.

[0044] In some embodiments, when the identification unit 702 performs the step of tracking the complete business process from initiation to termination for the business process, identifying key data nodes in the process, and determining the business source table corresponding to the data generated by each key data node, it is specifically used for: Based on the execution sequence of the business process, trace its complete business flow from initiation, intermediate transitions to termination; identify the core links in the complete business flow that directly affect the business results as key data nodes; the key data nodes include, but are not limited to, business initiation nodes, status transition nodes, user interaction nodes, and result feedback nodes; based on the execution identifier and user identifier, determine the business source table corresponding to each key data node; the business source table is a multi-user batch record table under the corresponding key data node.

[0045] In some embodiments, when the design unit 703 performs the step of designing a data processing architecture based on preset data construction specifications and indicator logic models, it is specifically used for: Based on the data construction specifications and the indicator logic model, a data construction planning model is constructed; according to the data construction planning model, a data processing architecture containing multiple logical levels is designed; wherein, the data processing architecture includes at least a source layer, a detail layer and a summary layer.

[0046] In some embodiments, after performing the step of designing a data processing architecture comprising multiple logical levels based on the data construction planning model, the design unit 703 is further configured to: In response to business changes, the specific logical levels affected by the business changes are identified; the data processing logic of the specific logical levels is adaptively adjusted to integrate the business changes into the data processing architecture.

[0047] In some embodiments, when the output unit 704 performs the step of sequentially associating, cleaning, and integrating the identified business source tables based on the execution identifier according to the data processing architecture and business analysis requirements, and outputting the corresponding detailed data table, it is specifically used for: Data processing rules are determined based on the business analysis requirements; the data processing rules include at least time period rules, business limitation rules, and data association logic; based on the logical positioning of the corresponding level in the data processing architecture, cross-table joins, quality cleaning, and structural integration are sequentially performed on the data of the business source table based on the execution identifier and the data processing rules to generate the corresponding detailed data table.

[0048] In some embodiments, when the wide table generation unit 705 performs the step of aggregating and summarizing the detailed data table according to business requirements to generate the corresponding core wide table, it is specifically used for: Based on the analysis theme and aggregation dimensions corresponding to the business requirements; perform grouping and aggregation operations on the detailed data table to obtain the corresponding relevant dimension fields; perform pre-calculation operations on the detailed data table to obtain the corresponding calculation results; wherein, the pre-calculation operation is performed based on the derived indicator calculation logic defined in the indicator logic model; merge the calculation results with the relevant dimension fields to generate a core wide table that directly supports business queries and analysis.

[0049] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned data processing device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0050] The aforementioned data processing device can be implemented as a computer program, which can, for example... Figure 3 It runs on the electronic device shown.

[0051] Please see Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 800 can be a terminal or a server. The terminal can be an electronic device with communication functions. The server can be a standalone server or a server cluster composed of multiple servers.

[0052] See Figure 3 The electronic device 800 includes a processor 802, a memory, and a network interface 805 connected via a system bus 801. The memory may include a non-volatile storage medium 803 and internal memory 804.

[0053] The non-volatile storage medium 803 may store an operating system 8031 ​​and a computer program 8032. The computer program 8032 includes program instructions that, when executed, cause the processor 802 to perform a data processing method.

[0054] The processor 802 provides computing and control capabilities to support the operation of the entire electronic device 800.

[0055] The internal memory 804 provides an environment for the execution of the computer program 8032 in the non-volatile storage medium 803. When the computer program 8032 is executed by the processor 802, the processor 802 can perform a data processing method.

[0056] This network interface 805 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device 800 to which the present invention is applied. The specific electronic device 800 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0057] The processor 802 is used to run a computer program 8032 stored in the memory to perform the following steps: A unique execution identifier is generated for each business process; for each business process, its complete business flow from initiation to termination is tracked, key data nodes in the process are identified, and the business source table corresponding to the data generated by each key data node is determined; a data processing architecture is designed according to the preset data construction specifications and indicator logic model; based on the data processing architecture and business analysis requirements, the identified business source tables are sequentially associated, cleaned, and integrated based on the execution identifier, and the corresponding detailed data tables are output; the detailed data tables are aggregated and summarized according to business requirements to generate the corresponding core wide table.

[0058] In some embodiments, when the processor 802 generates a unique execution identifier step for each business process, it specifically implements the following steps: Generate a corresponding business level code segment based on the business level and data entity type of the business process; generate a sequence number segment for the business process based on a preset identifier generation algorithm; and connect the business level code segment and the sequence number segment to form the execution identifier of the corresponding business process.

[0059] In some embodiments, when the processor 802 implements the steps of tracing the complete business process from initiation to termination for the business process, identifying key data nodes in the process, and determining the business source table corresponding to the data generated by each key data node, the specific implementation is as follows: Based on the execution sequence of the business process, trace its complete business flow from initiation, intermediate transitions to termination; identify the core links in the complete business flow that directly affect the business results as key data nodes; the key data nodes include, but are not limited to, business initiation nodes, status transition nodes, user interaction nodes, and result feedback nodes; based on the execution identifier and user identifier, determine the business source table corresponding to each key data node; the business source table is a multi-user batch record table under the corresponding key data node.

[0060] In some embodiments, when the processor 802 designs the data processing architecture based on a preset data construction specification and indicator logic model, it specifically implements the following steps: Based on the data construction specifications and the indicator logic model, a data construction planning model is constructed; according to the data construction planning model, a data processing architecture containing multiple logical levels is designed; wherein, the data processing architecture includes at least a source layer, a detail layer and a summary layer.

[0061] In some embodiments, after implementing the step of designing a data processing architecture comprising multiple logical levels based on the data construction planning model, the processor 802 further implements the following steps: In response to business changes, the specific logical levels affected by the business changes are identified; the data processing logic of the specific logical levels is adaptively adjusted to integrate the business changes into the data processing architecture.

[0062] In some embodiments, when the processor 802 implements the step of sequentially associating, cleaning, and integrating the identified business source tables based on the execution identifier according to the data processing architecture and business analysis requirements, and outputting the corresponding detailed data table, the specific implementation is as follows: Data processing rules are determined based on the business analysis requirements; the data processing rules include at least time period rules, business limitation rules, and data association logic; based on the logical positioning of the corresponding level in the data processing architecture, cross-table joins, quality cleaning, and structural integration are sequentially performed on the data of the business source table based on the execution identifier and the data processing rules to generate the corresponding detailed data table.

[0063] In some embodiments, when the processor 802 aggregates and summarizes the detailed data table according to business requirements to generate the corresponding core wide table, it specifically implements the following steps: Based on the analysis theme and aggregation dimensions corresponding to the business requirements; perform grouping and aggregation operations on the detailed data table to obtain the corresponding relevant dimension fields; perform pre-calculation operations on the detailed data table to obtain the corresponding calculation results; wherein, the pre-calculation operation is performed based on the derived indicator calculation logic defined in the indicator logic model; merge the calculation results with the relevant dimension fields to generate a core wide table that directly supports business queries and analysis.

[0064] It should be understood that, in this embodiment of the invention, the processor 802 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0065] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0066] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps: A unique execution identifier is generated for each business process; for each business process, its complete business flow from initiation to termination is tracked, key data nodes in the process are identified, and the business source table corresponding to the data generated by each key data node is determined; a data processing architecture is designed according to the preset data construction specifications and indicator logic model; based on the data processing architecture and business analysis requirements, the identified business source tables are sequentially associated, cleaned, and integrated based on the execution identifier, and the corresponding detailed data tables are output; the detailed data tables are aggregated and summarized according to business requirements to generate the corresponding core wide table.

[0067] In one embodiment, when the processor executes the program instructions to generate a unique execution identifier for each business process, it specifically implements the following steps: Generate a corresponding business level code segment based on the business level and data entity type of the business process; generate a sequence number segment for the business process based on a preset identifier generation algorithm; and connect the business level code segment and the sequence number segment to form the execution identifier of the corresponding business process.

[0068] In one embodiment, when the processor executes the program instructions to implement the complete business process from initiation to termination for the business process, identify key data nodes in the process, and determine the business source table steps corresponding to the data generated by each key data node, the processor specifically implements the following steps: Based on the execution sequence of the business process, trace its complete business flow from initiation, intermediate transitions to termination; identify the core links in the complete business flow that directly affect the business results as key data nodes; the key data nodes include, but are not limited to, business initiation nodes, status transition nodes, user interaction nodes, and result feedback nodes; based on the execution identifier and user identifier, determine the business source table corresponding to each key data node; the business source table is a multi-user batch record table under the corresponding key data node.

[0069] In one embodiment, when the processor executes the program instructions to design the data processing architecture based on preset data construction specifications and indicator logic models, it specifically implements the following steps: Based on the data construction specifications and the indicator logic model, a data construction planning model is constructed; according to the data construction planning model, a data processing architecture containing multiple logical levels is designed; wherein, the data processing architecture includes at least a source layer, a detail layer and a summary layer.

[0070] In one embodiment, after executing the program instructions to design a data processing architecture comprising multiple logical levels based on the data construction planning model, the processor further performs the following steps: In response to business changes, the specific logical levels affected by the business changes are identified; the data processing logic of the specific logical levels is adaptively adjusted to integrate the business changes into the data processing architecture.

[0071] In one embodiment, when the processor executes the program instructions to perform the steps of sequentially associating, cleaning, and integrating the identified business source tables based on the execution identifier according to the data processing architecture and business analysis requirements, and outputting the corresponding detailed data table, the specific implementation is as follows: Data processing rules are determined based on the business analysis requirements; the data processing rules include at least time period rules, business limitation rules, and data association logic; based on the logical positioning of the corresponding level in the data processing architecture, cross-table joins, quality cleaning, and structural integration are sequentially performed on the data of the business source table based on the execution identifier and the data processing rules to generate the corresponding detailed data table.

[0072] In one embodiment, when the processor executes the program instructions to aggregate and summarize the detailed data table according to business requirements and generate the corresponding core wide table, the specific steps are as follows: Based on the analysis theme and aggregation dimensions corresponding to the business requirements; perform grouping and aggregation operations on the detailed data table to obtain the corresponding relevant dimension fields; perform pre-calculation operations on the detailed data table to obtain the corresponding calculation results; wherein, the pre-calculation operation is performed based on the derived indicator calculation logic defined in the indicator logic model; merge the calculation results with the relevant dimension fields to generate a core wide table that directly supports business queries and analysis.

[0073] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0075] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0076] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0077] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0078] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Generate a unique execution identifier for each business process; For the aforementioned business process, trace its complete business flow from initiation to termination, identify the key data nodes in the process, and clarify the business source table corresponding to the data generated by each key data node; Design the data processing architecture based on the pre-defined data construction specifications and indicator logic model; Based on the data processing architecture and business analysis requirements, and based on the execution identifier, the identified business source tables are sequentially associated, cleaned, and integrated to output the corresponding detailed data tables. The detailed data tables are aggregated and summarized according to business requirements to generate the corresponding core wide table.

2. The data processing method according to claim 1, characterized in that, The process of generating a unique execution identifier for each business process includes: Generate corresponding business layer coding segments based on the business layer and data entity type to which the business process belongs; A sequence number segment is generated for the business process according to a preset identifier generation algorithm; The business level code segment is concatenated with the sequence number segment to form the execution identifier of the corresponding business process.

3. The data processing method according to claim 1, characterized in that, The process involves tracing the complete business flow from initiation to termination, identifying key data nodes within the flow, and clarifying the business source table corresponding to the data generated by each key data node, including: Based on the execution sequence of the business process, trace its complete business flow from initiation, intermediate transitions to termination; Identify the core links in the complete business process that directly affect the business outcome as key data nodes; the key data nodes include, but are not limited to, business initiation nodes, status transition nodes, user interaction nodes, and result feedback nodes. Based on the execution identifier and user identifier, the business source table corresponding to each key data node is determined; the business source table is a multi-user batch record table under the corresponding key data node.

4. The data processing method according to claim 1, characterized in that, The step of designing a data processing architecture based on preset data construction specifications and indicator logic models includes: Based on the aforementioned data construction specifications and the aforementioned indicator logic model, a data construction planning model is constructed. Based on the data construction planning model, a data processing architecture with multiple logical layers is designed; wherein, the data processing architecture includes at least a source layer, a detail layer, and a summary layer.

5. The data processing method according to claim 4, characterized in that, After designing a data processing architecture with multiple logical levels based on the data construction planning model, the process further includes: In response to business changes, determine the specific logical level affected by the business changes; Adaptive adjustments are made to the data processing logic at the specific logical level to integrate the business changes into the data processing architecture.

6. The data processing method according to claim 1, characterized in that, Based on the data processing architecture and business analysis requirements, and using the execution identifier, the identified business source tables are sequentially associated, cleaned, and integrated to output corresponding detailed data tables, including: Data processing rules are determined based on the aforementioned business analysis requirements; the data processing rules include at least time period rules, business limitation rules, and data association logic. Based on the logical positioning of the corresponding layer in the data processing architecture, cross-table joins, quality cleaning, and structural integration are sequentially performed on the data of the business source table according to the execution identifier and the data processing rules to generate the corresponding detailed data table.

7. The data processing method according to claim 1, characterized in that, The process of aggregating and summarizing the detailed data table according to business needs to generate a corresponding core wide table includes: Based on the analysis topics and aggregation dimensions corresponding to the aforementioned business requirements; Perform grouping and aggregation operations on the detailed data table to obtain the corresponding relevant dimension fields; A pre-calculation operation is performed on the detailed data table to obtain the corresponding calculation results; wherein, the pre-calculation operation is performed based on the derived indicator calculation logic defined in the indicator logic model; The calculation results are merged with the relevant dimension fields to generate a core wide table that directly supports business queries and analysis.

8. A data processing apparatus, characterized in that, The device includes: The generation unit is used to generate a unique execution identifier for each business process; The identification unit is used to track the complete business process from initiation to termination for the business process, identify the key data nodes in the process, and determine the business source table corresponding to the data generated by each key data node. The design unit is used to design the data processing architecture based on the preset data construction specifications and indicator logic model. The output unit is used to sequentially associate, clean and integrate the identified business source tables based on the execution identifier according to the data processing architecture and business analysis requirements, and output the corresponding detailed data table. The wide table generation unit is used to aggregate and summarize the detailed data table according to business needs, and generate the corresponding core wide table.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the data processing method as described in any one of claims 1-7.

Images