Data processing method based on hybrid modeling and program product

Abstract

The invention discloses a data processing method based on hybrid modeling and a program product. The method comprises the following steps: constructing a new model by taking activation data of a bottom layer model as an input source; and customizing a data uploading process of the application model according to a business scene through a processing chain technology, and automatically scheduling a data uploading task. And according to the data uploading task, dynamically switching the new model to replace the underlying model for data uploading, and realizing loose coupling of data uploading and data extraction between the underlying model and the application model. According to the method, the effect that model data uploading and model data extraction do not interfere with each other is achieved in a hybrid modeling mode, so that the data delay time is shortened, and the data updating efficiency and the data updating stability of the data warehouse are remarkably improved by improving the system operation efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular, to a data processing method and program product based on hybrid modeling. Background Art

[0002] In the traditional mode, the data flow of the data model in the commercial data warehouse has strict processing flow specifications. During the data upload process of a single data model, for example, when it is being updated or not activated, the data extraction operation cannot be performed on this data model anymore, otherwise data model locking will occur, resulting in data errors.

[0003] With the wide application of big data platforms for data warehouses and data lakes, the data application scenarios of the application model for the underlying model are becoming more and more extensive, and data calls are becoming more and more frequent. And the application model needs to upload all the data of all the underlying models associated with it before it can start a complete data flow update, resulting in a relatively long delay time for the application model to update data, and the operation and maintenance are difficult. Once the data extraction of a single underlying model fails, it will cause the data update of the application model to fail, greatly increasing the operation and maintenance workload. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art, and provide a data processing method and program product based on hybrid modeling. By means of hybrid modeling and model replacement, loose coupling between data upload and data extraction between the underlying model and the application model is realized, thereby improving the update efficiency and update stability of the application model and reducing the operation and maintenance workload of the system.

[0005] The technical solution of the present invention provides a data processing method based on hybrid modeling, including:

[0006] Construct a new model with the activation data of the underlying model as the input source;

[0007] Customize the data upload process of the application model according to the business scenario through the processing chain technology, and automatically schedule the data upload task;

[0008] According to the data upload task, dynamically switch the new model to replace the underlying model for data upload, and realize loose coupling between data upload and data extraction between the underlying model and the application model.

[0009] In one of the optional technical solutions, the constructing a new model with the activation data of the underlying model as the input source includes:

[0010] Taking the activation data of the underlying model as the input source, generate a data provider through visualization technology;

[0011] Obtain a data source through the data provider to make the metadata structure of the newly created model compatible with the underlying model;

[0012] Build a data interface and processing logic for the newly created model based on the data provider so that the newly created model can independently complete data upload.

[0013] In one alternative technical solution, the newly created model extracts data from the underlying model to achieve activation.

[0014] In one alternative technical solution, a HANA view is generated through visualization technology in the underlying model. The underlying model and the newly created model are BW models with line item tables and activation tables. The activation data of the underlying model is the activation table, and the data provider assigns values to it through the HANA view.

[0015] In one alternative technical solution, the data provider is a composite provider, and the composite provider is used to integrate input data from multiple data sources. The data sources include at least one of a database, an API interface, and an external file.

[0016] In one alternative technical solution, the data upload process of the application model is customized according to the business scenario through the processing chain technology, and the data upload task is automatically scheduled, including:

[0017] Obtain the types of the application model and the business scenario;

[0018] Create a processing chain according to the application model and the business scenario;

[0019] Define and schedule the task trigger conditions, task priorities, and exception handling strategies of the data upload process in the processing chain.

[0020] In one alternative technical solution, the task trigger conditions include:

[0021] When a single underlying model uploads data to multiple application models simultaneously, switch the upload object of each data upload task to a newly created model respectively to relieve the bottleneck of the data upload speed of a single underlying model;

[0022] When multiple underlying models upload data simultaneously, switch the upload object of the data upload task to the newly created model to relieve the reconciliation relationship between multiple underlying models.

[0023] In one of the optional technical solutions, the dynamic switching of the newly created model to replace the underlying model for data uploading is achieved through a hot standby mechanism, the newly created model and the underlying model share the same data provider, and high availability is maintained between the newly created model and the underlying model through heartbeat detection.

[0024] In one of the optional technical solutions, the loose coupling of data uploading and data extraction between the underlying model and the application model is achieved by:

[0025] In the data upload task, only the newly created model is relied upon to complete the data upload operation;

[0026] In the data extraction task, it is implemented by asynchronously calling the underlying model and the newly created model, and is decoupled from the data upload task.

[0027] The technical solution of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the aforementioned data processing methods based on hybrid modeling.

[0028] The above technical solution has the following beneficial effects:

[0029] The data processing method based on hybrid modeling provided by the present invention replaces the underlying model with a newly created model in the process of data uploading by means of hybrid modeling and model replacement, so that the data uploading and data extraction of the underlying model are decoupled, and a single underlying model can upload data to multiple application models at the same time, so that the model data uploading and model data extraction do not interfere with each other, and the data delay time is shortened. On the other hand, the underlying model can also extract data in the process of data uploading, which can improve the system operation efficiency, significantly improve the data update efficiency and data update stability, and thus improve the data output efficiency of the data model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It should be understood that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings:

[0031] Figure 1 A flowchart of a data processing method for hybrid modeling provided by an embodiment of the present invention;

[0032] Figure 2 A workflow diagram of a data processing method for hybrid modeling provided by another embodiment of the present invention;

[0033] Figure 3 A workflow diagram of a data processing method for hybrid modeling provided by yet another embodiment of the present invention;

[0034] Figure 4 Schematic diagram of the hardware structure of a computer device provided by an embodiment of the present invention. Specific embodiments

[0035] The following further illustrates the specific embodiments of the present invention with reference to the accompanying drawings.

[0036] As Figure 1 shown is the workflow diagram of a data processing method for hybrid modeling provided by an embodiment of the present invention. The embodiment of the present invention provides a data processing method based on hybrid modeling, including the following steps:

[0037] Step S101: Construct a new model with the activation data of the underlying model as the input source.

[0038] Step S102: Customize the data upload process of the application model according to the business scenario through the processing chain technology, and automate the scheduling of data upload tasks.

[0039] Step S103: Dynamically switch the new model to replace the underlying model for data upload according to the data upload task, so as to realize the loose coupling of data upload and data extraction between the underlying model and the application model.

[0040] Among them, the data processing method provided by the embodiment of the present invention includes steps of hybrid modeling and model replacement, and is mainly applied to a commercial data warehouse. The commercial data warehouse supports an in-memory computing platform in enterprise-premise deployment and cloud deployment modes, and provides high-performance data query functions. Users can directly query and analyze a large amount of real-time business data without modeling, aggregating, etc. of the business data, especially applied to enterprise-level data warehouses such as SAP BW / 4HANA. The model in the present invention is a data container for receiving, storing, and processing data, such as a BW (Business Warehouse) model like ADSO (Advanced Data Store Object). Structurally, the ADSO model has a line item table and an activation table. The line item table is used to temporarily store the data extracted from the data source and the intermediate data during the data conversion process, and the activation table is used to store the valid data after verification and cleaning. At this time, the processing chain is a BW processing chain (Business Warehouse Process Chain), and the BW processing chain is a tool for automating and coordinating data processing tasks in the SAP BW environment.

[0041] With the wide application of big data platforms in data warehouses and data lakes, the data application scenarios of application models for underlying models are becoming increasingly extensive, and data calls are becoming more and more frequent. Therefore, there often appears a scenario where one underlying model needs to be uploaded to multiple application models, and the data updates of these application models all depend on the completion of the data updates of the underlying model before they can start. In this case, the data upload of the underlying model has become a bottleneck for data stream to update data.

[0042] The underlying model is the source layer, and there is a large amount of redundant data of the operational transaction system in the underlying model. Therefore, the data upload time of the underlying model will be relatively long. And application models often associate with multiple underlying models, making the reconciliation relationship between the underlying model and the application model complex. Therefore, when updating data in the application model, it is necessary to upload all the data of all underlying models associated with the application model before a relatively complete data stream update can be carried out. This makes the data update delay time of the application model relatively long, and the operation and maintenance difficulty is large. Once the data extraction or data upload of one of the underlying models fails, it will cause the data update of the application model to fail, increasing the workload of the operation and maintenance personnel.

[0043] Although real-time calculation can be achieved by using HANA views, when the business logic is complex and the data volume is large, the calculation time of HANA views will be relatively long, resulting in a longer waiting time for users when using the data warehouse, seriously affecting the user experience.

[0044] To this end, step S101 of the embodiment of the present invention constructs a new model based on the original underlying model in the data warehouse. Specifically, the activation data in the underlying model is used as the input source to construct the new model, and the activation of the new model is realized by extracting the intermediate data of the underlying model into the new model, so that the new model can independently upload data to the application model, and release the upload waiting or job deadlock in the data upload process of the underlying model. In step S102, the process control of automated coordinated data processing is carried out through the processing chain technology. The data upload process of the application model is independently controlled by configuring the processing chain, so that the data upload processes of multiple underlying models associated with the application model are executed asynchronously, or the underlying model can upload data to multiple application models simultaneously without interference. In step S103, in response to the data upload task, the new model is automatically and dynamically switched to replace the underlying model for data upload during the data upload process, and the loose coupling of data upload and data extraction between the underlying model and the application model is realized through the methods of hybrid modeling and model replacement. After loose coupling, data upload and data extraction can be carried out independently of each other. During the process of the underlying model uploading data replaced by the new model, the underlying model itself can also be executed for data extraction operations, which significantly improves the overall processing efficiency and response efficiency of the system, and also improves the update efficiency and overall stability of the data warehouse, and significantly improves the user experience.

[0045] In summary, the data processing method based on hybrid modeling provided by the embodiment of the present invention, through the methods of hybrid modeling and model replacement, constructs a new model based on the underlying model, and can replace the underlying model with the new model during the data upload process, decoupling the data upload and data extraction of the underlying model, enabling a single underlying model to upload data to multiple application models simultaneously, realizing that model data upload and model data extraction do not interfere with each other, and shortening the data delay time. On the other hand, the underlying model can also perform data extraction during the data upload process, which can improve the system operation efficiency, significantly improve the data update efficiency and data update stability, and thus improve the data output efficiency of the data model.

[0046] As Figure 2 shown, in one of the embodiments, step S101 includes the following sub-steps:

[0047] Step S201: Using the activation data of the underlying model as the input source, generate a data provider through visualization technology.

[0048] Step S202: Obtain the data source through the data provider, and make the metadata structure of the new model compatible with the underlying model.

[0049] Step S203: Based on the data provider, construct the data interface and processing logic of the new model, enabling the new model to independently complete data upload.

[0050] Furthermore, the new model extracts data from the underlying model to achieve activation.

[0051] Preferably, in the underlying model, a HANA View is generated through visualization technology. The underlying model and the new model are BW models with line item tables and activation tables. The activation data of the underlying model is the activation table, and the data provider assigns values to it through the HANA View.

[0052] Preferably, the data provider is a Composite Provider, which is used to integrate the input data of multiple data sources. The data sources include at least one of a database, an API interface, and an external file.

[0053] In this embodiment, by proposing a new transaction processing protocol for the locking problem in data upload of traditional data models, it is possible to reduce the locking time and conflict probability of transactions while ensuring data consistency. Then, through the combination of the BW model and the HANA View, loose coupling of data upload between the underlying model and the application model is achieved. When dealing with complex business logic, the output service efficiency of the data warehouse is improved through an improved BW processing chain.

[0054] Therefore, the overall process of the above steps is as follows:

[0055] Establish a HANA View on the activation table of the advanced data storage object in the underlying model. The HANA View is established through a graphical mode and is directly associated with the activation table. If SQLScript is used for HANA View modeling, graphical mode encapsulation is also required.

[0056] Create a Composite Provider based on the HANA View, and integrate the real-time data logic of the HANA View through the Composite Provider.

[0057] Construct a new model with the Composite Provider as the data source, extract data from the underlying model to activate the new model, and replace the dependency relationship of the corresponding underlying model with the new model. Enable the new model to perform data upload of the application model at any time, thereby improving the high availability of the underlying model.

[0058] Configure the BW processing chain. The BW processing chain independently controls the data upload process of the application model and executes asynchronously with the data upload process of the underlying model.

[0059] Through this method, it is only necessary to improve the efficiency of the HANA view itself, and then use the activation table of the model as a replacement view to output data services. By combining the HANA view and the BW model, data landing is achieved, greatly improving the data reading efficiency. The embodiment of the present invention solves the problems of traditional BW modeling data upload waiting and job deadlocks, improves the system operation efficiency, reduces the workload of operation and maintenance personnel, and solves the problem of too long report refresh time in the case of complex business logic of the HANA view through hybrid modeling and model replacement, thereby improving the data output service efficiency.

[0060] In one embodiment, step S102 includes the following sub-steps:

[0061] Step S301: Obtain the types of application models and business scenarios.

[0062] Step S302: Create a processing chain according to the application model and the business scenario.

[0063] Step S303: Define and schedule the task trigger conditions, task priorities, and exception handling strategies of the data upload process in the processing chain.

[0064] Further, the task trigger conditions include:

[0065] When a bottom layer model uploads data to multiple application models at the same time, the upload objects of each data upload task are respectively switched to a new model to relieve the bottleneck of the data upload speed of a single bottom layer model.

[0066] When multiple bottom layer models upload data at the same time, the upload object of the data upload task is switched to the new model to relieve the reconciliation relationship between the multiple bottom layer models.

[0067] Therefore, through the specific types of application models and business scenarios, the embodiment of the present invention dynamically switches between the bottom layer model and the new model during the data upload process, which can reduce the lock time and conflict probability of transactions on the premise of ensuring data consistency. Through the hybrid modeling solution of the BW model and the HANA view, transactions are allowed to read the old version of the data without blocking other transactions. At the same time, by optimizing the version management mechanism, a technology for improving the efficiency of fast data services is provided, improving the availability of the data warehouse system and the efficiency of data updates.

[0068] Preferably, the dynamic switching of the new model to replace the bottom layer model for data upload is realized through a hot standby mechanism. The new model and the bottom layer model share the same data provider, and high availability is maintained between the new model and the bottom layer model through heartbeat detection.

[0069] In one embodiment, step S103 is implemented as follows:

[0070] In the data upload task, only rely on the newly created model to complete the data upload operation.

[0071] In the data extraction task, it is implemented by asynchronously calling the underlying model and the newly created model, and is decoupled from the data upload task.

[0072] As a specific example, when real-time transaction data reports need to be generated in financial business, but the data upload time of the underlying model is too long, the existing data processing methods result in a long delay in updating the application model and cannot meet the real-time requirements. The method provided by the present invention can meet the real-time requirements, and the specific steps are as follows:

[0073] Establish a HANA view (graphical mode modeling) on the activation table of the underlying model, and define real-time data filtering rules (such as transaction timestamp, amount threshold).

[0074] Integrate this HANA view with other business dimension tables (such as customer information table) through a composite provider.

[0075] Construct a newly created model with the composite provider as the information provider, and use this newly created model as a highly available data source to replace the original underlying model.

[0076] Build a BW processing chain based on the relationship between the application model, the underlying model, and the newly created model, set the application model to extract data from the composite provider every 5 minutes in incremental mode, and activate the newly created model.

[0077] The reporting system directly accesses the activation table of the new application model to replace the original HANA view query.

[0078] Through the above steps, it is possible to completely decouple the data upload of the underlying model from the extraction of the application model, greatly shortening the data delay. For example, the data delay is reduced from several hours to 5 minutes. And for the complex calculation logic of the original HANA view (such as multi-table association), it is implemented through model pre-computation, greatly shortening the report response time. For example, it is shortened from 3 minutes to 10 seconds.

[0079] In another specific example, if the write frequency of manufacturing Internet of Things data is high (thousands of records per second), the traditional BW model has a data extraction failure rate exceeding 15% due to table locking. At this time, the method provided by the present invention can be used to reduce the data extraction failure rate. The specific application steps are as follows:

[0080] Create a redundant HANA view on the activation table of the underlying model, and use graphical mode partitioning (sharding by timestamp).

[0081] Establish dual-active composite providers (Composite Provider A / B), and bind them to HANA views in different partitions respectively.

[0082] Create new dual-application models (ADSO_A, ADSO_B), bind them to Composite Provider A and Composite Provider B respectively, and set the alternate activation strategy.

[0083] Design a BW processing chain that includes a primary chain and a backup chain. Specifically: Primary chain: The underlying model is written in real time at the second level and activated independently. Backup chain: The application model is batch-extracted at the minute level through the composite provider, and the A / B models are alternately activated to avoid table locking.

[0084] The reporting system accesses the latest activated application model through dynamic routing.

[0085] Through the above method, it is possible to completely isolate high-concurrency writing and batch extraction, significantly improve the system throughput, and significantly reduce the failure rate of data extraction. For example, the data extraction failure rate is reduced from 15% to 0.3%.

[0086] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0087] As Figure 4 shown in the schematic diagram of the hardware structure of a computer device according to the present invention, it includes a memory 402, a processor 401, and a computer program on the memory 402. The processor 401 executes the computer program to implement the steps of the data processing method for hybrid modeling in any of the above embodiments.

[0088] Figure 4 Taking one processor 401 as an example.

[0089] The computer device may further include: an input device 403 and a display device 404.

[0090] The processor 401, the memory 402, the input device 403, and the display device 404 may be connected through a bus or other means. In the figure, it is shown as being connected through a bus as an example.

[0091] The memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the data processing method for hybrid modeling in the embodiments of the present application. The processor 401 executes various functional applications and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 402, that is, implements the data processing method for hybrid modeling in the above embodiments.

[0092] The memory 402 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function. The data storage area may store data created according to the use of the data processing method of hybrid modeling, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 402 may optionally include a memory remotely provided with respect to the processor 401, and these remote memories may be connected to the device for executing the data processing method of hybrid modeling through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0093] The input device 403 may receive input user clicks and generate signal inputs related to user settings and function controls of the data processing method of hybrid modeling. The display device 404 may include a display device such as a display screen.

[0094] When the one or more modules are stored in the memory 402 and run by the one or more processors 401, the data processing method of hybrid modeling in any of the above method embodiments is executed.

[0095] When the computer device disclosed in the present invention is running, it can execute all steps of the above data processing method of hybrid modeling. Through the data processing method of hybrid modeling, by means of hybrid modeling and model replacement, the underlying model is replaced with a newly created model during the data upload process, decoupling the data upload and data extraction of the underlying model, enabling a single underlying model to upload data to multiple application models simultaneously, achieving the effect that the model data upload and model data extraction do not interfere with each other, thereby achieving the purpose of shortening the data latency time and significantly improving the response efficiency and stability of the data warehouse.

[0096] An embodiment of the present invention provides a computer-readable storage medium, which stores a computer program / instructions, and when the computer program / instructions are executed by the processor 401, all steps of the data processing method of hybrid modeling as described above are implemented.

[0097] In the context of the present disclosure, a storage medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The storage medium may be a machine-readable signal medium or a machine-readable storage medium. Optionally, the storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be ROM, Random Access Memory (RAM), Compact Disc ROM (CD-ROM), magnetic tape, floppy disk, and optical data storage devices, etc.

[0098] An embodiment of the present invention provides a computer program product, including a computer program / instructions, which when executed by a processor, implement the steps of the data processing method for hybrid modeling as described above.

[0099] By running the above computer program product, all the steps of the data processing method for hybrid modeling as described above can be executed. The data processing method for hybrid modeling decouples the data uploading and data extraction of the underlying model by replacing the underlying model with a newly created model during the data uploading process through hybrid modeling and model replacement, enabling a single underlying model to upload data to multiple application models simultaneously, achieving the effect that the model data uploading and model data extraction do not interfere with each other, thereby achieving the purpose of shortening the data latency time and significantly improving the response efficiency and stability of the data warehouse.

[0100] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

1. A data processing method based on hybrid modeling, characterized in that: include: Build a new model using the activation data of the underlying model as input source; The data upload process of the application model is customized according to the business scenario through the processing chain technology, and the data upload task is automatically scheduled; According to the data upload task, the newly created model is dynamically switched to replace the underlying model for data upload, thereby achieving loose coupling of data upload and data extraction between the underlying model and the application model.

2. The data processing method based on hybrid modeling according to claim 1, characterized in that: The step of constructing a new model using the activation data of the underlying model as an input source includes: Taking the activation data of the underlying model as input source, generating a data provider through visualization technology; Acquire the data source through the data provider, so that the metadata structure of the newly created model is compatible with the underlying model; The data interface and processing logic of the newly created model are constructed based on the data provider, so that the newly created model can independently complete data uploading.

3. The data processing method based on hybrid modeling according to claim 2 is characterized in that: The newly created model extracts data from the underlying model to achieve activation.

4. The data processing method based on hybrid modeling according to claim 3 is characterized in that: A HANA view is generated in the underlying model by using a visualization technology. The underlying model and the newly created model are BW models having a line item table and an activation table. The activation data of the underlying model is the activation table, and the data provider assigns values ​​to it through the HANA view.

5. The data processing method based on hybrid modeling according to any one of claims 2 to 4, characterized in that: The data provider is a composite provider, which is used to integrate input data from multiple data sources, and the data source includes at least one of a database, an API interface, and an external file.

6. The data processing method based on hybrid modeling according to claim 1, characterized in that: The processing chain technology is used to customize the data upload process of the application model according to the business scenario, and automatically schedule the data upload task, including: Get the types of application models and business scenarios; Creating a processing chain according to the application model and the business scenario; The task triggering conditions, task priorities and exception handling strategies of the data upload process are defined and scheduled in the processing chain.

7. The data processing method based on hybrid modeling according to claim 6, characterized in that: The task triggering conditions include: When one of the underlying models uploads data to multiple application models at the same time, the upload object of each data upload task is switched to one of the newly created models respectively, so as to remove the bottleneck of the data upload speed of a single underlying model; When multiple underlying models are uploading data at the same time, the upload object of the data upload task is switched to the newly created model to release the cross-reference relationship between the multiple underlying models.

8. The data processing method based on hybrid modeling according to claim 1, characterized in that: The dynamic switching of the newly created model to replace the underlying model for data uploading is achieved through a hot standby mechanism. The newly created model and the underlying model share the same data provider, and high availability is maintained between the newly created model and the underlying model through heartbeat detection.

9. The data processing method based on hybrid modeling according to claim 1, characterized in that: The loose coupling of data upload and data extraction between the underlying model and the application model is achieved by: In the data upload task, only the newly created model is relied upon to complete the data upload operation; In the data extraction task, it is implemented by asynchronously calling the underlying model and the newly created model, and is decoupled from the data upload task.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data processing method based on hybrid modeling described in any one of claims 1 to 9 are implemented.

Images