Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first data flow to be transmitted to an OLAP system in an OLTP system, enabling the first data flow to comprise the current service data of the OLTP system, carrying out the association processing of the first data flow and a second data flow based on a preset association rule, and enabling the first data flow to be transmitted to the OLAP system; the second data stream comprises service data to be sent to the OLAP system in the target service system at the current moment, so that in the process of performing data analysis and query through the OLAP system, the current data stream is obtained, and the current data stream is sent to the OLAP system. Real-time service data of the OLTP system and real-time data streams of other service systems can be integrated and then sent to the OLAP system, and the data analysis and query efficiency of the OLAP system is effectively improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the increasing demand for digitalization and data-driven approaches, OLTP (Online Transaction Processing) systems and OLAP (Online Analytical Processing) systems have wide applications. OLTP systems are mainly used for real-time business operations, while OLAP systems are mainly used for complex data analysis and querying. In particular, OLAP systems require data generated by OLTP systems during business operations for data analysis and querying.

[0003] Traditional methods typically use ETL (Extract, Transform, Load) to synchronize data from OLTP systems to OLAP systems. This approach suffers from poor data real-time performance and the inability to integrate real-time data streams sent to the OLAP system from other business systems. Consequently, during high-performance, high-volume analysis and querying in the OLAP system, synchronous feedback from the OLTP system cannot be achieved, thus compromising the efficiency of data analysis and querying in the OLAP system. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a data processing method, apparatus, electronic device, and storage medium to resolve the issue of the inability to achieve synchronous feedback operations with OLTP systems during high-performance, massive-scale analysis and querying in OLAP systems.

[0005] To achieve the above technical objectives, the embodiments of this application provide the following technical solutions:

[0006] Firstly, the embodiments of this specification provide a data processing method, including:

[0007] Acquire the first data stream in the OLTP system to be sent to the OLAP system, the first data stream including the current service data of the OLTP system;

[0008] The first data stream and the second data stream are associated based on a preset association rule to obtain the current data stream. The second data stream includes the business data in the target business system that is to be sent to the OLAP system at the current time.

[0009] The current data stream is sent to the OLAP system.

[0010] Optionally, obtaining the first data stream to be sent from the OLTP system to the OLAP system includes:

[0011] Obtain the current change log of the database corresponding to the OLTP system, and extract the changed data from the current business data of the OLTP system from the current change log;

[0012] The changed data is processed to obtain the first data stream.

[0013] Optionally, the step of processing the changed data to obtain the first data stream includes:

[0014] The data format of the changed data is converted into a target data format, which is compatible with the data format of the second data stream.

[0015] The changed data after data format conversion is identified as the first data stream.

[0016] Optionally, converting the data format of the changed data into the target data format includes:

[0017] Based on the target table structure information, the data format of the changed data is converted into the target data format. The target table structure information includes the table structure information of the target data table in the database corresponding to the OLTP system. The target data table is used to store the changed data.

[0018] Optionally, sending the current data stream to the OLAP system includes:

[0019] Cache multiple first target data in the current data stream into a first preset storage unit;

[0020] When it is determined that all the first target data have been cached, the first target data are combined based on the preset order corresponding to each first target data to obtain the second target data;

[0021] The second target data is sent to the OLAP system.

[0022] Optionally, sending the current data stream to the OLAP system includes:

[0023] Based on the comparison results between the current data stream and the historical data cached in the second preset storage unit, a third target data is determined. The historical data includes all data sent to the OLAP system within the current lifecycle, and the third target data includes updated data in the current data stream compared to the historical data.

[0024] The third target data is sent to the OLAP system.

[0025] Optionally, the second preset storage unit includes a first sub-storage unit and a second sub-storage unit, wherein the first sub-storage unit is a non-external storage unit and the second sub-storage unit is an external storage unit;

[0026] After determining the third target data, the process also includes:

[0027] The field value corresponding to the first field in the third target data is cached in the first sub-storage unit, and the field value corresponding to the second field in the third target data is cached in the second sub-storage unit, wherein the importance of the first field is greater than that of the second field.

[0028] Secondly, embodiments of this specification provide a data processing apparatus, including:

[0029] The first processing module is used to acquire the first data stream in the OLTP system to be sent to the OLAP system, the first data stream including the current business data of the OLTP system;

[0030] The second processing module is used to perform association processing on the first data stream and the second data stream based on preset association rules to obtain the current data stream. The second data stream includes the business data in the target business system to be sent to the OLAP system at the current time.

[0031] The third processing module is used to send the current data stream to the OLAP system.

[0032] Thirdly, embodiments of this specification provide an electronic device, including a memory and a processor;

[0033] The memory is connected to the processor and is used to store programs;

[0034] The processor is used to implement the data processing method as described above by running a program in the memory.

[0035] Fourthly, embodiments of this specification provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data processing method described in any of the preceding claims.

[0036] Fifthly, embodiments of this specification provide a computer program product or a computer program, the computer program product including a computer program stored in a computer-readable storage medium; a processor of the computer device reads the computer program from the computer-readable storage medium, and when the processor executes the computer program, it implements the data processing method as described in any of the preceding claims.

[0037] As can be seen from the above technical solutions, the embodiments of this application provide a data processing method, apparatus, electronic device, and storage medium. The data processing method obtains a first data stream from the OLTP system to be sent to the OLAP system. The first data stream includes the current business data of the OLTP system. Based on preset association rules, the first data stream and a second data stream are associated to obtain a current data stream, which is then sent to the OLAP system. The second data stream includes the business data from the target business system to be sent to the OLAP system at the current moment. Thus, during the data analysis and query process through the OLAP system, the real-time business data of the OLTP system can be integrated with the real-time data streams of other business systems and sent to the OLAP system, effectively improving the data analysis and query efficiency of the OLAP system. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a data processing method provided for one embodiment of this specification;

[0040] Figure 2 A flowchart illustrating another data processing method provided as one embodiment of this specification;

[0041] Figure 3 A schematic diagram of the structure of a data processing apparatus provided for one embodiment of this specification;

[0042] Figure 4 This is a schematic diagram of an electronic device provided for one embodiment of this specification. Detailed Implementation

[0043] Unless otherwise defined, the technical or scientific terms used in the embodiments of this specification shall have the ordinary meaning understood by one of ordinary skill in the art to which this specification pertains. The terms "first," "second," and similar terms used in the embodiments of this specification do not indicate any order, quantity, or importance, but are merely used to avoid confusion of constituent elements.

[0044] Unless the context otherwise requires, throughout this specification, "a plurality of" means "at least two," and "including" is interpreted as open-ended or encompassing, that is, "including, but not limited to." In the description of this specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this specification. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example.

[0045] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0046] Overview

[0047] As described in the background section, with the increasing demand for digitalization and data-driven approaches, OLTP and OLAP systems have wide applications. OLTP systems are mainly used for real-time business operations, while OLAP systems are mainly used for complex data analysis and querying. In particular, OLAP systems require data generated by OLTP systems during business operations for data analysis and querying.

[0048] Traditional methods typically use ETL to synchronize data from OLTP systems to OLAP systems. ETL processes business data generated in the OLTP system offline in batches and loads it into the OLAP system. This approach suffers from poor data real-time performance and the inability to integrate real-time data streams sent to the OLAP system from other business systems. Consequently, during high-performance, high-volume analysis and querying in the OLAP system, synchronous feedback from the OLTP system cannot be achieved. Therefore, after the OLTP system updates data, it cannot provide real-time feedback to the OLAP system, thus compromising the efficiency of data analysis and querying in the OLAP system.

[0049] To address the issue that traditional methods cannot achieve synchronous feedback from OLTP systems during high-performance, high-volume analysis and querying in OLAP systems, the technical solution in this application obtains a first data stream from the OLTP system to be sent to the OLAP system. This first data stream includes the current business data of the OLTP system. Based on preset association rules, the first data stream and a second data stream are associated to obtain the current data stream, which is then sent to the OLAP system. The second data stream includes the business data from the target business system to be sent to the OLAP system at the current moment. Thus, during data analysis and querying through the OLAP system, the real-time business data of the OLTP system can be integrated with the real-time data streams of other business systems before being sent to the OLAP system, effectively improving the data analysis and querying efficiency of the OLAP system.

[0050] Based on the above inventive concept, the data processing method provided in the embodiments of this specification will be described exemplarily below.

[0051] Exemplary methods

[0052] This specification provides a data processing method, such as... Figure 1 As shown, it includes:

[0053] S101. Obtain the first data stream in the OLTP system to be sent to the OLAP system, the first data stream including the current service data of the OLTP system.

[0054] Specifically, an OLTP system can be an online business operating system, enabling the insertion, updating, and deletion of business data. For example, an OLTP system can be an order processing system. There can be one or more OLTP systems, which can be used for different online business operations.

[0055] OLAP systems can be data analysis and query systems used to analyze and query business data from multiple business systems to provide decision support, etc.

[0056] During implementation, the first data stream to be sent from the OLTP system to the OLAP system can be obtained in real time. The first data stream may include the current business data of the OLTP system, which may include the business data generated in the OLTP system at the current moment, thereby enabling the data in the OLTP system to be synchronized to the OLAP system in real time.

[0057] This process can generate a first data stream from all business data generated in the OLTP system at the current moment. It can also extract changed data from the business data generated in the OLTP system at the current moment and generate a first data stream from the changed data. Additionally, it can generate a first data stream from specified data within the changed data. The specified data can be the data required by the OLAP system for data analysis and querying. In implementation, the data category of the specified data can be preset. The first data stream can be a Kafka data stream; for example, the current business data from the OLTP system can be processed and written to a Kafka message queue to obtain the first data stream.

[0058] S102. Based on preset association rules, the first data stream and the second data stream are associated to obtain the current data stream. The second data stream includes the business data in the target business system to be sent to the OLAP system at the current time.

[0059] Specifically, the second data stream may include business data in the target business system that is to be sent to the OLAP system at the current moment. The target business system may be a business system outside the OLTP system in the data source system of the OLAP system, such as a warehouse system. The data source system of the OLAP system is the system that provides the data source for the OLAP system to perform data analysis and query.

[0060] The second data stream can be a Kafka data stream, and there can be one or more second data streams. For example, each target business system can correspond to one or more second data streams, which can be set according to actual needs.

[0061] Preset association rules can be pre-defined rules for associating data sent to the OLAP system. For example, preset association rules can include association fields, such as order number and business ID (Identification).

[0062] In practice, the first data stream and the second data stream can be correlated based on preset association rules to obtain the current data stream. The current data stream can include each data in the first data stream, each data in the second data stream, and the corresponding association rules.

[0063] S103. Send the current data stream to the OLAP system.

[0064] Specifically, the current data stream can be sent directly to the OLAP system, or it can be processed by deduplication and / or reorganization, and the processed data can be written into a new Kafka message queue and sent to the OLAP system. This allows business data generated in the OLTP system to be integrated with real-time data streams from other business systems that are to be sent to the OLAP system before being sent to the OLAP system, ensuring the real-time performance and integrity of the data sent to the OLAP system, thereby improving the data analysis and query efficiency of the OLAP system.

[0065] Therefore, the data processing method provided in this specification obtains a first data stream from the OLTP system to be sent to the OLAP system. The first data stream includes the current business data of the OLTP system. Based on preset association rules, the first data stream and a second data stream are associated to obtain the current data stream, which is then sent to the OLAP system. The second data stream includes the business data from the target business system to be sent to the OLAP system at the current moment. Thus, during the data analysis and query process through the OLAP system, the real-time business data of the OLTP system can be integrated with the real-time data streams of other business systems and sent to the OLAP system, effectively improving the data analysis and query efficiency of the OLAP system.

[0066] To quickly and effectively acquire the first data stream in the OLTP system to be sent to the OLAP system, in one embodiment of this specification, acquiring the first data stream in the OLTP system to be sent to the OLAP system includes:

[0067] Obtain the current change log of the database corresponding to the OLTP system, and extract the changed data from the current business data of the OLTP system from the current change log;

[0068] The changed data is processed to obtain the first data stream.

[0069] Specifically, the current change log of the OLTP system's database can be extracted in real time. The current change log is the change log generated by the OLTP system's database at the current moment. The change log can be a Binlog (Binary log file), and the OLTP system's database can be a relational database that supports Binlog.

[0070] During implementation, relevant read permissions can be configured. Only after the permission verification is passed can the change logs of the corresponding database in the OLTP system be extracted.

[0071] After extracting the current change log of the database corresponding to the OLTP system, the current change log can be parsed to extract the changed data in the current business data of the OLTP system based on the parsing results. The changed data can include the fields that have been changed and the field values ​​of those fields.

[0072] On the one hand, by extracting changed data from the current business data generated by the OLTP system based on the current change log of the database corresponding to the OLTP system, it is possible to ensure real-time acquisition of business data generated by the OLTP system while effectively reducing the need to modify the structure of the OLTP system, thereby reducing the intrusion and development workload of the OLTP system. At the same time, it can also quickly access business data generated by multiple OLTP systems.

[0073] On the other hand, by extracting changed data from current business data, the amount of data transmission and processing can be effectively reduced, further improving the real-time synchronization of business data generated in the OLTP system to the OLAP system.

[0074] After extracting changed data from the current business data of the OLTP system, the changed data can be processed and written to a Kafka message queue to obtain the first data stream. Data processing can include data format conversion, noise reduction, and validation, etc., which can be configured according to actual needs.

[0075] In one feasible implementation, the step of processing the changed data to obtain the first data stream includes:

[0076] The data format of the changed data is converted into a target data format, which is compatible with the data format of the second data stream.

[0077] The changed data after data format conversion is identified as the first data stream.

[0078] Specifically, the change data is extracted from the current change log of the database corresponding to the OLTP system. Therefore, the change data can be in text format.

[0079] The target data format can be compatible with the data format of the second data stream. For example, the target data format can be row data, DTO (Data Transfer Object), etc. This allows for fast and efficient association processing of the first and second data streams. Simultaneously, the current data stream in the target data format enables rapid and accurate updates to the corresponding database in the OLAP system.

[0080] After converting the changed data format to the target data format, the converted changed data can be written to the Kafka message queue to obtain the first data stream, thereby enabling the quick and efficient acquisition of the first data stream to be sent from the OLTP system to the OLAP system.

[0081] In one feasible implementation, converting the data format of the changed data into the target data format includes:

[0082] Based on the target table structure information, the data format of the changed data is converted into the target data format. The target table structure information includes the table structure information of the target data table in the database corresponding to the OLTP system. The target data table is used to store the changed data.

[0083] Specifically, the target data table is the data table in the database corresponding to the OLTP system used to store changed data. During implementation, the data tables that need to be monitored in the database corresponding to the OLTP system can be pre-configured, and the table structure information of each monitored data table can be registered. The table structure information is used to characterize the attributes of the corresponding data table; for example, it can include the table name, column names, data types, etc. This allows for quick and accurate determination of the corresponding table structure information when changed data is obtained, and enables data format conversion of the changed data based on the corresponding table structure information.

[0084] The version of the table structure information of each data table that needs to be monitored can be the DDL (Data Definition Language) version of the database corresponding to the OLTP system. The DDL version of the database corresponding to the OLTP system can be used to track changes in the database structure, thereby effectively managing the evolution process of the database. Therefore, when the table structure information of each data table that needs to be monitored changes, it will not affect the changed data consumed in the early and later stages.

[0085] In implementation, when the target data format is row data, the changed data can be converted into row data based on the column names and data types in the target table structure information. When the target data format is DTO, a DTO class can be determined based on the target table structure information (e.g., column names). This class includes fields and attributes, and the values ​​of the fields in the changed data that have undergone data changes are assigned to the attributes of the corresponding fields in the DTO class to obtain the changed data in DTO format.

[0086] To improve the reliability of data sent to the OLAP system, in one embodiment of this specification, sending the current data stream to the OLAP system includes:

[0087] Cache multiple first target data in the current data stream into a first preset storage unit;

[0088] When it is determined that all the first target data have been cached, the first target data are combined based on the preset order corresponding to each first target data to obtain the second target data;

[0089] The second target data is sent to the OLAP system.

[0090] Specifically, due to factors such as data transmission rate, the order in which multiple data streams are received simultaneously cannot be guaranteed. The first target data can be data that is not allowed to be out of order in the current data stream. That is, if multiple first target data streams are out of order, it will significantly affect the accuracy of subsequent data analysis and querying in the OLAP system. For example, during the order creation process, if order details and order change information are sent simultaneously, and the order change information is received first, an error will occur during data analysis and querying in the OLAP system.

[0091] In implementation, fields that are not allowed to be out of order, and the order of these fields, can be pre-defined. When multiple corresponding fields exist in the current data stream, they can be used as the first target data and cached in the first preset storage unit. The type of the first preset storage unit is not specifically limited here, as long as it can reliably store each first target data. For example, a storage unit built into an electronic device can be used. The electronic device is the device executing the data processing method of this application. The storage unit built into the electronic device has high data compatibility and data consistency, thus enabling reliable storage of the first target data.

[0092] Once it is confirmed that all first target data have been cached, the first target data can be combined based on the preset order of the fields corresponding to each first target data to obtain a complete data record, which is then used as the second target data. This second target data is then sent to the OLAP system. This ensures that no key data and / or status is lost within a certain degree of out-of-order tolerance, thereby guaranteeing the accuracy of the data analysis query results of the OLAP system.

[0093] When sending the second target data to the OLAP system, the second target data can be directly written to the Kafka message queue to send the second target data to the OLAP system via data stream. Alternatively, the second target data can be deduplicated and / or reassembled, and the processed data can be written to the Kafka message queue and sent to the OLAP system.

[0094] In addition, after combining each first target data into second target data, the cached first target data in the first preset storage unit can be deleted, and only the second target data can be retained, so as to ensure the validity of the cached data in the first preset storage unit and reduce the storage space requirement of the first preset storage unit.

[0095] To improve the data analysis and query efficiency of OLAP systems, in one embodiment of this specification, sending the current data stream to the OLAP system includes:

[0096] Based on the comparison results between the current data stream and the historical data cached in the second preset storage unit, a third target data is determined. The historical data includes all data sent to the OLAP system within the current lifecycle, and the third target data includes updated data in the current data stream compared to the historical data.

[0097] The third target data is sent to the OLAP system.

[0098] Specifically, the second preset storage unit is used to cache all data sent to the OLAP system within the current lifecycle. The current lifecycle can be the current data update cycle of the OLAP system or the current business processing cycle of the OLAP system, and can be set according to actual needs.

[0099] The second preset storage unit may be the same as or different from the first preset storage unit; this embodiment does not impose any specific limitations on this.

[0100] The historical data cached in the second preset storage unit includes all data sent to the OLAP system within the current lifecycle. The data in the current data stream can be compared with the historical data cached in the second preset storage unit. Based on the comparison results, when there is updated data in the current data stream compared with the historical data, the third target data can be determined based on the updated data. This effectively avoids sending duplicate data to the OLAP system, reduces the amount of data processing in the OLAP system, and thus effectively improves the data analysis and query efficiency of the OLAP system.

[0101] The third target data can be determined based on the updated data and the type of database corresponding to the OLAP system. For example, if the database supports field-by-field modification based on its type, the updated data and the value of the corresponding identifier field can be used as the third target data. The identifier field could be an order number, business ID, etc. If the database does not support field-by-field modification, meaning only entire rows can be modified, the target field value can be obtained from the historical data cached in the second preset storage unit based on the table structure information of the table corresponding to the updated data in the OLAP system's database. This target field value is then merged with the updated data to obtain the third target data. The target field is the field in the target row of the table corresponding to the updated data in the OLAP system's database, excluding the fields in the updated data itself. The target row is the row that includes the fields corresponding to the updated data.

[0102] After determining the third target data, the third target data can be written into the Kafka message queue so that it can be sent to the OLAP system via data stream.

[0103] Understandably, at the end of the current lifecycle, the data in the second preset storage unit can be cleared. That is, at the beginning of the current lifecycle, the historical data is empty, which can effectively reduce the storage space requirement of the second preset storage unit. At the same time, when sending the current data stream to the OLAP system, it is only necessary to compare the data in the current data stream with the data sent to the OLAP system in this lifecycle, which improves the real-time performance of sending data to the OLAP system.

[0104] In one feasible implementation, the second preset storage unit includes a first sub-storage unit and a second sub-storage unit, wherein the first sub-storage unit is a non-external storage unit and the second sub-storage unit is an external storage unit;

[0105] After determining the third target data, the process also includes:

[0106] The field value corresponding to the first field in the third target data is cached in the first sub-storage unit, and the field value corresponding to the second field in the third target data is cached in the second sub-storage unit, wherein the importance of the first field is greater than that of the second field.

[0107] Specifically, the second preset storage unit may include a first sub-storage unit and a second sub-storage unit. The first sub-storage unit may be a non-external storage unit, that is, a storage unit built into the electronic device. The second sub-storage unit may be an external storage unit, that is, a storage unit connected to the electronic device. Therefore, the data compatibility and data consistency of the first sub-storage unit are higher than those of the second sub-storage unit, and data storage can be performed more reliably.

[0108] The electronic device can utilize cloud service resources to perform data processing methods. These cloud service resources can include computing resources and storage space, and the ratio of computing resources to storage space in the cloud service resources is typically a fixed value. In implementation, the storage space of the first sub-storage unit can be determined based on the computing resource requirements of the electronic device and the ratio of computing resources to storage space in the cloud service resources. The storage space of the second sub-storage unit is then determined based on the storage space of the first sub-storage unit and the required storage space. For example, the storage space of the second sub-storage unit can be greater than or equal to the difference between the required storage space and the storage space of the first sub-storage unit. It is understood that the first preset storage unit can utilize the first sub-storage unit.

[0109] In practice, after determining the third target data, the updated data in the third target data can be cached in the second preset storage unit. It is understood that all the data in the third target data can also be cached in the second preset storage unit.

[0110] In the process of caching the third target data, the field value corresponding to the first field in the third target data can be cached in the first sub-storage unit. For example, the field value corresponding to the first field in the third target data can be added to the corresponding field in the first sub-storage unit. That is, the first sub-storage unit can record the change process of the field value of each first field in the current life cycle for easy traceability. In addition, the field value of the corresponding field in the first sub-storage unit can also be updated based on the field value corresponding to the first field in the third target data. That is, the latest value of each first field is recorded through the first sub-storage unit to reduce the storage space requirement of the first sub-storage unit.

[0111] At the same time, the field value corresponding to the second field can be cached in the second sub-storage unit. For example, the field value corresponding to the second field in the third target data can be added to the corresponding field in the second sub-storage unit. That is, the second sub-storage unit can record the change process of the value of each second field in the current life cycle. In addition, the field value of the corresponding field in the second sub-storage unit can also be updated based on the field value corresponding to the second field in the third target data. That is, the latest value of each second field is recorded through the second sub-storage unit.

[0112] In this context, the first field is more important than the second field. For example, the first field may include fields related to calculations and / or key values ​​used for aggregation, while the second field may include fields other than the first field in the OLAP system's required fields. Thus, when the ratio of the electronic device's required computing resources to its required storage space is less than the ratio of computing resources to storage space in the cloud service resources, setting up a second sub-storage unit can ensure reliable storage of data in the current data stream while effectively avoiding waste of computing resources.

[0113] The implementation process of the data processing method of the present invention is illustrated below through a feasible embodiment. (Refer to...) Figure 2 The method includes:

[0114] S201. Obtain the current change log of the database corresponding to the OLTP system, and extract the changed data from the current business data of the OLTP system from the current change log;

[0115] S202. Process the changed data to obtain the first data stream;

[0116] S203. Based on preset association rules, perform association processing on the first data stream and the second data stream to be sent to the OLAP system in the target business system at the current time to obtain the current data stream;

[0117] S204. Based on the updated data in the current data stream compared to the historical data cached in the second preset storage unit, determine the third target data;

[0118] S205. Send the third target data to the OLAP system, and at the same time, cache the third target data in the second preset storage unit.

[0119] Exemplary device

[0120] refer to Figure 3 This specification also provides a data processing apparatus, comprising:

[0121] The first processing module 301 is used to acquire a first data stream in the OLTP system to be sent to the OLAP system, the first data stream including the current service data of the OLTP system;

[0122] The second processing module 302 is used to perform association processing on the first data stream and the second data stream based on a preset association rule to obtain the current data stream. The second data stream includes business data in the target business system to be sent to the OLAP system at the current moment.

[0123] The third processing module 303 is used to send the current data stream to the OLAP system.

[0124] For example, the first processing module 301 is specifically used for:

[0125] Obtain the current change log of the database corresponding to the OLTP system, and extract the changed data from the current business data of the OLTP system from the current change log;

[0126] The changed data is processed to obtain the first data stream.

[0127] For example, the first processing module 301 is specifically used for:

[0128] The data format of the changed data is converted into a target data format, which is compatible with the data format of the second data stream.

[0129] The changed data after data format conversion is identified as the first data stream.

[0130] For example, the first processing module 301 is specifically used for:

[0131] Based on the target table structure information, the data format of the changed data is converted into the target data format. The target table structure information includes the table structure information of the target data table in the database corresponding to the OLTP system. The target data table is used to store the changed data.

[0132] For example, the third processing module 303 is specifically used for:

[0133] Cache multiple first target data in the current data stream into a first preset storage unit;

[0134] When it is determined that all the first target data have been cached, the first target data are combined based on the preset order corresponding to each first target data to obtain the second target data;

[0135] The second target data is sent to the OLAP system.

[0136] For example, the third processing module 303 is specifically used for:

[0137] Sending the current data stream to the OLAP system includes:

[0138] Based on the comparison results between the current data stream and the historical data cached in the second preset storage unit, a third target data is determined. The historical data includes all data sent to the OLAP system within the current lifecycle, and the third target data includes updated data in the current data stream compared to the historical data.

[0139] The third target data is sent to the OLAP system.

[0140] For example, the second preset storage unit includes a first sub-storage unit and a second sub-storage unit, wherein the first sub-storage unit is a non-external storage unit and the second sub-storage unit is an external storage unit; the device further includes a fourth processing module, the fourth processing module being used for:

[0141] The field value corresponding to the first field in the third target data is cached in the first sub-storage unit, and the field value corresponding to the second field in the third target data is cached in the second sub-storage unit, wherein the importance of the first field is greater than that of the second field.

[0142] The data processing apparatus provided in this embodiment belongs to the same concept as the data processing method provided in the above embodiments of this application. It can execute the data processing method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the data processing method. Technical details not described in detail in this embodiment can be found in the specific processing content of the data processing method provided in the above embodiments of this application, and will not be repeated here.

[0143] Exemplary electronic devices

[0144] This specification also provides an electronic device, see [link to embodiment]. Figure 4 As shown, the device includes a memory 401 and a processor 402;

[0145] The memory 401 is connected to the processor 402 and is used to store programs;

[0146] The processor 402 is used to implement the data processing method as described in any of the above embodiments by running the program in the memory 401.

[0147] Specifically, the electronic device may also include: a bus, a communication interface 403, an input device 404, and an output device 405.

[0148] The processor 402, memory 401, communication interface 403, input device 404, and output device 405 are interconnected via a bus. Among them:

[0149] A bus can include a pathway for transmitting information between various components of a computer system.

[0150] Processor 402 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0151] The memory 401 stores a program that executes the technical solution of the present invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 401 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0152] Input device 404 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0153] Output device 405 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0154] The communication interface 403 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0155] The processor 402 executes the program stored in the memory 401 and calls other devices, which can be used to implement the various steps of any of the data processing methods provided in the above embodiments of this application.

[0156] Exemplary computer program products and storage media

[0157] In addition to the methods and devices described above, the data processing methods provided in the embodiments of this specification can also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the data processing methods according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0158] The computer program product described herein can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments described herein. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0159] Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps in the data processing methods according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this specification can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The embodiments described above are merely illustrative of several implementation methods outlined in this specification. While the descriptions are specific and detailed, they should not be construed as limiting the scope of the solutions provided in this specification. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this specification, and these all fall within the scope of protection of this specification. Therefore, the scope of protection for this patent should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, include: Acquire the first data stream in the OLTP system to be sent to the OLAP system, the first data stream including the current service data of the OLTP system; The first data stream and the second data stream are associated based on a preset association rule to obtain the current data stream. The second data stream includes the business data in the target business system that is to be sent to the OLAP system at the current time. The current data stream is sent to the OLAP system.

2. The method according to claim 1, characterized in that, The acquisition of the first data stream to be sent from the OLTP system to the OLAP system includes: Obtain the current change log of the database corresponding to the OLTP system, and extract the changed data from the current business data of the OLTP system from the current change log; The changed data is processed to obtain the first data stream.

3. The method according to claim 2, characterized in that, The step of processing the changed data to obtain the first data stream includes: The data format of the changed data is converted into a target data format, which is compatible with the data format of the second data stream. The changed data after data format conversion is identified as the first data stream.

4. The method according to claim 3, characterized in that, The step of converting the changed data format into the target data format includes: Based on the target table structure information, the data format of the changed data is converted into the target data format. The target table structure information includes the table structure information of the target data table in the database corresponding to the OLTP system. The target data table is used to store the changed data.

5. The method according to any one of claims 1 to 4, characterized in that, Sending the current data stream to the OLAP system includes: Cache multiple first target data in the current data stream into a first preset storage unit; When it is determined that all the first target data have been cached, the first target data are combined based on the preset order corresponding to each first target data to obtain the second target data; The second target data is sent to the OLAP system.

6. The method according to any one of claims 1 to 4, characterized in that, Sending the current data stream to the OLAP system includes: Based on the comparison results between the current data stream and the historical data cached in the second preset storage unit, a third target data is determined. The historical data includes all data sent to the OLAP system within the current lifecycle, and the third target data includes updated data in the current data stream compared to the historical data. The third target data is sent to the OLAP system.

7. The method according to claim 6, characterized in that, The second preset storage unit includes a first sub-storage unit and a second sub-storage unit, wherein the first sub-storage unit is a non-external storage unit and the second sub-storage unit is an external storage unit; After determining the third target data, the process also includes: The field value corresponding to the first field in the third target data is cached in the first sub-storage unit, and the field value corresponding to the second field in the third target data is cached in the second sub-storage unit, wherein the importance of the first field is greater than that of the second field.

8. A data processing apparatus, characterized in that, include: The first processing module is used to acquire the first data stream in the OLTP system to be sent to the OLAP system, the first data stream including the current business data of the OLTP system; The second processing module is used to perform association processing on the first data stream and the second data stream based on preset association rules to obtain the current data stream. The second data stream includes the business data in the target business system to be sent to the OLAP system at the current time. The third processing module is used to send the current data stream to the OLAP system.

9. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the data processing method as described in any one of claims 1 to 7 by running a program in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the data processing method as described in any one of claims 1 to 7.

Images