Business data processing method and device, computer device and storage medium

Abstract

The application relates to a business data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving first changed business data in a source database; the first changed business data is determined by a business end through a change data capture tool; converting the data format of the first changed business data to obtain first target business data; obtaining target dimension business data having a first correlation relationship with the first target business data in a target database according to the data identifier of the first target business data; the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data; generating to-be-executed business data according to the first target business data and the target dimension business data, and executing a business processing operation corresponding to the to-be-executed business data. The method can improve the business data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a business data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Data processing is a fundamental aspect of systems engineering and automatic control. It permeates all areas of social production and life and is widely applied in various enterprises and institutions. Its basic purpose is to extract and derive valuable and meaningful data from large amounts of potentially chaotic and incomprehensible data.

[0003] Traditional technologies typically employ a master-slave database model, where business analysis and processing operations on business data are performed in the slave database. However, this model can only determine changed business data by periodically polling the master database, making it impossible to efficiently perform corresponding business processing on changed business data.

[0004] Therefore, traditional technologies suffer from low efficiency in processing business data. Summary of the Invention

[0005] Therefore, it is necessary to provide a business data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of business data processing in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a business data processing method. The method includes:

[0007] Receive the first changed business data from the source database; the first changed business data is determined by the business end through a change data capture tool.

[0008] The data format of the first modified business data is converted to obtain the first target business data; the data format of the first target business data matches the data format that the target database can recognize.

[0009] Based on the data identifier of the first target business data, target dimension business data with a first association relationship with the first target business data is obtained from the target database; the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data.

[0010] Based on the first target business data and the target dimension business data, generate business data to be executed, and execute the business processing operation corresponding to the business data to be executed.

[0011] In one of the embodiments, the obtaining, according to the data identifier of the first target business data, target dimension business data having a first association relationship with the first target business data in the target database comprises: determining, according to a preset query condition, to-be-associated variable business data in the first target business data; a field value of the to-be-associated variable business data satisfies the preset query condition; determining, according to the data identifier of the to-be-associated variable business data, to-be-associated dimension business data in the target database; the to-be-associated dimension business data has the first association relationship with the to-be-associated variable business data; the generating, according to the first target business data and the target dimension business data, to-be-executed business data, and executing a business processing operation corresponding to the to-be-executed business data comprises: generating the to-be-executed business data according to the to-be-associated variable business data and the to-be-associated dimension business data, and executing the business processing operation corresponding to the to-be-executed business data.

[0012] In one of the embodiments, the generating, according to the to-be-associated variable business data and the to-be-associated dimension business data, the to-be-executed business data, and executing the business processing operation corresponding to the to-be-executed business data comprises: parsing the to-be-associated variable business data to determine a data field name of the to-be-associated variable business data; the data field name has a corresponding data field value; generating the to-be-executed business data according to the data field name of the to-be-associated variable business data, the data field value of the to-be-associated variable business data, and the to-be-associated dimension business data; and executing the business processing operation corresponding to the to-be-executed business data.

[0013] In one of the embodiments, before the step of obtaining, according to the data identifier of the first target business data, target dimension business data having a first association relationship with the first target business data in the target database, the method further comprises: determining a business type corresponding to the first target business data; classifying, according to a data variable degree, first target business data belonging to a same business type to obtain classified first target business data; and combining the classified first target business data, and sending the combined first target business data to different data processing channels.

[0014] In one of the embodiments, the method further comprises: receiving second changed business data in the source database; the changed degree of the second changed business data is less than that of the first changed business data, and the business type corresponding to the second changed business data is the same as that of the dimension business data in the target database; converting the data format of the second changed business data to obtain second target business data; the data format of the second target business data matches the data format identifiable by the target database; synchronizing the second target business data to the target database according to the data identifier of the second target business data to establish a second association relationship between the second target business data and the dimension business data in the target database; the second target business data and the dimension business data corresponding to the second association relationship have the same data identifier.

[0015] In one of the embodiments, the method further comprises: analyzing the second target business data to determine the data field name of the second target business data; the data field name has a corresponding data field value; determining the data operation type of the second target business data; the data operation type includes at least one of a data deletion type, a data addition type and a data modification type; synchronizing the data field name and the data field value of the second target business data to the target database according to the data identifier of the second target business data to establish the second association relationship between the second target business data and the dimension business data in the target database; and performing a data update operation on the dimension business data having the second association relationship with the second target business data according to the data operation type.

[0016] In a second aspect, the application further provides a business data processing device. The device comprises:

[0017] A data receiving module is configured to obtain first changed business data in a source database; the first changed business data is determined by a business end through a change data capture tool.

[0018] A data conversion module is configured to convert the data format of the first changed business data to obtain first target business data; the data format of the first target business data matches the data format identifiable by a target database.

[0019] The dimension data acquisition module is configured to acquire target dimension business data having a first correlation relationship with the first target business data from the target database according to a data identifier of the first target business data; the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and a business type corresponding to the target dimension business data is different from a business type corresponding to the first target business data.

[0020] The execution module is configured to generate to-be-executed business data according to the first target business data and the target dimension business data, and execute a business processing operation corresponding to the to-be-executed business data.

[0021] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0022] receiving first changed business data in a source database; the first changed business data is determined by a business end through a change data capture tool;

[0023] converting a data format of the first changed business data to obtain first target business data; the data format of the first target business data matches a data format identifiable by a target database;

[0024] acquiring target dimension business data having a first correlation relationship with the first target business data from the target database according to a data identifier of the first target business data; the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and a business type corresponding to the target dimension business data is different from a business type corresponding to the first target business data;

[0025] generating to-be-executed business data according to the first target business data and the target dimension business data, and executing a business processing operation corresponding to the to-be-executed business data.

[0026] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0027] receiving first changed business data in a source database; the first changed business data is determined by a business end through a change data capture tool;

[0028] converting a data format of the first changed business data to obtain first target business data; the data format of the first target business data matches a data format identifiable by a target database;

[0029] According to the data identifier of the first target business data, target dimension business data having a first correlation relationship with the first target business data is acquired in the target database; the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data;

[0030] According to the first target business data and the target dimension business data, to-be-executed business data is generated, and a business processing operation corresponding to the to-be-executed business data is executed.

[0031] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the following steps:

[0032] Receiving first changed business data in a source database; the first changed business data is determined by a business end through a change data capture tool;

[0033] Converting a data format of the first changed business data to obtain first target business data; the data format of the first target business data matches a data format identifiable by a target database;

[0034] According to the data identifier of the first target business data, target dimension business data having a first correlation relationship with the first target business data is acquired in the target database; the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data;

[0035] According to the first target business data and the target dimension business data, to-be-executed business data is generated, and a business processing operation corresponding to the to-be-executed business data is executed.

[0036] The aforementioned business data processing method, apparatus, computer equipment, storage medium, and computer program product receive first changed business data from a source database; wherein the first changed business data is determined by the business end through a change data capture tool; then, the data format of the first changed business data is converted to obtain first target business data; wherein the data format of the first target business data matches the data format recognizable by the target database; then, based on the data identifier of the first target business data, target dimension business data with a first association relationship with the first target business data is obtained from the target database; wherein the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data; finally, based on the first target business data and the target dimension business data, business data to be executed is generated. It executes the business processing operations corresponding to the pending business data; thus, by receiving the first changed business data captured from the source database by the business end through the change data capture tool, it can obtain the changed business data in the source database in real time, and convert the data format of the first changed business data into a data format that the target database can recognize, to obtain the first target business data. This makes the data format of the first target business data the same as the data format of the target dimension business data that is related to the first target business data obtained in the target database. This can support the fusion of business data between relational databases of different data types in heterogeneous environments, and can execute the business processing operations corresponding to the first target business data and the target dimension business data in real time. This solves the problem of low data acquisition efficiency caused by periodic polling to obtain changed data in the master-slave database mode, and can thus process business data efficiently in heterogeneous environments. Attached Figure Description

[0037] Figure 1 This is an application environment diagram of a business data processing method in one embodiment;

[0038] Figure 2 This is a flowchart illustrating a business data processing method in one embodiment;

[0039] Figure 3 This is a flowchart illustrating a business data processing method in another embodiment;

[0040] Figure 4 This is a physical deployment diagram of a business data processing method implemented in another embodiment;

[0041] Figure 5 This is a schematic diagram of the data processing process of the stream processing unit in another embodiment;

[0042] Figure 6A structural block diagram of an embodiment of a service data processing apparatus is shown in the figure.

[0043] Figure 7 An internal structural diagram of a computer device in an embodiment is shown in the figure. DETAILED DESCRIPTION

[0044] For the purposes of the present application, the technical solutions and advantages thereof are more clearly apparent, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0045] The service data processing method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 . In the application environment, a service end 102 communicates with a server 104 through a network. A data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or can be placed on a cloud or other network server. The server 104 receives first changed service data in a source database. The first changed service data is determined by the service end 102 through a change data capture tool, and the degree of change of the first changed service data meets a preset condition. Then, the server 104 converts a data format of the first changed service data to obtain first target service data. The data format of the first target service data matches a data format recognizable by a target database. Then, the server 104 acquires target dimension service data having a first correlation relationship with the first target service data in the target database according to a data identifier of the first target service data. The data identifier of the target dimension service data is the same as the data identifier of the first target service data, and a business type corresponding to the target dimension service data is different from a business type corresponding to the first target service data. Finally, the server 104 generates to-be-executed service data according to the first target service data and the target dimension service data, and executes a business processing operation corresponding to the to-be-executed service data. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers. The target database can be, but is not limited to, an open-source HTAP (Hybrid Transactional / Analytical Processing) library.

[0046] In an embodiment, as shown in Figure 2 , a service data processing method is provided. The method is described below by taking the server 104 in Figure 1 as an example, and includes the following steps.

[0047] In step S210, first changed service data in a source database is received.

[0048] The first changed business data is determined by the business end through a change data capture tool.

[0049] The source database can be a relational database.

[0050] The first changed business data can be, but is not limited to, billing data.

[0051] In a specific implementation, the business end can analyze the log in the source database in real time to obtain the DML (Data Manipulation Language) operation record in the source database through the CDC (Change Data Capture) method, and format the DML information in the Json (JavaScript Object Notation) structure. The DML information includes all field information of the changed data table in the source database, the values before and after the change of the field values, and adds information such as the data operation type, the data change time, and the data self-increment unique offset. At the same time, the business end also needs to store the DML information in binary text, generate the first changed business data, and send the first changed business data to the receiving program of the server in real time in binary text, so that the server can receive the first changed business data from the source database. The first changed business data can be business data in a data table in the source database that frequently changes a large amount of data, such as business data in a billing data table.

[0052] In step S220, the data format of the first changed business data is converted to obtain the first target business data.

[0053] The data format of the first target business data matches the data format that can be recognized by the target database.

[0054] In a specific implementation, since the data format of the business data in the source database can be different from the data format that can be received by the target database, the data format of the first changed business data needs to be converted to the data format that can be recognized by the target database to obtain the first target business data and send it to the message queue. Specifically, the first changed business data stored in binary text can be parsed to obtain the first target business data in Json format.

[0055] In step S230, the target dimension business data having a first association relationship with the first target business data is obtained from the target database according to the data identifier of the first target business data.

[0056] The data identifier of the target dimension business data is the same as the data identifier of the first target business data.

[0057] The service type corresponding to the target dimension service data is different from the service type corresponding to the first target service data.

[0058] The target dimension service data can be source-pasted data synchronized with the source database in quasi real time.

[0059] In a specific implementation, the server can determine the data identifier of the first target service data, and query, according to the data identifier of the target service data, the target dimension service data having the first association relationship with the first target service data in the target database, that is, query, in the target database, the dimension service data having the same data identifier as the first target service data as the target dimension service data; wherein the service type corresponding to the first target service data is different from the service type corresponding to the target dimension service data, for example, the service type corresponding to the first target service data can be a bill data type, and the service type corresponding to the target dimension service data can be a user personal information data type.

[0060] In a specific implementation, the server can determine the data identifier of the first target service data, and query, according to the data identifier of the target service data, the target dimension service data having the first association relationship with the first target service data in the target database, that is, query, in the target database, the dimension service data having the same data identifier as the first target service data as the target dimension service data; wherein the service type corresponding to the first target service data is different from the service type corresponding to the target dimension service data, for example, the service type corresponding to the first target service data can be a bill data type, and the service type corresponding to the target dimension service data can be a user personal information data type.

[0061] In a specific implementation, the server can determine the data identifier of the first target service data, and query, according to the data identifier of the target service data, the target dimension service data having the first association relationship with the first target service data in the target database, that is, query, in the target database, the dimension service data having the same data identifier as the first target service data as the target dimension service data; wherein the service type corresponding to the first target service data is different from the service type corresponding to the target dimension service data, for example, the service type corresponding to the first target service data can be a bill data type, and the service type corresponding to the target dimension service data can be a user personal information data type.

[0062] In the business data processing method, the first changed business data in the source database is received, wherein the first changed business data is determined by the business end through the change data capture tool; then, the data format of the first changed business data is converted to obtain the first target business data; wherein the data format of the first target business data matches the data format recognizable by the target database; then, the target dimension business data having the first correlation relationship with the first target business data is obtained in the target database according to the data identifier of the first target business data; wherein the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data; finally, the to-be-executed business data is generated according to the first target business data and the target dimension business data, and the business processing operation corresponding to the to-be-executed business data is executed; in this way, the changed business data in the source database can be obtained in real time by receiving the first changed business data captured from the source database by the business end through the change data capture tool, and the data format of the first changed business data is converted into the data format recognizable by the target database to obtain the first target business data, so that the data format of the first target business data is the same as the data format of the target dimension business data having the correlation relationship with the first target business data obtained in the target database, thereby supporting the business data fusion between different data types of relational databases in a heterogeneous environment, and the business processing operation corresponding to the first target business data and the target dimension business data can be executed in real time, thereby solving the problem of low data acquisition efficiency caused by periodic polling of changed data in the master-slave library mode, and further efficiently processing business data in a heterogeneous environment.

[0063] In one embodiment, according to the data identifier of the first target business data, the target dimension business data having the first correlation relationship with the first target business data is obtained in the target database, including: determining the to-be-associated changed business data in the first target business data according to the preset query condition; the field value of the to-be-associated changed business data satisfies the preset query condition; determining the to-be-associated dimension business data in the target database according to the data identifier of the to-be-associated business data; the to-be-associated dimension business data has the first correlation relationship with the to-be-associated changed business data; generating the to-be-executed business data according to the first target business data and the target dimension business data, and executing the business processing operation corresponding to the to-be-executed business data, including: generating the to-be-executed business data according to the to-be-associated changed business data and the to-be-associated dimension business data, and executing the business processing operation corresponding to the to-be-executed business data.

[0064] According to the to-be-associated change business data and the to-be-associated dimension business data, the to-be-executed business data is generated, and a business processing operation corresponding to the to-be-executed business data is performed, including: analyzing the to-be-associated change business data, and determining a data field name of the to-be-associated change business data; the data field name has a corresponding data field value; according to the data field name of the to-be-associated change business data, the data field value of the to-be-associated change business data, and the to-be-associated dimension business data, the to-be-executed business data is generated; and the business processing operation corresponding to the to-be-executed business data is performed.

[0065] The preset query condition is obtained through a where statement of an sql (Structured Query Language) mode.

[0066] The to-be-associated change business data is source-pasted data synchronized with the source database in quasi real time.

[0067] In the specific implementation, the server can filter, according to the preset query condition, business data in the first target business data that meets the preset query condition as the to-be-associated change business data; for example, the where statement of the sql mode is used to filter, as the to-be-associated change business data, business data in the first target business data that has a data field value greater than a certain threshold.

[0068] Then, a RPC (Remote Procedure Call) short connection is established with the target database, dimension business data having the same data identifier as the to-be-associated change business data is queried in the target database according to the data identifier of the to-be-associated business data as the to-be-associated dimension business data, that is, dimension business data having a first association relationship with the to-be-associated change business data, and the to-be-associated dimension business data is extracted from the target database; for example, the data identifier can be a user number corresponding to the to-be-associated business data, and the to-be-associated dimension business data having the same user number is queried in the target database according to the user number.

[0069] Finally, the server can consume the first target business data existing in the message queue through a stream procedure, parse the first target business data, extract the data field name and data field value of the first target business data, and determine the data operation type of the first target business data, such as the data operation type being a data addition type, a data deletion type, a data modification type, and extract the key information in the to-be-associated dimension business data; generate to-be-executed business data according to the data field name and data field value of the first target business data and the key information in the to-be-associated dimension business data, and execute a business processing operation corresponding to the to-be-executed business data; for example, the data field name and data field value can be the fee name and fee value in the bill change information, the key information can be the contact information of the user, the to-be-executed business data includes the fee name and fee value in the bill change information and the contact information of the user, and the business processing operation can be to send the fee name and fee value in the bill change information to the contact information of the corresponding user. Finally, the processing result of the business processing operation can be stored in a DB (database) manner or a message queue manner, and interacted with a downstream application.

[0070] The technical scheme of the embodiment, by determining the to-be-associated change business data in the first target business data according to the preset query condition; wherein the field value of the to-be-associated change business data satisfies the preset query condition; then determining the to-be-associated dimension business data in the target database according to the data identifier of the to-be-associated business data; wherein the to-be-associated dimension business data has a first association relationship with the to-be-associated change business data; finally, generating to-be-executed business data according to the to-be-associated change business data and the to-be-associated dimension business data, and executing a business processing operation corresponding to the to-be-executed business data; in this way, a part of the first target business data can be filtered out as the to-be-associated change business data through the preset query condition, and the to-be-associated dimension business data having a first association relationship with the to-be-associated change business data is obtained, and a business processing operation corresponding to the to-be-executed business data generated according to the to-be-associated change business data and the to-be-associated dimension business data is executed; thereby avoiding the problem of large resource consumption caused by loading and caching a large amount of business data, and improving the business data processing efficiency.

[0071] In one embodiment, before the step of obtaining the target dimension business data having the first association relationship with the first target business data in the target database according to the data identifier of the first target business data, the method further comprises: determining the business type corresponding to the first target business data; classifying the first target business data belonging to the same business type according to the data change degree to obtain the first target business data after classification; and combining the first target business data after classification, and sending the combined first target business data to different data processing channels.

[0072] In a particular implementation, the server can determine the business type of the first target business data stored in the message queue, and classify the first target business data of the same business type according to the data variation degree. Specifically, the server can iteratively count the first target business data of multiple business systems of the same business type, such as the marketing domain system and the production domain system of the same business type, and classify the first target business data according to the business data volume and the business data importance. Then, the server can reasonably combine the classified first target business data, such as combining the classified first target business data according to the business data importance and the corresponding daily variation record level, and send the combined first target business data to different data processing channels for the next step of generating the to-be-executed business data according to the first target business data and the target dimension business data, and executing the business processing operation corresponding to the to-be-executed business data.

[0073] The technical solution of the embodiment determines the business type corresponding to the first target business data, classifies the first target business data of the same business type according to the data variation degree to obtain the classified first target business data, combines the classified first target business data, and sends the combined first target business data to different data processing channels. In this way, the first target business data of the same business type is classified and combined before being sent to the data processing channels, thereby reducing the business data skew and ensuring the business data processing efficiency.

[0074] In one embodiment, the method further includes receiving second variation business data in the source database, converting the data format of the second variation business data to obtain second target business data, and synchronizing the second target business data to the target database according to the data identifier of the second target business data to establish a second association relationship with the dimension business data in the target database.

[0075] The variation degree of the second variation business data is less than that of the first variation business data, and the business type corresponding to the second variation business data is the same as that of the dimension business data in the target database.

[0076] The dimension business data can be source data that is synchronized with the source database in quasi real time.

[0077] The data format of the second target business data matches the data format that can be recognized by the target database.

[0078] The second target business data corresponding to the second association relationship has the same data identifier as the dimension business data.

[0079] In a specific implementation, the service end can also generate the second changed service data based on the same method as in step S210, and send the second changed service data to the receiving program of the server. The server can identify the first changed service data and the second changed service data according to the data table names corresponding to the first changed service data and the second changed service data in the source database, and the change degree of the second changed service data is smaller than that of the first changed service data, and the corresponding service type is the same as the service type corresponding to the dimension service data in the target database. For example, the service type corresponding to the second changed service data can be the user contact information, and the data change degree of the user contact information is smaller. Then, the server performs data format conversion on the second changed service data stored in the binary text mode to obtain the second target service data whose data format can be recognized by the target database. Specifically, the data format of the second target service data can be the Json format.

[0080] Then, the server can query the dimension service data having the same data identifier as the second target service data in the target database according to the data identifier of the second target service data, and synchronize the second target service data after disorder processing to the corresponding dimension service data in the target database to establish a second association relationship with the dimension service data having the same data identifier. Specifically, the data identifier of the second target service data can be the user identity identifier corresponding to the second target service data, such as user number, user name, etc.

[0081] The technical scheme of the embodiment receives the second changed service data in the source database. The change degree of the second changed service data is smaller than that of the first changed service data, and the service type corresponding to the second changed service data is the same as the service type corresponding to the dimension service data in the target database. Then, the data format of the second changed service data is converted to obtain the second target service data, and the second target service data is sent to the message queue. The data format of the second target service data matches the data format recognizable by the target database. Finally, the second target service data is synchronized to the target database according to the data identifier of the second target service data to establish a second association relationship with the dimension service data in the target database. The second target service data corresponding to the second association relationship has the same data identifier as the dimension service data. In this way, when the second changed service data having the same service type as the dimension service data in the target database in the source database is generated, the second changed service data can be synchronized to the target database to establish a second association relationship with the dimension service data in the target database. The dimension service data in the target database is the source data synchronized with the source database in real time, so as to ensure the consistency of the dimension service data in the source database and the target database.

[0082] In an embodiment, according to the data identifier of the second target business data, the second target business data is synchronized into the target database, and a second association relationship is established between the second target business data and the dimensional business data in the target database, including: analyzing the second target business data to determine the data field name of the second target business data; determining the data operation type of the second target business data; according to the data identifier of the second target business data, the data field name of the second target business data and the data field value of the second target business data are synchronized into the target database, and a second association relationship is established between the second target business data and the dimensional business data in the target database; according to the data operation type, the dimensional business data having the second association relationship with the second target business data is subjected to a data update operation.

[0083] Among them, the data field name has a corresponding data field value.

[0084] Among them, the data operation type includes at least one of a data deletion type, a data addition type and a data modification type.

[0085] Among them, the data update operation includes at least one of a dimensional business data addition operation, a dimensional business data deletion operation and a dimensional business data modification operation.

[0086] In a specific implementation, the server can consume the second target business data existing in the message queue through a stream program, analyze the second target business data, extract the data field name and data field value of the second target business data, and determine the data operation type of the second target business data, such as at least one of a data addition type, a data deletion type and a data modification type; then, according to the data identifier of the second target business data, the dimensional business data having the same data identifier is queried in the target database; specifically, the data identifier can be a user identity identifier corresponding to the second target business data, such as a user number, a user name, etc.; after the second target business data is subjected to a disorder processing, the data field name and data field value of the second target business data are synchronized into the target database, and a second association relationship is established between the corresponding dimensional business data, and according to the data operation type of the second target business data, the dimensional business data having the second association relationship with the second target business data is subjected to a data update operation; for example, if the data operation type is a data update operation, the data field value corresponding to the dimensional business data can be updated to the data field value of the second target business data; if the data operation type is a data addition operation, the data field value corresponding to the dimensional business data can be increased by a new field value, i.e. the data field value of the second target business data; if the data operation type is a data deletion operation, the data field value corresponding to the dimensional business data can be deleted.

[0087] The technical scheme of the embodiment is as follows: the second target service data is parsed to determine the data field name of the second target service data; the data field name has a corresponding data field value; then, the data operation type of the second target service data is determined; the data operation type includes at least one of a data deletion type, a data addition type and a data modification type; then, according to the data identifier of the second target service data, the data field name of the second target service data and the data field value of the second target service data are synchronized to the target database to establish a second association relationship with the dimension service data in the target database; finally, according to the data operation type, the dimension service data having the second association relationship with the second target service data is subjected to a data update operation; in this way, the second changed service data in the source database, which has the same service type as the dimension service data in the target database, can be synchronized to the target database, and according to the data operation type of the second changed service data, the dimension service data having the same data identifier as the second changed service data is subjected to a corresponding data operation, so that the data update operation of the dimension service data in the target database is completed, thereby ensuring the consistency of the dimension service data in the source database and the target database.

[0088] In another embodiment, as shown in Figure 3 a service data processing method is provided, and the method is applied to a Figure 1Taking server 104 as an example, the process includes the following steps: Step S302: Receive first changed business data from the source database; the first changed business data is determined by the business end through a change data capture tool. Step S304: Convert the data format of the first changed business data to obtain first target business data; the data format of the first target business data matches the data format that the target database can recognize. Step S306: Determine the business type corresponding to the first target business data. Step S308: Classify the first target business data belonging to the same business type according to the degree of data change to obtain classified first target business data. Step S310: Combine the classified first target business data and send the combined first target business data to different data processing channels. Step S312: Determine the changed business data to be associated in the first target business data according to preset query conditions; the field values ​​of the changed business data to be associated satisfy the preset query conditions. Step S314: Determine the dimension business data to be associated in the target database according to the data identifier of the business data to be associated; the dimension business data to be associated has the first association relationship with the changed business data to be associated. Step S316: Parse the business data to be associated and determine the data field names of the business data to be associated; each data field name has a corresponding data field value. Step S318: Generate the business data to be executed based on the data field names, data field values, and the business data to be associated. Step S320: Execute the business processing operation corresponding to the business data to be executed. It should be noted that the specific limitations of the above steps can be found in the specific limitations of a business data processing method described above.

[0089] For the ease of understanding of those skilled in the art, Figure 4 A physical deployment diagram is provided for another method of implementing business data processing, such as Figure 4As shown, it includes a change data capture unit 410 (corresponding to the business end in the above embodiment), a data receiving unit 420 of a server, a distributed message unit 430, a stream processing unit 440, and an HTAP unit 450. Specifically, the change data capture unit 410 is deployed on the business system side, and the change data capture unit 410 can be deployed on the machine of the relational database of the business system or on the intermediate machine. The change data capture unit 410 analyzes the log of the relational database (corresponding to the source database in the above embodiment) of the business system by the CDC method to obtain the changed DML; formats the DML information in the Json structure, contains all field information of the data table, the values before and after the change of the field, and adds the operation type, the occurrence time, the unique offset, and the like; then, stores the DML information formatted in the Json structure in the binary text mode to generate the changed business data; the changed business data includes the changed business data in the main table (corresponding to the first changed business data in the above embodiment) and the dimension business data in the dimension table (corresponding to the second changed business data in the above embodiment); and the changed business data is transmitted to the data receiving unit 420 in real time.

[0090] The analysis program of the data receiving unit 420 analyzes the changed business data to obtain the target business data in the Json format; wherein the target business data includes the target changed business data in the main table (corresponding to the first target business data) and the target dimension business data in the dimension table (corresponding to the second target business data); and the target business data is sent to the distributed message unit 430.

[0091] In the distributed message unit 430, the distributed message queue stores data in a certain period, which is used for buffering data in the business peak period and for the data playback function in the data checking scene. If the target business data needs to be re-consumed on the business, the target business data can be re-consumed. At the same time, in order to ensure the processing efficiency and reduce the data skew, the data tables of the same business system (referring to multiple systems of a type of business, such as the marketing domain and the production domain, and each domain is divided into multiple systems) are classified according to the change, the tables are reasonably combined and sent to different data processing channels. Specifically, the target business data can be iteratively counted, and the first target business data is classified according to the business data volume and the business data importance; then, the first target business data after the classification is reasonably combined, such as combined according to the business data importance, the daily change record level corresponding to the business data, and the like, and sent to different data processing channels.

[0092] The stream processing unit 440 can consume the target business data in the message queue through the stream program, and the data processing process of the stream processing unit 440 is as shown in Figure 5As shown, the change data capturing unit 410 obtains the change business data from the relational database by the CDC mode, and generates the change business data, the analysis program of the data receiving unit 420 analyzes the change business data, obtains the target business data in the Json format, and sends to the distributed message unit 430; the stream processing unit 440 stream program consumes the target dimension business data of the dimension table in the first message queue, extracts the data field name and data field value of the target dimension business data, after disorder processing, writes into the HTAP unit 450 according to the data operation type (add, modify, delete) of the target dimension business data, and guarantees the data quasi-real-time consistency between the HTAP database and the relational database; and the stream processing unit 440 stream program also consumes the target change business data of the main table in the second message queue, extracts the data field name and data field value of the target change business data, and determines the data operation type (add, modify, delete) of the target change business data; then, according to the preset query condition (which can be set by the where statement of the sql mode), the to-be-associated change business data in the target change business data is screened out, a RPC short connection is established with the HTAP unit 450, the to-be-associated dimension business data having a first association relationship with the to-be-associated change business data in the HTAP unit 450 is queried in real time, and the business processing operation related to the to-be-associated dimension business data and the to-be-associated change business data is executed, and the business processing operation result is stored in the DB mode or stored in the message queue mode and interacts with the downstream application.

[0093] The HTAP unit 450 is a new sql database supporting OLTP (On-Line Transaction Processing) and OLAP (On-Line Analytical Processing) at the same time, which carries the unified storage of the structured data of the data center in the technical scheme, and the main characteristics are: 1, rich field types, compatible with the field types commonly used in the relational database. 2, supporting ACID (atomicity, consistency, isolation, durability), supporting SQL, having row-level modification capability. 3, friendly to the main big data analysis engine, supporting Spark (computing engine), Flink (open source stream processing framework) and the like. 4, RPC short connection type, can support ten thousand level connections.

[0094] It should be understood that although each step in the flowchart involved in the embodiments described above is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0095] Based on the same inventive concept, the embodiments of the present application also provide a business data processing apparatus for implementing the above-mentioned business data processing method. The problem-solving implementation scheme provided by the apparatus is similar to the implementation scheme described in the above method, so the specific limitations in one or more business data processing apparatus embodiments provided below can refer to the limitations of the business data processing method described above, and will not be repeated here.

[0096] In one embodiment, as shown in Figure 6 a business data processing apparatus is provided, comprising: a data receiving module 610, a data conversion module 620, a data acquisition module 630 and an execution module 640, wherein:

[0097] The data receiving module 610 is configured to acquire first changed business data in a source database; the first changed business data is determined by a change data capture tool through a business end.

[0098] The data conversion module 620 is configured to convert the data format of the first changed business data to obtain first target business data; the data format of the first target business data matches the data format that can be recognized by a target database.

[0099] The data acquisition module 630 is configured to acquire target dimension business data having a first association relationship with the first target business data in the target database according to the data identifier of the first target business data; the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and the business type corresponding to the target dimension business data is different from the business type corresponding to the first target business data.

[0100] The execution module 640 is configured to generate to-be-executed business data according to the first target business data and the target dimension business data, and execute a business processing operation corresponding to the to-be-executed business data.

[0101] In one of the embodiments, the data obtaining module 630 is specifically configured to determine to-be-associated variable business data in the first target business data according to a preset query condition; a field value of the to-be-associated variable business data satisfies the preset query condition; determine to-be-associated dimension business data in the target database according to a data identifier of the to-be-associated variable business data; the to-be-associated dimension business data has the first association relationship with the to-be-associated variable business data; and the execution module 640 is specifically configured to generate the to-be-executed business data according to the to-be-associated variable business data and the to-be-associated dimension business data, and execute a business processing operation corresponding to the to-be-executed business data.

[0102] In one of the embodiments, the execution module 640 is specifically configured to parse the to-be-associated variable business data to determine a data field name of the to-be-associated variable business data; the data field name has a corresponding data field value; generate the to-be-executed business data according to the data field name of the to-be-associated variable business data, the data field value of the to-be-associated variable business data, and the to-be-associated dimension business data; and execute the business processing operation corresponding to the to-be-executed business data.

[0103] In one of the embodiments, the business data processing apparatus further includes a business type determining module configured to determine a business type corresponding to the first target business data; a hierarchical classification module configured to hierarchically classify first target business data belonging to the same business type according to a data variable degree to obtain hierarchically classified first target business data; and a combination module configured to combine the hierarchically classified first target business data and send the combined first target business data to different data processing channels.

[0104] In one of the embodiments, the business data processing apparatus further includes a second data receiving module configured to receive second variable business data in the source database; the second variable business data has a smaller variable degree than the first variable business data, and a business type corresponding to the second variable business data is the same as a business type corresponding to dimension business data in the target database; a second data conversion module configured to convert a data format of the second variable business data to obtain second target business data; the second target business data has a data format that matches a data format identifiable by the target database; and a data synchronization module configured to synchronize the second target business data to the target database according to a data identifier of the second target business data to establish a second association relationship between the second target business data and the dimension business data in the target database; the second association relationship corresponds to second target business data and dimension business data having the same data identifier.

[0105] In one of the embodiments, the data synchronization module is specifically configured to parse the second target business data, determine a data field name of the second target business data, the data field name having a corresponding data field value, determine a data operation type of the second target business data, the data operation type including at least one of a data deletion type, a data addition type and a data modification type, synchronize the data field name of the second target business data and the data field value of the second target business data to the target database according to a data identifier of the second target business data, establish the second association relationship between the second target business data and the dimensional business data in the target database, and perform a data update operation on the dimensional business data having the second association relationship with the second target business data according to the data operation type.

[0106] The modules in the business data processing apparatus can be implemented by software, hardware or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by the processor to perform operations corresponding to the modules.

[0107] In one embodiment, a computer device is provided, which can be a server. An internal structure diagram of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store dimensional business data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a business data processing method.

[0108] Those skilled in the art can understand that Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] In one embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the steps in the above-mentioned method embodiments.

[0110] In an embodiment, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0111] In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0112] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0113] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0114] Any combination of the technical features in the above embodiments can be made. For the sake of brevity, the foregoing description has not described all possible combinations of the technical features in the above embodiments. However, as long as the combination of the technical features does not contradict, it should be considered within the scope of the present disclosure.

[0115] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A service data processing method characterized by, The method comprises: receiving first and second changed business data in a source database; the first changed business data is determined by a business end through a change data capture tool, and the second changed business data has a smaller degree of change than the first changed business data; converting a data format of the first changed business data to obtain first target business data; the data format of the first target business data matches a data format recognizable by a target database; converting a data format of the second changed business data to obtain second target business data; the data format of the second target business data matches the data format recognizable by the target database; synchronizing the second target business data to the target database according to a data identifier of the second target business data, and establishing a second association relationship between the second target business data and dimension business data in the target database, so that the dimension business data in the source database is consistent with the dimension business data in the target database; the dimension business data is source-attached data that is quasi-real-time synchronized with the source database; in a case where the second target business data establishes the second association relationship with the dimension business data in the target database, obtaining target dimension business data having a first association relationship with the first target business data in the target database according to a data identifier of the first target business data; the data identifier of the target dimension business data is the same as the data identifier of the first target business data, and a business type corresponding to the target dimension business data is different from a business type corresponding to the first target business data; extracting changed information in the first target business data, and generating to-be-executed business data based on the changed information and the target dimension business data, and performing a business processing operation corresponding to the to-be-executed business data; the business processing operation at least includes pushing the changed information to a user corresponding to the changed information.

2. The method of claim 1, wherein, The method comprises: determining to-be-associated changed business data in the first target business data according to a preset query condition; a field value of the to-be-associated changed business data satisfies the preset query condition; determining to-be-associated dimension business data in the target database according to a data identifier of the to-be-associated changed business data; the to-be-associated dimension business data has the first association relationship with the to-be-associated changed business data; The method comprises: generating the to-be-executed business data based on the to-be-associated changed business data and the to-be-associated dimension business data, and performing the business processing operation corresponding to the to-be-executed business data.

3. The method of claim 2, wherein, The step of generating the business data to be executed based on the business data to be associated and the business data of the dimension to be associated, and executing the business processing operation corresponding to the business data to be executed, includes: The business data to be associated with changes is parsed to determine the data field names of the business data to be associated with changes; each data field name has a corresponding data field value. The business data to be executed is generated based on the data field names of the business data to be associated with, the data field values ​​of the business data to be associated with, and the business data of the dimension to be associated. Execute the business processing operation corresponding to the business data to be executed.

4. The method of claim 1, wherein, Before the step of obtaining target dimension business data with a first association relationship to the first target business data from the target database based on the data identifier of the first target business data, the method further includes: Determine the business type corresponding to the first target business data; The primary target business data belonging to the same business type are classified into different levels according to the degree of data change, resulting in the primary target business data after classification. The first target business data after being classified and categorized is combined, and the combined first target business data is sent to different data processing channels.

5. The method of claim 1, wherein, The business type corresponding to the second changed business data is the same as the business type corresponding to the dimension business data in the target database; The second target business data and the dimension business data corresponding to the second association have the same data identifier.

6. The method of claim 5, wherein, The step of synchronizing the second target business data to the target database based on the data identifier of the second target business data, and establishing a second association relationship with the dimensional business data in the target database, includes: The second target business data is parsed to determine the data field names of the second target business data; each data field name has a corresponding data field value. Determine the data operation type of the second target business data; the data operation type includes at least one of data deletion type, data addition type, and data modification type. Based on the data identifier of the second target business data, the data field names and data field values ​​of the second target business data are synchronized to the target database, and the second association relationship is established with the dimension business data in the target database. According to the data operation type, perform a data update operation on the dimension business data that has the second association relationship with the second target business data.

7. A service data processing apparatus characterized by comprising: The device includes: The data receiving module is used to acquire first changed business data and second changed business data from the source database; the first changed business data is determined by the business end through a change data capture tool, and the degree of change of the second changed business data is less than that of the first changed business data. The data conversion module is configured to convert a data format of the first changed business data to obtain first target business data, wherein the data format of the first target business data matches a data format identifiable by the target database; convert a data format of the second changed business data to obtain second target business data, wherein the data format of the second target business data matches the data format identifiable by the target database; and synchronize the second target business data to the target database according to a data identifier of the second target business data, and establish a second association relationship between the second target business data and dimension business data in the target database, so as to make the dimension business data in the source database consistent with the dimension business data in the target database, wherein the dimension business data is source-attached data that is synchronized with the source database in quasi real time. The data acquisition module is configured to acquire target dimension business data having a first association relationship with the first target business data from the target database according to a data identifier of the first target business data, in a case where the second target business data and the dimension business data in the target database establish the second association relationship, wherein a data identifier of the target dimension business data is the same as the data identifier of the first target business data, and a business type corresponding to the target dimension business data is different from a business type corresponding to the first target business data. The execution module is configured to extract changed information in the first target business data, generate to-be-executed business data based on the changed information and the target dimension business data, and execute a business processing operation corresponding to the to-be-executed business data, wherein the business processing operation at least includes pushing the changed information to a user corresponding to the changed information. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images