Business data processing system, method, server and storage medium

Abstract

The embodiment of the present invention discloses a business data processing system, method, server and storage medium, which includes: a data warehouse mart data layer, which is used to synchronize the full data of each business data source to the data warehouse data mart according to a preset frequency, and process the full data to obtain the full data of the business wide table; an incremental data layer, which is used to connect with each business data source, obtain the real-time incremental data of each business data source, and obtain the incremental data of the business wide table; a business processing layer, which is used to verify the full data of the business wide table, and fuse the verified full data of the business wide table with the incremental data of the business wide table to obtain the target business wide table data. Through the technical solution of the embodiment of the present invention, the decoupling of the cumbersome docking process between multiple business departments can be achieved, and the consistency of data when docking multiple data sources is guaranteed.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a business data processing system, method, server, and storage medium. Background Art

[0002] At present, with the popularization of intelligent data management needs, business parties are not only concerned with the analysis and display of real-time data, but also concerned with comparative data recorded at the same time on historical dates. By comparing and analyzing these two parts of data, business personnel can better understand the data trends of various indicators, grasp business dynamics, and thus better implement plans and guide actual production operations.

[0003] At present, the conventional business docking solution is through Figure 1 This is accomplished using the data processing framework shown in the figure. Different data sources synchronize all data to the business side's database at once through interface calls to generate full data for the business wide table.

[0004] However, in the process of implementing the present invention, the inventors found that there are at least the following technical problems in the existing technology: as the business model develops, the wide table of its own business will be connected to multiple systems to obtain basic wide table data, and the full amount of data will be connected at one time, which cannot guarantee the transaction consistency after data heterogeneity. Summary of the Invention

[0005] Embodiments of the present invention provide a business data processing system, method, server, and storage medium to reduce coupling between different business data sources when synchronizing full data and ensure data consistency and accuracy.

[0006] In a first aspect, an embodiment of the present invention provides a business data processing system, the system comprising:

[0007] The data warehouse mart data layer is used to synchronize the full data of each business data source to the data warehouse data mart at a preset frequency, and process the full data to obtain the full data of the business wide table;

[0008] The incremental data layer is used to connect with the business data sources, obtain real-time incremental data from each business data source, and obtain incremental data of the business wide table;

[0009] The business processing layer is used to verify the full data of the business wide table, and fuse the verified full data of the business wide table with the incremental data of the business wide table to obtain target business wide table data.

[0010] Optionally, the data warehouse mart data layer is specifically used for:

[0011] On a daily basis, synchronize the full amount of data from each business data source before the current data synchronization time to the data warehouse data mart and record the data synchronization timestamp;

[0012] Integrate the full data according to the current business model to obtain the full data of the wide table of the business on that day;

[0013] The full amount of data of the broad table of business on that day and the timestamp are pushed to the business database of the current business party.

[0014] Optionally, the business processing layer verifies the full amount of data in the business wide table, including:

[0015] Determine whether the full amount of data of each business data source has changed;

[0016] If so, the full data of the broad table of the day's business is updated according to the changed full data.

[0017] Optionally, when the business processing layer performs data fusion, the Job task is called at a preset time to de-duplicate and merge the full data of the business wide table for the day and the incremental data after the timestamp to obtain the target business wide table data for the day.

[0018] Optionally, the incremental data layer is connected to each business data source via a message queue.

[0019] Optionally, the business processing layer is further used to perform data verification and filtering on the incremental data of the business wide table before data fusion.

[0020] Optionally, the business processing layer stores the incremental data of the business wide table in a MySQL database, and uses distributed locks to ensure the transaction consistency of the incremental data of each business data source.

[0021] Optionally, the system further includes: a business storage layer, which is used to store the target business wide table data of the day in the business database, and archive the target business wide table data before the target business wide table data of the day into the ClickHouse database for partitioned storage.

[0022] In a second aspect, an embodiment of the present invention further provides a business data processing method, which is applied to a business data processing system. The method includes:

[0023] Obtaining full data of the current day's business wide table from the data warehouse data mart, wherein the data warehouse data mart includes full data of each business data source updated daily, and the full data of the current day's business wide table is obtained by processing the full data of each business data source in the data warehouse data mart;

[0024] Acquire incremental data from each business data source in real time to obtain incremental data of the business wide table;

[0025] The full data of the business wide table on the day and the incremental data of the business wide table are merged to obtain the target business wide table data on the day.

[0026] Optionally, processing the full amount of data of each business data source in the data warehouse data mart includes:

[0027] Every day, the full amount of data of each business data source before the current data synchronization time is synchronized to the data warehouse data mart and the timestamp of the data synchronization is recorded;

[0028] The full amount of data is integrated according to the business model of the current business party to obtain the full amount of data of the broad table of the business on that day.

[0029] Optionally, the method further includes:

[0030] The target business wide table data of the day is stored in the MySQL database, and the target business wide table data before the target business wide table data of the day is archived in the ClickHouse database for partition storage.

[0031] Optionally, the method further includes:

[0032] The incremental data of the business wide table is stored in a MySQL database, and a distributed lock is used to ensure the transaction consistency of the incremental data of each business data source.

[0033] In a third aspect, an embodiment of the present invention further provides a server, comprising:

[0034] one or more processors;

[0035] a memory for storing one or more programs;

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the business data processing method provided by any embodiment of the present invention.

[0037] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the business data processing method provided by any embodiment of the present invention.

[0038] The embodiments of the above invention have the following advantages or beneficial effects:

[0039] The business data processing system is composed of the data warehouse mart data layer, the incremental data layer and the business processing layer. In the data warehouse mart data layer, the data warehouse is used as a data mart and the full data of multiple data sources are synchronized at a preset frequency. The full data of each data source is processed to obtain the full data of the business wide table of the business party. The business processing layer then verifies the full data of the business wide table, and merges the verified full data of the business wide table with the incremental data of the business wide table obtained by the incremental data layer to obtain the target business wide table. In the process of obtaining the target business wide table, the business processing layer does not need to read data from the corresponding business data source according to the business logic, which realizes the decoupling of the cumbersome docking processes between multiple business departments. In addition, the system synchronizes data at a certain frequency and verifies the full data of the business wide table to ensure data consistency when docking multiple data sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a structural diagram of the business data processing architecture in the prior art;

[0041] Figure 2 This is a schematic diagram of the structure of a business data processing system provided by the first embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of a business data processing system provided by Embodiment 2 of the present invention;

[0043] Figure 4 This is an example architecture diagram of a business data processing system provided by Embodiment 2 of the present invention;

[0044] Figure 5 This is a flowchart of a business data processing method provided by Embodiment 3 of the present invention;

[0045] Figure 6 This is a schematic diagram of a data query page provided by the third embodiment of the present invention;

[0046] Figure 7 This is a structural diagram of a server provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0048] Example 1

[0049] Figure 2This is a schematic diagram of the structure of a business data processing system provided in Example 1 of the present invention. This embodiment is applicable to situations where a business party integrates business data from multiple data sources. The system can be implemented using software and / or hardware and integrated into a device with application development capabilities. The system specifically includes the following structure:

[0050] Data warehouse mart data layer 110 , incremental data layer 120 and business processing layer 130 .

[0051] The data warehouse mart data layer 110 is used to synchronize the full data of each business data source to the data warehouse data mart at a preset frequency, and process the full data to obtain the full data of the business wide table.

[0052] Specifically, the data warehouse mart data layer 110 uses the Hive data warehouse as a Hive data mart (data warehouse data mart) that stores a large amount of data. It is a mechanism that can store, query and analyze large-scale data stored in Hadoop. The data is synchronized from the business data source related to the current business party at a preset frequency. The current business party can be a sales merchant or a platform, and the business data source is the source of specific sales information of various commodities sold by the current business party. The preset frequency can be to synchronize data every few hours, or to synchronize data on a daily basis (one day, two days, or any other number of days). Usually, data synchronization is performed once a day.

[0053] Furthermore, the data warehouse mart data layer 110 performs preliminary processing on the synchronized data to obtain the full set of business-wide table data that conforms to the current business entity's business model, and then stores the full set of business-wide table data in the business entity's business database. This eliminates the need for the current business entity to retrieve the corresponding business data from each relevant business data source according to business processing logic when sorting business data. This decouples the cumbersome docking processes of various business departments, achieving good business decoupling and reducing the overhead of excessive interface docking and joint debugging when the full set is heterogeneous.

[0054] For example, in one specific implementation, historical data from each business data source, i.e., all data before the current time (T+1 data), can be synchronized to the Hive marketplace at a fixed time each day (e.g., 00:00) and the synchronization timestamp recorded. This full data is then consolidated according to the current business model to generate the full data for the business wide table for that day, and pushed daily to the business database, which can be a MySQL database.

[0055] The incremental data layer 120 is used to connect with each business data source to obtain real-time incremental data from each business data source, thereby obtaining incremental data for the business wide table. Specifically, the incremental data layer connects with each business data source through a point-to-point message queue (MQ) or publish-subscribe messaging (Kafka) to obtain incremental data.

[0056] Typically, when analyzing and processing daily business data, there will be a time delay relative to the historical full data obtained. For example, all historical full data before the day is obtained at 00:00 every day, and the current business party needs to complete the business data statistics of the day at 15:00 on the same day. In this case, the business data will change during the time period from 00:00 to 15:00. This requires collecting all dynamic change information before collating the data for the day. For example, during the sales process of goods, the inventory of goods changes, which can be replenishment to increase inventory or sales to reduce inventory. Further, the business processing layer 130 performs data processing.

[0057] The business processing layer 130 is used to verify the full data of the business wide table, and fuse the verified full data of the business wide table with the incremental data of the business wide table to obtain target business wide table data.

[0058] Specifically, the business processing layer verifies the full data of the business wide table, including determining whether the full data of each business data source has changed, such as whether some data is still valid. If so, the full data of the business wide table is updated based on the changed full data. Alternatively, data processing can be performed directly using the new data after daily full data synchronization. This ensures that the business data after daily data processing remains consistent with the data source data.

[0059] Since the overlay of full and incremental data may result in data duplication and disordered order, the accuracy of subsequent aggregate queries on the data will be affected. Therefore, data deduplication is necessary to ensure data order and integrity. When the business database receives the full data from the business wide table, a status table records the successful data reception. At a preset time, a job task in the business processing layer triggers verification of the received full data from the business wide table. Then, based on the timestamp version, the latest daily full data from the business wide table and the incremental messages are deduplicated and merged to obtain the daily target business wide table data that achieves data accuracy. For example, assume that the incremental information is the inventory information of product P. At the first moment, P's inventory information becomes zero. At the second moment, P's inventory information becomes 10. At the third moment, P's inventory information becomes 2. At the fourth moment, P's inventory information becomes 10 again. During the data merge, only the inventory data of P at the fourth moment is updated. The data at the first, second, and third moments can be filtered out. This way, the target business wide table data obtained is the most up-to-date data.

[0060] The technical solution of this embodiment is to form a business data processing system through the Hive mart data layer, the incremental data layer and the business processing layer. In the Hive mart data layer, the Hive data warehouse is used as the data mart and the full data of multiple data sources are synchronized at a preset frequency. The full data of each data source is processed to obtain the full data of the business wide table of the business party. The business processing layer then verifies the full data of the business wide table, and merges the verified full data of the business wide table with the incremental data of the business wide table obtained by the incremental data layer to obtain the target business wide table. In the process of obtaining the target business wide table, the business processing layer does not need to read data from the corresponding business data source according to the business logic, thereby realizing the decoupling of the cumbersome docking process between multiple business departments. In addition, the system synchronizes data at a certain frequency and verifies the full data of the business wide table to ensure data consistency when docking multiple data sources.

[0061] Example 2

[0062] Figure 3 This is a schematic diagram of the structure of a business data processing system provided in Example 2 of the present invention. This example further improves the business data processing system based on the above examples by adding a business data storage layer. Explanations of terms that are identical or corresponding to those in the above examples are not repeated here.

[0063] See also Figure 3 The business data processing system provided in this embodiment specifically includes the following structure:

[0064] The data warehouse includes a mart data layer 110 , an incremental data layer 120 , a business processing layer 130 and a business storage layer 140 .

[0065] Among them, the data warehouse mart data layer 110 is used to synchronize the full data of each business data source to the Hive data mart according to a preset frequency, and process the full data to obtain the full data of the business wide table; the incremental data layer 120 is used to connect with the business data sources, obtain the real-time incremental data of each business data source, and obtain the incremental data of the business wide table; the business processing layer 130 is used to verify the full data of the business wide table, and fuse the verified full data of the business wide table with the incremental data of the business wide table to obtain the target business wide table data; the business storage layer 140 is used to store the target business wide table data of the day in the business database, and archive the target business wide table data before the target business wide table data of the day into the ClickHouse database for partitioned storage.

[0066] Furthermore, the business processing layer 130 may filter incremental data according to preset constraints. For example, if the preset condition is to only display products with inventory quantities less than or equal to 1, then information with inventory quantities greater than 1 in incremental information from multiple data sources will be filtered out. In actual applications, corresponding constraints can be set according to business needs.

[0067] In a preferred embodiment, in order to ensure the transactional nature of incremental data, the incremental data is stored in a Mysql database, and the transactional nature and consistency of the data are ensured by a distributed lock. The distributed lock can be a Redis distributed lock or other forms of distributed locks.

[0068] Furthermore, the business storage layer 140 is used to store the daily target business wide table data in the business database, and archive the target business wide table data before the target business wide table data of the day into the ClickHouse database for partitioned storage. Among them, the business database is a MySQL database. The selection of Mysql and ClickHouse is mainly based on the characteristics of the data. Since daily data includes a large amount of incremental data and full data updates, the transactional requirements and insert update performance are relatively high, so Mysql is a good choice. The target business wide table data before the target business wide table data of the day (i.e. historical data) is stored in the ClickHouse database because of the large amount of data. The main reason for not using the conventional Mysql sharding solution is that if Mysql sharding is used, the storage performance and the subsequent aggregation query performance of different fields will be affected (in actual scenario applications, 20 million data, using Mysql database aggregation query will be delayed by more than 10s), and this part of the data also has the following characteristics: the data volume is large, no changes are required, and complex aggregation queries are required. Therefore, this system introduces the ClickHouse database to realize partitioned storage of archived data. ClickHouse is a columnar database suitable for large-scale data aggregation operations, with excellent query performance (queries in seconds). It solves the storage and query problems of the above business scenarios and can support the increase in business data volume in the future. Specifically, when the wide table data of the target business on that day becomes historical data, it is pushed to the Hive data mart, and then the archived data in the data mart is synchronized to the ClickHouse partition through a scheduling task.

[0069] Figure 4The business data processing system architecture in a specific example includes a Hive mart data layer (data warehouse mart data layer), an incremental data layer, a business processing layer, and a business data storage layer. In this figure, only three different business data sources are shown as representatives of multiple business data sources, namely Business 1, Business 2, and Business 3. At a fixed time every day (such as 00:00), the historical data of each business data source, that is, all data before the current time (T+1 data), is synchronized to the Hive data mart and the timestamp of the data synchronization is recorded. According to the business model of the current business party, the full amount of synchronized data is integrated to obtain the full amount of business wide table data (business wide table T+1 data) and pushed to the business database. Among them, the business database can be a MySQL database. It is understandable that synchronizing the historical data of the business data source once a day completes the daily verification of the historical data. Furthermore, the business data processing system calls the corresponding interface to connect with each business data source, and receives incremental data through MQ or Binlog to obtain incremental data of the business wide table. In the business processing layer, data indicators are processed for incremental data, including business verification, data supplementation, etc., and the idempotence and transactional guarantees of the interface are realized (i.e. Figure 4 The core logic processing in the business processing layer is then performed (the target business wide table data is merged with the full data). This layer also records data processing logs and monitors data processing. Furthermore, the target business wide table data obtained by the business processing layer is stored in a MySQL database. The next day, when the target business wide table data becomes a historical record, it is archived in the ClickHouse database. This can be provided to business parties for historical data joint table queries and historical data comparisons.

[0070] The technical solution of this embodiment is to form a business data processing system through a data warehouse mart data layer, an incremental data layer and a business processing layer. In the data warehouse mart data layer, the Hive data warehouse is used as a data mart and the full data of multiple data sources are synchronized at a preset frequency. The full data of each data source is processed to obtain the full data of the business wide table of the business party. The business processing layer then verifies the full data of the business wide table, and merges the verified full data of the business wide table with the incremental data of the business wide table obtained by the incremental data layer to obtain the target business wide table. In the process of obtaining the target business wide table, the business processing layer does not need to read data from the corresponding business data source according to the business logic, thereby realizing the decoupling of the cumbersome docking process between multiple business departments. In addition, the system synchronizes data at a certain frequency and verifies the full data of the business wide table to ensure the consistency of data when docking multiple data sources. In addition, the target business wide table of the day is stored in the MySQL database, and the historical target business wide table is archived in the ClickHouse database to improve data query efficiency.

[0071] The following is an embodiment of a business data processing method provided by an embodiment of the present invention. This method and the business data processing system of each of the aforementioned embodiments are based on the same inventive concept and can be executed by the business data processing system of each of the aforementioned embodiments. For details not fully described in the embodiments of the business data processing method, reference can be made to the embodiments of the aforementioned business data processing system.

[0072] Example 3

[0073] Figure 5 This is a flowchart of a business data processing method provided in Example 3 of the present invention. This embodiment is applicable to situations where a business party integrates business data by connecting to multiple data sources. The method specifically includes the following steps:

[0074] S210. Obtain the full data of the current day's business wide table from the data warehouse data mart, wherein the data warehouse data mart includes the full data of each business data source updated daily, and the full data of the current day's business wide table is obtained by processing the full data of each business data source in the data warehouse data mart.

[0075] Specifically, the business data processing system includes a data warehouse mart data layer. This layer uses the Hive data warehouse as a Hive data mart to store large amounts of data. This layer is a mechanism for storing, querying, and analyzing large amounts of data stored in Hadoop. This data is synchronized daily from the business data source associated with the current business entity, and also records the timestamp of the synchronized data (i.e., a snapshot of the data at a specific moment). The current business entity can be a sales merchant or a platform, and the business data source is the source of specific sales information for the various products sold by the current business entity.

[0076] Furthermore, the data warehouse mart data layer performs preliminary processing on the daily synchronized data to obtain the full set of data from the current business wide table that matches the current business model. This data is then stored in the business database. This eliminates the need for the current business side to retrieve the corresponding business data from each relevant business data source according to business processing logic when sorting business data. This decouples the cumbersome integration processes between business departments, achieving good business decoupling and reducing the overhead of excessive interface integration and joint debugging when the full heterogeneous data is used.

[0077] S220: Acquire incremental data from each business data source in real time to obtain business wide table incremental data.

[0078] Specifically, in the business data processing system, the incremental data layer connects to various business data sources through a point-to-point messaging system (MQ) or a publish-subscribe messaging system (Kafka) to obtain incremental data from the business wide table. This incremental data is stored in a MySQL database, using distributed locks to ensure transactional consistency of the incremental data from each business data source.

[0079] Typically, when analyzing and processing daily business data, there is a time delay relative to the acquisition of the full historical data. For example, if all historical full data prior to the current day is acquired at 12:00 AM each day, and the current business side needs to complete the business data statistics for the current day at 3:00 PM, then the business data will change between 12:00 AM and 3:00 PM. This requires collecting all dynamic change information before compiling the daily data. For example, during the sales process, the inventory of goods will change, either due to replenishment or sales.

[0080] S230: Fusing the full data of the business wide table on the current day with the incremental data of the business wide table to obtain the target business wide table data on the current day.

[0081] Because the overlay of the full and incremental data from the daily business wide table may result in data duplication and disordered order, the accuracy of subsequent aggregate queries on the data will be affected. Therefore, data deduplication is necessary to ensure data order and integrity. When the business database receives the full data from the business wide table, a status table records the successful data reception. At a preset time, a job task in the business processing layer triggers verification of the received full data from the business wide table. Then, based on the timestamp version, the latest full data from the business wide table and the incremental messages are deduplicated and merged to obtain the daily target business wide table data with accurate data. For example, assume the incremental information is the inventory information for product P. At the first moment, P's inventory information becomes zero. At the second moment, P's inventory information becomes 10. At the third moment, P's inventory information becomes 2. At the fourth moment, P's inventory information becomes 10 again. During the data merge, only the inventory data for P at the fourth moment is updated. The data from the first, second, and third moments can be filtered out. This way, the target business wide table data for the day is the most up-to-date data.

[0082] Furthermore, the business data processing system will also store the target business wide table data of the day in the MySQL database, and archive the target business wide table data before the target business wide table data of the day into the ClickHouse database for partition storage, so that users can query and compare historical data. This is because ClickHouse is a columnar database, which is suitable for large-scale data aggregation operations and has good query performance (query in seconds). It solves the storage and query problems in the business scenario of this embodiment and can support the increase in business data volume in the future.

[0083] For example, let's take the out-of-stock analysis business scenario of a shopping mall app as an example:

[0084] Specifically, the APP out-of-stock analysis mainly hopes to display two parts of data. One part is the real-time out-of-stock data analysis at the moment, and the other part is the comparative analysis of historical data at the same time (such as the current time Time1, compared with the snapshot data of Time1 yesterday).

[0085] In the business data processing system, full data is processed using business models such as the inventory model and product model in the Hive data mart to generate the app out-of-stock business model. This data is the T+1 daily full data and is pushed from the Hive data mart to the MySQL database via a scheduled task. Incremental data is obtained through omni-channel inventory MQ messages and stored in the MySQL database.

[0086] To ensure high concurrency and data consistency for incremental data, Redis distributed locks are used to provide transactional protection for each primary key data point. For example, in this business scenario, storeId+skuId+status+qty (store ID+product ID+delisting and listing status+inventory quantity) is verified and guaranteed. The verification process refers to the content recorded in Table 1. That is, only data with changes in listing and delisting status or inventory (inventory changes from 0 to non-zero or non-zero to 0) on the same date needs to be added to the database. Furthermore, through grouped aggregation queries, the products (SKUs) with the latest out-of-stock status can be identified daily for statistical and detailed query. At the same time, the JOB task triggered after the successful execution of the full data synchronization task can merge the full data and incremental data to deduplicate data based on chronological order.

[0087] Table 1

[0088] SKU in stock time channel state date Whether to add sku1 0 10:00 APP Shelves 02-19 Yes, new data sku1 0 10:00 APP Shelves 02-20 Yes, the dates are different sku1 0 10:20 APP Shelves 02-20 No, inventory remains unchanged sku1 3 11:00 APP Shelves 02-20 Yes, inventory changes sku1 2 11:10 APP Shelves 02-20 No, inventory is still not 0 sku1 0 12:00 APP Shelves 02-20 Yes, inventory changes sku1 0 13:00 APP Removed from shelves 02-20 Yes, status change

[0089] Furthermore, the task data processing system can also push the daily out-of-stock analysis data from MySQL to the Hive data mart through scheduling tasks. After the push is successful, the data archive data will be synchronized to ClickHouse and stored by date partition. It can also provide aggregate query and page data support. The query page is as follows: Figure 6 As shown. Figure 6 A page of out-of-stock statistics is displayed, which includes product category information. You can select data under a category such as catering, fresh food, or daily necessities. The interface also displays real-time out-of-stock monitoring data. For example, the number of out-of-stock products is 1,807. The real-time out-of-stock number is 94.6% lower than the data at the same time on 2020-2-28. The data comparison period can be selected from any historical data of the past day. In addition, Figure 6 The interface can also display the number of out-of-stock items in different stores on the same day and the number of out-of-stock items on the comparison day, and can further obtain data such as single item out-of-stock orders.

[0090] The technical solution of this embodiment is to obtain the full data of the business wide table of the day from the Hive market every day through the business data processing system, and then obtain incremental data through the message queue. The full data of the business wide table of the day is merged with the incremental data of the business wide table obtained from the incremental data layer to obtain the target business wide table of the day. In the process of obtaining the target business wide table, there is no need to read data from the corresponding business data source according to the business logic, which realizes the decoupling of the cumbersome docking process between multiple business departments. In addition, the system synchronizes data at a certain frequency and verifies the full data of the business wide table to ensure data consistency when docking multiple data sources. In addition, the target business wide table of the day is stored in the MySQL database, and the historical target business wide table is archived in the ClickHouse database to improve data query efficiency.

[0091] Example 4

[0092] Figure 7 A schematic diagram of the structure of a server provided in embodiment 4 of the present invention. Figure 7 A block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present invention is shown. Figure 7 The server 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0093] like Figure 7 As shown, server 12 is implemented as a general-purpose computing device. Components of server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing units 16).

[0094] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0095] The server 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 12, including volatile and non-volatile media, removable and non-removable media.

[0096] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive"). Although Figure 7 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0097] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0098] The server 12 may also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the server 12, and / or any device that enables the server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22. Furthermore, the server 12 may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the server 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with the server 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0099] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a business data processing method provided by an embodiment of the present invention, which includes:

[0100] Obtaining full data of the current day's business wide table from the data warehouse data mart, wherein the data warehouse data mart includes full data of each business data source updated daily, and the full data of the current day's business wide table is obtained by processing the full data of each business data source in the data warehouse data mart;

[0101] Acquire incremental data from each business data source in real time to obtain incremental data of the business wide table;

[0102] The full data of the business wide table on the day and the incremental data of the business wide table are merged to obtain the target business wide table data on the day.

[0103] Of course, those skilled in the art will understand that the processor may also implement the technical solution of the business data processing method provided by any embodiment of the present invention.

[0104] Example 5

[0105] This fifth embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the service data processing method provided in any embodiment of the present invention are implemented. The method includes:

[0106] Obtaining full data of the current day's business wide table from the data warehouse data mart, wherein the data warehouse data mart includes full data of each business data source updated daily, and the full data of the current day's business wide table is obtained by processing the full data of each business data source in the data warehouse data mart;

[0107] Acquire incremental data from each business data source in real time to obtain incremental data of the business wide table;

[0108] The full data of the business wide table on the day and the incremental data of the business wide table are merged to obtain the target business wide table data on the day.

[0109] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0110] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0111] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0112] The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computer device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

[0114] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A business data processing system, characterized in that: include: The data warehouse mart data layer is used to synchronize the full data of each business data source to the data warehouse data mart at a preset frequency, and process the full data to obtain the full data of the business wide table, wherein the data warehouse data mart is a Hive data mart; The incremental data layer is used to connect with the business data sources, obtain real-time incremental data from each business data source, and obtain incremental data of the business wide table; The business processing layer is configured to verify the full data of the business wide table and fuse the verified full data of the business wide table with the incremental data of the business wide table to obtain target business wide table data, wherein the target business wide table data includes the target business wide table data for the day; When the business processing layer performs data fusion, the business wide table incremental data corresponding to the latest time in the business wide table incremental data corresponding to the same data item is fused with the verified business wide table full data; The business storage layer is used to store the target business wide table data of the day into the business database, and archive the target business wide table data before the target business wide table data of the day into the ClickHouse database for partition storage, wherein the business database includes a MySQL database; The business storage layer is specifically used to push the target business wide table data before the target business wide table data of the day to the Hive data mart, and then synchronize the archived data to the ClickHouse partition in the Hive data mart through scheduling tasks.

2. The system according to claim 1, wherein: The data warehouse mart data layer is specifically used for: On a daily basis, synchronize the full amount of data from each business data source before the current data synchronization time to the data warehouse data mart and record the data synchronization timestamp; Integrate the full data according to the current business model to obtain the full data of the wide table of the business on that day; The full amount of data of the broad table of business on that day and the timestamp are pushed to the business database of the current business party.

3. The system according to claim 2, characterized in that The business processing layer verifies the full amount of data in the business wide table, including: Determine whether the full amount of data of each business data source has changed; If so, the full data of the broad table of the day's business is updated according to the changed full data.

4. The system according to claim 2, wherein: When the business processing layer performs data fusion, the Job task is called at a preset time to de-duplicate and merge the full data of the business wide table of the day and the incremental data after the timestamp to obtain the target business wide table data of the day.

5. The system according to claim 1, wherein: The incremental data layer is connected to each business data source through a message queue.

6. The system according to any one of claims 1 to 5, characterized in that: The business processing layer is also used to perform data verification and filtering on the incremental data of the business wide table before data fusion.

7. The system according to claim 6, characterized in that The business processing layer stores the incremental data of the business wide table in a MySQL database, and uses distributed locks to ensure the transaction consistency of the incremental data of each business data source.

8. A business data processing method, applied to a business data processing system, characterized in that: include: Obtaining full data from the current day's business wide table from a data warehouse data mart, wherein the data warehouse data mart includes full data from each business data source updated daily, and the current day's business wide table full data is obtained by processing the full data from each business data source in the data warehouse data mart, wherein the data warehouse data mart is a Hive data mart; Acquire incremental data from each business data source in real time to obtain incremental data of the business wide table; The full data of the business wide table on the day and the incremental data of the business wide table are merged to obtain the target business wide table data on the day; The data fusion of the full data of the business wide table on the day and the incremental data of the business wide table to obtain the target business wide table data on the day includes: Merge the incremental data of the business wide table corresponding to the latest time in the incremental data of the business wide table corresponding to the same data item with the full data of the business wide table on that day; The target business wide table data of the day is stored in the MySQL database, and the target business wide table data before the target business wide table data of the day is archived in the ClickHouse database for partition storage; Archiving the target business wide table data before the target business wide table data of the day into the ClickHouse database for partition storage includes: Push the target business wide table data before the target business wide table data of the day to the Hive data mart, and then synchronize the archived data to the ClickHouse partition in the Hive data mart through scheduling tasks.

9. The method according to claim 8, characterized in that The processing of the full amount of data of each business data source in the data warehouse data mart includes: Every day, the full amount of data of each business data source before the current data synchronization time is synchronized to the data warehouse data mart and the timestamp of the data synchronization is recorded; The full amount of data is integrated according to the business model of the current business party to obtain the full amount of data of the broad table of the business on that day.

10. The method according to claim 8, characterized in that The method further comprises: The incremental data of the business wide table is stored in a MySQL database, and a distributed lock is used to ensure the transaction consistency of the incremental data of each business data source.

11. A server, characterized in that: The server includes: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the business data processing method as described in any one of claims 8 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the business data processing method according to any one of claims 8 to 10 is implemented.

Images