Data processing method and device applied to data warehouse, equipment and medium

Abstract

The invention provides a data processing method and device applied to a data warehouse, equipment and a medium. The method comprises the following steps: acquiring change data of a service, and determining a data type of the change data; wherein the change data represents changed data in the business; the data type is at least one of main data and operation data, the main data represents attribute information of the business, and the operation data represents data in a transaction activity of the business; determining format information of the service report according to the data type of the changed data; wherein the service report represents the change condition of the service, and the format information represents the content required to be contained in the service report and the format of the content; and generating a service report according to the change data and the format information, and storing and displaying the service report. According to the method, the data processing efficiency of the data warehouse is improved.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a data processing method, apparatus, equipment, and medium for use in data warehouses. Background Technology

[0002] In business environments characterized by large volumes of business data, high rates of incremental data generation, and complex business changes, enterprises often have weak data analysis capabilities. Therefore, a data warehouse can help business decision-makers analyze business trends by extracting and transforming data from multiple data sources.

[0003] Data warehouses store massive amounts of data, which leads to problems such as low data processing efficiency and high resource consumption. Summary of the Invention

[0004] This application provides a data processing method, apparatus, equipment, and medium for use in data warehouses, in order to solve the technical problem of low data processing efficiency in data warehouses.

[0005] In a first aspect, this application provides a data processing method applied to a data warehouse, including:

[0006] Obtain change data for the business and determine the data type of the change data; wherein, change data represents data that has changed in the business; the data type is at least one of master data and operational data, master data represents the attribute information of the business, and operational data represents the data in the business's transaction activities;

[0007] Based on the data type of the changed data, determine the format information of the business report; whereby the business report represents the changes in the business, and the format information represents the content that needs to be included in the business report and the format of that content;

[0008] Based on the changed data and format information, generate a business report, and store and display the business report.

[0009] Optionally, using the method described above, a business report is generated based on the changed data and format information, including:

[0010] If the data type of the changed data is operation data, then the master data corresponding to the changed data is searched from the preset master data cache pool;

[0011] Based on the changed data and the corresponding master data, and using the format information of the business report, a business report is generated.

[0012] Optionally, the method described above retrieves the master data corresponding to the changed data from a preset master data cache pool, including:

[0013] Retrieve the business ID from the changed data;

[0014] Search the master data corresponding to the business ID from the preset master data cache pool, which is the master data corresponding to the changed data.

[0015] Optionally, using the method described above, a business report is generated based on the changed data and the corresponding master data, according to the format information of the business report. This report includes:

[0016] The changed data and the corresponding master data are merged to obtain the merged data.

[0017] Based on the format information of the business report, the merged data is converted to a new format to obtain the business report.

[0018] Optionally, using the method described above, a business report is generated based on the changed data and format information, including:

[0019] If the data type of the changed data is primary data, then the changed data is converted to the correct format according to the format information of the business report to obtain the business report.

[0020] Optionally, using the method described above, the format information of the business report is determined based on the data type of the changed data, including:

[0021] Based on the preset association relationships, the format information corresponding to the changed data is determined, which is the format information of the business report; whereby the preset association relationships represent the relationship between different data types and format information.

[0022] Optionally, in the method described above, a message queue is pre-set, which is divided into multiple topics, and each topic is divided into multiple partitions; the method also includes:

[0023] Based on the identifier information located at the preset first position in the changed data, the subject to which the changed data belongs is determined; whereby the subject represents the object that has changed in the business.

[0024] Based on the identifier information located at the preset second position in the changed data, determine the partition to which the changed data belongs;

[0025] The changed data is stored in the message queue based on the topic and partition to which it belongs.

[0026] Optionally, the method described above determines the partition to which the changed data belongs based on the identifier information located at a preset second position in the changed data, including:

[0027] Based on a preset hash algorithm, determine the hash value corresponding to the identifier information located at the preset second position;

[0028] The partition corresponding to the hash value is determined as the partition to which the changed data belongs.

[0029] Alternatively, the method described above may also include:

[0030] Retrieve changed data from multiple preset partitions in the message queue;

[0031] Based on the changed data in each preset partition, the system determines the format information of the business report according to the data type of the changed data; based on the changed data and format information, the system generates the business report and stores and displays the business report.

[0032] Alternatively, the method described above may also include:

[0033] If the data type of the changed data is primary data, then the changed data will be cached in the preset primary data cache pool.

[0034] Optionally, the method described above caches the changed data to a preset master data cache pool, including...

[0035] Determine the target location of the master data in the message queue; where the target location is the topic and partition to which the master data belongs in the message queue;

[0036] Retrieve the stored master data from the target location and cache the stored master data in a preset master data cache pool.

[0037] Alternatively, the method described above may also include:

[0038] Traverse the stored master data. If the master data being traversed meets the preset filtering conditions, then filter the master data being traversed.

[0039] Secondly, this application provides a data processing apparatus for use in a data warehouse, comprising:

[0040] The first determining unit is used to acquire change data of the business and determine the data type of the change data; wherein, the change data represents the data that has changed in the business; the data type is at least one of master data and operational data, where master data represents the attribute information of the business and operational data represents the data in the transaction activities of the business;

[0041] The second determining unit is used to determine the format information of the business report based on the data type of the changed data; wherein, the business report represents the changes in the business, and the format information represents the content that needs to be included in the business report and the format of the content;

[0042] The generation unit is used to generate business reports based on the changed data and format information, and to store and display the business reports.

[0043] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0044] The memory stores the instructions that the computer executes;

[0045] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0046] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0047] Fifthly, this application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0048] The data processing method, apparatus, equipment, and medium for data warehouses provided in this application acquire change data from business processes and determine the data type of the change data. The data type is at least one of master data and operational data. Master data represents the attribute information of the business, and operational data represents data from business transaction activities. Based on the different data types of the change data, the corresponding business report format is determined. The format information represents the content and format to be included in the business report, thus separating the processing of master data and operational data, ensuring data processing efficiency. Furthermore, a business report is generated based on different change data and their corresponding format information. The data processing method for data warehouses provided in this application improves the efficiency of data processing in data warehouses. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] Figure 1 A flowchart illustrating a data processing method applied to a data warehouse, as provided in this application;

[0051] Figure 2 A flowchart illustrating another data processing method for data warehouses provided in this application;

[0052] Figure 3 A flowchart illustrating another data processing method applied to a data warehouse provided in this application;

[0053] Figure 4A schematic diagram of a data processing device applied to a data warehouse is provided in this application;

[0054] Figure 5 A schematic diagram of another data processing device for use in a data warehouse provided in this application;

[0055] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0056] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0058] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0059] In a business environment characterized by large volumes of business data, high rates of incremental data generation, and complex business changes, enterprises often have weak data analysis capabilities. They need to quickly integrate and analyze data and adjust their strategies in a timely manner to cope with the highly competitive business environment.

[0060] Therefore, a data warehouse can extract and transform data from multiple data sources and present it to enterprises, helping decision-makers analyze business trends.

[0061] In a data warehouse, the stored data mainly consists of master data and operational data. Master data represents the attribute information of the business, while operational data represents data from the business's transaction activities. However, most of the incremental data in the data source is frequently changing operational data. Considering the correlation with operational data, it is still necessary to frequently extract related master data from the data source, which leads to the problem of high overhead in master data extraction.

[0062] Therefore, data warehouses suffer from problems such as low data processing efficiency and high resource consumption.

[0063] The data processing method, apparatus, equipment, and medium provided in this application for data warehouses acquire change data from business processes and determine the data type of the change data. The data type is at least one of master data and operational data. Master data represents the attribute information of the business, and operational data represents data from business transaction activities. If the change data is master data, the master data cache pool is updated, and a business report is generated based on the master data and the format information of the business report. If the change data is operational data, the associated master data is retrieved from the master data cache pool based on the operational data, and a business report is generated based on the operational data, the associated master data, and the format information of the business report. Simultaneously, a message queue is set up, divided into multiple topics, and each topic is further divided into multiple partitions for distributing change data, thus separating the processing of master data and operational data and ensuring data processing efficiency. Furthermore, the master data cache pool is updated based on the master data in the message queue, so that the master data required for the business report is retrieved from the master data cache pool instead of from the data source of the change data. This not only ensures data processing efficiency but also saves the overhead of revisiting the data source and conserves computing resources. The data processing method for data warehouses provided in this application improves the efficiency of data processing in data warehouses.

[0064] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0065] Figure 1 This application provides a flowchart illustrating a data processing method applied to a data warehouse. The execution entity of this method can be a server, host, or other device. This method can be applied to electronic devices that have a data warehouse deployed within them. The data warehouse is used for data processing and storage. Figure 1 As shown, the method may include:

[0066] S101. Obtain change data for the business and determine the data type of the change data; wherein, change data represents data that has changed in the business; the data type is at least one of master data and operational data, master data represents the attribute information of the business, and operational data represents the data in the business's transaction activities.

[0067] Specifically, business change data refers to data derived from business operations, such as adding, modifying, or deleting records. These operations may involve updating customer information, changing order status, adjusting inventory, etc. Business change data can originate from multiple data sources. For example, the data source could be data actively entered by the user, or data transmitted from a third-party system.

[0068] In one possible implementation, change data for the business can be obtained from multiple data sources using CDC (Change Data Capture) technology.

[0069] Preferably, a listening device can be used to customize the filtering of the service change data obtained by the CDC, such as filtering only the newly added and modified data records related to the service in the change data obtained by the CDC, so that the service change data better meets the user's needs.

[0070] In one possible implementation, the listener retrieves business change data at certain time intervals, such as every minute.

[0071] In one possible implementation, multiple data sources are multiple data tables. Changes to the data tables will generate corresponding change data. The data types of the change data include master data and operational data.

[0072] Master data can refer to business entity data that does not change frequently, representing the attribute information of the business, such as product information or customer information; operational data can refer to dynamic data that changes frequently during business activities and transactions, representing transaction data in business transactions, such as order quantity and order amount.

[0073] For example, if the data source consists of two tables, namely the Orders table and the Products table, the Orders table contains detailed information about each order, and the content of the Orders table is frequently updated; the Orders table is the data table for operational data. The Products table contains detailed attribute information about the products, and the product information is not frequently updated; the Products table is the data table for main data.

[0074] The Orders table can include the following fields: OrderID, CustomerID, ProductID, OrderDate, and Status. OrderID represents a unique identifier for an order, CustomerID represents a unique identifier for the customer, ProductID represents a unique identifier for the product, OrderDate represents the order date, and Status represents the order status, such as "paid" or "shipped."

[0075] The Products table can include the following fields: ProductID, ProductName, Category, and Price. ProductID represents the unique identifier of the product, ProductName represents the product name, Category represents the product category, and Price represents the product price.

[0076] If the monitoring device detects that a row of data has been inserted into the order table, indicating that a customer has placed an order, the changed data can be represented as: {"operation": "INSERT", "table": "Orders", "timestamp": "2024-10-15T10:30:00", "data": {"OrderID": 12345, "CustomerID": 1001, "ProductID": 987, "OrderDate": "2024-10-15", "Status": "Paid"}}, where operation indicates that the operation is inserting data, the corresponding data table is the order table, timestamp indicates the time the customer placed the order, and data indicates the data of the field in the order table.

[0077] The "table": "Orders" indicates that the changed data is located in the Orders table, which is a table for operation data. Therefore, the data type of the changed data is operation data.

[0078] S102. Determine the format information of the business report based on the data type of the changed data; wherein, the business report represents the changes in the business, and the format information represents the content that needs to be included in the business report and the format of the content.

[0079] Business reports can refer to documents used to present and analyze changes in business operations. By reviewing business reports, managers and decision-makers can understand business dynamics, assess business performance, and identify business problems or opportunities.

[0080] The format information of a business report can refer to the content that needs to be included in the business report and how this content is presented. The business report can be a user-defined template. For example, the format information of a business report can include data fields, data arrangement, visualization charts, text descriptions, etc.

[0081] It is understandable that knowing the data type of the change data allows us to determine the format information of the business report. For example, the change data is the master data, which is the content of the product table. The format of the business report can be a chart showing the relationship between product categories and product prices, used to observe the relationship between product categories and product prices.

[0082] For example, the change data is the operation data, which is the content of the order table. The business report can be a bar chart of the monthly sales of the product with product number 987, used to observe the sales performance of the product.

[0083] In one alternative implementation, step S102 may include:

[0084] Based on the preset association relationships, the format information corresponding to the changed data is determined, which is the format information of the business report; whereby the preset association relationships represent the relationship between different data types and format information.

[0085] The preset relationships can be those pre-set by staff, and these relationships can be defined rules or mappings to data tables. In one possible implementation, the preset relationships can be implemented using an API (Application Programming Interface).

[0086] For example, if the changed data is related to the order table, the format information of the mapping to the order table can be a sales report, which may include at least one of the following: tables, charts, or dashboards.

[0087] For example, if the changed data is {"OrderID": 12345, "CustomerID": 1001, "ProductID": 987, "OrderDate": "2024-10-15", "Status": "Paid"}, and the dashboard in the sales report displays the sales quantity of the product, then the format of the changed data can be a dashboard. The data in the dashboard includes the sales quantity and product name. The product name can be obtained from the product table using the ProductID.

[0088] The advantage of this setup is that by pre-setting multiple business report formats that are related to the data table, it can provide rich presentation formats for business reports from various perspectives, thereby improving the efficiency and accuracy of business report generation.

[0089] S103. Generate a business report based on the changed data and format information, and store and display the business report.

[0090] It is understandable that a business report can be compiled based on the changed data and the corresponding format information.

[0091] In one possible implementation, historical business report data can be retrieved and combined with current change data to generate the current business report. That is, the current business report can retain historical business report data.

[0092] In one possible implementation, the changed data can be inserted into the content corresponding to the format information to generate a business report. For example, the sales report of the monthly sales of the product with product number 987 is the content of the business report.

[0093] In one possible implementation, business reports can be stored in the file system in file formats such as PDF, Excel, and HTML for easy sharing and distribution.

[0094] This application provides a data processing method for data warehouses that improves the efficiency of data warehouse data processing by processing master data and operational data separately, generating business reports based on change data and the corresponding format information.

[0095] Figure 2 This application provides a flowchart illustrating another data processing method applied to a data warehouse. The execution entity of this method can be a server, host, or other device. This method can be applied to electronic devices that have a data warehouse deployed within them. The data warehouse is used for data processing and storage. Figure 2 As shown, the method may include:

[0096] S201. Obtain change data for the business and determine the data type of the change data; wherein, change data represents data that has changed in the business; the data type is at least one of master data and operational data, master data represents the attribute information of the business, and operational data represents the data in the business's transaction activities.

[0097] For example, this step can refer to step S101 above, and will not be repeated here.

[0098] S202. Determine the format information of the business report based on the data type of the changed data; wherein, the business report represents the changes in the business, and the format information represents the content that needs to be included in the business report and the format of the content.

[0099] For example, this step can refer to step S102 above, and will not be repeated here.

[0100] S203. If the data type of the changed data is operation data, then search for the master data corresponding to the changed data from the preset master data cache pool; based on the changed data and the master data corresponding to the changed data, generate a business report based on the format information of the business report.

[0101] The pre-defined master data cache pool can refer to a storage area specifically used to store master data. In one possible implementation, the master data cache pool can organize and index master data based on its identifier, such as a product ID, to quickly find the detailed information corresponding to the product ID.

[0102] In one optional implementation, retrieving the master data corresponding to the changed data from a preset master data cache pool may include:

[0103] Retrieve the business ID from the changed data; search for the master data corresponding to the business ID from the preset master data cache pool, which is the master data corresponding to the changed data.

[0104] The business ID can refer to a master data identifier, such as a product ID. The master data corresponding to the business ID can be retrieved from a pre-defined master data cache pool.

[0105] For example, the change data with data type operation data is {"OrderID": 12345, "CustomerID": 1001, "ProductID": 987, "OrderDate": "2024-10-15", "Status": "Paid"}, where ProductID is a field in the main data table Products, i.e., the main data identifier. Based on ProductID, the product name, product unit price, and other information corresponding to ProductID 987 can be found in the main data cache pool, which is the main data corresponding to this change data.

[0106] In one optional implementation, a business report is generated based on the changed data and the corresponding master data, according to the format information of the business report. This may include:

[0107] The changed data and the corresponding master data are merged to obtain merged data; the merged data is then converted to the correct format based on the format information of the business report to obtain the business report.

[0108] In this context, fusion processing refers to combining changed data with its corresponding master data to form a complete data set. One possible implementation may include, but is not limited to, operations such as data matching, merging, deduplication, and supplementation to ensure data integrity and consistency.

[0109] For example, the changed data is {"OrderID": 12345, "CustomerID": 1001, "ProductID": 987, "OrderDate": "2024-10-15", "Status": "Paid"}, {"OrderID": 12346, "CustomerID": 1002, "ProductID": 987, "OrderDate": "2024-10-15", "Status": "Paid"}, and the corresponding master data is {"ProductID": 987, "ProductName": Jeans, "Category": "Bottoms", "Price": 200}. Merging the changed data and the corresponding master data results in the merged data, which can be represented as {"OrderID": 12345, "ProductName": Jeans, "Price": 200, "Sales Quantity": 2, "Total Sales": 400}.

[0110] Based on the format information of the business report, the merged data is converted to a new format to obtain the business report, which may include:

[0111] Based on the format information mapped from the order table, i.e., the format corresponding to the sales report (e.g., a bar chart showing daily sales revenue), the merged data is converted into a bar chart with the horizontal axis representing the date and the vertical axis representing sales revenue. As can be seen from the business report, the bar chart for the date 2024-10-15 shows an increase of 400 bars compared to the last generated sales report. This yields the business report.

[0112] The advantage of this setup is that by retrieving the master data corresponding to the changed data as the operation data from the preset master data cache pool, instead of retrieving the master data from the data source, the back-access to the master data is avoided, computing resources are saved, and the efficiency of data warehouse processing is improved.

[0113] S204. If the data type of the changed data is primary data, then the changed data is converted to a new format according to the format information of the business report to obtain the business report.

[0114] It is understandable that if the data type is changed to master data, there is no need to obtain the operation data related to the master data. The master data can be directly converted according to the format information of the business report to obtain the business report.

[0115] For example, the change data is to insert a row of information into the Orders table, indicating that a new product and its corresponding details have been added. The change data can be represented as {"ProductID": 999, "ProductName": cotton and linen pants, "Category": "bottoms", "Price": 300}.

[0116] Based on the format information mapped from the product table, i.e., the format corresponding to the product report, for example, if the product report format is a pie chart showing the price distribution of products categorized as "bottoms," with the pie chart size representing the proportion of product prices within that category and the pie chart labels being the product names, we can see from the product report that cotton and linen pants have been added to the pie chart, and this product occupies a certain area on the pie chart. Thus, the business report is obtained.

[0117] This application provides another data processing method for data warehouses, which improves data processing efficiency by processing master data and operational data separately. At the same time, if the data is changed to operational data, the operational data is obtained from the preset master data cache instead of the data source, avoiding the need to revisit the master data, saving computing resources, and improving the efficiency of data warehouse processing.

[0118] Figure 3 This application provides a flowchart illustrating another data processing method applied to a data warehouse. The executing entity of this method can be a server, host, or other device. This method can be applied to electronic devices, where a data warehouse is deployed for data processing and storage. A message queue is pre-configured, divided into multiple topics, and each topic is further divided into multiple partitions. Figure 3 As shown, the method may include:

[0119] S301. Determine the subject to which the changed data belongs based on the identification information located at the preset first position in the changed data; wherein, the subject represents the object that has changed in the business.

[0120] A topic can refer to a pre-defined topic applied to a message queue for staff. A topic is a classification unit in a message queue used to organize and manage data. A topic represents an object that changes in the business.

[0121] For example, the topic can be set as CustomerTopic, OrdersTopic, ProductTopic, etc., where CustomerTopic is used to process customer data, OrdersTopic is used to process order data, and ProductTopic is used to process product data.

[0122] The identifier information at the preset first position can refer to the name of the data table or the ID of the data table. The first position is used to store the name of the data table or the ID of the data table.

[0123] For example, the changed data can be represented as: {"operation": "INSERT", "table": "Orders", "timestamp": "2024-10-15T10:30:00", "data": {"OrderID": 12345, "CustomerID": 1001, "ProductID": 987, "OrderDate": "2024-10-15", "Status": "Paid"}}, where "table": "Orders" indicates that a row of order information has been inserted into the Orders table. Orders is the name of the data table, and based on Orders, the topic to which the changed data belongs can be determined as the OrdersTopic.

[0124] S302. Determine the partition to which the changed data belongs based on the identifier information located at the preset second position in the changed data.

[0125] Partitions can refer to partitions that are pre-defined by staff and applied to a topic within a message queue. Partitions enable parallel data processing and improve data processing efficiency.

[0126] The identifier in the pre-defined second position can refer to a unique field identifier in the data table. For example, the OrderID in the Orders table can be the identifier in the second position, and the ProductID in the Products table can be the identifier in the second position. The second position is used to store the unique field identifier in the data table.

[0127] In one possible implementation, OrderID in the Orders table is the business key of the Orders table, and ProductID in the Products table is the primary key of the Products table. It can be understood that the business key or primary key is a unique field identifier in the data table.

[0128] In an optional implementation, step S302 may include:

[0129] Based on the preset hash algorithm, determine the hash value corresponding to the identifier information located at the preset second position; determine the partition corresponding to the hash value as the partition to which the changed data belongs.

[0130] The preset hash algorithm can refer to a pre-set hash algorithm, that is, to determine the hash value corresponding to the identifier information located at the preset second position based on the preset number of partitions.

[0131] For example, suppose there are 5 partitions under the OrdersTopic, that is, the preset number of partitions is 5, and the partition numbers are 1, 2, 3, 4 and 5 respectively. The identifier information located in the preset second position is OrderID, which is 1001. The hash value corresponding to the identifier information located in the preset second position can be represented as hash(1001)%5=1. Therefore, it can be seen that the partition with partition number 1 is the partition to which the changed data belongs.

[0132] The advantage of this setup is that by setting up multiple topics and multiple partitions, the parallel processing capability of data processing is improved, thereby increasing the efficiency of data processing.

[0133] S303. Store the changed data in the message queue according to the topic and partition to which the changed data belongs.

[0134] A message queue can refer to a data structure used to store and transmit messages. Messages can be any type of data, such as text, JSON (JavaScript Object Notation), XML (Extensible Markup Language), binary data, etc.

[0135] Message queues can be divided into multiple topics, and each topic can be further divided into multiple partitions. Topics are used to classify messages, while partitions are used to improve the parallel processing capabilities of message queues.

[0136] In one possible implementation, changed data is converted into messages and stored in a message queue, where a message can refer to the smallest unit of transmission in the message queue. A message may include a message ID, a partition ID, and a topic identifier. For example, if the changed data is {"operation": "INSERT", "table": "Orders", "timestamp": "2024-10-15T10:30:00", "data": {"OrderID": 12345, "CustomerID": 1001, "ProductID": 987, "OrderDate": "2024-10-15", "Status": "Paid"}}, the message can be {"numberID": 007, "operation": "INSERT", "table": "Orders", "data": {"OrderID": 12345, "CustomerID": 1001, "ProductID": 987, "OrderDate": "2024-10-15", "Status": "Paid"}, "Topic": "OrdersTopic", "partitionKey": 3}, where "numberID": 007 indicates that the message ID of this message is 007, "Topic": "OrdersTopic" indicates that the topic identifier is OrdersTopic, and "partitionKey": 3 indicates that the partition ID is 3.

[0137] S304. Retrieve change data from multiple preset partitions in the message queue.

[0138] In the message queue, messages are arranged in a first-in, first-out (FIFO) order, meaning that the earliest message to enter the queue is processed first, and the messages that enter the queue later are processed later.

[0139] In one possible implementation, the message queue includes pointers for efficient management and access to messages. Pointers can be used to mark specific positions within the queue, such as the head or tail. By setting pointers, it's possible to quickly insert or delete messages without having to traverse the entire message queue for each operation.

[0140] In one alternative implementation, each message has a corresponding timestamp, and the time of message insertion is recorded when the message is inserted, so that the newly inserted message can be determined by comparing the timestamps.

[0141] For example, if the last time the message queue retrieved the changed data from multiple preset partitions was at 15:21, then all messages inserted after 15:21 are new messages. When traversing from the tail pointer of the message queue, if the timestamp of the inserted message is earlier than 15:21, then the traversal stops and the changed data retrieved from the message queue is obtained.

[0142] S305. If the data type of the changed data is primary data, then the changed data will be cached in the preset primary data cache pool.

[0143] For example, if the data type of the changed data is primary data, that is, the data type of the changed data in the message queue is primary data, such as messages under the product topic ProductTopic are all primary data related messages.

[0144] Messages that change data to primary data are cached in a preset primary data cache pool to update the preset primary data cache pool and ensure the consistency of primary data in the data warehouse.

[0145] In one alternative implementation, step S305 may include:

[0146] Determine the target location of the master data in the message queue; where the target location is the topic and partition to which the master data belongs in the message queue; retrieve the stored master data from the target location and cache the stored master data in a preset master data cache pool.

[0147] The target location can refer to the topic and partition to which the changed data, i.e. the master data, belongs in the message queue. The target location can be understood as the location where the most recently inserted message was placed in the message queue.

[0148] For example, if the last time the message queue retrieved the changed data from multiple preset partitions was 15:21, then all messages inserted after 15:21 are new messages. When traversing from the tail pointer of the message queue, if the timestamp of the inserted message is earlier than 15:21, then the traversal stops, and the pointer position at this time is the target position.

[0149] Messages following the target location are all modified data to be cached in the preset master data cache pool.

[0150] It should be noted that if there is no stored data in the preset main data cache pool, the tail pointer traversal method is not required, as long as all messages in the message queue can be traversed.

[0151] In one possible implementation, the master data is inserted into the master data cache pool in the form of an object, such as a List. <record>The structure consists of Record, which represents an object corresponding to a single data record and contains the values ​​of each field in the table that need to be inserted into the main data cache pool, and List, which represents a list of multiple objects.

[0152] In one optional implementation, if the device hosting the data warehouse experiences a system outage, potentially leading to the loss of master data, the security of the master data can be ensured by restoring a snapshot of the master data based on its distribution in the message queue. Specifically, the steps for restoring a snapshot of the master data are consistent with the steps described above: determining the target location of the master data in the message queue; retrieving the stored master data from the target location; and caching the stored master data in a preset master data cache pool. These steps will not be repeated here.

[0153] In one optional implementation, the stored master data is traversed, and if the currently traversed master data meets the preset filtering conditions, the currently traversed master data is filtered.

[0154] It is understandable that by using preset filtering conditions, some data that is irrelevant to business reports can be filtered out, thereby reducing unnecessary computing resources and memory usage.

[0155] For example, the preset filtering conditions can be met by satisfying the filtering key, such as customer ID, product ID, timestamp, etc.

[0156] The timestamp serves as a filter key, ensuring that only data within a certain time range is processed. For example, it filters out historical data that has expired or no longer needs to be processed, and only processes data from the most recent time period.

[0157] It is understandable that the preset filtering conditions can be customized by staff in advance, and there are no restrictions on the preset filtering conditions here.

[0158] The advantage of this setup is that by using filter keys, unnecessary data processing can be reduced, computing resources can be saved, and the master data cache can retain valid business-related data.

[0159] S306. Based on the change data in each preset partition, determine the format information of the business report according to the data type of the change data; generate the business report according to the change data and format information, and store and display the business report.

[0160] It is understandable that when a large amount of incremental data is generated frequently, the acquired change data is not just one piece of data, but can include multiple pieces of data. These multiple pieces of data can all be master data, all be operation data, or a combination of some master data and some operation data. Regardless of the case, the steps are as follows: determine the format information of the business report based on the data type of the change data, where the data type is at least one of master data and operation data; generate the business report based on the change data and the format information; and store and display the business report.

[0161] This application provides another data processing method for data warehouses. By dividing data into multiple topics and partitions and utilizing message queues for data organization and management, it improves the parallel processing capability and efficiency of data processing. Specifically, this method identifies the topics and partitions of changed data, stores them in a message queue, and retrieves changed data from the message queue for processing. Filtering data using preset filtering conditions reduces unnecessary computational resource consumption and ensures that business-related valid data is retained in the master data cache. Finally, this method can generate business reports based on data types and store and display them. This data processing method for data warehouses improves the efficiency and accuracy of data processing while ensuring data consistency and security.

[0162] Figure 4 A schematic diagram of a data processing device for use in a data warehouse is provided in this application, as shown below. Figure 4 As shown, the data processing device 40 applied to the data warehouse includes: a first determining unit 401, a second determining unit 402, and a generating unit 403.

[0163] The first determining unit 401 is used to acquire change data of the business and determine the data type of the change data; wherein, the change data represents the data that has changed in the business; the data type is at least one of master data and operation data, master data represents the attribute information of the business, and operation data represents the data in the transaction activities of the business;

[0164] The second determining unit 402 is used to determine the format information of the business report based on the data type of the changed data; wherein, the business report represents the changes in the business, and the format information represents the content that needs to be included in the business report and the format of the content;

[0165] The generation unit 403 is used to generate a business report based on the changed data and format information, and to store and display the business report.

[0166] Figure 5 A schematic diagram of another data processing device for use in a data warehouse provided in this application is shown below. Figure 5 As shown, the data processing device 50 applied to the data warehouse includes: a first determining unit 501, a second determining unit 502, and a generating unit 503. The second determining unit 502 further includes an operation data processing module 5021 and a main data processing module 5022.

[0167] The operation data processing module 5021 is used to, if the data type of the changed data is operation data, search for the master data corresponding to the changed data from the preset master data cache pool; and generate a business report based on the format information of the business report according to the changed data and the master data corresponding to the changed data.

[0168] In an optional example, the operation data processing module 5021 is specifically used to obtain the business ID from the changed data; and to search for the master data corresponding to the business ID from the preset master data cache pool, which is the master data corresponding to the changed data.

[0169] In an optional example, the operational data processing module 5021 is specifically used to merge the changed data and the master data corresponding to the changed data to obtain merged data; and to convert the format of the merged data according to the format information of the business report to obtain the business report.

[0170] The master data processing module 5022 is used to convert the format of the changed data into a business report if the data type of the changed data is master data.

[0171] In an optional example, the generation unit 503 is further configured to determine the format information corresponding to the changed data, which is the format information of the business report, based on a preset association relationship; wherein the preset association relationship represents the association relationship between different data types and format information.

[0172] In an optional example, the generation unit 503 is further configured to determine the topic to which the changed data belongs based on the identification information located at a preset first position in the changed data; wherein, the topic represents the object that has changed in the business; determine the partition to which the changed data belongs based on the identification information located at a preset second position in the changed data; and store the changed data in a message queue based on the topic and partition to which the changed data belongs.

[0173] In an optional example, the generation unit 503 is further configured to determine the hash value corresponding to the identification information located at a preset second position according to a preset hash algorithm; and to determine the partition corresponding to the hash value as the partition to which the changed data belongs.

[0174] In an optional example, generation unit 503 is further configured to retrieve change data from multiple preset partitions in a message queue; based on the change data in each preset partition, determine the format information of the business report according to the data type of the change data; generate a business report based on the change data and the format information, and store and display the business report.

[0175] In an optional example, generation unit 503 is also configured to cache the changed data in a preset main data cache pool if the data type of the changed data is main data.

[0176] In an optional example, generation unit 503 is further configured to determine the target location of master data in the message queue; wherein the target location is the topic and partition to which the master data belongs in the message queue; retrieve the stored master data from the target location, and cache the stored master data in a preset master data cache pool.

[0177] In an optional example, the generation unit 503 is also used to traverse the stored master data. If the master data currently being traversed meets the preset filtering conditions, the master data currently being traversed is then filtered.

[0178] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0179] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0180] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0181] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0182] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0183] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0184] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0185] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0186] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0187] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0188] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0191] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0193] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0194] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0195] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0196] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0197] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0198] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0199] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.< / record>

Claims

1. A data processing method applied to a data warehouse, characterized in that, The method is applied to an electronic device, which has a data warehouse deployed thereon for data processing and storage; the method includes: Obtain change data for the business and determine the data type of the change data; wherein the change data represents data that has changed in the business; the data type is at least one of master data and operational data, wherein the master data represents the attribute information of the business, and the operational data represents the data in the business's transaction activities; Based on the data type of the changed data, the format information of the business report is determined; wherein, the business report represents the changes in the business, and the format information represents the content that needs to be included in the business report and the format of the content; Based on the changed data and the format information, the business report is generated, and the business report is stored and displayed.

2. The method according to claim 1, characterized in that, Based on the changed data and the format information, the business report is generated, including: If the data type of the changed data is operation data, then the master data corresponding to the changed data is searched from the preset master data cache pool; Based on the changed data and the corresponding master data, and using the format information of the business report, the business report is generated.

3. The method according to claim 2, characterized in that, From the preset master data cache pool, retrieve the master data corresponding to the changed data, including: Obtain the business ID from the changed data; From the preset master data cache pool, find the master data corresponding to the business ID, which is the master data corresponding to the changed data.

4. The method according to claim 2, characterized in that, Based on the changed data and the corresponding master data, and according to the format information of the business report, the business report is generated, including: The changed data and the master data corresponding to the changed data are fused together to obtain the fused data; Based on the format information of the business report, the merged data is converted to a new format to obtain the business report.

5. The method according to claim 1, characterized in that, Based on the changed data and the format information, the business report is generated, including: If the data type of the changed data is primary data, then the changed data is formatted according to the format information of the business report to obtain the business report.

6. The method according to claim 1, characterized in that, Based on the data type of the changed data, determine the format information of the business report, including: Based on a preset association relationship, the format information corresponding to the changed data is determined as the format information of the business report; wherein, the preset association relationship represents the association relationship between different data types and format information.

7. The method according to any one of claims 1-6, characterized in that, A message queue is pre-configured, wherein the message queue is divided into multiple topics, and each topic is further divided into multiple partitions; the method also includes: Based on the identifier information located at a preset first position in the changed data, the topic to which the changed data belongs is determined; wherein, the topic represents the object that has changed in the business. Based on the identifier information located at a preset second position in the changed data, determine the partition to which the changed data belongs; The changed data is stored in the message queue according to the topic and partition to which it belongs.

8. The method according to claim 7, characterized in that, Based on the identifier information located at a preset second position in the changed data, the partition to which the changed data belongs is determined, including: According to a preset hash algorithm, determine the hash value corresponding to the identifier information located at the preset second position; The partition corresponding to the hash value is determined as the partition to which the changed data belongs.

9. The method according to claim 7, characterized in that, Also includes: Retrieve change data from multiple preset partitions from the message queue; Based on the changed data in each preset partition, the process of determining the format information of the business report according to the data type of the changed data is executed. Based on the changed data and the format information, the business report is generated, and the business report is stored and displayed.

10. The method according to claim 7, characterized in that, Also includes: If the data type of the changed data is primary data, then the changed data is cached in a preset primary data cache pool.

11. The method according to claim 7, characterized in that, The changed data is cached in a preset main data cache pool, including... Determine the target location of the master data in the message queue; wherein, the target location is the topic and partition to which the master data belongs in the message queue; The stored master data is retrieved from the target location and cached in the preset master data cache pool.

12. The method according to claim 11, characterized in that, Also includes: Traverse the stored master data. If the master data being traversed meets the preset filtering conditions, then the master data being traversed will be filtered.

13. A data processing apparatus for use in a data warehouse, characterized in that, include: The first determining unit is used to acquire change data of the business and determine the data type of the change data; wherein the change data represents data that has changed in the business; the data type is at least one of master data and operational data, wherein the master data represents the attribute information of the business, and the operational data represents the data in the transaction activities of the business; The second determining unit is used to determine the format information of the business report based on the data type of the changed data; wherein the business report represents the changes in the business, and the format information represents the content that needs to be included in the business report and the format of the content; The generation unit is used to generate the business report based on the change data and the format information, and to store and display the business report.

14. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-12.

Images