Data warehouse model simplification and data analysis method and device, equipment and medium

Abstract

The invention relates to the technical field of big data and the field of financial science and technology, and discloses a data warehouse model simplification and data analysis method, which comprises the following steps: initializing a data warehouse model, and determining to-be-processed data in a time dimension and a code value dimension; converting the to-be-processed data of the time dimension into a time bitmap; converting the to-be-processed data of the code value dimension into a code value bitmap; performing combination processing on the time bitmap and the code value bitmap to generate a bitmap column for recording and counting index information in a service scene; and storing the bitmap column into a field of the data warehouse model, and performing data query and analysis based on the bitmap column in the data warehouse model. The structure of the data warehouse model is simplified through bitmap conversion, redundant data storage of time and code value dimensions is reduced, a data link is effectively shortened, and complexity and maintenance cost of the model are reduced. In the statistics and analysis links, the use of the bitmap accelerates the data query and change identification, and improves the operation efficiency of the data warehouse.

Description

Technical Field

[0001] The present invention relates to the fields of big data technology and financial technology, and in particular to a data warehouse model simplification and data analysis method, device, equipment and storage medium. Background Art

[0002] In the financial field, data, as a core production factor, has become an important resource for enterprises to enhance their competitiveness and achieve data-driven decision-making. In order to maximize the value of data, many financial companies have built a data middle platform based on their own business forms, aiming to enable business processes through efficient data management and analysis and complete data-driven digital transformation. The structure of the data middle platform usually consists of four main parts: basic technology, data governance system, data warehouse, and data application layer. However, although the construction of the data middle platform has great potential, enterprises face many challenges in the process of building the data warehouse model, which makes it difficult for data to effectively empower the business and causes high governance costs.

[0003] First, the core task of a financial enterprise's data warehouse is to build a data warehouse model system through a reasonable modeling method to ensure that the model has high scalability, high reusability and low maintenance cost. However, in reality, model construction often has a "chimney-style" development, that is, each team develops data warehouse models in isolation, lacking unified design and planning. This approach not only leads to repeated construction of data warehouse models, but also makes the model design unreasonable and cannot fully meet the flexibility and scalability requirements of the business.

[0004] Secondly, due to the lack of unified data governance and standards, the construction of data warehouse models in financial enterprises is prone to get out of control, resulting in long data links and complex data lineage relationships. When data anomalies occur, developers and operations teams often need to spend a lot of time troubleshooting data problems. This not only increases maintenance costs, but also hinders the efficient circulation and utilization of data to a certain extent, delaying the release of the value of data elements.

[0005] Although the common practice in the industry is to constrain the design and use of data warehouse models through institutional norms and manual review, these measures can only reduce the occurrence of problems to a certain extent, and do not fundamentally solve the irrationality of model design. Models that lack underlying design optimization will gradually expose more problems in long-term operation, resulting in poor business empowerment effects and high data governance costs. Summary of the invention

[0006] The main purpose of the present invention is to provide a data warehouse model simplification and data analysis method, device, equipment and storage medium, aiming to solve the technical problems in the prior art that the data warehouse model design is complex, resulting in data redundancy and complex data links, increasing maintenance costs and reducing data utilization efficiency.

[0007] To achieve the above object, the present invention provides a data warehouse model simplification and data analysis method, comprising:

[0008] Initialize the data warehouse model and determine the data to be processed in the time dimension and the code value dimension in the data warehouse model;

[0009] Converting the data to be processed in the time dimension into a time bitmap, wherein the time bitmap is used to identify the validity and invalidity of a time point;

[0010] Converting the to-be-processed data of the code value dimension into a code value bitmap, wherein the code value bitmap is used to identify a change status of the code value;

[0011] The time bitmap and the code value bitmap are combined and processed to generate a bitmap column, where the bitmap column is used to record and count indicator information in the business scenario;

[0012] Storing the bitmap column in a corresponding field of a data warehouse model;

[0013] Data query and data analysis are performed based on the bitmap columns in the data warehouse model.

[0014] Furthermore, to achieve the above-mentioned purpose, the present invention provides a data warehouse model simplification and data analysis device, comprising:

[0015] A data initialization module is used to initialize the data warehouse model and determine the data to be processed in the time dimension and the code value dimension in the data warehouse model;

[0016] A time dimension conversion module, used to convert the to-be-processed data in the time dimension into a time bitmap, wherein the time bitmap is used to identify the validity and invalidity of a time point;

[0017] A code value dimension conversion module, used to convert the to-be-processed data of the code value dimension into a code value bitmap, wherein the code value bitmap is used to identify the change status of the code value;

[0018] A bitmap column generation module, used to combine the time bitmap and the code value bitmap to generate a bitmap column, wherein the bitmap column is used to record and count indicator information in a business scenario;

[0019] A data storage module, used for storing the bitmap column into a corresponding field of a data warehouse model;

[0020] The data query and analysis module is used to perform data query and data analysis based on the bitmap columns in the data warehouse model.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data warehouse model simplification and data analysis program stored in the memory and executable on the processor, and the data warehouse model simplification and data analysis program, when executed by the processor, implements the steps of the data warehouse model simplification and data analysis method as described above.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data warehouse model simplification and data analysis program is stored. When the data warehouse model simplification and data analysis program is executed by a processor, the steps of the data warehouse model simplification and data analysis method as described above are implemented.

[0023] Beneficial effects: The present invention relates to the fields of big data technology and financial technology, and discloses a data warehouse model simplification and data analysis method, which initializes the data warehouse model and determines the data to be processed in the time dimension and the code value dimension; converts the data to be processed in the time dimension into a time bitmap for identifying the validity of the time point; converts the data to be processed in the code value dimension into a code value bitmap for identifying the change status of the code value; combines the time bitmap and the code value bitmap to generate a bitmap column for recording and counting indicator information in business scenarios; stores the bitmap column in the field of the data warehouse model, and performs data query and analysis based on the bitmap column in the data warehouse model. The present invention simplifies the structure of the data warehouse model through bitmap conversion, reduces redundant data storage in the time and code value dimensions, effectively shortens the data link, and reduces the complexity and maintenance cost of the model. In the statistical and analysis links, the use of bitmaps accelerates data query and change identification, and improves the operating efficiency of the data warehouse. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 A schematic diagram of an application environment of a data warehouse model simplification and data analysis method in an embodiment of the present invention;

[0026] Figure 2 It is a flow chart of an embodiment of a method for simplifying a data warehouse model and analyzing data according to the present invention;

[0027] Figure 3 It is a functional module diagram of a preferred embodiment of the data warehouse model simplification and data analysis device of the present invention;

[0028] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0029] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0031] The data warehouse model simplification and data analysis method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can initialize the data warehouse model through the user end, and determine the data to be processed in the time dimension and the code value dimension; convert the data to be processed in the time dimension into a time bitmap for identifying the validity of the time point; convert the data to be processed in the code value dimension into a code value bitmap for identifying the change status of the code value; combine the time bitmap and the code value bitmap to generate a bitmap column for recording and counting the indicator information in the business scenario; store the bitmap column in the field of the data warehouse model, and perform data query and analysis based on the bitmap column in the data warehouse model. The present invention simplifies the structure of the data warehouse model through bitmap conversion, reduces the redundant data storage of the time and code value dimensions, effectively shortens the data link, and reduces the complexity and maintenance cost of the model. In the statistical and analysis links, the use of bitmaps accelerates data query and change identification, and improves the operating efficiency of the data warehouse. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a data warehouse model simplification and data analysis method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0033] like Figure 2 As shown, the data warehouse model simplification and data analysis method proposed in the present invention includes the following steps:

[0034] S10, initializing the data warehouse model, and determining the data to be processed in the time dimension and the code value dimension in the data warehouse model;

[0035] In this embodiment, initializing the data warehouse model refers to building a data warehouse structure for storing, processing and analyzing large-scale business data to support enterprise business analysis. The data warehouse model usually includes fact tables, dimension tables, etc., which are used to store various types of business-related data. The initialization of the data warehouse model is a basic step, which aims to provide storage space and structured data support for subsequent data processing.

[0036] The process of initializing the data warehouse model can be achieved through database management tools (such as MySQL, PostgreSQL, etc.) or data modeling tools (such as Erwin). First, design the logical structure of the warehouse, define the basic architecture such as data tables, columns, indexes, etc., and ensure that the time dimension, code value dimension and other business data can be stored correctly. In the process of creating database tables, you can pre-define the primary key, foreign key and index structure to optimize subsequent query and data processing operations.

[0037] The time dimension is a very important dimension in the data warehouse, which is used to store records related to the time when business events occur. Determining the data to be processed in the time dimension means identifying the time points or time ranges that need to be analyzed in the business, and loading the data at these time points into the data warehouse model. Usually, these time data are used for subsequent statistical analysis (such as time series analysis, trend analysis, etc.).

[0038] In the data warehouse model, the time dimension is usually stored in date, timestamp or other time formats. When determining the data to be processed in the time dimension, the system needs to extract the time information of business events from the original data source or historical database through SQL query. For example, the timestamps of customer behavior data can be extracted through SQL statements and mapped to the time dimension. At the same time, a unique identifier is assigned to each time point in the model for subsequent bitmap conversion operations.

[0039] The code value dimension is used to record information related to classification and coding in the business (such as organization code, agent code, product classification, etc.). Determining the data to be processed in the code value dimension means extracting these classification or coding data from the business and using them for subsequent data processing and analysis. These code value dimensions are usually used for classification statistics or status change tracking in analysis.

[0040] Through data integration tools or SQL queries, extract organization codes, customer codes or product classification information from business systems such as CRM and ERP. These code value data need to be cleaned and standardized to ensure data consistency and integrity. After that, the key fields in these code value dimensions are loaded into the data warehouse, usually stored in the form of integers or strings for subsequent analysis operations and bitmap conversion. Coded data can be indexed to speed up retrieval and improve system performance.

[0041] Example: In the financial field, the time dimension in the data warehouse model can be used to record the time when customer behavior occurs (such as loan application, repayment, etc.). The code value dimension can store customer classification information (such as customer risk level, account status, etc.). For example, a financial institution needs to analyze the risk change trend of customers within a certain period of time, or count the sales performance of a certain product in different time periods.

[0042] By extracting and preparing the data to be processed in the time dimension and code value dimension in the data warehouse model, the system can effectively integrate scattered business data and avoid repeated modeling and data redundancy. The standardized processing of the time dimension ensures the accuracy of subsequent data analysis, and the reasonable loading of the code value dimension can optimize the efficiency of subsequent data query and classification statistics, thereby greatly reducing the subsequent maintenance and processing costs.

[0043] S20, converting the data to be processed in the time dimension into a time bitmap, where the time bitmap is used to identify the validity and invalidity of a time point;

[0044] In this embodiment, converting the to-be-processed data of the time dimension into a time bitmap means converting each data point in the time dimension into a binary bit to form a bitmap. Each bit in the bitmap represents a specific time point, and 0 and 1 are used to indicate whether the time point is valid. For example, 1 in the time bitmap can indicate that an event occurs at the time point, and 0 indicates that no event occurs or is invalid at the time point.

[0045] First, the time dimension data to be processed can be extracted from the data warehouse, which is usually stored in date or timestamp format. The extraction method can be completed through SQL query statements or predefined time dimension tables.

[0046] Through program logic (such as Python, Java and other programming languages), traverse the data at each time point. According to business rules, set the qualified time point data to 1, and other time points to 0. For example, if the business rule is to determine whether the customer pays the premium on time, the time point in the payment record can be set to 1, and the time point of non-payment can be set to 0.

[0047] The generated bitmap can be stored as a binary file or loaded into memory, depending on whether subsequent operations require frequent queries or statistical analysis. The database can store the bitmap in a field to optimize query performance.

[0048] The core function of the time bitmap is to distinguish the validity and invalidity of a time point. This distinction is crucial for efficient data analysis. For example, in the insurance business, the time bitmap can quickly identify the customer's payment history, 1 represents a valid time point when payment is made on time, and 0 represents an invalid time point when payment is not made.

[0049] When designing a time bitmap, you need to set rules based on business needs. For example, in insurance renewal, each payment record of a customer will generate a time point, and the validity of the time point will be marked based on whether the payment has occurred. During the analysis process, the system can quickly read the bitmap and perform statistics and analysis based on the 0s and 1s in the bitmap. For example, by counting all time points that are 1, you can calculate the number of payment periods for a customer or the transaction frequency within a certain period of time.

[0050] Example description: In the premium payment scenario in the financial field, the system needs to track each premium payment made by the customer. The traditional method requires recording a piece of data for each payment, but through the time bitmap method, the system only needs to record whether the customer has completed the payment in each payment period. Assuming that the customer's payment period is 10 years, the traditional storage method needs to record 10 payment records, while through the bitmap method, only a bitmap with a length of 10 is required. Each payment is 1, and non-payment is 0. The system can quickly determine the customer's payment status and calculate the cumulative number of payment periods. This method not only saves a lot of storage space, but also improves the efficiency of query and analysis.

[0051] By converting the data to be processed in the time dimension into a time bitmap, the redundant storage and query time of the data are significantly reduced. In conventional time series data processing, the traditional method needs to store complete information at each time point, resulting in data expansion. Through the bitmap method, the system only needs to use binary bits to represent the validity of the time point, which greatly reduces the storage space. At the same time, through the efficient processing method of the bitmap, the query and statistics of the data can be accelerated, and the performance of data analysis and business response efficiency can be improved.

[0052] S30, converting the to-be-processed data of the code value dimension into a code value bitmap, where the code value bitmap is used to identify a change status of the code value;

[0053] In this embodiment, the code value dimension represents a specific code or classification related to the business (such as customer status, product code, organization code, etc.). By converting the data to be processed in the code value dimension into a code value bitmap, it is possible to effectively indicate whether these codes have changed. Each binary bit indicates whether the code value has changed at a certain point in time, 1 indicates a change, and 0 indicates no change.

[0054] Extract historical data of the code value dimension from the database or business system to determine the code value that needs to be monitored. The data of the code value dimension may involve frequently changing elements such as customer status and organizational changes.

[0055] By traversing these data, determine whether a change in status or code has occurred based on the time dimension or other business requirements. For each time period or status change record, generate a binary bit mark, 1 represents a change, and 0 represents no change. For example, if the customer's institution code changes, the bitmap bit corresponding to that time point is set to 1, and if there is no change, it is set to 0.

[0056] The generated bitmap can be stored in the database as an auxiliary tool for quickly retrieving customer change status or other business information. Bitmap storage can be optimized to binary or compressed form to reduce space usage.

[0057] The function of the code value bitmap is to quickly identify whether a certain business code has changed. For example, whether a certain institution code has changed at a certain point in time, whether a certain agent code has changed, etc. Through the code value bitmap, the system can effectively track the change history of business entities for subsequent statistical analysis and query operations.

[0058] According to specific business rules, define the criteria for whether the code value has changed. When the system traverses business data, it can identify whether a bit in the code value bitmap needs to be updated based on the change rules.

[0059] Once the code value bitmap is generated, the system can quickly query based on the change status. For example, you can query the historical change records of a customer to identify the time points at which changes occurred, or count the number of changes made by the agent within a certain period of time. This process greatly improves the speed of business processing.

[0060] For example, when the system is initialized, the basic framework of the data warehouse model is defined, including setting the code value dimension to record the coding information of business entities (such as organization code, agent code). The code value dimension stores the status change information of these business entities at different time points. The system extracts the data to be processed related to the code value dimension from the business system, which usually includes agent code, organization code, customer status code, etc. The purpose of extracting this data is to prepare for the subsequent bitmap conversion.

[0061] Convert the extracted business entity encoding data into a bitmap. By converting the change status of the business entity at each time point into a binary bit, "1" in the bitmap indicates that a change has occurred at that time point, and "0" indicates that no change has occurred. This process can simplify data storage and efficiently represent change information.

[0062] When the change frequency of a business entity exceeds a certain range (for example, 10 changes), the system can accumulate the first 10 bits to indicate that the number of changes exceeds 10. This processing method can achieve unlimited expansion of the change frequency and avoid waste of storage space due to excessive data volume. If an agent or institution has no business activities or changes in a certain period of time, the system can quickly identify such vacant business entities through the bitmap. For example, if an agent has no change operations for several months, the bitmap will be all "0". The system can identify the agent as an agent for an "orphan" policy based on this information to facilitate subsequent management. The generated code value bitmap is stored in the corresponding field of the data warehouse model. The bitmap can be compressed using a compression algorithm (such as run-length encoding, Huffman encoding, etc.) to save storage space.

[0063] In business scenarios, the system can perform quick queries based on the code value bitmap. For example, query the frequency of changes of a certain agent in a certain period of time, calculate the number of changes through bitwise operations; or filter out all business entities that have not changed for further analysis.

[0064] Example: In the financial sector, a customer's agent or institution often changes, and traditionally a new record needs to be recorded for each change. In the insurance business, the agent of a customer's policy may change multiple times. If a customer's agent changes more than 10 times, the traditional method requires recording 10 change data. However, through the code value bitmap, the system only needs to generate a bitmap for the customer, and each change is recorded as 1 in the binary bit, and no change record is 0, which greatly saves storage space and simplifies change queries. For example, if the customer's agent has not changed in a certain year, the bitmap bit for that year is 0, otherwise it is 1. The system can quickly count the number of times each customer's agent has changed through simple bit operations, simplifying the query process and improving efficiency.

[0065] By converting the data to be processed in the code value dimension into a code value bitmap, the recording and query process of business status changes is greatly simplified. In traditional methods, status changes often require recording a large amount of redundant data, and each change requires a new record. Through the bitmap method, the system only needs to store binary bit information to quickly identify change events, greatly reducing the redundancy of data storage. In addition, the efficient representation of the bitmap enables the system to quickly query and count change data, improving data processing efficiency and business responsiveness.

[0066] S40, combining the time bitmap and the code value bitmap to generate a bitmap column, where the bitmap column is used to record and count indicator information in the business scenario;

[0067] In this embodiment, the time bitmap and the code value bitmap represent the status information of the time dimension and the code value dimension respectively. Combination processing refers to combining two independent bitmaps together through bit operations to generate a composite bitmap column containing multiple dimensional information. Through combination processing, the changes in time and business status can be reflected in one bitmap at the same time, simplifying the complexity of data query and statistical analysis.

[0068] Define the combination rules of the time bitmap and the code value bitmap, usually through bit operations (such as AND, OR, XOR) to perform combination processing. For example, for a certain business scenario, it may be necessary for both the time dimension and the code value dimension to be valid at the same time to perform statistics, in which case the AND operation can be used.

[0069] Perform bitwise operations on the time bitmap and the code value bitmap to generate a new bitmap column. Each combined bitmap bit represents business information that has both time and state validity. For example, when using AND operations, the corresponding bit of the bitmap column is 1 only when the time dimension is valid (1) and the code value dimension has changed (1), otherwise it is 0.

[0070] The generated bitmap column can be stored in the corresponding field of the database for subsequent data statistics, query and analysis operations.

[0071] The purpose of generating bitmap columns is to simplify the data model and efficiently record the indicator information in the business scenario through a compressed data structure. The indicator information in the business scenario may involve multiple dimensions, such as the time node of customer behavior, the frequency of business status changes, etc. The generated bitmap columns can quickly provide this information and support efficient statistical analysis.

[0072] The generated bitmap column can be used for indicator statistics in various business scenarios. For example, it can be used to calculate the frequency of effective business behaviors that occur within a certain period of time, or to count whether a customer's status changes within a certain period of time.

[0073] After the generated bitmap column is stored, it can be quickly retrieved through the index structure in the database. For example, when doing business statistics, the system can quickly count the indicator information of "both time validity and code value change" in a certain business scenario, greatly improving query efficiency.

[0074] Example description: In customer account management in the financial field, financial institutions need to track both the status changes and transaction activities of customer accounts. By combining the time bitmap and the code value bitmap, a bitmap column can be generated to simultaneously record whether a customer has transaction behavior and account status changes within a certain period of time. For example, the time bitmap records the customer's monthly transaction behavior, and the code value bitmap records the freezing or unfreezing status of the customer's account. Through combined processing, the system can quickly query whether a customer has both transaction activities and account status changes within a certain period of time. This not only greatly simplifies data storage and query logic, but also improves query performance, making customer behavior analysis more efficient.

[0075] By combining the time bitmap and the code value bitmap to generate a bitmap column, the system can effectively reduce the storage redundancy of dimensional data in business scenarios and accelerate multi-dimensional data queries. In traditional methods, it is necessary to query time and status information separately and merge the two through complex SQL queries. However, through bitmap combination processing, time and status information can be reflected in one bitmap column at the same time, greatly reducing the complexity of data query. The storage method of bitmap columns is simple and takes up less space. At the same time, it supports efficient bitwise operation queries, which improves the speed of data processing and system performance.

[0076] S50, storing the bitmap column into a corresponding field of a data warehouse model;

[0077] In this embodiment, after the bitmap column is generated, it needs to be stored in the corresponding field of the data warehouse model for subsequent query, statistics and analysis operations. The bitmap column is an efficient data representation form and can be used as a specific field type in the data warehouse model to store compressed multi-dimensional business information. By storing the bitmap column in a specific field, the system can quickly access these efficient binary representation data and reduce the storage space occupied.

[0078] Assign a storage field to the bitmap column in the data warehouse model. This field needs to have a predefined data type (such as BLOB (large binary object), VARBINARY (variable binary), etc.) to ensure efficient storage of bitmap data. You can create a corresponding field in the database table to store the results of the bitmap combination processing.

[0079] Since bitmap columns may be large, they are usually compressed before storage to reduce space usage. Common compression algorithms include run-length encoding (RLE) and Huffman coding. In addition, bitmap columns can also be encoded into binary streams or other data formats suitable for storage so that they can be quickly read during queries.

[0080] The compressed and encoded bitmap columns are stored in the corresponding fields through database write operations. The storage operation can be implemented through standard SQL statements, such as using INSERT or UPDATE statements to write the bitmap columns to the specified fields of the data table. In order to ensure the efficiency and security of data storage, the database can set up appropriate storage engines and indexes.

[0081] Example description: In data analysis in the financial field, the system needs to store customer behavior and account status information at the same time. Through the bitmap column storage method, the system can combine the customer's transaction behavior (such as whether a transaction has occurred) and account status (such as whether the account is frozen) within a certain period of time, and store them in the customer table of the data warehouse in the form of a bitmap column. By assigning a BLOB field to the bitmap column in the customer table, these combined data can be efficiently stored and managed. In this way, the system can quickly read the comprehensive behavior and status information of the customer within a certain period of time, greatly improving the query efficiency and reducing the storage of redundant data.

[0082] By storing bitmap columns in the corresponding fields in the data warehouse model, data storage efficiency can be significantly improved and storage space can be reduced. Compared with the traditional multiple record method, bitmap columns, as a compressed and efficient data representation, can store combined information of multiple dimensions in a single field. This not only reduces redundant data, but also improves the speed of data reading and query. In data analysis scenarios, storing bitmap columns enables the system to quickly complete complex queries through simple bitwise operations, reduce the number of database accesses, and improve overall performance.

[0083] S60, performing data query and data analysis based on the bitmap column in the data warehouse model.

[0084] In this embodiment, data query refers to retrieving data related to a specific business scenario from a data warehouse. Through the bitmap column, the system can quickly locate records that meet the conditions. By utilizing the efficiency of the bitmap, the query operation no longer needs to traverse the entire data set, but directly filter out the required data through bitwise operations. For example, using the bitmap column, you can quickly query which time points the business status is valid, or how many times a certain business event has changed its status.

[0085] Set query conditions based on business needs, such as filtering valid records within a certain period of time, or querying changes in specific code value dimensions. Use SQL query statements or API calls to locate bitmap columns in the data warehouse model based on these conditions.

[0086] Perform bitwise operations based on the set query conditions. For example, if you are querying the account status changes of a certain customer, you can use the bitwise AND or OR operations in the bitmap column to quickly locate the records that meet the conditions. Bitwise operations can be implemented using the database's built-in bitwise operation functions or custom program logic.

[0087] Once the bitwise operation is completed, the system will return the records that meet the conditions. The result can be all valid records within a specified time period or the change statistics of a specific business dimension.

[0088] Data analysis refers to further processing and statistics of query results to generate indicators and insights required by the business. Through the combination and bitwise operation of bitmap columns, the system can quickly perform complex analysis operations such as time series analysis, trend analysis, and aggregate statistics. For example, based on bitmap columns, you can quickly calculate the number of customer payments, calculate the frequency of agent changes, or generate a trend chart for a certain business dimension.

[0089] Based on the query results, set analysis conditions, such as cumulatively counting the number of customer payments, calculating the frequency of changes in customer account status, etc. At this point, the system needs to further analyze and process the results in the bitmap column.

[0090] Based on the query results of the bitmap column, the system performs predefined analysis operations. For example, by counting the number of bits marked as "1" in the bitmap column, the cumulative number of a customer's behavior can be calculated, or by performing a bitwise "OR" operation on the bitmap, the business status changes within a certain period of time can be counted.

[0091] After the analysis is completed, the system can generate an analysis report, providing detailed statistical information to support business decisions or provide further operational suggestions. The analysis results can be in the form of reports, charts, or exported to CSV, Excel and other formats.

[0092] Example description: In the financial field, customers' transaction behavior and account status are key analysis objects. Through the bitmap column in the data warehouse, financial institutions can quickly query a customer's transaction behavior in a specific time period and generate detailed trend reports through data analysis. For example, financial institutions can use the time bitmap to query customers' monthly transaction behavior and use the code value bitmap to query customers' account status changes. Based on the bitwise operation of the bitmap column, the system can quickly filter out whether the customer has transaction behavior in each month and calculate the transaction frequency. In addition, through time series analysis, the system can generate a trend analysis chart of customer transaction behavior to help institutions predict customers' future transaction activity. This query and analysis method based on bitmap columns not only speeds up the query speed, but also improves the accuracy and efficiency of data analysis.

[0093] Through data query and data analysis operations based on bitmap columns, the system can greatly improve the efficiency of data processing. Bitmap columns simplify the traditional multi-dimensional data screening process through bitwise operations, reduce the complexity of query and analysis, and improve response speed. At the same time, the compression and efficient representation of bitmap columns can significantly reduce data storage occupancy and improve data processing performance. Ultimately, data query and analysis results can be directly applied to business decisions, helping enterprises quickly gain business insights and improve operational efficiency.

[0094] The present invention relates to the fields of big data technology and financial technology, and discloses a data warehouse model simplification and data analysis method, which initializes the data warehouse model and determines the data to be processed in the time dimension and the code value dimension; converts the data to be processed in the time dimension into a time bitmap for identifying the validity of the time point; converts the data to be processed in the code value dimension into a code value bitmap for identifying the change status of the code value; combines the time bitmap and the code value bitmap to generate a bitmap column for recording and counting indicator information in a business scenario; stores the bitmap column in a field of the data warehouse model, and performs data query and analysis based on the bitmap column in the data warehouse model. The present invention simplifies the structure of the data warehouse model through bitmap conversion, reduces redundant data storage in the time and code value dimensions, effectively shortens the data link, and reduces the complexity and maintenance cost of the model. In the statistical and analysis links, the use of bitmaps accelerates data query and change identification, and improves the operating efficiency of the data warehouse.

[0095] In one embodiment, the above S10 includes:

[0096] S101, constructing a basic data warehouse model framework, wherein the basic data warehouse model framework includes a time dimension and a code value dimension, wherein the time dimension is used to record time point information of business data, and the code value dimension is used to record coding information of business entities;

[0097] S102, determining time point data to be processed in the time dimension, wherein the time point data is used for time bitmap conversion;

[0098] S103, determining code value change information that needs to be recorded in the code value dimension, where the code value change information is used for code value bitmap conversion.

[0099] In this embodiment, building a basic data warehouse model framework refers to designing and building a data warehouse infrastructure for storing and managing large amounts of business data. The framework needs to support multi-dimensional data storage and analysis, and common dimensions include time dimension and code value dimension. The time dimension is used to store the time information of business events, such as transaction occurrence time, state change time, etc.; the code value dimension is used to store the classification information of business entities, such as customer code, product code or organization code.

[0100] In actual operation, the basic model framework of the data warehouse is usually designed through database modeling tools (such as Erwin or PowerDesigner). First, define the table structure in the data warehouse, including the time dimension table and the code value dimension table, and specify their fields, indexes, and foreign key relationships. In this way, business data can be stored in a structured manner and can support subsequent query, analysis, and bitmap conversion operations.

[0101] The time dimension records the time point information related to the business, such as transaction time, event occurrence time, etc. In the data warehouse, the processed time point data is usually used for subsequent time dimension analysis. By filtering or querying the time point data to be processed, the system can perform time-based statistics and trend analysis. The time point data to be processed is the data that will participate in the time bitmap conversion in the future.

[0102] The extraction of time point data can be achieved through SQL queries or data integration tools. For example, relevant time point data can be extracted from a transaction system or historical database through a timestamp or date field and loaded into a time dimension table. The determined time point data needs to be consistent with business rules to ensure that each time point can participate in bitmap conversion and be used for time series analysis or trend prediction in subsequent analysis.

[0103] The code value dimension is used to record the coding information of business entities, such as customer codes, organization codes, or product codes. Code value change information indicates the state change or other important operation changes of business entities. The purpose of determining code value change information is to track and record these changes in a bitmap manner. Code value change information will participate in the subsequent code value bitmap conversion to identify the frequency and impact of the change.

[0104] Extract code value information from the business system through data query, such as obtaining the change history of customers or institutions through SQL query. Change information usually needs to be cleaned and standardized to ensure data consistency. The confirmed code value change information will be loaded into the code value dimension table, and the time and specific code of each change will be recorded through bitmap conversion to facilitate subsequent statistics and analysis.

[0105] By constructing a basic data warehouse model framework, this embodiment enables the system to effectively store and manage multi-dimensional business data. The separation design of the time dimension and the code value dimension can simplify the data model and improve the efficiency of query and analysis. By determining the time point data and code value change information, the system can achieve efficient data tracking, change management and trend analysis. The standardized design of the data warehouse model can also reduce subsequent maintenance costs and support data analysis requirements with high scalability and reusability.

[0106] In one embodiment, the above S20 includes:

[0107] S201, extracting the data to be processed in the time dimension, where the data to be processed is a time point record of the business data;

[0108] S202: Convert the time point information into a time bitmap, where the time bitmap is used to represent the state of each time point, including a state indicating that the time point is valid and a state indicating that the time point is invalid.

[0109] In this embodiment, the data to be processed in the time dimension refers to the key information related to time in the business process, which usually exists in the form of time point records (such as transaction time, event occurrence time, etc.). Extracting the data to be processed in the time dimension is the first step in screening and processing these records, aiming to obtain all time point data that need to be further analyzed or processed.

[0110] In actual operation, time dimension data can be extracted from business systems through SQL queries or data integration tools. Common data sources include transaction databases, customer management systems, etc. The extracted data is usually a time point record in the form of a date or timestamp. The system filters out the time point data that needs to be processed through specific query conditions (such as by business category, time range, etc.) and loads it into the time dimension table in memory or data warehouse.

[0111] Converting time point information into a time bitmap means representing each time point with a binary bit, that is, using "1" and "0" to identify the validity or invalidity of the time point. The time bitmap is an efficient data structure that can be used to quickly represent the information status of a large number of time points.

[0112] The conversion process of the bitmap is usually implemented through program logic. For each extracted time point, its status is judged according to its business rules (such as whether an event occurs or a task is completed). If the business event corresponding to the time point occurs, the bitmap bit is marked as "1", indicating that the time point is valid; otherwise, it is marked as "0", indicating an invalid time point. The generation of the bitmap can be achieved through bit operations in programming languages ​​such as Python and Java, and the results are stored in binary format or loaded into memory for subsequent query and statistical analysis.

[0113] The time bitmap uses binary bits to represent the validity or invalidity status of each time point. Through this efficient bitmap representation, the system can quickly perform query, screening, statistics and analysis operations. For example, 1 indicates that the time point is valid (such as the customer pays on time), and 0 indicates that the time point is invalid (such as the customer does not pay on time).

[0114] In application scenarios, the time bitmap can be used for quick statistics and analysis through bitwise operations. The system can query all valid time points within a certain time period based on the bitmap, or count how many valid events occurred within a certain time period. When the bitmap is used for query and statistical operations, the program directly filters the time points through bitwise operations without traversing each record one by one, which greatly improves the query efficiency.

[0115] By converting the time point information into a time bitmap, the system can greatly improve the processing and analysis efficiency of the time dimension data. The use of the time bitmap not only simplifies the representation of the time point, but also speeds up the query and statistical operations within the time period through bitwise operations. This efficient bitmap representation can significantly reduce the storage space of the data and avoid the problem of recording redundant information in the traditional table method. Ultimately, the system can quickly complete the judgment, statistics and trend analysis of the time point status, improving the performance of data processing.

[0116] In one embodiment, the above S30 includes:

[0117] S301, extracting the data to be processed in the code value dimension, where the data to be processed is the code record of the business entity;

[0118] S302, converting the coded data of the business entity into a code value bitmap, wherein the code value bitmap is used to indicate a change status of a code value, wherein the change status includes a status indicating that the code value has changed and a status indicating that the code value has not changed.

[0119] In this embodiment, the data to be processed in the code value dimension refers to the code records of the business entity (such as customer code, organization code, product code, etc.), which are usually used to represent the status or classification of the business entity. Extracting the data of the code value dimension is the premise of bitmap conversion. The system identifies the changes of the business entity by extracting these code records from the database or business system.

[0120] Extracting code value dimension data is usually done through database query or API call. For example, the system can extract the status code record of customers or the change record of institutions within a certain period of time from the business system through SQL query. The extracted data will be used for subsequent bitmap conversion operations and is usually stored in memory or temporary tables in the form of codes or identifiers for bitmap conversion and further processing.

[0121] Converting the coded data of a business entity into a code value bitmap means converting the coded state change of each business entity into binary bits. Each bit is used to indicate whether the code of the business entity has changed at a certain point in time. In this way, the change state of the business entity can be efficiently represented and provide a basis for subsequent analysis.

[0122] The system traverses the extracted business entity code records and determines whether each code has changed at a certain point in time according to business rules. If the business entity code has changed at a certain point in time, the corresponding binary bit in the code value bitmap is set to "1" to indicate a change; if no change has occurred, it is set to "0". This process can be achieved through bit operations, and the converted code value bitmap is stored in the database or memory for subsequent statistical analysis.

[0123] The code value bitmap uses binary bits to represent the change status of the business entity code. Through the structure of the bitmap, the system can quickly identify the time points at which the business entity status changes and the time points at which the status does not change. This bitmap representation method can greatly simplify complex status change queries and improve query and statistical efficiency.

[0124] The code value bitmap can be queried and analyzed through bit operations in the database. For example, when the system needs to query the coding changes of a business entity over a period of time, it can quickly filter out the time points when the changes occurred through bit operations. By accumulating "1" in the code value bitmap, the system can calculate the number of times the business entity has changed in a certain period of time, or identify the change status at a specific time point.

[0125] This embodiment converts the coded data of the business entity into a code value bitmap, so that the system can efficiently manage and track the state changes of the business entity. The bitmap representation method not only reduces the redundancy of data storage, but also speeds up the query and statistics of state changes. Through bitwise operations, the system can easily identify the state changes of business entities within a certain period of time, reducing the complexity of one-by-one retrieval in traditional table queries, and significantly improving the query efficiency and analysis performance of the system.

[0126] In one embodiment, the above S40 includes:

[0127] S401, aligning data in the time bitmap and the code value bitmap based on the same timestamp or service identifier;

[0128] S402, determining a combination rule of the time bitmap and the code value bitmap according to the requirements of the business scenario;

[0129] S403: According to the combination rule, the time bitmap and the code value bitmap are combined to generate a bitmap column, where the bitmap column is used to reflect the validity of the time point and the change status of the code value.

[0130] In this embodiment, the purpose of aligning the time bitmap and the code value bitmap is to synchronize the data of the two dimensions to ensure that they are combined at the same time point or business identifier (such as customer ID, transaction ID, etc.). The time bitmap indicates the validity of a time point, and the code value bitmap indicates the state change of a business entity. By aligning their timestamps or identifiers, accurate combination processing can be ensured on the same time period or business entity.

[0131] The alignment process can be performed through program logic. First, the system checks the timestamps or business identifiers in each time bitmap and code value bitmap to ensure that they are consistent. If inconsistent time points or identifiers are found, they can be adjusted by filling 0 or deleting redundant data to align the data in the time dimension and code value dimension in structure. The aligned data structure can then be further used for combination operations.

[0132] The combination rule refers to how to combine the time bitmap and the code value bitmap to reflect the relationship between the two. The combination rules may be different in different business scenarios. For example, in some scenarios, both the validity of the time point and the change of the code value may be required to be counted; in other scenarios, as long as the time point is valid or the code value changes, it can be counted. Therefore, the system needs to determine the combination logic of the bitmap according to the business scenario (such as bitwise "AND" operation, bitwise "OR" operation, etc.).

[0133] Select the appropriate bit operation logic according to business needs. For example, in an insurance scenario, if you need to determine whether the customer's payment time and agent change status meet a certain condition at the same time, you can use the bitwise "AND" operation to combine the two bitmaps; if you only need to meet one of the conditions, you can use the bitwise "OR" operation. After determining the combination rule, the system will perform bit-by-bit operations on the time bitmap and code value bitmap according to this rule to generate the final bitmap column.

[0134] The combined bitmap column generates a result that reflects multi-dimensional information by combining the time bitmap and the code value bitmap according to a predetermined rule. The bitmap column can simultaneously represent the validity of a time point and the change status of the code value. Through the combined operation, the system can effectively simplify the storage and query of multi-dimensional data, making subsequent statistical and analytical operations more efficient.

[0135] The system performs bitwise operations (such as bitwise AND or bitwise OR) according to the previously set combination rules, operates the time bitmap and code value bitmap bit by bit, and generates a combined bitmap column. The binary bits in each combined bitmap column reflect both the validity of the time point and the change status of the code value. The generated bitmap column can be stored in the data warehouse model for subsequent business scenario statistics and analysis.

[0136] In this embodiment, by combining the time bitmap and the code value bitmap for processing, the system can effectively simplify the storage and management of multi-dimensional data. The bitmap column not only saves storage space, but also significantly improves the efficiency of query and analysis through bitwise operations. Traditional table storage methods require complex query operations in multiple dimensions, but through bitmap combination processing, joint queries of time and status can be quickly implemented. Ultimately, this efficient data processing method enables the system to quickly complete multi-dimensional statistical and analysis tasks, greatly improving data processing performance and business response speed.

[0137] In one embodiment, the above S50 includes:

[0138] S501, determining a compression method and a storage format according to the size and sparsity of the bitmap column and the requirements of the business scenario;

[0139] S502, allocating storage fields for the bitmap columns in the data warehouse model, and defining corresponding data types;

[0140] S503, establishing an index structure for the storage field;

[0141] S504, compressing the bitmap according to the compression method and storage format;

[0142] S505, writing the compressed bitmap column into the storage field.

[0143] In this embodiment, the size, sparsity and business scenario requirements of the bitmap column affect its storage efficiency and performance. If the bitmap column is very sparse (i.e., most of the bits are 0), a specific compression algorithm can be used to reduce storage space. According to the business scenario (such as high-frequency query or batch analysis), choosing the appropriate compression method and storage format to optimize the storage and access efficiency of data is a key step.

[0144] The system first evaluates the size and sparsity of the bitmap column. If the sparsity of the bitmap column is high, compression algorithms such as Run-Length Encoding (RLE) can usually be used to reduce storage space. For dense bitmap columns, you can choose to store directly or use light compression. In addition, the requirements in the business scenario (such as high-concurrency queries or large batch processing) also affect the choice of storage format, such as choosing a BLOB (large binary object) or VARBINARY (variable-length binary) type field.

[0145] In order to store the processed bitmap column, the system needs to allocate a suitable storage field for it in the data warehouse model. The data type of the field needs to be selected according to the compressed data structure. Usually, a binary data type such as BLOB or VARBINARY is selected to ensure that the data can be stored efficiently and read quickly later.

[0146] The system can create new fields in the data table through database management tools or SQL commands to store bitmap columns. The field type is selected according to the compression method of the bitmap column, such as VARBINARY type for uncompressed or lightly compressed data, and BLOB type for data with a higher compression rate. When creating a field, the system can define the length or limit of the field to accommodate bitmap columns of different sizes.

[0147] Indexing stored bitmap columns can speed up data query and retrieval operations. The index structure allows the database to quickly locate relevant data when processing queries, especially when performing bitwise operations on bitmap columns, indexes can significantly improve query performance.

[0148] According to the needs of business queries, the system can create appropriate indexes for storage fields. Common index structures include B-tree and hash index. When creating an index, the system needs to consider the query mode, such as giving priority to creating indexes for high-frequency queries to improve data access speed. In addition, index creation can also be optimized based on the sparsity of bitmap columns and data size.

[0149] The compression process is to reduce the space occupied by the bitmap column when it is stored, while maintaining its structural integrity for subsequent query and operation. Different compression methods are suitable for different data types and sparsity. Choosing the right compression algorithm can significantly reduce storage overhead.

[0150] The system processes the bitmap column according to the compression method determined previously. For data with high sparsity, run-length encoding (RLE) is a common compression method that compresses consecutive identical bits into a short representation. For other types of bitmap columns, compression algorithms such as Huffman coding or dictionary encoding can also be used. The compressed bitmap column is re-encoded into a format suitable for storage and is ready to be written to the storage field in the data warehouse model.

[0151] After compression and encoding, the bitmap column will be written to the corresponding field in the data warehouse model. This operation ensures that the data is stored persistently and can be accessed and operated by query and analysis tools when needed.

[0152] The system writes the processed bitmap columns into the storage fields of the data warehouse model through SQL commands or database operation interfaces. Usually, the INSERT or UPDATE statement is used to write the bitmap column data into the allocated BLOB or VARBINARY field. In order to ensure the integrity of the data, exception handling and error checking are required during the writing process to ensure that the data can be successfully stored.

[0153] By compressing the bitmap columns and storing them in the data warehouse model, this embodiment can effectively reduce the storage space occupied by the system and significantly improve the efficiency of data query and analysis. Compression processing can reduce storage costs, and the index structure can accelerate data retrieval, especially in high-frequency query and analysis scenarios. The overall process design ensures a balance between efficient data compression and fast query, providing a flexible and efficient storage solution.

[0154] In one embodiment, the above S60 includes:

[0155] S601, according to the business scenario, set the query condition based on the time point dimension and / or the code value change status dimension;

[0156] S602, using the bitmap column and the predefined index structure, performing data retrieval in the database of the data warehouse model to determine target data that meets the query condition;

[0157] S603: Analyze the target data based on the needs of the business scenario to generate analysis indicator data.

[0158] In this embodiment, the business scenario determines the setting of the query conditions. The time point dimension is used to indicate the time when the business event occurs, and the code value change status dimension is used to track the status changes of business entities (such as customers, institutions, etc.). By combining the time and status dimensions, the system can set complex query conditions to filter the changes of business entities within a specific time period.

[0159] The system selects the time point dimension and / or the code value change status dimension based on the specific business scenario (such as analyzing the customer's behavior or status changes within a certain period of time). Based on the time point dimension, the system can query data within a certain time range, and based on the code value change status dimension, the system can filter out the status changes of business entities. The query conditions finally set can be applied to the bitmap columns in the data warehouse through SQL statements or database query tools.

[0160] The combination of bitmap columns and index structures can speed up data retrieval. Through bitwise operations on bitmap columns, the system can quickly filter data that meets time point and status conditions. Predefined index structures (such as B-trees, hash indexes, etc.) help optimize database query paths and speed up the retrieval process. Especially in a big data environment, indexes can significantly improve query performance.

[0161] The system searches the bitmap column and the index structure to retrieve the target data that meets the conditions in the database of the data warehouse model. The bitmap column quickly matches the query conditions through bit operations (such as bitwise "AND" or "OR" operations), while the index structure optimizes the search path to help the system find the target data efficiently. The result is that the records or business entity information that meet the query conditions are quickly returned.

[0162] Data analysis is the process of further extracting valuable business information based on query results. Analysis of indicator data includes frequency statistics, trend analysis, aggregate statistics, etc., which help business users understand business changes or behavior trends and support business decision-making.

[0163] The system further analyzes the query results based on business needs. For example, it can collect statistics on the transaction behaviors or status change frequencies of qualified customers, or analyze the trend of customer status changes within a certain period of time. The analysis results can generate statistical reports, charts, or be exported as analysis data for subsequent use. The data analysis module usually executes specific analysis algorithms or business logic based on the target data to generate useful business indicators.

[0164] In this embodiment, through data query and analysis based on bitmap columns, the system can quickly locate and filter business entities or records that meet specific conditions, significantly improving the efficiency of data retrieval and analysis. Combined with the index structure, the system can maintain efficient query performance in a large-scale data environment, especially in frequent business scenarios, and can quickly generate analysis results to support business decision-making and management.

[0165] In one embodiment, a data warehouse model simplification and data analysis device is provided, and the data warehouse model simplification and data analysis device corresponds one-to-one to the data warehouse model simplification and data analysis method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the data warehouse model simplification and data analysis device of the present invention. Data initialization module 10, time dimension conversion module 20, code value dimension conversion module 30, bitmap column generation module 40, data storage module 50 and data query and analysis module 60. The functional modules are described in detail as follows:

[0166] The data initialization module 10 is used to initialize the data warehouse model and determine the data to be processed in the time dimension and the code value dimension in the data warehouse model;

[0167] A time dimension conversion module 20, used to convert the data to be processed in the time dimension into a time bitmap, wherein the time bitmap is used to identify the validity and invalidity of a time point;

[0168] A code value dimension conversion module 30, used to convert the to-be-processed data of the code value dimension into a code value bitmap, wherein the code value bitmap is used to identify the change status of the code value;

[0169] A bitmap column generation module 40, used for combining the time bitmap and the code value bitmap to generate a bitmap column, wherein the bitmap column is used to record and count indicator information in a business scenario;

[0170] A data storage module 50, used to store the bitmap column into a corresponding field of a data warehouse model;

[0171] The data query and analysis module 60 is used to perform data query and data analysis based on the bitmap columns in the data warehouse model.

[0172] In one embodiment, the data initialization module 10 is specifically used for:

[0173] Constructing a basic data warehouse model framework, wherein the basic data warehouse model framework includes a time dimension and a code value dimension, wherein the time dimension is used to record time point information of business data, and the code value dimension is used to record coding information of business entities;

[0174] Determining time point data to be processed in the time dimension, wherein the time point data is used for time bitmap conversion;

[0175] The code value change information to be recorded is determined in the code value dimension, and the code value change information is used for code value bitmap conversion.

[0176] In one embodiment, the time dimension conversion module 20 is specifically used for:

[0177] Extracting the data to be processed in the time dimension, where the data to be processed is a time point record of the business data;

[0178] The time point information is converted into a time bitmap, where the time bitmap is used to represent the state of each time point, where the state includes a state representing that the time point is valid and a state representing that the time point is invalid.

[0179] In one embodiment, the code value dimension conversion module 30 is specifically used to:

[0180] Extracting the data to be processed in the code value dimension, where the data to be processed is a coded record of a business entity;

[0181] The coded data of the business entity is converted into a code value bitmap, wherein the code value bitmap is used to indicate a change state of the code value, wherein the change state includes a state indicating that the code value has changed and a state indicating that the code value has not changed.

[0182] In one embodiment, the bitmap column generation module 40 is specifically configured to:

[0183] Aligning data in the time bitmap and the code value bitmap based on the same timestamp or service identifier;

[0184] Determine the combination rule of the time bitmap and the code value bitmap according to the needs of the business scenario;

[0185] According to the combination rule, the time bitmap and the code value bitmap are combined to generate a bitmap column, and the bitmap column is used to reflect the validity of the time point and the change status of the code value.

[0186] In one embodiment, the data storage module 50 is specifically used for:

[0187] Determine the compression method and storage format according to the size, sparsity and business scenario requirements of the bitmap column;

[0188] Allocating storage fields for the bitmap columns in the data warehouse model and defining corresponding data types;

[0189] Establishing an index structure for the storage field;

[0190] According to the compression method and storage format, compressing the bitmap;

[0191] The compressed bitmap column is written into the storage field.

[0192] In one embodiment, the data query and analysis module 60 is specifically used to:

[0193] According to the business scenario, set query conditions based on the time point dimension and / or the code value change status dimension;

[0194] Using the bitmap column and the predefined index structure, data is retrieved in the database of the data warehouse model to determine target data that meets the query condition;

[0195] Based on the needs of the business scenario, the target data is analyzed to generate analysis indicator data.

[0196] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a data warehouse model simplification and data analysis method.

[0197] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a function or step on the user side of a data warehouse model simplification and data analysis method

[0198] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0199] Initialize the data warehouse model and determine the data to be processed in the time dimension and the code value dimension in the data warehouse model;

[0200] Converting the data to be processed in the time dimension into a time bitmap, wherein the time bitmap is used to identify the validity and invalidity of a time point;

[0201] Converting the to-be-processed data of the code value dimension into a code value bitmap, wherein the code value bitmap is used to identify a change status of the code value;

[0202] The time bitmap and the code value bitmap are combined and processed to generate a bitmap column, where the bitmap column is used to record and count indicator information in the business scenario;

[0203] Storing the bitmap column in a corresponding field of a data warehouse model;

[0204] Data query and data analysis are performed based on the bitmap columns in the data warehouse model.

[0205] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0206] Initialize the data warehouse model and determine the data to be processed in the time dimension and the code value dimension in the data warehouse model;

[0207] Converting the data to be processed in the time dimension into a time bitmap, wherein the time bitmap is used to identify the validity and invalidity of a time point;

[0208] Converting the to-be-processed data of the code value dimension into a code value bitmap, wherein the code value bitmap is used to identify a change status of the code value;

[0209] The time bitmap and the code value bitmap are combined and processed to generate a bitmap column, where the bitmap column is used to record and count indicator information in the business scenario;

[0210] Storing the bitmap column in a corresponding field of a data warehouse model;

[0211] Data query and data analysis are performed based on the bitmap columns in the data warehouse model.

[0212] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0213] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0214] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0215] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data warehouse model simplification and data analysis method, characterized in that: The following steps are involved: Initialize the data warehouse model and determine the data to be processed in the time dimension and the code value dimension in the data warehouse model; Converting the data to be processed in the time dimension into a time bitmap, wherein the time bitmap is used to identify the validity and invalidity of a time point; Converting the data to be processed in the code value dimension into a code value bitmap, wherein the code value bitmap is used to identify a change status of the code value; The time bitmap and the code value bitmap are combined and processed to generate a bitmap column, where the bitmap column is used to record and count indicator information in the business scenario; Storing the bitmap column in a corresponding field of a data warehouse model; Data query and data analysis are performed based on the bitmap columns in the data warehouse model.

2. The data warehouse model simplification and data analysis method according to claim 1, characterized in that: Initialize the data warehouse model and determine the data to be processed in the time dimension and code value dimension of the data warehouse model, including: Constructing a basic data warehouse model framework, wherein the basic data warehouse model framework includes a time dimension and a code value dimension, wherein the time dimension is used to record time point information of business data, and the code value dimension is used to record coding information of business entities; Determining time point data to be processed in the time dimension, wherein the time point data is used for time bitmap conversion; The code value change information to be recorded is determined in the code value dimension, and the code value change information is used for code value bitmap conversion.

3. The data warehouse model simplification and data analysis method according to claim 1, characterized in that: Convert the data to be processed in the time dimension into a time bitmap, where the time bitmap is used to identify the validity and invalidity of a time point, including: Extracting the data to be processed in the time dimension, where the data to be processed is a time point record of the business data; The time point information is converted into a time bitmap, where the time bitmap is used to represent the state of each time point, where the state includes a state representing that the time point is valid and a state representing that the time point is invalid.

4. The data warehouse model simplification and data analysis method according to claim 1, characterized in that: Converting the data to be processed in the code value dimension into a code value bitmap, where the code value bitmap is used to identify the change status of the code value, including: Extracting the data to be processed in the code value dimension, where the data to be processed is a coded record of a business entity; The coded data of the business entity is converted into a code value bitmap, wherein the code value bitmap is used to indicate a change state of the code value, wherein the change state includes a state indicating that the code value has changed and a state indicating that the code value has not changed.

5. The data warehouse model simplification and data analysis method as claimed in claim 1, characterized in that: The time bitmap and the code value bitmap are combined and processed to generate a bitmap column, which is used to record and count indicator information in the business scenario, including: Aligning data in the time bitmap and the code value bitmap based on the same timestamp or service identifier; Determine the combination rule of the time bitmap and the code value bitmap according to the needs of the business scenario; According to the combination rule, the time bitmap and the code value bitmap are combined to generate a bitmap column, and the bitmap column is used to reflect the validity of the time point and the change status of the code value.

6. The data warehouse model simplification and data analysis method according to claim 1, characterized in that: Storing the bitmap column in a corresponding field of the data warehouse model includes: Determine the compression method and storage format according to the size, sparsity and business scenario requirements of the bitmap column; Allocating storage fields for the bitmap columns in the data warehouse model and defining corresponding data types; Establishing an index structure for the storage field; According to the compression method and storage format, compressing the bitmap; The compressed bitmap column is written into the storage field.

7. The data warehouse model simplification and data analysis method according to claim 1, characterized in that: Based on the bitmap column in the data warehouse model, data query and data analysis are performed, including: According to the business scenario, set query conditions based on the time point dimension and / or the code value change status dimension; Using the bitmap column and the predefined index structure, data is retrieved in the database of the data warehouse model to determine target data that meets the query condition; Based on the needs of the business scenario, the target data is analyzed to generate analysis indicator data.

8. A data warehouse model simplification and data analysis device, characterized in that: The data warehouse model simplification and data analysis device comprises: A data initialization module is used to initialize the data warehouse model and determine the data to be processed in the time dimension and the code value dimension in the data warehouse model; A time dimension conversion module, used to convert the to-be-processed data in the time dimension into a time bitmap, wherein the time bitmap is used to identify the validity and invalidity of a time point; A code value dimension conversion module, used to convert the to-be-processed data of the code value dimension into a code value bitmap, wherein the code value bitmap is used to identify the change status of the code value; A bitmap column generation module, used for combining the time bitmap and the code value bitmap to generate a bitmap column, wherein the bitmap column is used to record and count indicator information in a business scenario; A data storage module, used for storing the bitmap column into a corresponding field of a data warehouse model; The data query and analysis module is used to perform data query and data analysis based on the bitmap columns in the data warehouse model.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data warehouse model simplification and data analysis program stored in the memory and executable on the processor. When the data warehouse model simplification and data analysis program is executed by the processor, the steps of the data warehouse model simplification and data analysis method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a data warehouse model simplification and data analysis program, which, when executed by a processor, implements the steps of the data warehouse model simplification and data analysis method as described in any one of claims 1 to 7.

Images