Data hierarchical classification method and device

Abstract

The invention relates to a data hierarchical classification method and device. The method comprises the steps of collecting data to enter a data warehouse; loading the collected data into the ODS of the data warehouse; performing data warehouse preprocessing on the data in the ODS; transmitting the processed data to an ODSP (Optical Digital Signal Processor) for processing to obtain service detail data; performing data modeling according to the business detail data, and determining the classification of the business detail data; and performing hierarchical processing on the data after data modeling to obtain a hierarchical result of the business detail data. According to the scheme of the invention, clear data management and flow are displayed, the data processing flow is very flexible, the data quality and consistency are improved, and from the perspective of service-driven data, diversified service requirements are supported, the data access and use efficiency is optimized, and the data autonomy of a service department is enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of data warehouses, and in particular to a method and device for hierarchical classification of data. Background Art

[0002] At present, in modern enterprise management, especially in the steel industry Internet industry, data stratification and classification methods are crucial for data management and decision support. Traditional data management faces problems such as diversified data sources, low data processing efficiency, and low data reuse rate.

[0003] Traditional data warehouses often use batch processing to load and convert data. The specific defects are as follows:

[0004] 1. Complex data model design:

[0005] Traditional data warehouses usually use star or snowflake models to organize data, which need to be carefully designed and optimized to meet different analysis needs. However, as the amount and complexity of data increase, model design becomes more complex and difficult.

[0006] 2. Insufficient ability to process semi-structured and unstructured data:

[0007] Traditional data warehouses are mainly oriented towards structured data, and have limited processing capabilities for semi-structured data, such as JSON (JavaScript Object Notation) and XML (Extensible Markup Language). In the current era of information explosion, this limits the potential of data warehouses in fully utilizing enterprise data.

[0008] 3. Insufficient flexibility and scalability:

[0009] Traditional data warehouses usually use static models and predefined data structures, which are difficult to quickly adapt to changes in business needs and the access of new data sources. This results in data warehouses lacking sufficient flexibility and scalability in the face of rapidly changing business environments.

[0010] 4. Difficult to support real-time analysis and decision-making:

[0011] Due to data update delays and complex data processing processes, traditional data warehouses often have difficulty supporting real-time data analysis and instant decision-making. This becomes a significant limitation in application scenarios that require rapid response and real-time adjustment of strategies. Summary of the invention

[0012] In view of the problems existing in the prior art, the present application provides a data stratification and classification scheme and proposes a new data processing flow to achieve stratification and classification of data.

[0013] According to a first aspect of the present application, a method for hierarchical data classification is provided, characterized by comprising:

[0014] Collect data to be entered into the data warehouse;

[0015] Loading the collected data into the ODS of the data warehouse;

[0016] Preprocessing the data in the ODS for data warehouse;

[0017] The processed data is transmitted to ODSP for processing to obtain business detail data;

[0018] Performing data modeling based on the business detail data to determine the classification of the business detail data; and

[0019] The data after data modeling is layered to obtain the layered results of the business detail data.

[0020] According to a second aspect of the present application, a device for hierarchical classification of data is provided, characterized in that it includes:

[0021] The collection module is used to collect data to be entered into the data warehouse;

[0022] A loading module, used for loading the collected data into the ODS of the data warehouse;

[0023] A preprocessing module, used for preprocessing the data in the ODS for data warehouse;

[0024] The first acquisition module is used to transmit the processed data to the ODSP for processing to obtain business detail data;

[0025] A determination module, configured to perform data modeling based on the business detail data and determine the classification of the business detail data; and

[0026] The second acquisition module is used to perform hierarchical processing on the data after data modeling to obtain the hierarchical results of the business detail data.

[0027] According to a third aspect of the present application, there is provided an electronic device, comprising:

[0028] Processor; and

[0029] A memory storing computer instructions, which, when executed by the processor, causes the processor to execute the method described in the first aspect.

[0030] According to a fourth aspect of the present application, a non-transitory computer storage medium is provided, storing a computer program, which, when executed by multiple processors, enables the processors to execute the method described in the first aspect.

[0031] The new data processing flow provided by the data hierarchical classification scheme provided in this application shows clear data management and process, the data processing flow is very flexible, the data quality and consistency are improved, and from a business-driven data perspective, it supports diversified business needs, optimizes data access and usage efficiency, and enhances the data autonomy of business departments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by the present application.

[0033] Figure 1 is a schematic diagram of a system for executing a data hierarchical classification scheme according to an embodiment of the present application.

[0034] Figure 2 The figure is a flowchart of a method for hierarchical data classification according to an embodiment of the present application.

[0035] Figure 3 The present invention is a flowchart of a method for hierarchical data classification according to another embodiment of the present application.

[0036] Figure 4 It is a schematic diagram of a device for hierarchical classification of data according to an embodiment of the present application.

[0037] Figure 5 It is a schematic diagram of a device for hierarchical classification of data according to another embodiment of the present application.

[0038] Figure 6 It is a structural diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0040] Figure 1Schematic diagram of a system for executing a data stratification and classification scheme according to an embodiment of the present application. According to a specific embodiment, the system adopts a new open source MPP (Massively Parallel Processing) architecture, a CBO (Cost-Based Optimization) optimizer, and a columnar storage engine analytical database StarRocks that can be updated in real time. Figure 1 As shown, the source data is the data to be entered into the data warehouse, including offline data and real-time data. According to a specific embodiment, the data to be entered into the data warehouse includes data generated by the steel Internet industry. For offline data, batch processing is adopted, and offline data collection tools such as ETL (Extract, Transform, Load) tools are used to collect offline data. The open source tools corresponding to the ETL tools may include sqoop and datax. After configuring the IP address, database name, account number and other information of the business database, the collected data is loaded into the offline data warehouse and enters the operational data layer, including ODS (Operational Data Store) and ODSP (Operational Data Store Processing), and business mirror data is generated, which is consistent with the data table content of the source end collected. For real-time data, the collection method of log synchronization or embedded point activation is adopted, and the collected real-time data is transmitted to a real-time computing engine such as Flink for real-time processing through a message middleware such as kafka to obtain real-time processing results. Among them, for the data landing part, a processing method similar to that of offline data is adopted, and finally stored in the offline data warehouse.

[0041] Next, the data in the ODS is preprocessed for the data warehouse, including one or more of data cleaning, data verification, missing value processing, and format error processing. The processed data is transferred to the ODSP for processing to obtain business detail data as the data basis for data application stratification (ADS). Data modeling is performed based on the business detail data to determine the classification of the business detail data. According to a specific embodiment, dimension tables and fact tables of the data warehouse are formed in the process of data modeling based on the business detail data, wherein the classification of the business detail data is recorded in the dimension table and / or the fact table. Subsequently, the data after data modeling is layered to obtain the layered results of the business detail data.

[0042] According to some embodiments, for data classification, in the process of forming dimension tables and fact tables, the classification of business detail data is recorded or identified in the dimension tables and / or the fact tables according to the classification of source data. According to a specific implementation, data classification includes member data classification, transaction data classification, logistics data classification, financial data classification, etc. Among them, for member data classification, member information is collected, including registration forms, purchase records, user external credit information, etc.; member data from different sources are integrated into a unified data storage system; member-related information, including personal information, purchase history, preferences, etc., is stored and managed to support personalized marketing and customer relationship management. For transaction data classification, transaction data is collected from the platform, including order information, transaction amount, product information; and transaction data is cleaned, verified and formatted. For logistics data classification, logistics data, distribution information, transportation status, inventory changes, etc. are collected from the logistics system, and a real-time monitoring system is established to track key indicators and abnormal conditions in the logistics transportation process, and timely adjustments and processing are carried out. Financial data classification includes collecting financial data from the financial middle office, payment platform and financial software, general ledger, payment records, tax declaration information, and analyzing financial data such as profit and loss, cash flow, cost structure, etc. to provide support for financial decision-making and forecasting.

[0043] According to some embodiments, the hierarchical processing of the data after data modeling includes standardizing the data after data modeling to obtain the standardized data; performing data aggregation processing on the standardized data to generate data sets for different user levels, and obtaining the hierarchical results of the business detail data. Figure 1 As shown in the figure, DWD (data warehouse layer) is used to store unprocessed data extracted from the original data source, which retains the integrity and historical changes of the data, and is helpful for data reproduction and data quality control. DWS (data integration layer) cleans, converts and integrates the original data to ensure the consistency and quality of the data so that it can be used for subsequent analysis and processing. The data in DWD is standardized to ensure the consistency of the data between different business functions and departments. Among them, standardization includes unified naming, unified data format, etc. Data aggregation includes data aggregation and summary according to business needs to generate data sets suitable for different user levels.

[0044] According to some embodiments, after obtaining data sets of different user levels, these data sets may be optimized to obtain optimized data, wherein the optimization is to optimize and process the data according to specific application scenarios, including data simplification and pre-calculation.

[0045] After obtaining the hierarchical results of business detail data, the hierarchical results of business detail data can be distributed to applications or users, so that application platforms or business users can access them through data services or API interfaces. ADS (application data layer) processes data according to user needs and application scenarios, and finally provides it to business users and application data layers. The data has been processed, optimized and adapted to specific application scenarios in ADS to support various analysis and decision-making needs.

[0046] according to Figure 1 The system and its data processing flow shown can bring the following effects:

[0047] 1. Clear data management and processes:

[0048] Data warehouse layer (DWD): used to store unprocessed data extracted from the original data source, retaining the integrity and historical changes of the data, and facilitating data reproduction and data quality control.

[0049] Data Integration Layer (DWS): At this layer, the raw data is cleaned, converted, and integrated to ensure the consistency and quality of the data so that it can be used for subsequent analysis and processing.

[0050] Application Data Layer (ADS): The data layer that is ultimately provided to business users and applications, where data has been processed, optimized, and adapted to specific application scenarios to support various analysis and decision-making needs.

[0051] 2. Flexible data processing flow:

[0052] Each layer in DWD, DWS and ADS has its own specific data processing and management goals, making the entire data processing process more flexible and efficient. Data is processed and processed layer by layer, gradually changing from its original state to a format that can be used for business analysis and decision-making.

[0053] 3. Improve data quality and consistency:

[0054] The data cleaning and conversion process at the DWS layer helps improve data quality and reduce data errors and redundancy. At the same time, through standardized data models and architectures, data consistency can be ensured across different systems and business functions.

[0055] 4. Support diverse business needs:

[0056] Different levels of data stratification allow for customized data processing and management for specific business functions and user needs. For example, the finance department may need to focus on precision and accuracy, while the operations department may be more concerned with the real-time and operational nature of the data.

[0057] 5. Business-driven data perspective:

[0058] This classification method is directly related to business functions and business processes, making data management closer to business needs and actual operations. Different business departments can more easily obtain and understand data related to their business.

[0059] 6. Optimize data access and usage efficiency:

[0060] The data layer of each business function or business process can optimize data access and usage efficiency. For example, the member data layer may focus on the management and analysis of personal information and user behavior data, while the transaction data layer focuses on the processing and analysis of order and transaction data.

[0061] 7. Enhance data autonomy for business departments:

[0062] Each business unit can manage its own data layer according to its own needs and priorities, thereby enhancing its autonomy and flexibility in data management and use.

[0063] Based on the above system, according to one aspect of the present application, a method for hierarchical classification of data is provided. Figure 2 FIG. 1 is a flow chart of a method for hierarchical classification of data according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0064] Step S201, collecting data to be entered into the data warehouse;

[0065] Step S202, loading the collected data into the ODS of the data warehouse;

[0066] Step S203, preprocessing the data in the ODS for data warehouse;

[0067] Step S204, transmitting the processed data to the ODSP for processing to obtain business detail data;

[0068] Step S205, performing data modeling according to the business detail data to determine the classification of the business detail data; and

[0069] Step S206, performing layering processing on the data after data modeling to obtain the layering result of the business detail data.

[0070] According to an optional embodiment, step S206 may specifically include:

[0071] Standardize the data after data modeling to obtain standardized data;

[0072] The standardized data is subjected to data aggregation processing to generate data sets for different user levels, thereby obtaining a hierarchical result of the business detail data.

[0073] According to an optional embodiment, step S206 may further specifically include:

[0074] The data set is optimized to obtain optimized data.

[0075] According to an optional embodiment, step S205 may specifically include:

[0076] In the process of data modeling based on the business detail data, dimension tables and fact tables of the data warehouse are formed, wherein the classification of the business detail data is recorded in the dimension tables and / or the fact tables.

[0077] According to an optional embodiment, step S201 may specifically include:

[0078] Collect offline data through ETL tools; and / or

[0079] Collect real-time data through log synchronization tools or point-of-sale push.

[0080] Based on the above system, according to one aspect of the present application, a method for hierarchical classification of data is provided, such as Figure 3 As shown. Figure 2 compared to, Figure 3 Steps S301 to S306 are Figure 2 Steps S201 to S206 are the same or similar, except that: Figure 3 The method shown also includes:

[0081] Step S307: Distribute the hierarchical results of the business detail data to applications or users.

[0082] Based on the above system, according to one aspect of the present application, a device for hierarchical classification of data is provided. Figure 4 FIG. 1 is a schematic diagram of a device for hierarchical classification of data according to an embodiment of the present application. Figure 4As shown, the device includes a collection module 401, a loading module 402, a preprocessing module 403, a second acquisition module 404, a determination module 405, and a second acquisition module 406. The collection module 401 is used to collect data to be entered into the data warehouse; the loading module 402 is used to load the collected data into the ODS of the data warehouse; the preprocessing module 403 is used to preprocess the data in the ODS for the data warehouse; the first acquisition module 404 is used to transmit the processed data to the ODSP for processing to obtain business detail data; the determination module 405 is used to perform data modeling according to the business detail data and determine the classification of the business detail data; the second acquisition module 406 is used to perform hierarchical processing on the data after data modeling to obtain the hierarchical results of the business detail data.

[0083] According to an optional embodiment, the second obtaining module 406 may be specifically configured to:

[0084] Standardize the data after data modeling to obtain standardized data;

[0085] The standardized data is subjected to data aggregation processing to generate data sets for different user levels, thereby obtaining a hierarchical result of the business detail data.

[0086] According to an optional embodiment, the second obtaining module 406 may also be specifically configured to:

[0087] The data set is optimized to obtain optimized data.

[0088] According to an optional embodiment, the determination module 405 may be specifically configured to:

[0089] In the process of data modeling based on the business detail data, dimension tables and fact tables of the data warehouse are formed, wherein the classification of the business detail data is recorded in the dimension tables and / or the fact tables.

[0090] According to an optional embodiment, the acquisition module 401 may be specifically used for:

[0091] Collect offline data through ETL tools; and / or

[0092] Collect real-time data through log synchronization tools or point-of-sale push.

[0093] Based on the above system, according to another aspect of the present application, a device for hierarchical classification of data is provided, such as Figure 5 As shown. Figure 4 compared to, Figure 5 Modules 501 to 506 and Figure 4Modules 401 to 406 are the same or similar, except that: Figure 5 The device shown also includes:

[0094] The distribution module 507 is used to distribute the hierarchical results of the business detail data to applications or users.

[0095] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical connection or other forms.

[0098] See also Figure 6 , Figure 6 An electronic device is provided, comprising a processor and a memory. The memory stores computer instructions, and when the computer instructions are executed by the processor, the processor executes the computer instructions to achieve the following Figure 2 and Figure 3 The method and refinement scheme shown.

[0099] It should be understood that the above device embodiments are only illustrative, and the device disclosed in the present invention can also be implemented in other ways. For example, the division of units / modules described in the above embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0100] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present invention may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0101] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not otherwise specified, the processor or chip may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not otherwise specified, the on-chip cache, the off-chip memory, and the memory may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0102] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer electronic device (which can be a personal computer, a server or a network electronic device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0103] The present application also provides a non-transitory computer storage medium storing a computer program. When the computer program is executed by a plurality of processors, the processors execute the following Figure 2 and Figure 3The method and refinement scheme shown.

[0104] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. A method for hierarchical classification of data, characterized in that: include: Collect data to be entered into the data warehouse; Loading the collected data into the ODS of the data warehouse; Preprocessing the data in the ODS for data warehouse; The processed data is transmitted to ODSP for processing to obtain business detail data; Performing data modeling based on the business detail data to determine the classification of the business detail data; as well as The data after data modeling is layered to obtain the layered results of the business detail data.

2. The method according to claim 1, characterized in that The step of performing layered processing on the data after data modeling to obtain a layered result of the business detail data includes: Standardize the data after data modeling to obtain standardized data; The standardized data is subjected to data aggregation processing to generate data sets for different user levels, thereby obtaining a hierarchical result of the business detail data.

3. The method according to claim 2, characterized in that The layering of the data after data modeling to obtain the layering result of the business detail data also includes: The data set is optimized to obtain optimized data.

4. The method according to claim 1, characterized in that The performing of data modeling according to the business detail data to determine the classification of the business detail data includes: In the process of data modeling based on the business detail data, dimension tables and fact tables of the data warehouse are formed, wherein the classification of the business detail data is recorded in the dimension tables and / or the fact tables.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: The hierarchical results of the business detail data are distributed to applications or users.

6. The method according to any one of claims 1 to 4, characterized in that: The data to be collected into the data warehouse includes: Collect offline data through ETL tools; and / or Collect real-time data through log synchronization tools or point-of-sale push.

7. The method according to any one of claims 1 to 4, characterized in that The preprocessing of the data warehouse includes one or more of data cleaning, data verification, missing value processing and format error processing. The data to be entered into the data warehouse includes data generated by the steel Internet industry.

8. A device for hierarchical classification of data, characterized in that: include: The collection module is used to collect data to be entered into the data warehouse; A loading module, used for loading the collected data into the ODS of the data warehouse; A preprocessing module, used for preprocessing the data in the ODS for data warehouse; The first acquisition module is used to transmit the processed data to the ODSP for processing to obtain business detail data; A determination module, used to perform data modeling according to the business detail data and determine the classification of the business detail data; as well as The second acquisition module is used to perform hierarchical processing on the data after data modeling to obtain the hierarchical results of the business detail data.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images