Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment, and relates to the technical field of data management and analysis. The method comprises the steps that a fact table and a dimension table are obtained, the fact table is used for storing business events and data corresponding to the business events, and the dimension table is used for storing description information of the business events; registering an interface service for the fact table and the dimension table, the interface service being used for performing data operation on the fact table and the dimension table by using a network protocol; a dimension model is determined according to the interface service of the dimension table and the dimension table, and the dimension model is used for describing dimension information of the business event; a measurement model is determined according to the interface service of the fact table, the fact table and the dimension model, and the measurement model is a model for processing measurement data related to the business event; and generating a target index according to the measurement model. The technical problems of low efficiency and low accuracy of index creation caused by index creation based on manual operation in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the technical field of data management and analysis, and in particular to a data processing method, device, storage medium, and electronic device. Background Art

[0002] Traditional metric management software generally follows a fixed, multi-step process for business metric modeling. This process consists of three main phases: connecting to data sources, manually identifying dimensions and metrics, and finally creating metrics. However, this traditional metric modeling approach presents the following key issues:

[0003] Complex data source connections: Due to the diversity of internal enterprise data sources, including various database tables and data sets in the data center, connecting data sources often consumes a lot of time and resources. Data engineers must understand the structure and characteristics of each data source in detail and perform complex configuration and debugging to ensure accurate data reading and subsequent processing.

[0004] Manual identification of dimensions and metrics: For each data source, business personnel must manually identify indicator dimensions and metrics. This process is not only time-consuming and labor-intensive, but also relies on their expertise and experience. In a multi-source, heterogeneous data environment, the complexity of manual identification is further exacerbated, which can easily lead to incorrect or missed labeling of indicator dimensions and metrics, affecting the accuracy and consistency of subsequent indicators.

[0005] Inefficient indicator creation: Creating indicators based on manually identified dimensions and metrics is still a manual process. This not only increases the workload of business personnel, but also easily introduces errors when creating and modifying indicators, reducing the efficiency of indicator creation and user experience.

[0006] In summary, traditional indicator modeling approaches suffer from significant efficiency and accuracy issues in areas such as data source connection, dimension measurement identification, and indicator creation. Furthermore, their cumbersome procedures often lead to high learning and usage costs. These issues are particularly pronounced when dealing with large-scale, multi-source, and heterogeneous data, hindering the rapid iteration and efficient management of business indicators. Therefore, a more efficient, accurate, and automated indicator modeling approach is urgently needed to address the shortcomings of existing technologies.

[0007] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0008] The present application provides a data processing method, device, storage medium and electronic device to at least solve the technical problem in the prior art that indicators are created based on manual operations, resulting in low efficiency and low accuracy in indicator creation.

[0009] According to one aspect of the present application, a data processing method is provided, including: obtaining a fact table and a dimension table, wherein the fact table is used to store business events and data corresponding to the business events, and the dimension table is used to store descriptive information of the business events; registering an interface service for the fact table and the dimension table, wherein the interface service is used to perform data operations on the fact table and the dimension table using a network protocol; determining a dimensional model based on the interface service and the dimension table of the dimension table, wherein the dimensional model is used to describe dimensional information of the business events; determining a measurement model based on the interface service, the fact table, and the dimensional model of the fact table, wherein the measurement model is a model for processing measurement data related to the business events; and generating target indicators based on the measurement model.

[0010] Optionally, registering interface services for the fact table and the dimension table includes: registering a first query interface service and a second query interface service for the dimension table, wherein the first query interface service is used to query the description information of the dimension table, and the second query interface service is used to query the data in the dimension table; registering a third query interface service and a fourth query interface service for the fact table, wherein the third query interface service is used to query the description information of the fact table, and the fourth query interface service is used to query the data in the fact table.

[0011] Optionally, determining the dimension model based on the interface service of the dimension table and the dimension table includes: obtaining metadata information in the dimension table based on the first query interface service of the dimension table; marking the field corresponding to the field information included in the metadata information in the dimension table as the first dimension field; and determining the dimension model based on the dimension table and the first dimension field.

[0012] Optionally, determining a measurement model based on the interface service of the fact table, the fact table, and the dimensional model includes: obtaining metadata information in the fact table based on the third query interface service of the fact table; determining the field type of the field information included in the metadata information in the fact table, wherein the field type includes a measurement type and a dimension type, wherein the measurement type refers to a type that describes measurable business data, and the dimension type refers to a type that describes the classification attributes or dimensional information of the business data; determining the measurement model based on the fact table, the field type of the field information included in the metadata information in the fact table, and the dimensional model.

[0013] Optionally, the measurement model is determined based on the fact table, the field type of the field information included in the metadata information in the fact table, and the dimensional model, including: when the fact table includes a first part of metadata information and a second part of metadata information, and the field type of the first part of metadata information is a measurement type and the field type of the second part of metadata information is a dimension type, the field corresponding to the first part of metadata information is determined as a measurement field, and the field corresponding to the second part of metadata information is determined as a second dimension field; when it is detected that the first dimension field and the second dimension field in the dimensional model are the same, the first dimension field and the second dimension field are matched to obtain a matching result; and the measurement model is determined based on the fact table, the measurement field, and the matching result.

[0014] Optionally, after registering the interface services for the fact table and the dimension table respectively, the data processing method further includes: storing configuration information of the interface services in a database, wherein the configuration information is settings and parameters required for the interface services to operate normally and be called.

[0015] Optionally, after generating the target indicator according to the measurement model, the data processing method also includes: querying the corresponding data according to the second query interface service of the dimension table and the fourth query interface service of the fact table to obtain a data set; loading the data set into the memory, and performing parallel processing and indicator calculation on the data in the data set in the memory, wherein the indicator calculation is used to calculate the numerical value corresponding to the business indicator according to the measurement model.

[0016] According to another aspect of the present application, a data processing device is also provided, including: a first acquisition unit, which acquires a fact table and a dimension table, wherein the fact table is used to store business events and data corresponding to the business events, and the dimension table is used to store description information of the business events; a first registration unit, which registers an interface service for the fact table and the dimension table, wherein the interface service is used to perform data operations on the fact table and the dimension table using a network protocol; a first determination unit, which determines a dimension model based on the interface service and the dimension table of the dimension table, wherein the dimension model is used to describe the dimension information of the business event; a second determination unit, which determines a measurement model based on the interface service, the fact table and the dimension model of the fact table, wherein the measurement model is a model for processing measurement data related to the business event; and a first generation unit, which generates target indicators based on the measurement model.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program runs, the device where the computer-readable storage medium is located executes the above-mentioned data processing method.

[0018] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned data processing method.

[0019] As can be seen from the above content, in this application, the required dimension tables and fact tables are first obtained from different data sources, wherein the fact table is used to store business events and data corresponding to the business events, and the dimension table is used to store descriptive information of the business events. Then, the interface service is registered for the obtained dimension table and fact table, wherein the interface service can be used to perform data operations on the fact table and dimension table using network protocols. With the support of the interface service, the corresponding dimensional model is automatically generated according to the interface service and dimension table of the dimension table, wherein the dimensional model is used to describe the dimensional information of the business event. Similarly, the corresponding measurement model is automatically generated according to the interface service, fact table and dimensional model of the fact table, wherein the measurement model is a model for processing measurement data related to the business event. Finally, the target indicator is generated based on the created measurement model mapping. That is, by integrating multi-source heterogeneous data, registering service interfaces, automatically generating models and generating indicator mappings, the purpose of efficient, accurate and automated creation of indicators is achieved, thereby achieving the technical effect of improving the efficiency and accuracy of indicator creation, thereby solving the technical problem of low efficiency and low accuracy of indicator creation caused by manual operation in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a flowchart of an optional data processing method according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of an optional data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] It should also be noted that the information and data collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and the relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain the relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0026] According to an embodiment of the present application, an embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] It should be noted that an intelligent processing system can be used as the execution subject of the data processing method of the embodiment of the present application. It is understandable that the data processing method provided in the embodiment of the present application can also be executed by other systems or devices, and the embodiment of the present application does not specifically limit this.

[0028] Figure 1 is a flow chart of an optional data processing method according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps:

[0029] Step S101: Obtain fact tables and dimension tables.

[0030] In step S101 , the fact table is used to store business events and data corresponding to the business events, and the dimension table is used to store description information of the business events.

[0031] Optionally, the intelligent processing system can obtain multiple fact tables and multiple dimension tables corresponding to business needs, where one fact table can correspond to multiple dimension tables. For example, the fact table of sales data can be associated with multiple dimension tables such as time dimension table, product dimension table, customer dimension table, etc.

[0032] It's important to note that fact tables and dimension tables are two important types of tables in a data warehouse. Fact tables typically contain specific data about transactions or business events, such as sales records and production events. For example, in a sales business, a fact table might include fields such as sales date, product ID (Identifier), sales quantity, and sales amount to record the details of each sale. Dimension tables store additional information describing the business events in the fact table, such as product name, customer information, and region. This information facilitates more detailed analysis and classification of the fact data. For example, for sales data, a dimension table might include fields such as product name, product category, customer name, and region to provide detailed context for the sales data.

[0033] Optionally, the intelligent processing system uses front-end technology to develop corresponding front-end components to centrally manage the data set and provide it for use in subsequent dimensional modeling and metric modeling.

[0034] Optionally, the intelligent processing system can support data sources of different types, structures and formats, wherein fact tables and dimension tables can also come from multiple data sources of different types, structures and formats, and register interface services for data from different data sources.

[0035] Step S102: registering interface services for the fact table and dimension table.

[0036] In step S102 , the interface service is used to perform data operations on the fact table and the dimension table using the network protocol.

[0037] Optionally, the intelligent processing system registers the fact and dimension tables with a RESTful API (Representational State Transfer Application Programming Interface). This encapsulates the data access capabilities of the fact and dimension tables into an API (Application Programming Interface), enabling external systems to call these APIs over the network to retrieve or manipulate data. This provides a unified access point for subsequent data operations and model creation. Interface service registration not only involves querying data but also obtaining data metadata, providing data structure information for model creation.

[0038] It should be noted that the Restful API is a network application programming interface designed based on the REST architecture (Representational State Transfer) style. It is used to implement interaction between clients and servers. Especially in distributed environments, such as web services and microservices architectures, the Restful API interface service provides a simple and standardized communication method. The Restful API interface service is an API that provides services through the HTTP (Hypertext Transfer Protocol) protocol. This means that all data operations (such as query, create, update, and delete) will be implemented through standard HTTP methods. For example, the GET method (a request method defined in HTTP) is used to obtain information, the POST method (a request method defined in HTTP) is used to create resources, the PUT method (a request method defined in HTTP) is used to update resources, and the DELETE method (a request method defined in HTTP) is used to delete resources.

[0039] Optionally, the intelligent processing system registers a Restful API interface service for each data set (including fact tables and dimension tables), which usually means implementing and exposing a series of APIs in the backend system for calling by the frontend or other systems.

[0040] Step S103: determining a dimensional model according to the interface service of the dimension table and the dimension table.

[0041] In step S103 , the dimensional model is used to describe dimensional information of the business event.

[0042] Optionally, the dimensional model is used to describe the dimensional information of the data, including dimension tables and the relationships between them. By calling the interface service of the dimension table, the structural information of the dimension table (field name, data type, etc.) can be automatically obtained, and the dimensional model can be created based on this information.

[0043] Optionally, a dimensional model can be created based on one dimension table or multiple dimension tables.

[0044] Step S104: determining a measurement model based on the interface service of the fact table, the fact table, and the dimensional model.

[0045] In step S104 , the measurement model is a model for processing measurement data related to business events.

[0046] Optionally, a measure model can be flexibly associated with one or more dimensional models.

[0047] Optionally, the intelligent processing system determines the measurement model according to the interface service of the fact table, the fact table, and one or more dimensional models.

[0048] Step S105: Generate target indicators according to the measurement model.

[0049] Optionally, the intelligent processing system generates indicators by batch mapping based on the created measurement model.

[0050] Based on the contents of steps S101 to S105 above, it can be seen that in this application, the required dimension tables and fact tables are first obtained from different data sources, wherein the fact table is used to store business events and data corresponding to the business events, and the dimension table is used to store descriptive information of the business events. Then, the interface service is registered for the obtained dimension table and fact table, wherein the interface service can be used to perform data operations on the fact table and dimension table using network protocols. With the support of the interface service, a corresponding dimensional model is automatically generated based on the interface service and dimension table of the dimension table, wherein the dimensional model is used to describe the dimensional information of the business event. Similarly, a corresponding measurement model is automatically generated based on the interface service, fact table, and dimensional model of the fact table, wherein the measurement model is a model for processing measurement data related to the business event. Finally, target indicators are generated based on the created measurement model mapping. That is, by integrating multi-source heterogeneous data, registering service interfaces, automatically generating models, and generating indicator mappings, the purpose of efficient, accurate, and automated indicator creation is achieved, thereby achieving the technical effect of improving the efficiency and accuracy of indicator creation, thereby solving the technical problem of low efficiency and low accuracy of indicator creation caused by manual operation in the existing technology.

[0051] In an optional embodiment, the intelligent processing system registers a first query interface service and a second query interface service for the dimension table, wherein the first query interface service is used to query the descriptive information of the dimension table, and the second query interface service is used to query the data in the dimension table, and registers a third query interface service and a fourth query interface service for the fact table, wherein the third query interface service is used to query the descriptive information of the fact table, and the fourth query interface service is used to query the data in the fact table.

[0052] Optionally, the intelligent processing system registers a corresponding Restful API interface service for each data set based on the microservice architecture, including a metadata query API interface service and a data query API interface service.

[0053] Optionally, the intelligent processing system registers a metadata query interface (first query interface service) and a data query interface (second query interface service) for the dimension table, wherein the metadata query interface is used to obtain descriptive information of the dimension table, including metadata such as field name, data type, field description, and table structure. For example, for the "product" dimension table, the metadata query interface of the dimension table is used to obtain detailed information on fields such as product ID, product name, and product category; the data query interface is used to directly obtain specific data in the dimension table. For example, for the "product" dimension table, the data query interface of the dimension table is used to obtain information such as product ID, product name, product category, and product description.

[0054] Optionally, the intelligent processing system registers a metadata query interface (third query interface service) and a data query interface (fourth query interface service) for the fact table, wherein the metadata query interface is used to obtain descriptive information of the fact table. For example, for the "sales" fact table, the metadata query interface of the fact table is used to obtain metadata information of fields such as sales ID, sales date, product ID, customer ID, sales quantity, and sales amount; the data query interface is used to obtain data recorded in the fact table. For example, for the "sales" fact table, the data query interface of the fact table is used to obtain information such as sales ID, sales date, product ID, customer ID, sales quantity, and sales amount.

[0055] From the above content, we can see that by registering a specific query interface service, the intelligent processing system can obtain descriptive information of dimension tables and fact tables, including field names, data types, and field descriptions, providing a basis for subsequent automated identification and creation of dimensional models and measurement models. At the same time, the system can quickly and securely access the required data, improving the efficiency of data queries and the speed of data processing.

[0056] In an optional embodiment, the intelligent processing system first obtains metadata information in the dimension table according to the first query interface service of the dimension table, then marks the field corresponding to the field information included in the metadata information in the dimension table as the first dimension field, and then determines the dimension model based on the dimension table and the first dimension field.

[0057] Optionally, the intelligent processing system calls the metadata query interface service registered with the dimension table to obtain the metadata information of the dimension table. After obtaining the metadata information of the dimension table, the intelligent processing system will automatically identify which fields are suitable for use as dimension fields in the dimensional model and mark these fields as dimension fields (first dimension fields), where dimension fields are fields used to describe and classify measurement data. For example, product ID, product name, and product category fields will be marked as dimension fields because they provide specific information about the product, such as unique identification, name, and category. After completing the field marking, the intelligent processing system will determine the dimensional model based on the dimension table and the marked dimension fields. For example, based on the product dimension table and dimension fields (product ID, product name, product category), a product dimension model including the attributes and relationships of these fields will be created.

[0058] From the above content, we can see that the intelligent processing system greatly improves the efficiency, accuracy and flexibility of dimensional model creation by automatically obtaining dimension table metadata, automatically identifying and marking dimension fields, and automatically determining the dimensional model, reducing human errors and simplifying user operations.

[0059] In an optional embodiment, the intelligent processing system obtains metadata information in the fact table according to the third query interface service of the fact table, and then determines the field type of the field information included in the metadata information in the fact table, wherein the field type includes a measurement type and a dimension type, wherein the measurement type refers to a type that describes measurable business data, and the dimension type refers to a type that describes the classification attributes or dimension information of the business data. Finally, the measurement model is determined according to the fact table, the field type of the field information included in the metadata information in the fact table, and the dimensional model.

[0060] Optionally, the intelligent processing system obtains metadata information in the fact table by calling the metadata query interface service registered with the fact table. The metadata information covers the structural description of the fact table, including field names, field types, etc. The system then automatically analyzes the metadata information of the fact table and identifies which fields have measurement type characteristics and which fields have dimension type characteristics. Measurement type fields usually represent business data that can be mathematically calculated, such as sales, costs, quantities, etc.; while dimension type fields represent attributes used to describe and classify measurement data, such as product categories, regions, time, etc. Based on the analyzed fact table metadata information and the existing dimension model, the intelligent processing system determines the measurement model.

[0061] For example, let's build a sales data analysis system in which the sales fact table contains fields such as sales quantity, sales amount, product ID, date, and region. After calling the metadata query interface service registered for the sales fact table, the system will identify sales quantity and sales amount as metric type fields, and product ID, date, and region as dimension type fields. Next, the system will create a measurement model, define sales quantity and sales amount as metric fields, and determine measurement operations such as total sales amount and average sales quantity based on business needs. At the same time, the system will automatically match dimension fields such as product ID, date, and region with the corresponding dimensional models that have been created to achieve the association between the measurement model and the dimensional model, thereby building a measurement model that can support the analysis of sales data by different dimensions such as product, time, and region.

[0062] As can be seen from the above, the intelligent processing system can automatically call the third query interface service of the fact table to obtain metadata information about the fact table. Based on this metadata, it automatically identifies field types, specifically which fields are metric types and which are dimensional types. This automated process significantly reduces manual effort, avoids errors that can occur with traditional manual field identification and classification, and improves the accuracy and efficiency of data processing. After identifying metric and dimension fields, the intelligent processing system automatically determines the metric model based on the fact table, metric field type, dimension field type, and the established dimensional model. This means that the system can automatically establish the connection between metric data and dimensional data. By automating the creation of the metric model, the data analysis process is automated, significantly improving the efficiency, accuracy, and responsiveness of data processing.

[0063] In an optional embodiment, the intelligent processing system includes a first part of metadata information and a second part of metadata information in a fact table, and when the field type of the first part of metadata information is a measurement type and the field type of the second part of metadata information is a dimension type, the field corresponding to the first part of metadata information is determined as a measurement field, and the field corresponding to the second part of metadata information is determined as a second dimension field. When it is detected that the first dimension field and the second dimension field in the dimensional model are the same, the first dimension field and the second dimension field are matched to obtain a matching result, and then the measurement model is determined based on the fact table, the measurement field and the matching result.

[0064] Optionally, the metadata information of the fact table includes dimension information (the first part of metadata information) and measurement information (the second part of metadata information), and then the fields of the first part of metadata information are determined as measurement fields, and the fields of the second part of metadata information are determined as second dimension fields (i.e., representing dimension fields). The system detects whether the first dimension field and the second dimension field in the dimensional model are the same. If they are the same, they are matched. For example, if the existing product dimension model in the dimensional model contains a product ID field, and the second part of metadata information in the fact table also contains a product ID field, the system will match these two fields, and based on the matching results of the fact table, measurement fields, and dimension fields, the system determines the measurement model. It should be noted that the measurement model not only includes measurement fields and calculation methods, but also includes the specific association method between measurement fields and dimension fields in the dimensional model, so as to facilitate subsequent data analysis and indicator generation.

[0065] Optionally, creation of a metric model may include defining metric fields (i.e., fields identified as metric types in the first part of metadata information), metric operations (such as sum, average, etc.), and how metric fields are associated with dimension fields in the dimensional model.

[0066] Optionally, the intelligent processing system can process multi-source heterogeneous data more flexibly through automatic identification of metric fields and dimension fields, as well as intelligent matching mechanisms, and quickly adapt to changes in business scenarios without the need for frequent manual adjustments to the model.

[0067] As can be seen from the above, the intelligent processing system automatically distinguishes the metadata information in the fact table into metric and dimension types, thereby achieving automatic identification of metric and dimension fields. This step greatly reduces the time and errors associated with manual classification, improving the efficiency and accuracy of model building. The system then detects whether the first dimension field in the dimensional model is identical to the second dimension field (metadata information from the fact table). If so, it automatically matches them. This intelligent matching mechanism ensures accurate association between the metric and dimensional models, laying a solid foundation for subsequent data query and analysis.

[0068] In an optional embodiment, the intelligent processing system stores configuration information of the interface service in a database, wherein the configuration information is settings and parameters required for the interface service to operate normally and be called.

[0069] Optionally, the intelligent processing system first collects the configuration information of the interface service, which includes but is not limited to the URL of the interface, request method (such as GET, POST), parameter format, response format, authentication method, permission control, etc. The system analyzes this configuration information to ensure that the interface service can be called correctly and used effectively. The system then stores the analyzed interface service configuration information in a database and establishes a configuration information database, which is used to store the configuration information of all registered interface services for subsequent query and use.

[0070] Optionally, the intelligent processing system can call the interface query service of each data set online by loading the configuration information of the interface service corresponding to each data set stored in the database, and query the metadata information and data information of the corresponding data set based on the interface query service to provide it for use in dimensional modeling or metric modeling.

[0071] Optionally, the intelligent processing system supports dynamic updating of configuration information. When the configuration of the interface service changes, the system can promptly update the configuration information in the database to ensure the timeliness and accuracy of the information. At the same time, the system provides a management interface for the interface service configuration information, and users can query, modify and delete the configuration information.

[0072] Optionally, the intelligent processing system can also take security measures such as encrypted storage to protect the security of configuration information. In addition, the system can also provide permission control to limit access to configuration information to ensure that only authorized users can view and modify configuration information.

[0073] From the above content, it can be seen that the intelligent processing system stores the configuration information of the interface service in the database, so that the system can quickly retrieve and load the configuration information of the interface service, avoiding manual configuration before each call, and improving the efficiency and accuracy of the interface service call. The establishment of the configuration information database provides the system with a platform for centralized management of interface service configuration information, which facilitates the unified management and maintenance of configuration information. At the same time, it also supports the system in future expansion and upgrades, and can easily add new interface service configuration information.

[0074] In an optional embodiment, the intelligent processing system queries the corresponding data based on the second query interface service of the dimension table and the fourth query interface service of the fact table to obtain a data set, and then loads the data set into the memory, and performs parallel processing and indicator calculation on the data in the data set in the memory, wherein the indicator calculation is used to calculate the numerical value corresponding to the business indicator according to the measurement model.

[0075] Optionally, the intelligent processing system calls the second query interface service of the dimension table and the fourth query interface service of the fact table to obtain dimension data and business fact data respectively, and generate a data set. This process allows the system to access the latest data directly from the data source without having to load the data into local storage in advance, thereby improving the flexibility and timeliness of data queries. The system then loads the obtained data set directly into the memory. Memory processing has the advantages of high-speed access and parallel computing, and is particularly suitable for large data volumes and complex computing requirements. By aggregating data in memory, frequent disk reading and writing is avoided, and the efficiency of data processing is improved. The data set is then used in memory to process and calculate the business indicator values.

[0076] Optionally, the intelligent processing system creates a corresponding measurement model based on business needs, and then when calculating specific indicators, calls the corresponding dataset API interface service online to query the data of the corresponding dataset, and uses spark technology (an open source distributed computing framework) to load the dataset data into memory, and completes the calculation of the specific values ​​of the corresponding indicators in memory.

[0077] It should be noted that Spark technology is an open source distributed computing framework. Spark is designed to solve the problems of processing and analyzing large-scale data sets. It provides a variety of functions such as data processing, machine learning, and graphics processing. It is suitable for various data processing scenarios, including batch processing, stream processing, interactive query and machine learning. Spark technology has the characteristics of high performance, fault tolerance, ease of use, unified data processing model, and support for multiple data sources.

[0078] Optionally, the intelligent processing system can automatically or semi-automatically register and configure the Restful API interface service according to the user-specified dimension requirements (for example, product classification, time range, geographic location, etc.) or measurement model requirements (for example, sales measurement model, cost-effectiveness evaluation measurement model, etc.), and then create the dimension model and measurement model according to the interface service, and generate indicators based on the created measurement model mapping.

[0079] Optionally, the intelligent processing system first registers the fact table and dimension table for which indicators are to be generated with the RESTful API interface service. It then directly creates the indicator dimension model based on the dimension table and its interface service. It then directly creates the dataset through the fact table's interface service. During the creation process, it supports automatic tagging of field attributes (dimension fields / metric fields). Finally, when creating indicators, the user can select the Generate Indicator Based on Model Mapping option. The user specifies the datasets for which indicators need to be generated, and then batch maps the generated indicators.

[0080] Optionally, the business front end in the intelligent processing system will perform Restful processing on the received and processed fact tables and dimension tables. API interface service registration ensures that data sources can be accessed by the system in a standardized, service-oriented manner, laying the foundation for subsequent indicator construction and data analysis. When users need to create new indicator dimensions, the system provides dimension model construction through the registered dimension table interface service. Users can intuitively select and configure the required dimension fields, and the system will automatically create the corresponding dimension model, simplifying the complexity of dimension management. The creation of measurement models also relies on the dataset interface service. When creating a measurement model, users can directly select the corresponding dataset from the fact table interface service, and the system automatically identifies field attributes (dimension fields or measurement fields), thus efficiently completing the measurement model construction and avoiding the tedious and error-prone manual configuration. During the indicator creation process, the system provides the option of automatically generating indicators based on model mapping. Users only need to specify the datasets for which indicators are to be generated, and the system will automatically map and generate indicators in batches, greatly reducing manual configuration steps and improving the efficiency and accuracy of indicator creation. Finally, the system provides an indicator query function. Users can quickly obtain the data for the required indicators through the query interface. Whether it is real-time query or historical data analysis, the system provides accurate and timely feedback.

[0081] From the above content, we can see that the intelligent processing system significantly improves the speed of indicator calculation and reduces calculation delay by performing parallel calculations in memory. The effect is particularly obvious when processing large-scale data sets. The combination of the high-speed read and write capabilities of memory and the high efficiency of parallel computing greatly improves the response speed and overall performance of the system. In addition, by automatically calculating business indicators according to the measurement model without human intervention, it avoids the errors that may be caused by manual operation and improves the accuracy and consistency of indicator calculation.

[0082] According to another aspect of the present application, a data processing device is provided. Figure 2 is a schematic diagram of an optional data processing device according to an embodiment of the present application, such as Figure 2 As shown, the data processing device includes: a first acquiring unit 201 , a first registering unit 202 , a first determining unit 203 , a second determining unit 204 and a first generating unit 205 .

[0083] Optionally, the first acquisition unit 201 is used to obtain a fact table and a dimension table, wherein the fact table is used to store business events and data corresponding to the business events, and the dimension table is used to store descriptive information of the business events; the first registration unit 202 is used to register an interface service for the fact table and the dimension table, wherein the interface service is used to perform data operations on the fact table and the dimension table using a network protocol; the first determination unit 203 is used to determine a dimension model based on the interface service and the dimension table of the dimension table, wherein the dimension model is used to describe the dimension information of the business event; the second determination unit 204 is used to determine a measurement model based on the interface service, the fact table and the dimension model of the fact table, wherein the measurement model is a model for processing measurement data related to the business event; the first generation unit 205 is used to generate target indicators based on the measurement model.

[0084] Optionally, the first registration unit 202 includes: a first registration sub-unit and a second registration sub-unit, wherein the first registration sub-unit is used to register a first query interface service and a second query interface service for the dimension table, wherein the first query interface service is used to query the description information of the dimension table, and the second query interface service is used to query the data in the dimension table; the second registration sub-unit is used to register a third query interface service and a fourth query interface service for the fact table, wherein the third query interface service is used to query the description information of the fact table, and the fourth query interface service is used to query the data in the fact table.

[0085] Optionally, the first determination unit 203 includes: a first acquisition subunit, a first marking subunit and a first determination subunit, wherein the first acquisition subunit is used to obtain metadata information in the dimension table according to the first query interface service of the dimension table; the first marking subunit is used to mark the field corresponding to the field information included in the metadata information in the dimension table as a first dimension field; and the first determination subunit is used to determine the dimension model based on the dimension table and the first dimension field.

[0086] Optionally, the second determination unit 204 includes: a second acquisition subunit, a first determination subunit, and a second determination subunit. The second acquisition subunit is configured to acquire metadata information in the fact table according to the third query interface service of the fact table; the first determination subunit is configured to determine the field type of field information included in the metadata information in the fact table, wherein the field type includes a metric type and a dimension type. The metric type refers to a type that describes measurable business data, and the dimension type refers to a type that describes classification attributes or dimensional information of the business data; and the second determination subunit is configured to determine the metric model based on the fact table, the field type of the field information included in the metadata information in the fact table, and the dimensional model.

[0087] Optionally, the second determination subunit includes: a first determination module, a first matching module, and a second determination module. The first determination module is configured to, when the fact table includes the first portion of metadata information and the second portion of metadata information, and when the field type of the first portion of metadata information is a measurement type and the field type of the second portion of metadata information is a dimension type, determine the field corresponding to the first portion of metadata information as a measurement field and the field corresponding to the second portion of metadata information as a second dimension field; the first matching module is configured to, when detecting that the first dimension field and the second dimension field in the dimensional model are the same, match the first dimension field with the second dimension field to obtain a matching result; and the second determination module is configured to determine the metric model based on the fact table, the metric field, and the matching result.

[0088] Optionally, the first registration unit 202 includes: a first storage submodule, configured to store configuration information of the interface service in a database, wherein the configuration information is settings and parameters required for the interface service to operate normally and be called.

[0089] Optionally, the first generating unit 205 includes: a third determining subunit and a first processing subunit. The third determining subunit is configured to query corresponding data using the second query interface service of the dimension table and the fourth query interface service of the fact table to obtain a data set; and the first processing subunit is configured to load the data set into memory and perform parallel processing and indicator calculation on the data in the data set in the memory, wherein the indicator calculation is configured to calculate a numerical value corresponding to a business indicator according to a measurement model.

[0090] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the device where the computer-readable storage medium is located executes the above-mentioned data processing method.

[0091] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned data processing method.

[0092] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0093] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0095] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0098] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Obtain a fact table and a dimension table, wherein the fact table is used to store business events and data corresponding to the business events, and the dimension table is used to store description information of the business events; registering an interface service for the fact table and the dimension table, wherein the interface service is used to perform data operations on the fact table and the dimension table using a network protocol; Determining a dimensional model according to the interface service of the dimensional table and the dimensional table, wherein the dimensional model is used to describe dimensional information of the business event; Determining a measurement model according to the interface service of the fact table, the fact table, and the dimensional model, wherein the measurement model is a model for processing measurement data related to the business event; A target indicator is generated according to the measurement model.

2. The data processing method according to claim 1, characterized in that: Registering interface services for the fact table and the dimension table includes: Registering a first query interface service and a second query interface service for the dimension table, wherein the first query interface service is used to query description information of the dimension table, and the second query interface service is used to query data in the dimension table; A third query interface service and a fourth query interface service are registered for the fact table, wherein the third query interface service is used to query description information of the fact table, and the fourth query interface service is used to query data in the fact table.

3. The data processing method according to claim 2, characterized in that: Determining a dimensional model according to the interface service of the dimensional table and the dimensional table includes: Acquire metadata information in the dimension table according to a first query interface service of the dimension table; Marking a field corresponding to the field information included in the metadata information in the dimension table as a first dimension field; The dimensional model is determined according to the dimension table and the first dimension field.

4. The data processing method according to claim 2, characterized in that: Determining a measurement model according to the interface service of the fact table, the fact table, and the dimensional model includes: Acquire metadata information in the fact table according to a third query interface service of the fact table; Determine the field type of the field information included in the metadata information in the fact table, wherein the field type includes a metric type and a dimension type, wherein the metric type refers to a type that describes measurable business data, and the dimension type refers to a type that describes a classification attribute or dimensional information of the business data; The measurement model is determined according to the fact table, the field type of the field information included in the metadata information in the fact table, and the dimensional model.

5. The data processing method according to claim 4, characterized in that: Determining the measurement model according to the fact table, the field type of the field information included in the metadata information in the fact table, and the dimensional model includes: In a case where the fact table includes the first part of metadata information and the second part of metadata information, and the field type of the first part of metadata information is a metric type and the field type of the second part of metadata information is a dimension type, determining the field corresponding to the first part of metadata information as a metric field, and determining the field corresponding to the second part of metadata information as a second dimension field; When it is detected that the first dimension field in the dimensional model is the same as the second dimension field, matching the first dimension field with the second dimension field to obtain a matching result; The metric model is determined according to the fact table, the metric field, and the matching result.

6. The data processing method according to claim 1, characterized in that: After registering interface services for the fact table and the dimension table respectively, the data processing method further includes: The configuration information of the interface service is stored in a database, wherein the configuration information is the settings and parameters required for the interface service to run normally and be called.

7. The data processing method according to claim 2, characterized in that: After generating the target indicator according to the measurement model, the data processing method further includes: Querying corresponding data according to the second query interface service of the dimension table and the fourth query interface service of the fact table to obtain a data set; The data set is loaded into a memory, and data in the data set are processed in parallel and indexes are calculated in the memory, wherein the index calculation is used to calculate a numerical value corresponding to a business index according to the measurement model.

8. A data processing device, characterized in that: include: A first acquisition unit is configured to acquire a fact table and a dimension table, wherein the fact table is used to store business events and data corresponding to the business events, and the dimension table is used to store description information of the business events; A first registration unit registers an interface service for the fact table and the dimension table, wherein the interface service is used to perform data operations on the fact table and the dimension table using a network protocol; a first determining unit, which determines a dimensional model according to the interface service of the dimensional table and the dimensional table, wherein the dimensional model is used to describe dimensional information of the business event; a second determining unit, determining a measurement model according to the interface service of the fact table, the fact table, and the dimensional model, wherein the measurement model is a model for processing measurement data related to the business event; The first generating unit generates a target indicator according to the measurement model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the data processing method described in any one of claims 1 to 7.

Images