Data processing method, device, equipment, medium and program product

Abstract

This specification discloses a data processing method, apparatus, device, medium, and program product, which can improve the reliability of data processing. The method includes: registering an offline indicator database to a unified data service in response to a database upload request; processing the offline indicator database through the unified data service to obtain an online indicator database; and querying corresponding indicator data from the offline indicator database and / or the online indicator database through the unified data service in response to an indicator data query request.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] The processing requirements for offline data analysis and online decision-making often differ. Offline analysis typically involves in-depth analysis of large amounts of historical data, while online decision-making focuses on making rapid decisions using the latest data.

[0003] To adapt to both offline and online data processing scenarios, the data processing phase typically involves pre-processing data from the data source offline to obtain offline indicator data, which is then stored in an offline data warehouse. Next, the offline indicator data undergoes further processing to obtain online indicator data, which is stored in an online database. In the data application phase, users can write scripts to access the offline data warehouse for offline analysis or access the online database for online decision-making.

[0004] However, due to the different underlying storage engines and technology stacks used in offline and online processing, current data processing solutions are prone to inconsistencies in the data caliber of the same metric, affecting the consistency and accuracy of data processing. A more reliable data processing method is needed. Summary of the Invention

[0005] This specification provides a data processing method, apparatus, device, medium, and program product that can improve the reliability of data processing.

[0006] Firstly, embodiments of this specification provide a data processing method, including:

[0007] In response to the database listing request, the offline indicator database is registered with the unified data service;

[0008] An online indicator database is obtained by processing the offline indicator database through a unified data service.

[0009] In response to an indicator data query request, the system retrieves the corresponding indicator data from the offline indicator database and / or the online indicator database through a unified data service.

[0010] In one possible implementation, the database listing request carries indicator configuration information;

[0011] In response to the database listing request, the offline metrics database is registered with the unified data service, including:

[0012] In response to the database upload request, the offline indicator data stored in the offline indicator database is statistically analyzed;

[0013] Based on the indicator configuration information, generate offline indicator metadata corresponding to the offline indicator data; the offline indicator metadata includes the offline storage mapping information of each indicator.

[0014] Write offline indicator metadata into a metadata table that can be called by the unified data service.

[0015] In one possible implementation, the database listing request also carries data processing rules;

[0016] By processing offline indicator databases through a unified data service, an online indicator database is obtained, including:

[0017] Through unified data services, online indicator metadata corresponding to offline indicator metadata is generated in the metadata table based on indicator configuration information; online indicator metadata includes online storage mapping information for each indicator.

[0018] Through unified data services, the offline indicator database is processed according to online indicator metadata and data processing rules to generate an online indicator database.

[0019] In one possible implementation, both offline and online metric metadata also include status information for each metric.

[0020] The method also includes:

[0021] In response to the obsolescence request for the first indicator, the status information of the first indicator in the meta information table is updated to an obsolescence status applicable to the specified scenario.

[0022] In response to the request to remove the second indicator, the status information of the second indicator in the metadata table is updated to a prohibited removal status.

[0023] In one possible implementation, the above method also includes:

[0024] According to a preset time interval, the historical detailed data of the offline data source within the preset time period is processed in batches to obtain at least one offline table; each offline table stores the corresponding offline indicator data.

[0025] Build an offline metrics database based on at least one offline table.

[0026] In one possible implementation, in response to an indicator data query request, the corresponding indicator data is queried from an offline or online indicator database via a unified data service, including:

[0027] In response to offline indicator data query requests for offline analysis, the corresponding offline indicator data is retrieved from the offline indicator database through the unified data service;

[0028] In response to online indicator data query requests for online decision-making, the system retrieves the corresponding online indicator data from the online indicator database through a unified data service.

[0029] In one possible implementation, the offline indicator data query request carries the first indicator identifier of the first query indicator;

[0030] In response to offline indicator data query requests for offline analysis, the corresponding offline indicator data is retrieved from the offline indicator database through the unified data service, including:

[0031] In response to an offline indicator data query request for offline analysis, the first indicator metadata associated with the first indicator identifier is obtained from the metadata table through the unified data service; the first indicator metadata includes the offline storage mapping information of the first queried indicator.

[0032] Through unified data services, the corresponding offline indicator data is queried from the offline indicator database based on the offline storage mapping information of the first query indicator.

[0033] In one possible implementation, the online indicator data query request carries the second indicator identifier of the second query indicator;

[0034] In response to online indicator data query requests for online decision-making, the system retrieves relevant online indicator data from the online indicator database through a unified data service, including:

[0035] In response to an online indicator data query request for online decision-making, the second indicator metadata associated with the second indicator identifier is obtained from the metadata table through a unified data service; the second indicator metadata includes the online storage mapping information of the second queried indicator.

[0036] Through unified data services, the corresponding online indicator data is queried from the online indicator database based on the online storage mapping information of the second query indicator.

[0037] In one possible implementation, in response to an offline indicator data query request for offline analysis, the first indicator metadata associated with the first indicator identifier is obtained from the metadata table via a unified data service, including:

[0038] In response to an offline indicator data query request for offline analysis, the first domain-specific language query statement corresponding to the offline indicator data query request is determined through a unified data service.

[0039] Through unified data services, the metadata of the first indicator associated with the first indicator identifier is obtained from the metadata table based on the query statement of the first domain-specific language.

[0040] Through unified data services, based on the offline storage mapping information of the first queried indicator, the corresponding offline indicator data is retrieved from the offline indicator database, including:

[0041] Through unified data services, a structured query statement for the offline indicator database is generated based on the offline storage mapping information of the first query indicator.

[0042] Through unified data services, the corresponding offline indicator data can be retrieved from the offline indicator database based on the structured query statements of the offline indicator database.

[0043] In one possible implementation, in response to an online indicator data query request for online decision-making, second indicator metadata associated with the second indicator identifier is obtained from the metadata table via a unified data service, including:

[0044] In response to online indicator data query requests for online analysis, the second domain-specific language query statement corresponding to the online indicator data query request is determined through a unified data service;

[0045] Through unified data services, second indicator metadata associated with the second indicator identifier is obtained from the metadata table based on the second domain-specific language query statement.

[0046] Through unified data services, based on the online storage mapping information of the second query indicator, the corresponding online indicator data is queried from the online indicator database, including:

[0047] Through unified data services, a structured query statement for the online indicator database is generated based on the online storage mapping information of the second query indicator.

[0048] Through unified data services, the corresponding online indicator data can be retrieved from the online indicator database based on the structured query statements of the online indicator database.

[0049] Secondly, embodiments of this specification provide a data processing apparatus, including:

[0050] The registration module is used to register offline indicator databases to the unified data service in response to database listing requests.

[0051] The processing module is used to process offline indicator databases through unified data services to obtain online indicator databases;

[0052] The query module is used to respond to indicator data query requests and retrieve the corresponding indicator data from offline indicator databases and / or online indicator databases through a unified data service.

[0053] Thirdly, embodiments of this specification provide an electronic device, including: a processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the method steps provided in the first aspect of embodiments of this specification.

[0054] Fourthly, embodiments of this specification provide a computer storage medium storing multiple instructions adapted for loading by a processor and executing the method steps provided in the first aspect of embodiments of this specification.

[0055] Fifthly, embodiments of this specification provide a computer program product, including a computer program; when the computer program is executed by a processor, it implements the method steps provided in the first aspect of embodiments of this specification.

[0056] The aforementioned data processing methods, apparatus, electronic devices, computer storage media, and computer program products, in the data processing stage, respond to database upload requests by registering the offline indicator database to a unified data service. The unified data service then processes the offline indicator database to obtain an online indicator database, providing unified data management capabilities and ensuring that the offline and online databases store the same set of data corresponding to the same indicators. In the data application stage, in response to indicator data query requests, the unified data service queries the corresponding indicator data from the offline and / or online indicator databases, providing unified data query capabilities and ensuring consistency in the data definitions for the same indicator. The entire data processing process improves the consistency and accuracy of the data processing results, thereby achieving reliable data processing. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram illustrating the implementation process of a data processing method provided in related technologies;

[0059] Figure 2 A schematic diagram illustrating the implementation process of a data processing method provided for an exemplary embodiment of this specification;

[0060] Figure 3 A schematic diagram illustrating the application environment of a data processing method provided as an exemplary embodiment of this specification;

[0061] Figure 4A flowchart illustrating a data processing method provided for an exemplary embodiment of this specification;

[0062] Figure 5 This is a schematic diagram of the architecture of a data processing method in the data processing stage, provided as an example embodiment of this specification.

[0063] Figure 6 This is a schematic diagram of the architecture of a data processing method in the data application stage, provided as an example embodiment of this specification.

[0064] Figure 7 A flowchart illustrating another data processing method provided as an exemplary embodiment of this specification;

[0065] Figure 8 A flowchart illustrating another data processing method provided as an exemplary embodiment of this specification;

[0066] Figure 9 A schematic diagram of an indicator lifecycle provided for an exemplary embodiment of this specification;

[0067] Figure 10 A flowchart illustrating another data processing method provided as an exemplary embodiment of this specification;

[0068] Figure 11 A domain model diagram illustrating a data processing method in the data processing stage, provided as an exemplary embodiment of this specification;

[0069] Figure 12 A semantic transformation diagram of a data processing method in the data application stage, provided as an exemplary embodiment of this specification;

[0070] Figure 13 A schematic diagram of the structure of a data processing apparatus provided for an exemplary embodiment of this specification;

[0071] Figure 14 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this specification clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this specification.

[0073] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0074] Offline analysis and online decision-making are two common data processing methods. Offline analysis does not require real-time data processing, but it does require powerful data processing capabilities to analyze and calculate large-scale datasets. For example, it involves analyzing and calculating the historical details of a large number of users to understand the historical consumption behavior of different user groups and predict their future consumption trends. Online decision-making, on the other hand, demands high data processing speed. It typically involves quickly querying individual user data and then making rapid decisions based on the retrieved data. For instance, when a user submits a virtual resource request, a decision is made quickly based on that user's recent key performance indicators.

[0075] In related technologies, the implementation process of a data processing method is as follows: Figure 1 As shown. Please see below. Figure 1 In the data processing phase, the data processing platform performs batch processing on a large amount of detailed data stored in the offline data warehouse at preset time intervals to obtain offline indicator data. This offline indicator data is then stored in the offline indicator database. The platform further processes the offline indicator data to obtain online indicator data, which is then stored in the online indicator database. In the data application phase, in response to offline indicator data query requests for offline analysis, the data processing platform uses the structured query language corresponding to the offline indicator database to retrieve offline indicator data for offline analysis. Similarly, in response to online indicator data query requests for online decision-making, the platform uses the structured query language corresponding to the online indicator database to retrieve online indicator data for online decision-making.

[0076] Understandably, due to Figure 1 The data processing scheme shown uses different data storage engines and technology stacks in offline and online data processing. This can easily lead to inconsistencies in the data stored in the offline indicator database and the online indicator database, as well as inconsistencies in the data retrieved from the offline indicator database and the online indicator database. This can result in inconsistent data definitions for the same indicator, affecting the consistency and accuracy of data processing and causing unreliable data processing issues.

[0077] To address the above problems, this specification provides a data processing method such as... Figure 2 As shown. Please see below. Figure 2 During the data processing phase, the data processing platform performs batch processing on a large amount of detailed data stored in the offline data warehouse at preset time intervals to obtain an offline indicator database including at least one offline table. In response to a database deployment request, the offline indicator database is registered with a unified data service. The unified data service then processes the offline indicator database to obtain an online indicator database including at least one online table. During this process, indicator metadata (including offline indicator metadata corresponding to the offline indicator database and online indicator metadata corresponding to the online indicator database) is written to a metadata table. Specifically, the offline indicator metadata includes the storage mapping information of each indicator in the offline indicator database (i.e., the offline storage mapping information of each indicator); the online indicator metadata includes the storage mapping information of each indicator in the online indicator database (i.e., the online storage mapping information of each indicator).

[0078] In the data application phase, the data processing platform responds to offline indicator data query requests for offline analysis by determining the corresponding first domain-specific language query statement through a unified data service. Based on this query statement, it retrieves the first indicator metadata associated with the first indicator identifier from the metadata table. The first indicator metadata includes the offline storage mapping information of the first queried indicator. Then, through the unified data service, it generates a structured query statement for the offline indicator database based on the offline storage mapping information of the first queried indicator. Using this structured query statement, it queries the corresponding offline indicator data from the offline indicator database for offline analysis. Similarly, in response to online indicator data query requests for online analysis, the data processing platform determines the corresponding second domain-specific language query statement through a unified data service. Based on this query statement, it retrieves the second indicator metadata associated with the second indicator identifier from the metadata table. The second indicator metadata includes the online storage mapping information of the second queried indicator. Then, through the unified data service, it generates a structured query statement for the online indicator database based on the online storage mapping information of the second queried indicator. Using this structured query statement, it queries the corresponding online indicator data from the online indicator database for online decision-making.

[0079] Figure 2In the data processing scheme illustrated, during the data processing phase, the data processing platform processes the offline indicator database through a unified data service to obtain an online indicator database. It also generates corresponding indicator metadata for both the offline and online indicator databases, providing unified data management capabilities and ensuring that both databases store the same set of data corresponding to the same indicators. During the data application phase, the data processing platform uses the unified data service to access the metadata tables storing the offline and online indicator metadata, retrieving the corresponding indicator data from the offline and / or online indicator databases. This provides standardized data query capabilities, eliminating discrepancies between online and offline data processing at the data application stage. The entire process ensures consistent data definitions for the same indicator, improving the consistency and accuracy of data processing results and achieving reliable data processing.

[0080] It's worth noting that the aforementioned data processing platform is adaptable to various data processing scenarios. For example, in e-commerce data processing, the platform can build both offline and online indicator databases based on search and recommendation metrics. Offline indicator data can be queried from the offline database to generate user profiles, while online indicator data can be queried from the online database for real-time search and recommendation. In cybersecurity data processing, the platform can build both offline and online indicator databases based on threat intelligence metrics. Offline indicator data can be queried from the offline database for attack attribution, while online indicator data can be queried from the online database for abnormal behavior detection. In credit risk control data processing, the platform can build both offline and online indicator databases based on risk control metrics. Offline indicator data can be queried from the offline database for offline calculation and analysis, while online indicator data can be queried from the online database for online risk decision-making.

[0081] The data processing methods provided in the embodiments of this specification can be applied to, for example... Figure 3 In the application environment shown, the operator terminal 10 and the user terminal 20 communicate with the server 30, which provides unified data services, via a communication network. The data storage system may include, but is not limited to, offline and online indicator databases that the server 30 needs to process. The data storage system can be integrated onto the server 30 or located in the cloud or on other network servers.

[0082] In some possible embodiments, in response to a database listing operation, the operating terminal 10 sends a database listing request carried in the database listing operation to the server 30. The server 30 receives the database listing request from the operating terminal 10, and in response, registers the offline indicator database to the unified data service. The server then processes the offline indicator database through the unified data service to obtain the online indicator database. In response to an indicator data query operation, the operating terminal 10 and / or the user terminal 20 send an indicator data query request carried in the indicator data query operation to the server 30. The server 30 receives the indicator data query request from the operating terminal 10 and / or the user terminal 20, and in response, queries the corresponding indicator data from the offline indicator database and / or the online indicator database through the unified data service.

[0083] It is worth noting that the operating terminal 10 and the user terminal 20 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. The server 30 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0084] In one embodiment, such as Figure 4 As shown, a data processing method is provided, which can be applied to... Figure 3 Taking server 30 as an example, the explanation includes the following steps:

[0085] S402: In response to the database listing request, register the offline metrics database to the unified data service.

[0086] The database upload request includes metric configuration information and data processing rules. The metric configuration information may include, but is not limited to, the basic identifier of the metric and the metric's consumption logic. The data processing rules may include, but are not limited to, offline storage resources and online storage resources. The offline metric database may be constructed from historical detailed data stored in offline data sources. The offline metric database includes at least one offline table, each storing the corresponding offline metric data.

[0087] Optionally, operators send a database upload request carrying indicator configuration information and data processing rules to server 30 via operation terminal 10. Server 30 receives the database upload request and, in response, counts the offline indicator data stored in each offline table of the offline indicator database to obtain the indicator identifier corresponding to each piece of offline indicator data. Then, server 30 generates offline indicator metadata corresponding to the offline indicator data based on the basic identifier and consumption logic of the indicator in the indicator configuration information. The offline indicator metadata includes indicator identifier information, offline storage mapping information, status information, type information, and offline consumption logic information for each indicator. Finally, server 30 writes the offline indicator metadata into the metadata table for unified data service calls, thereby registering the offline indicator database to the unified data service.

[0088] For example, to meet the needs of analyzing and measuring massive amounts of data in offline scenarios (such as second-level analysis of hundreds of billions of data points), the high-performance real-time analysis database Hologress can be selected as an offline indicator database. During the data application phase, its powerful OLAP (Online Analytical Processing) capabilities can support high-throughput and low-latency offline analysis and measurement tasks.

[0089] In this embodiment, in response to the database listing request, the server 30 registers the offline indicator database to the unified data service according to the indicator configuration information carried in the database listing request. It can write the offline indicator metadata corresponding to the offline indicator database into the metadata table, which is convenient for calling the metadata table through the unified data service during the data application stage to automatically query the offline indicator data stored in the offline indicator database, thereby improving the efficiency and accuracy of offline analysis and realizing reliable data processing.

[0090] S404: The offline indicator database is processed through a unified data service to obtain an online indicator database.

[0091] Optionally, after registering the offline indicator database to the unified data service, server 30 further automatically generates an online indicator database based on the offline indicator database through the unified data service. Specifically, server 30 generates online indicator metadata corresponding to the offline indicator metadata in the metadata table according to the indicator configuration information carried in the database listing request. The online indicator metadata includes indicator identification information, online storage mapping information, status information, type information, and online consumption logic information of each indicator. Then, server 30 processes the offline indicator database through the unified data service according to the online indicator metadata and data processing rules to generate the online indicator database.

[0092] For example, to meet the online decision-making requirements of low latency and high availability in online scenarios, Lindorm, a database with high performance, high throughput and disaster recovery capabilities, can be selected as the main database for online metrics, and Tbase, a high-performance, high-throughput database, can be configured as a disaster recovery cache to achieve millisecond-level response and automatic fault recovery through a dual-storage architecture during the data application phase.

[0093] In this embodiment, server 30 uses a unified data service to generate online indicator metadata corresponding to offline indicator metadata in the metadata table based on the indicator configuration information carried in the database upload request. Then, based on the data processing rules and online indicator metadata carried in the database upload request, server 30 processes the offline indicator data to generate an online indicator database. On the one hand, during the data processing stage, the unified data service provides unified data management capabilities, enabling the offline and online databases to store the same set of data corresponding to the same indicators. On the other hand, it facilitates automated data query capabilities provided by the unified data service in the subsequent data application stage, ensuring consistency in the data caliber of the same indicator, thereby improving the consistency, accuracy, and efficiency of data processing and achieving reliable data processing.

[0094] S406: In response to an indicator data query request, query the corresponding indicator data from the offline indicator database and / or the online indicator database through the unified data service.

[0095] The indicator data query request carries the indicator identifier of at least one queried indicator.

[0096] Optionally, operators can send offline indicator data query requests for offline analysis to server 30 via operation terminal 10. These offline indicator data query requests carry the first indicator identifier of the first query indicator. Simultaneously, client users can also send online indicator data query requests for online decision-making to server 30 via user terminal 20. These online indicator data query requests carry the second indicator identifier of the second query indicator.

[0097] In response to the aforementioned offline indicator data query request, server 30 obtains the first indicator metadata associated with the first indicator identifier from the metadata table through the unified data service. The first indicator metadata includes the offline storage mapping information of the first queried indicator. Then, server 30 queries the corresponding offline indicator data from the offline indicator database based on the offline storage mapping information of the first queried indicator through the unified data service.

[0098] In response to the aforementioned online indicator data query request, server 30 obtains the second indicator metadata associated with the second indicator identifier from the metadata table through the unified data service. The second indicator metadata includes the online storage mapping information of the second query indicator. Then, server 30 queries the corresponding online indicator data from the online indicator database through the unified data service based on the online storage mapping information of the second query indicator.

[0099] In this embodiment, in response to offline indicator data query requests and / or online indicator data query requests, the server 30 calls the metadata table through the unified data service to query the corresponding indicator data from the offline indicator database and / or the online indicator database. The unified data service can handle indicator query requests in both offline analysis scenarios and online decision-making scenarios, improving the consistency, accuracy and efficiency of data processing, thereby achieving reliable data processing.

[0100] Please see Figure 5 This is a schematic diagram illustrating the architecture of a data processing method in the data processing stage, provided as an exemplary embodiment of this specification. Figure 5 As shown, during the data processing stage, the server providing unified data services responds to the database listing request sent by the operation terminal, registers the offline indicator database to the unified data service, processes the data of the offline indicator database through the unified data service, and obtains an online indicator database including at least one online table. In this process, the offline indicator metadata corresponding to the offline indicator database and the online indicator metadata corresponding to the online indicator database, i.e., indicator metadata, are written into the metadata table.

[0101] Please see Figure 6 This is a schematic diagram illustrating the architecture of a data processing method in the data application stage, as provided in an exemplary embodiment of this specification. Figure 6 As shown, in the data application phase, operators send offline indicator data query requests for offline analysis to the server providing unified data services via their operational terminals. The server responds to these requests by accessing the metadata table through the unified data service and retrieving the corresponding offline indicator data from the offline indicator database for offline analysis. Similarly, client users send online indicator data query requests for online analysis to the server via their user terminals. The server responds to these requests by accessing the metadata table through the unified data service and retrieving the corresponding online indicator data from the online indicator database for online decision-making.

[0102] In the above data processing method, during the data processing stage, the server responds to the database listing request by registering the offline indicator database to a unified data service. The unified data service then processes the offline indicator database to obtain the online indicator database, providing unified data management capabilities and ensuring that the offline and online indicator databases store the same set of data corresponding to the same indicators. During the data application stage, the server responds to indicator data query requests by retrieving the corresponding indicator data from the offline and / or online indicator databases through the unified data service. This provides unified data query capabilities, eliminating the differences between offline and online indicator data in application and addressing indicator data query requests in two different scenarios: offline analysis and online decision-making. The entire data processing process ensures consistency in the data definition for the same indicator, improving the consistency and accuracy of data processing results and achieving reliable data processing.

[0103] In one embodiment, such as Figure 7 As shown, another data processing method is provided, which can be applied to... Figure 3 Taking server 30 as an example, the explanation includes the following steps:

[0104] S702: Process the historical detailed data of the offline data source within the preset time period according to the preset time interval to obtain at least one offline table.

[0105] The offline data source can be, but is not limited to, an offline data warehouse. Each offline table stores the corresponding offline metric data.

[0106] Optionally, the server performs batch processing on the historical detailed data of the offline data warehouse on the previous day, such as ETL (Extraction-Transformation-Loading) or aggregation calculation, to generate at least one offline indicator table for use in subsequent data processing stages.

[0107] S704: Construct an offline metrics database based on at least one offline table.

[0108] Optionally, server 30 loads at least one offline table after batch processing into the high-performance real-time analysis database Hologress through ETL tools or data processing pipelines to form a structured offline metrics database.

[0109] In this embodiment, the server 30 processes the historical detailed data of the offline data source within a preset time period according to a preset time interval to obtain at least one offline table. Based on the at least one offline table, an offline indicator database is constructed, which can improve the efficiency and accuracy of subsequent data processing, thereby achieving reliable data processing.

[0110] S706: In response to a database upload request, calculate the offline indicator data stored in the offline indicator database.

[0111] The database deployment request carries indicator configuration information. This information may include, but is not limited to, at least one indicator identifier and the corresponding indicator consumption logic. The indicator identifier may be, but is not limited to, an indicator ID, and the indicator consumption logic includes offline consumption logic and online consumption logic.

[0112] Optionally, in response to the database listing request sent by the operating terminal 10, the server 30 counts the offline indicator data stored in each offline table of the offline indicator database to obtain the indicator identifier (such as indicator ID) of the indicator corresponding to each offline indicator data, so as to generate the offline indicator metadata corresponding to the offline indicator data in the future based on the indicator identifier of each offline indicator data and the pre-set indicator configuration information.

[0113] S708: Generate offline indicator metadata corresponding to the offline indicator data based on the indicator configuration information.

[0114] Optionally, the server 30 generates offline indicator metadata corresponding to the offline indicator data based on the indicator identifiers corresponding to each offline indicator data, the indicator identifiers in the indicator configuration information, and the offline consumption logic corresponding to each indicator identifier.

[0115] The offline metric metadata includes metric identification information, storage mapping information of each metric in the offline metric database, status information of each metric, type information of each metric, and offline consumption logic information of each metric. Specifically, the metric identification information may be, but is not limited to, the metric ID; the storage mapping information of each metric in the offline metric database, i.e., the offline storage mapping information of each metric, may include, but is not limited to, the offline metric database identifier, the offline table identifier, the offline partition information, and the offline field identifier; the status information of each metric is used to determine the lifecycle status of each metric, and may be, but is not limited to, at least one of the following: pending verification, listed, obsolete, and delisted; the type information of each metric may be, but is not limited to, the field-level data type of each metric; and the offline consumption logic information of each metric is used to define the query method and usage scenario of the metric (in this case, the offline analysis scenario).

[0116] S710: Write offline indicator metadata into the metadata table for unified data service calls.

[0117] Optionally, server 30 writes the aforementioned offline indicator metadata into a metadata table for unified data service calls, thereby registering the offline indicator database to the unified data service, so that server 30 can provide unified data management capabilities and unified data query capabilities through the unified data service and the indicator metadata in the metadata table.

[0118] In this embodiment, in response to the database listing request, the server 30 counts the offline indicator data stored in the offline indicator database, generates offline indicator metadata corresponding to the offline indicator data according to the indicator configuration information, and writes the offline indicator metadata into the metadata table for unified data service to call. This facilitates the automatic querying of the offline indicator data stored in the offline indicator database by calling the metadata table through the unified data service during the data application stage, thereby improving the accuracy and efficiency of offline analysis and realizing reliable data processing.

[0119] S712: Through unified data services, online indicator metadata corresponding to offline indicator metadata is generated in the metadata table based on indicator configuration information.

[0120] Understandably, the offline indicator metadata stored in the metadata table includes the indicator identification information of each indicator. Optionally, the server 30, through a unified data service, generates online indicator metadata corresponding to the offline indicator metadata in the metadata table based on the indicator identification information of each indicator in the metadata table, at least one indicator identifier in the indicator configuration information, and the online consumption logic corresponding to each indicator identifier. This is so that online indicator data sharing the same set of indicators as the offline indicator data can be generated subsequently based on the online indicator metadata.

[0121] The online indicator metadata includes indicator identification information, storage mapping information of each indicator in the online indicator database, status information, type information, and online consumption logic information for each indicator. Understandably, indicator identification information may include, but is not limited to, the indicator ID; storage mapping information in the online indicator database refers to the offline storage mapping information, which may include, but is not limited to, the online indicator database identifier, the online table identifier, the online partition information, and the online field identifier; status information determines the lifecycle status of each indicator, and may include, but is not limited to, at least one of the following: pending verification, listed, obsolete, or delisted; type information may include, but is not limited to, the field-level data type; and online consumption logic information defines the query method and usage scenario of the indicator (in this case, an online decision-making scenario).

[0122] S714: Through unified data services, the offline indicator database is processed according to online indicator metadata and data processing rules to generate an online indicator database.

[0123] Understandably, database deployment requests also carry data processing rules. These rules may include, but are not limited to, preset offline and online storage resources. For example, data processing rules may include the type of offline metrics database, offline table information within the offline metrics database, the type of online metrics database, and online table information within the online metrics database.

[0124] Optionally, server 30 uses a unified data service to process the offline indicator database according to the type of the offline indicator database, the offline table information in the offline indicator database, the type of the online indicator database, and the online table information of the online indicator database, to obtain online indicator data that shares the same set of indicators as the offline indicator data, and loads the online indicator data into a specified online storage engine (such as Lindorm or Tbase) to generate an online indicator database.

[0125] In this embodiment, server 30, through a unified data service, generates online indicator metadata corresponding to offline indicator metadata in the metadata table based on indicator configuration information. It then processes the offline indicator database according to the online indicator metadata and data processing rules to generate an online indicator database. This provides unified data management capabilities, allowing the offline and online indicator databases to store the same set of indicator data, while the metadata table stores the respective indicator metadata for both databases. Furthermore, it facilitates unified data query capabilities through the unified data service, addressing indicator query requests in both offline analysis and online decision-making scenarios. This embodiment helps improve the consistency, accuracy, and efficiency of data processing, thereby achieving reliable data processing.

[0126] S716: In response to an offline indicator data query request for offline analysis, query the corresponding offline indicator data from the offline indicator database through the unified data service.

[0127] The offline indicator data query request carries the first indicator identifier of the first queried indicator.

[0128] Optionally, in response to an offline indicator data query request for offline analysis, server 30 retrieves first indicator metadata associated with the first indicator identifier from the metadata table via a unified data service. The first indicator metadata includes offline storage mapping information for the first queried indicator. Then, server 30 queries the corresponding offline indicator data from the offline indicator database based on the offline storage mapping information of the first queried indicator via the unified data service.

[0129] S718: In response to an online indicator data query request for online decision-making, retrieve the corresponding online indicator data from the online indicator database through a unified data service.

[0130] Among them, the online indicator data query request carries the second indicator identifier of the second query indicator.

[0131] Optionally, in response to an online indicator data query request for online analysis, server 30 retrieves second indicator metadata associated with the second indicator identifier from the metadata table via a unified data service. The second indicator metadata includes online storage mapping information for the second queried indicator. Then, server 30 queries the corresponding online indicator data from the online indicator database based on the online storage mapping information of the second queried indicator using the unified data service.

[0132] In this embodiment, in response to offline indicator data query requests and online indicator data query requests, the server 30 calls the metadata table through a unified data service to query the corresponding indicator data from the offline indicator database and the online indicator database. This provides a unified data query capability, ensuring the consistency, accuracy, and efficiency of data processing in two different scenarios: offline analysis and online decision-making, thereby achieving reliable data processing.

[0133] In the above data processing method, during the data processing stage, server 30 processes historical detailed data from offline data sources within a preset time period in batches according to preset time intervals, constructs an offline indicator database, generates offline indicator metadata corresponding to the offline indicator data in response to database upload requests, and writes the offline indicator metadata into a metadata table for unified data service calls. Then, through the unified data service, online indicator metadata corresponding to the offline indicator metadata is generated in the metadata table according to indicator configuration information. The offline indicator database is then processed according to the online indicator metadata and data processing rules to generate an online indicator database, providing unified data management capabilities and ensuring that the offline and online indicator databases store the same set of data corresponding to the same indicators. During the data application stage, server 30 responds to offline indicator data query requests for offline analysis by querying the corresponding offline indicator data from the offline indicator database through the unified data service, and responds to online indicator data query requests for online decision-making by querying the corresponding online indicator data from the online indicator database through the unified data service. This provides unified data query capabilities, ensures consistency in data caliber for the same indicator, and improves data query efficiency. The entire process improves the consistency, accuracy, and efficiency of data processing results, achieving reliable data processing.

[0134] In one embodiment, such as Figure 8As shown, another data processing method is provided, which can be applied to... Figure 3 Taking server 30 as an example, the explanation includes the following steps:

[0135] S802: In response to the database upload request, calculate the offline indicator data stored in the offline indicator database.

[0136] Specifically, S802 is the same as S706, and will not be repeated here.

[0137] S804: Generate offline indicator metadata corresponding to the offline indicator data based on the indicator configuration information.

[0138] Specifically, S804 is the same as S708, and will not be repeated here.

[0139] S806: Write offline indicator metadata into the metadata table for unified data service calls.

[0140] Specifically, S806 is identical to S710, and will not be repeated here.

[0141] S808: Through unified data services, online indicator metadata corresponding to offline indicator metadata is generated in the metadata table based on indicator configuration information.

[0142] Specifically, S808 is identical to S712, and will not be repeated here.

[0143] S810: Through unified data services, the offline indicator database is processed according to online indicator metadata and data processing rules to generate an online indicator database.

[0144] Specifically, S810 is identical to S714, and will not be repeated here.

[0145] S812: In response to an offline indicator data query request for offline analysis, obtain the first indicator metadata associated with the first indicator identifier from the metadata table through the unified data service.

[0146] Optionally, server 30 receives an offline indicator data query request for offline analysis sent by the operating terminal. The offline indicator data query request carries a first indicator identifier (such as the indicator ID of the first query indicator) of the first query indicator. In response to the offline indicator data query request, server 30 retrieves the first indicator metadata associated with the first indicator identifier from the metadata table based on the first indicator identifier of the first query indicator through a unified data service.

[0147] The first metric metadata includes the offline storage mapping information of the first query metric. Understandably, the offline storage mapping information of the first query metric includes the offline metric database identifier where the first query metric resides, the offline table identifier where the first query metric resides, the offline partition information of the first query metric, and the offline field identifier of the first query metric, etc.

[0148] S814: Through unified data service, query the corresponding offline indicator data from the offline indicator database based on the offline storage mapping information of the first query indicator.

[0149] Optionally, the server 30 uses a unified data service to query the corresponding offline indicator data from the offline indicator database based on the field information of the first query indicator and the position index information of the field information of the first query indicator in the offline table.

[0150] Optionally, the server 30 generates a structured query statement for the offline indicator database based on the offline indicator database identifier where the first query indicator is located, the offline table identifier where the first query indicator is located, the offline partition information of the first query indicator, and the offline field identifier of the first query indicator through a unified data service. Then, the server 30 queries the corresponding offline indicator data from the offline indicator database based on the structured query language (SQL) statement for the offline indicator database through the unified data service.

[0151] In this embodiment, in response to an offline indicator data query request, the server 30 obtains the first indicator metadata associated with the first indicator identifier from the metadata table through a unified data service, and queries the corresponding offline indicator data from the offline indicator database based on the offline storage mapping information of the first query indicator in the first indicator metadata through the unified data service, thereby improving the accuracy and efficiency of data query and realizing reliable data processing.

[0152] S816: In response to an online indicator data query request for online decision-making, retrieve the second indicator metadata associated with the second indicator identifier from the metadata table through the unified data service.

[0153] Optionally, server 30 receives an online indicator data query request sent by the operating terminal for online decision-making. This online indicator data query request carries a second indicator identifier (such as the indicator ID of the second query indicator) of the second query indicator. In response to the online indicator data query request, server 30 retrieves the second indicator metadata associated with the second indicator identifier from the metadata table based on the second indicator identifier of the second query indicator through a unified data service.

[0154] The second metric metadata includes the online storage mapping information of the second query metric. Understandably, this online storage mapping information includes the online metric database identifier where the second query metric resides, the online table identifier where the second query metric resides, the online partition information of the second query metric, and the online field identifier of the second query metric, etc.

[0155] S818: Through unified data services, query the corresponding online indicator data from the online indicator database based on the online storage mapping information of the second query indicator.

[0156] Optionally, the server 30 uses a unified data service to query the corresponding offline indicator data from the offline indicator database based on the field information of the first query indicator and the position index information of the field information of the first query indicator in the offline table.

[0157] Optionally, the server 30 generates a structured query statement for the offline indicator database based on the offline indicator database identifier where the first query indicator is located, the offline table identifier where the first query indicator is located, the offline partition information of the first query indicator, and the offline field identifier of the first query indicator through a unified data service. Then, the server 30 queries the corresponding offline indicator data from the offline indicator database based on the structured query language (SQL) of the offline indicator database through the unified data service.

[0158] In this embodiment, in response to an offline indicator data query request, the server 30 obtains the first indicator metadata associated with the first indicator identifier from the metadata table through a unified data service, and queries the corresponding offline indicator data from the offline indicator database based on the offline storage mapping information of the first query indicator in the first indicator metadata through the unified data service, thereby improving the accuracy and efficiency of data query and realizing reliable data processing.

[0159] S820: In response to the obsolescence request for the first indicator, update the status information of the first indicator in the meta information table to the obsolescence status applicable to the specified scenario.

[0160] Optionally, in response to a request from the operating terminal to discard the first indicator, the server 30 updates the status of the first indicator in the metadata table to a discarded status applicable only to a specified scenario. The specified scenario can be pre-specified in a whitelist, and can be an old scenario.

[0161] Understandably, server 30 manages the lifecycle of metrics by storing offline and online metric metadata in the metadata table, in order to avoid the problem of ever-increasing complexity caused by metric expansion.

[0162] For example, please see Figure 9 A complete metric lifecycle includes the definition phase, the deployment phase, the obsolescence phase, and the delisting phase. Specifically, in the definition phase, the metric's basic metadata has been stored in the database, and its status in the metadata table is marked as pending verification. In the deployment phase, the metric has passed verification and is available for use in new scenarios; its status in the metadata table is marked as deployed. In the obsolescence phase, the metric can no longer be used in new scenarios and is only applicable to specified old scenarios; its status in the metadata table is marked as obsolescence. In the delisting phase, the metric is no longer used in any scenario; its status in the metadata table is delisted.

[0163] S822: In response to the removal request for the second indicator, update the status information of the second indicator in the meta information table to the prohibited removal status.

[0164] Optionally, in response to a delisting request for the second indicator sent by the operating terminal, the server 30 updates the status of the second indicator in the metadata table to a delisted state, prohibiting its use in any scenario. It is worth noting that after the second indicator is updated to a delisted state, the server 30 does not immediately delete the corresponding indicator metadata and indicator data from the metadata table and database. Instead, it deletes the corresponding indicator metadata and indicator data from the metadata table and database after the second indicator has been in the delisted state for a preset duration, in order to facilitate tracking and tracing the data processing results.

[0165] In this embodiment, in response to the obsolescence request sent by the operating terminal for the first indicator, the server 30 updates the status information of the first indicator in the metadata table to an obsolescence status applicable to the specified scenario; in response to the delisting request sent by the operating terminal for the second indicator, the server updates the status information of the second indicator in the metadata table to a delisting status that is prohibited from use; and after the duration of the second indicator being in the delisting status reaches a preset duration, the server deletes the indicator metadata and indicator data corresponding to the second indicator from the metadata table and the database. This effectively avoids the problem of increasing complexity caused by indicator expansion and achieves reliable data processing.

[0166] In the above data processing method, during the data application phase, the server responds to offline indicator data query requests by retrieving the first indicator metadata associated with the first indicator identifier from the metadata table through a unified data service. Based on the offline storage mapping information of the first queried indicator in the first indicator metadata, the server queries the corresponding offline indicator data from the offline indicator database, improving the accuracy and efficiency of data queries. In response to the obsolescence request for the first indicator and the delisting request for the second indicator sent by the operating terminal, the server updates the status information of the first indicator in the metadata table to an obsolescence status suitable for the specified scenario, and updates the status information of the second indicator to a delisting status that prohibits its use. After the second indicator remains in the delisting status for a preset duration, the server deletes the corresponding indicator metadata and indicator data from the metadata table and database. This effectively avoids the increasing complexity caused by indicator expansion and achieves reliable data processing.

[0167] In one embodiment, such as Figure 10 As shown, another data processing method is provided, which can be applied to... Figure 3 Taking server 30 as an example, the explanation includes the following steps:

[0168] S1002: In response to the database listing request, register the offline indicator database to the unified data service.

[0169] Specifically, S1002 is the same as S402, and will not be repeated here.

[0170] S1004: The offline indicator database is processed through a unified data service to obtain an online indicator database.

[0171] Specifically, S1004 is the same as S404, and will not be repeated here.

[0172] For example, please see Figure 11 This is a schematic diagram of a domain model for a data processing method in the data processing stage, provided as an exemplary embodiment of this specification. Figure 11 As shown, the left half represents the logic for generating indicator metadata, and the right half represents the logic for consuming indicator metadata.

[0173] Please see Figure 11In the left half, after the offline indicator database is registered with the unified data service, the unified data service abstracts the offline indicator data stored in each offline table into data assets defined by asset names and ODPS (Open Data Processing Service) table names. It then constructs consumption views defined by table fields and field types using these data assets. The consumption view integrates the decision view in online scenarios and the analysis view in offline scenarios. Specifically, the decision view in online scenarios is defined by online storage tables configured by Lindorm, and it originates from both real-time and offline indicator decision views. The real-time indicator decision view is defined by online storage tables configured by Lindorm and indicator consistency offline tables. This indicator consistency offline table ensures that online and offline tables share the same set of indicators. Through the decision view in online scenarios, online decision indicator features defined by feature names can be generated. Furthermore, the decision view in online scenarios can also query indicator data in the Lindorm database using SQL combined with task types to complete real-time calculation tasks. The analysis view in offline scenarios is defined by offline storage tables configured by Holo, and it is specifically constructed from the pre-configured indicator names of the analysis indicators.

[0174] Please see Figure 11 The right half of the document outlines the various types of metrics supported by the Unified Data Service, each defined by its name and value range type. Specifically, the metrics supported by the Unified Data Service can be either basic metrics defined by the original table fields and field types, or composite metrics defined by calculation logic and field types. These composite metrics can be constructed based on derived metrics and derived metrics defined by combining different dimension values. The dimension system supported by the metrics includes both basic dimensions defined by the original table fields and extended dimensions constructed from different dimension values ​​in derived metrics, while also being subject to the consumption conditions supported by the metric. The consumption conditions supported by an metric depend on its metric type (e.g., basic metric, derived metric) and its consumption rules. The consumption rules of an metric are defined by its label name and calculation logic, and are associated with its metric type (e.g., basic metric, derived metric) and the consumer group. The consumer group of an metric is identified by different group names, which determine the metric's available scope and usage. The consumer group can originate from physical groups defined based on a group table or logical groups defined based on group logic.

[0175] S1006: In response to an offline indicator data query request for offline analysis, determine the first domain-specific language query statement corresponding to the offline indicator data query request through the unified data service.

[0176] The offline indicator data query request carries the first indicator identifier (such as the first indicator ID) of the first query indicator, and the offline indicator data query request adopts the pre-configured service semantic definition.

[0177] Understandably, offline and online indicator data are stored using different storage engines. Since two separate sets of SQL queries are written for the offline and online indicator databases, the retrieved indicator data may be inconsistent. Therefore, a unified data service is needed to standardize the data query criteria and ensure data consistency.

[0178] Optionally, in response to an offline indicator data query request sent by an operational terminal, server 30 determines the first domain-specific language query statement corresponding to the offline indicator data query request through a unified data service. The aforementioned first domain-specific language query statement is implemented using Antlr4 and includes a first indicator identifier for the first queried indicator.

[0179] S1008: Through unified data services, obtain the first indicator metadata associated with the first indicator identifier from the metadata table based on the query statement of the first domain-specific language.

[0180] Optionally, server 30 obtains the first indicator metadata associated with the first indicator identifier from the metadata table through a unified data service based on a query statement in the first domain-specific language, so as to shield the SQL differences caused by different storage engines through a standardized domain-specific language.

[0181] S1010: Through unified data services, generate a structured query statement for the offline indicator database based on the offline storage mapping information of the first query indicator.

[0182] The offline storage mapping information of the first query indicator refers to the storage mapping information of the first query indicator in the offline indicator database, which includes, but is not limited to, the offline indicator database identifier where the first query indicator is located, the offline table identifier where the first query indicator is located, the offline partition information of the first query indicator, and the offline field identifier of the first query indicator.

[0183] Optionally, the server 30 generates a structured query statement for the offline indicator database based on the offline indicator database identifier where the first query indicator is located, the offline table identifier where the first query indicator is located, the offline partition information of the first query indicator, and the offline field identifier of the first query indicator through a unified data service.

[0184] S1012: Through unified data services, query the corresponding offline indicator data from the offline indicator database based on the structured query statement of the offline indicator database.

[0185] Optionally, server 30 uses a unified data service to query the corresponding offline indicator data from the offline indicator database (such as Hologress) based on the structured query statement of the aforementioned offline indicator database for offline analysis.

[0186] In this embodiment, server 30 uses a unified data service to transform the offline indicator data query request sent by the operation terminal into a standardized domain-specific language query statement to obtain the corresponding offline storage mapping information from the metadata table. Subsequently, based on the corresponding offline storage mapping information, it generates a structured query statement for the offline indicator database to query the corresponding offline indicator data from the offline indicator database. This provides standardized data query capabilities, helps to shield the differences between the underlying online indicator database and the offline indicator database in terms of different structured query languages, ensures the consistency of data in offline analysis and online decision-making scenarios, and eliminates the differences between online data processing and offline data processing in the data application stage.

[0187] S1014: In response to an online indicator data query request for online analysis, determine the second domain-specific language query statement corresponding to the online indicator data query request through the unified data service.

[0188] The online indicator data query request carries the second indicator identifier (such as the second indicator ID) of the second query indicator, and the online indicator data query request adopts a pre-configured service semantic definition.

[0189] Optionally, in response to an online indicator data query request sent by an operational terminal, server 30 determines the second domain-specific language query statement corresponding to the online indicator data query request through a unified data service. The aforementioned second domain-specific language query statement is implemented using Antlr4 and includes a second indicator identifier for the second queried indicator.

[0190] S1016: Through unified data services, retrieve the second indicator metadata associated with the second indicator identifier from the metadata table based on the second domain-specific language query statement.

[0191] Optionally, server 30 uses a unified data service to retrieve the second indicator metadata associated with the second indicator identifier from the metadata table based on a query statement in the second domain-specific language, so as to shield the SQL differences caused by different storage engines through a standardized domain-specific language.

[0192] S1018: Through unified data services, generate structured query statements for the online indicator database based on the online storage mapping information of the second query indicator.

[0193] The online storage mapping information of the second query indicator refers to the storage mapping information of the second query indicator in the online indicator database, which includes, but is not limited to, the online indicator database identifier where the second query indicator is located, the online table identifier where the second query indicator is located, the online partition information of the second query indicator, and the online field identifier of the second query indicator.

[0194] Optionally, the server 30 generates a structured query statement for the online indicator database based on the online indicator database identifier where the second query indicator is located, the online table identifier where the second query indicator is located, the online partition information of the second query indicator, and the online field identifier of the second query indicator through a unified data service.

[0195] S1020: Through unified data services, query the corresponding online indicator data from the online indicator database based on the structured query statements of the online indicator database.

[0196] Optionally, server 30 uses a unified data service to query corresponding offline indicator data from an online indicator database (such as Lindorm) based on the structured query statements of the aforementioned online indicator database for offline analysis.

[0197] In this embodiment, server 30 uses a unified data service to transform the online indicator data query request sent by the operation terminal into a standardized domain-specific language query statement to obtain the corresponding online storage mapping information from the metadata table. Subsequently, based on the corresponding online storage mapping information, it generates a structured query statement for the online indicator database to query the corresponding online indicator data from the online indicator database. This provides standardized data query capabilities, helps to shield the differences between the underlying online indicator database and the offline indicator database in terms of different structured query languages, ensures the consistency of data in offline analysis and online decision-making scenarios, and eliminates the differences between online data processing and offline data processing in the data application stage.

[0198] For example, please see Figure 12 This is a semantic transformation diagram of a data processing method in the data application stage, provided as an exemplary embodiment of this specification. Figure 12 As shown, server 30 transforms service semantics into a unified domain-specific language (DSL) through a unified data service, and then transforms the unified domain-specific language into Hologress SQL, the structured query language of the offline metrics database, and Lindorm SQL, the structured query language of the online metrics database, through a conversion layer.

[0199] In the aforementioned data processing method, during the data processing phase, the server, through a unified data service, extends various basic models by combining data assets, indicator models, consumption views, and actual scenarios. This allows for the registration and processing of an offline indicator database, including at least one offline table, generating online storage tables configured with Lindorm and offline storage tables configured with Holo. This ensures the consistency of indicator data used in online decision-making and offline analysis, providing unified data management capabilities. During the data application phase, the server, through the unified data service, transforms service semantics into a standardized domain-specific language implemented using Antlr4. Subsequently, based on the corresponding storage mapping information, it generates structured query statements for the corresponding indicator database, providing standardized data query capabilities and eliminating differences between online and offline data processing in the data application phase. The entire process ensures consistent data definitions for the same indicator, improving the consistency and accuracy of data processing results and achieving reliable data processing.

[0200] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0201] Based on the inventive concept of the above data processing method, such as Figure 13 As shown in the embodiments of this specification, a data processing apparatus 1300 for implementing the data processing method described above is also provided. The data processing apparatus 1300 includes:

[0202] Registration module 1301 is used to register the offline indicator database to the unified data service in response to the database listing request;

[0203] Processing module 1302 is used to process offline indicator database through unified data service to obtain online indicator database;

[0204] The query module 1303 is used to respond to indicator data query requests and query the corresponding indicator data from the offline indicator database and / or the online indicator database through the unified data service.

[0205] In one possible implementation, the database listing request carries indicator configuration information; the registration module 1301 is specifically used to respond to the database listing request, count the offline indicator data stored in the offline indicator database; generate offline indicator metadata corresponding to the offline indicator data according to the indicator configuration information; the offline indicator metadata includes the offline storage mapping information of each indicator; and write the offline indicator metadata into the metadata table for unified data service calls.

[0206] In one possible implementation, the database upload request also carries data processing rules; the processing module 1302 is specifically used to generate online indicator metadata corresponding to offline indicator metadata in the metadata table according to indicator configuration information through a unified data service; the online indicator metadata includes online storage mapping information of each indicator; and the offline indicator database is processed by the unified data service according to the online indicator metadata and data processing rules to generate the online indicator database.

[0207] In one possible implementation, the offline indicator metadata and the online indicator metadata also include the status information of each indicator; the data processing device 1300 also includes a status update module, which, in response to a request to discard the first indicator, updates the status information of the first indicator in the metadata table to a discarded state applicable to a specified scenario; and in response to a request to remove the second indicator, updates the status information of the second indicator in the metadata table to a removed state that is prohibited from use.

[0208] In one possible implementation, the data processing device 1300 further includes a database construction module, which is used to batch process the historical detailed data of the offline data source within a preset time period according to a preset time interval to obtain at least one offline table; each offline table stores corresponding offline indicator data; and an offline indicator database is constructed based on at least one offline table.

[0209] In one possible implementation, the query module 1303 is specifically used to query the corresponding offline indicator data from the offline indicator database through a unified data service in response to an offline indicator data query request for offline analysis; and to query the corresponding online indicator data from the online indicator database through a unified data service in response to an online indicator data query request for online decision-making.

[0210] In one possible implementation, the offline indicator data query request carries the first indicator identifier of the first query indicator; the query module 1303 is specifically used to respond to the offline indicator data query request for offline analysis, and obtain the first indicator metadata associated with the first indicator identifier from the metadata table through the unified data service; the first indicator metadata includes the offline storage mapping information of the first query indicator; and through the unified data service, query the corresponding offline indicator data from the offline indicator database according to the offline storage mapping information of the first query indicator.

[0211] In one possible implementation, the online indicator data query request carries the second indicator identifier of the second query indicator; the query module 1303 is specifically used to respond to the online indicator data query request for online decision-making, and obtain the second indicator metadata associated with the second indicator identifier from the metadata table through a unified data service; the second indicator metadata includes the online storage mapping information of the second query indicator; and through the unified data service, query the corresponding online indicator data from the online indicator database according to the online storage mapping information of the second query indicator.

[0212] In one possible implementation, the query module 1303 is specifically configured to respond to an offline indicator data query request for offline analysis by: determining a first domain-specific language query statement corresponding to the offline indicator data query request through a unified data service; obtaining first indicator metadata associated with the first indicator identifier from the metadata table based on the first domain-specific language query statement through the unified data service; generating a structured query statement for the offline indicator database based on the offline storage mapping information of the first queried indicator through the unified data service; and querying the corresponding offline indicator data from the offline indicator database based on the structured query statement for the offline indicator database through the unified data service.

[0213] In one possible implementation, the query module 1303 is specifically configured to respond to an online indicator data query request for online analysis by: determining a second domain-specific language query statement corresponding to the online indicator data query request through a unified data service; obtaining second indicator metadata associated with the second indicator identifier from the metadata table based on the second domain-specific language query statement through the unified data service; generating a structured query statement for the online indicator database based on the online storage mapping information of the second queried indicator through the unified data service; and querying the corresponding online indicator data from the online indicator database based on the structured query statement for the online indicator database through the unified data service.

[0214] Each module in the aforementioned data processing device 1300 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0215] This specification also provides an electronic device, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, this electronic device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The application database stores data for data processing. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. The processor executes computer programs to implement a data processing method.

[0216] Those skilled in the art will understand that Figure 14 The structures shown are merely block diagrams of a portion of the structure related to the scheme described in this specification, and do not constitute a limitation on the electronic devices to which the scheme described in this specification is applied. Specific electronic devices may include more or fewer components than those shown in the figures, or may combine certain components, or may have different component arrangements.

[0217] In one possible implementation, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0218] This specification also provides a computer storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer storage medium.

[0219] In one possible implementation, a computer storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps in the above method embodiments.

[0220] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0221] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0222] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0223] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims.

[0224] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. A data processing method, characterized in that, The method includes: In response to the database listing request, the offline indicator database is registered with the unified data service; The offline indicator database is processed using the unified data service to obtain an online indicator database. In response to an indicator data query request, the corresponding indicator data is queried from the offline indicator database and / or the online indicator database through the unified data service; The method further includes: According to a preset time interval, the historical detailed data of the offline data source within the preset time period are processed in batches to obtain at least one offline table; each offline table stores the corresponding offline indicator data. The offline indicator database is constructed based on the at least one offline table.

2. The method as described in claim 1, characterized in that, The database upload request carries indicator configuration information; The step of registering the offline metrics database to the unified data service in response to the database listing request includes: In response to the database listing request, the offline indicator data stored in the offline indicator database is statistically analyzed; Based on the indicator configuration information, generate offline indicator metadata corresponding to the offline indicator data; the offline indicator metadata includes offline storage mapping information for each indicator; The offline indicator metadata is written into the metadata table for use by the unified data service.

3. The method as described in claim 2, characterized in that, The database listing request also carries data processing rules; The process of processing the offline indicator database through the unified data service to obtain the online indicator database includes: Through the unified data service, online indicator metadata corresponding to the offline indicator metadata is generated in the metadata table according to the indicator configuration information; the online indicator metadata includes the online storage mapping information of each indicator; The online indicator database is generated by processing the offline indicator database using the unified data service based on the online indicator metadata and the data processing rules.

4. The method as described in claim 3, characterized in that, The offline indicator metadata and the online indicator metadata also include the status information of each indicator; The method further includes: In response to a request to discard the first indicator, the status information of the first indicator in the meta information table is updated to a discarded status applicable to the specified scenario. In response to the removal request for the second indicator, the status information of the second indicator in the meta information table is updated to a prohibited removal status.

5. The method as described in claim 1, characterized in that, The step of responding to an indicator data query request by querying the corresponding indicator data from the offline indicator database or the online indicator database through the unified data service includes: In response to an offline indicator data query request for offline analysis, the corresponding offline indicator data is queried from the offline indicator database through the unified data service; In response to an online indicator data query request for online decision-making, the corresponding online indicator data is queried from the online indicator database through the unified data service.

6. The method as described in claim 5, characterized in that, The offline indicator data query request carries the first indicator identifier of the first query indicator; The step of responding to an offline indicator data query request for offline analysis by querying the corresponding offline indicator data from the offline indicator database through the unified data service includes: In response to an offline indicator data query request for offline analysis, the first indicator metadata associated with the first indicator identifier is obtained from the metadata table through the unified data service; the first indicator metadata includes the offline storage mapping information of the first queried indicator. Through the unified data service, the corresponding offline indicator data is queried from the offline indicator database based on the offline storage mapping information of the first query indicator.

7. The method as described in claim 5, characterized in that, The online indicator data query request carries the second indicator identifier of the second query indicator; The step of responding to an online indicator data query request for online decision-making by querying the corresponding online indicator data from the online indicator database through the unified data service includes: In response to an online indicator data query request for online decision-making, the second indicator metadata associated with the second indicator identifier is obtained from the metadata table through the unified data service; the second indicator metadata includes the online storage mapping information of the second queried indicator. Through the unified data service, the corresponding online indicator data is queried from the online indicator database based on the online storage mapping information of the second query indicator.

8. The method as described in claim 6, characterized in that, The step of responding to an offline indicator data query request for offline analysis by retrieving the first indicator metadata associated with the first indicator identifier from the metadata table through the unified data service includes: In response to an offline indicator data query request for offline analysis, the unified data service determines the first domain-specific language query statement corresponding to the offline indicator data query request. Through the unified data service, the first indicator metadata associated with the first indicator identifier is obtained from the metadata table according to the first domain-specific language query statement. The step of querying corresponding offline indicator data from the offline indicator database through the unified data service, based on the offline storage mapping information of the first query indicator, includes: Through the unified data service, a structured query statement for the offline indicator database is generated based on the offline storage mapping information of the first query indicator. Through the unified data service, the corresponding offline indicator data is queried from the offline indicator database according to the structured query statement of the offline indicator database.

9. The method as described in claim 7, characterized in that, The step of responding to an online indicator data query request for online decision-making by retrieving the second indicator metadata associated with the second indicator identifier from the metadata table through the unified data service includes: In response to an online indicator data query request for online analysis, the unified data service determines the second domain-specific language query statement corresponding to the online indicator data query request. Through the unified data service, the second indicator metadata associated with the second indicator identifier is obtained from the metadata table according to the second domain-specific language query statement; The step of querying corresponding online indicator data from the online indicator database through the unified data service, based on the online storage mapping information of the second query indicator, includes: Through the unified data service, a structured query statement for the online indicator database is generated based on the online storage mapping information of the second query indicator; Through the unified data service, the corresponding online indicator data is queried from the online indicator database according to the structured query statement of the online indicator database.

10. A data processing apparatus, characterized in that, The device includes: The registration module is used to register offline indicator databases to the unified data service in response to database listing requests. The processing module is used to process the offline indicator database through the unified data service to obtain an online indicator database; The query module is used to respond to indicator data query requests and query corresponding indicator data from the offline indicator database and / or the online indicator database through the unified data service; The database construction module is used to batch process historical detailed data of offline data sources within a preset time period according to a preset time interval to obtain at least one offline table. Each offline table stores corresponding offline indicator data, and the offline indicator database is constructed based on the at least one offline table.

11. An electronic device, characterized in that, include: A processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of claims 1-9.

12. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-9.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-9.

Images