Data processing method and device, equipment, medium and program product

Abstract

The embodiment of the invention discloses a data processing method and device, equipment, a medium and a program product. The reliability of data processing can be improved. The method comprises the following steps: in response to a database putaway request, registering an offline index database to a unified data service; performing data processing on the offline index database through a unified data service to obtain an online index database; and in response to the index data query request, querying corresponding index data from the offline index database and / or the online index database through the unified data service.

Description

Technical Field

[0001] This specification relates to the technical field of data processing, and in particular, to a data processing method, apparatus, device, medium, and program product. Background Art

[0002] The processing requirements for offline analysis and online decision-making of data are often different. Offline analysis usually involves in-depth analysis of a large amount of historical data, while online decision-making focuses on making decisions quickly using the latest data.

[0003] To adapt to the two different data processing scenarios of offline and online, during the data processing stage, data in the data source is usually pre-analyzed offline in advance to obtain offline metric data, and the offline metric data is stored in the offline data warehouse; then the offline metric data is processed to obtain online metric data, and the online metric data is stored in the online database. During the data application stage, users can call the offline data warehouse through writing corresponding scripts for offline analysis, or call the online database for online decision-making.

[0004] However, due to the different underlying storage engines and technology stacks used for offline processing and online processing, the current data processing solutions are prone to inconsistent data calibers for the same metric, affecting the consistency and accuracy of data processing, and a more reliable data processing method needs to be provided. Summary of the Invention

[0005] Embodiments of this specification provide a data processing method, apparatus, device, medium, and program product, which can improve the reliability of data processing.

[0006] In a first aspect, embodiments of this specification provide a data processing method, including: In response to a database on-shelf request, register the offline metric database to the unified data service; Perform data processing on the offline metric database through the unified data service to obtain an online metric database; In response to a metric data query request, query the corresponding metric data from the offline metric database and / or the online metric database through the unified data service.

[0007] In a possible implementation, the database on-shelf request carries metric configuration information; In response to a database on-shelf request, registering the offline metric database to the unified data service includes: In response to a database on-shelf request, count the offline metric data stored in the offline metric database; Generate offline metric metadata corresponding to the offline metric data according to the metric configuration information; the offline metric metadata includes offline storage mapping information for each metric; Write the offline metric metadata into the meta-information table for the unified data service to call.

[0008] In a possible implementation, the database onboarding request also carries data processing rules; Perform data processing on the offline metric database through the unified data service to obtain an online metric database, including: Through the unified data service, generate online metric metadata corresponding to the offline metric metadata in the meta-information table according to the metric configuration information; the online metric metadata includes the online storage mapping information of each metric; Through the unified data service, perform data processing on the offline metric database according to the online metric metadata and data processing rules to generate an online metric database.

[0009] In a possible implementation, the offline metric metadata and the online metric metadata also include the status information of each metric; The method further includes: In response to a discard request for a first metric, update the status information of the first metric in the meta-information table to a discarded status applicable to a specified scenario; In response to a take-off request for a second metric, update the status information of the second metric in the meta-information table to a taken-off status where it is prohibited from use.

[0010] In a possible implementation, the above method further includes: At preset time intervals, perform batch processing on the historical detail data of the offline data source within a preset time period to obtain at least one offline table; each offline table stores the corresponding offline metric data; Build an offline metric database based on at least one offline table.

[0011] In a possible implementation, in response to a metric data query request, query the corresponding metric data from the offline metric database or the online metric database through the unified data service, including: In response to an offline metric data query request for offline analysis, query the corresponding offline metric data from the offline metric database through the unified data service; In response to an online metric data query request for online decision-making, query the corresponding online metric data from the online metric database through the unified data service.

[0012] In a possible implementation, the offline metric data query request carries the first metric identifier of the first query metric; In response to an offline metric data query request for offline analysis, query the corresponding offline metric data from the offline metric database through the unified data service, including: In response to an offline metric data query request for offline analysis, obtain the first metric metadata associated with the first metric identifier from the meta-information table through the unified data service; the first metric metadata includes the offline storage mapping information of the first query metric. Through the unified data service, query the corresponding offline metric data from the offline metric database according to the offline storage mapping information of the first query metric.

[0013] In a possible implementation, the online metric data query request carries the second metric identifier of the second query metric. In response to an online metric data query request for online decision-making, query the corresponding online metric data from the online metric database through the unified data service, including: In response to an online metric data query request for online decision-making, obtain the second metric metadata associated with the second metric identifier from the meta-information table through the unified data service; the second metric metadata includes the online storage mapping information of the second query metric. Through the unified data service, query the corresponding online metric data from the online metric database according to the online storage mapping information of the second query metric.

[0014] In a possible implementation, in response to an offline metric data query request for offline analysis, obtain the first metric metadata associated with the first metric identifier from the meta-information table through the unified data service, including: In response to an offline metric data query request for offline analysis, determine the first domain-specific language query statement corresponding to the offline metric data query request through the unified data service. Through the unified data service, obtain the first metric metadata associated with the first metric identifier from the meta-information table according to the first domain-specific language query statement. Through the unified data service, query the corresponding offline metric data from the offline metric database according to the offline storage mapping information of the first query metric, including: Through the unified data service, generate a structured query statement for the offline metric database according to the offline storage mapping information of the first query metric. Through the unified data service, query the corresponding offline metric data from the offline metric database according to the structured query statement of the offline metric database.

[0015] In a possible implementation, in response to an online metric data query request for online decision-making, obtain the second metric metadata associated with the second metric identifier from the meta-information table through the unified data service, including: In response to an online metric data query request for online analysis, determine the second domain-specific language query statement corresponding to the online metric data query request through the unified data service. Through the unified data service, obtain the second metric metadata associated with the second metric identifier from the meta-information table according to the second domain-specific language query statement; Through the unified data service, query the corresponding online metric data from the online metric database according to the online storage mapping information of the second query metric, including: Through the unified data service, generate a structured query statement for the online metric database according to the online storage mapping information of the second query metric; Through the unified data service, query the corresponding online metric data from the online metric database according to the structured query statement of the online metric database.

[0016] In a second aspect, an embodiment of this specification provides a data processing device, including: A registration module, configured to register the offline metric database to the unified data service in response to a database on-shelf request; A processing module, configured to perform data processing on the offline metric database through the unified data service to obtain an online metric database; A query module, configured to query the corresponding metric data from the offline metric database and / or the online metric database through the unified data service in response to a metric data query request.

[0017] In a third aspect, an embodiment of this specification provides an electronic device, including: a processor and a memory; the above memory stores a computer program, and when the processor executes the computer program, the method steps provided in the first aspect of the embodiment of this specification are implemented.

[0018] In a fourth aspect, an embodiment of this specification provides a computer storage medium, which stores multiple instructions, and the above instructions are suitable for being loaded and executed by a processor to implement the method steps provided in the first aspect of the embodiment of this specification.

[0019] In a fifth aspect, an embodiment of this specification provides a computer program product, including a computer program; when the above computer program is executed by a processor, the method steps provided in the first aspect of the embodiment of this specification are implemented.

[0020] The above data processing method, apparatus, electronic device, computer storage medium and computer program product, in the data processing stage, in response to a database online request, register an offline metric database to a unified data service, and perform data processing on the offline metric database through the unified data service to obtain an online metric database, which can provide unified data management capabilities, enabling the same set of metric-corresponding data to be stored in both the offline database and the online database; in the data application stage, in response to a metric data query request, query corresponding metric data from the offline metric database and / or the online metric database through the unified data service, which can provide unified data query capabilities, ensuring the consistency of the data caliber for the same metric. The entire data processing process improves the consistency and accuracy of the data processing results, thereby achieving reliable data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the technical solutions in the embodiments of this specification, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of this specification. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 Schematic diagram of the implementation process of a data processing method provided in the related art; Figure 2 Schematic diagram of the implementation process of a data processing method provided in an exemplary embodiment of this specification; Figure 3 Schematic diagram of the application environment of a data processing method provided in an exemplary embodiment of this specification; Figure 4 Schematic diagram of the flow of a data processing method provided in an exemplary embodiment of this specification; Figure 5 Schematic diagram of the architecture of a data processing method in the data processing stage provided in an exemplary embodiment of this specification; Figure 6 Schematic diagram of the architecture of a data processing method in the data application stage provided in an exemplary embodiment of this specification; Figure 7 Schematic diagram of the flow of another data processing method provided in an exemplary embodiment of this specification; Figure 8 Schematic diagram of the flow of another data processing method provided in an exemplary embodiment of this specification; Figure 9 Schematic diagram of a metric life cycle provided in an exemplary embodiment of this specification; Figure 10A schematic flowchart of another data processing method provided by an exemplary embodiment of this specification; Figure 11 A schematic diagram of a domain model in the data processing stage of a data processing method provided by an exemplary embodiment of this specification; Figure 12 A semantic transformation diagram in the data application stage of a data processing method provided by an exemplary embodiment of this specification; Figure 13 A schematic structural diagram of a data processing apparatus provided by an exemplary embodiment of this specification; Figure 14 A schematic structural diagram of an electronic device provided by an exemplary embodiment of this specification. Detailed implementation manners

[0023] In order to make the purpose, technical solutions and advantages of this specification clearer, the following further details this specification in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this specification and are not used to limit this specification.

[0024] In the description of this specification, it should be understood that terms such as "first", "second", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this specification can be understood according to specific situations. In addition, in the description of this specification, unless otherwise specified, "a plurality of" means two or more. " / ", describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0025] Offline analysis and online decision-making are two common data processing methods. Offline analysis does not require real-time data processing, but requires strong data processing capabilities to analyze and calculate large-scale data. For example, analyzing and calculating the historical detailed data of a large number of users to understand the historical consumption behaviors of different user groups and predict the future consumption trends of different user groups. Online decision-making has high requirements for data processing speed. It usually quickly queries single-user data and then quickly makes a decision based on the queried data. For example, when a user sends a virtual resource application request, a quick decision is made based on the user's recent key indicator data.

[0026] In the related art, the implementation process of a provided data processing method is as Figure 1 shown. Please refer to Figure 1, in the data processing stage, the data processing platform batch processes a large amount of detailed data stored in the offline data warehouse at preset time intervals to obtain offline metric data, stores the offline metric data in the offline metric database, and processes the offline metric data to obtain online metric data, and stores the online metric data in the online metric database. In the data application stage, the data processing platform responds to an offline metric data query request for offline analysis, and queries the offline metric data from the offline metric database through the structured query language corresponding to the offline metric database for offline analysis; responds to an online metric data query request for online decision-making, and queries the online metric data from the online metric database through the structured query language corresponding to the online metric database for online decision-making.

[0027] Understandably, since Figure 1 the data storage engines and technology stacks used in the data processing solutions shown are different in the offline data processing and online data processing processes, it is easy for the data stored in the offline metric database and the online metric database to be inconsistent, and the data queried from the offline metric database and the online metric database to be inconsistent, resulting in inconsistent data calibers for the same metric, affecting the consistency and accuracy of data processing, and there is a problem of unreliable data processing.

[0028] To address the above problems, this specification provides a data processing method as Figure 2 shown. Please refer to Figure 2 , in the data processing stage, the data processing platform batch processes a large amount of detailed data stored in the offline data warehouse at preset time intervals to obtain an offline metric database including at least one offline table. In response to a database on-shelf request, the offline metric database is registered with the unified data service, and the unified data service processes the offline metric database to obtain an online metric database including at least one online table, and writes the metric metadata (including the offline metric metadata corresponding to the offline metric database and the online metric metadata corresponding to the online metric database) into the meta-information table during this process. Among them, the offline metric metadata includes the storage mapping information of each metric in the offline metric database (i.e., the offline storage mapping information of each metric); the online metric metadata includes the storage mapping information of each metric in the online metric database (i.e., the online storage mapping information of each metric).

[0029] In the data application stage, in response to an offline metric data query request for offline analysis, the data processing platform determines, through the unified data service, a first domain-specific language query statement corresponding to the offline metric data query request, and obtains, from the meta-information table according to the first domain-specific language query statement, first metric metadata associated with a first metric identifier; the first metric metadata includes offline storage mapping information of a first query metric. Then, through the unified data service, according to the offline storage mapping information of the first query metric, a structured query statement for the offline metric database is generated, and according to the structured query statement for the offline metric database, corresponding offline metric data is queried from the offline metric database for offline analysis. In response to an online metric data query request for online analysis, the data processing platform determines, through the unified data service, a second domain-specific language query statement corresponding to the online metric data query request, and obtains, from the meta-information table according to the second domain-specific language query statement, second metric metadata associated with a second metric identifier; the second metric metadata includes online storage mapping information of a second query metric. Then, through the unified data service, according to the online storage mapping information of the second query metric, a structured query statement for the online metric database is generated, and according to the structured query statement for the online metric database, corresponding online metric data is queried from the online metric database for online decision-making.

[0030] Figure 2 In the shown data processing solution, in the data processing stage, the data processing platform processes the data of the offline metric database through the unified data service to obtain the online metric database, and generates metric metadata corresponding to the offline metric database and the online metric database respectively, which can provide unified data management capabilities, enabling the offline database and the online metric database to store data corresponding to the same set of metrics. In the data application stage, the data processing platform calls, through the unified data service, the meta-information table storing the offline metric metadata and the online metric metadata, and queries the corresponding metric data from the offline metric database and / or the online metric database, which can provide standardized data query capabilities and eliminate the difference problems in the data application stage between online data processing and offline data processing. The whole process can ensure the consistency of the data caliber of the same metric, improve the consistency and accuracy of the data processing results, and achieve reliable data processing.

[0031] It should be noted that the above data processing platform can be adapted to a variety of data processing scenarios. For example, in the e-commerce data processing scenario, the data processing platform can respectively construct an offline metric database and an online metric database based on search and recommendation metrics, query offline metric data from the offline metric database to generate user portraits, and query online metric data from the online metric database to perform real-time search and recommendation. In the network security data processing scenario, the data processing platform can respectively construct an offline metric database and an online metric database based on threat intelligence metrics, query offline metric data from the offline metric database to conduct attack traceability, and query online metric data from the online metric database to detect abnormal behaviors. In the credit risk control data processing scenario, the data processing platform can respectively construct an offline metric database and an online metric database based on risk control metrics, query offline metric data from the offline metric database to conduct offline measurement and analysis, and query online metric data from the online metric database to make online risk decisions.

[0032] The data processing method provided by the embodiments of this specification can be applied to an application environment as Figure 3 shown. Among them, the operation terminal 10 and the user terminal 20 communicate with the server 30 that provides unified data services through a communication network. The data storage system can but is not limited to include the offline metric database and the online metric database that the server 30 needs to process. The data storage system can be integrated on the server 30, or can be placed in the cloud or other network servers.

[0033] In some possible embodiments, in response to a database shelving operation, the operation terminal 10 sends a database shelving request carried by the database shelving operation to the server 30. The server 30 receives the database shelving request sent by the operation terminal 10, and in response to the database shelving request, registers the offline metric database to the unified data service, and processes the data of the offline metric database through the unified data service to obtain the online metric database. The operation terminal 10 and / or the user terminal 20, in response to a metric data query operation, send a metric data query request carried by the metric data query operation to the server 30. The server 30 receives the metric data query request sent by the operation terminal 10 and / or the user terminal 20, and in response to the metric data query request, queries the corresponding metric data from the offline metric database and / or the online metric database through the unified data service.

[0034] It should be noted that the operation terminal 10 and the user terminal 20 can be but are not limited to various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, etc. The server 30 can be implemented by an independent server or a server cluster composed of multiple servers.

[0035] In one embodiment, as Figure 4As shown, a data processing method is provided. Taking the server 30 shown in Figure 3 as an example for illustration, the method includes the following steps: S402: In response to a database on-shelf request, register the offline metric database to the unified data service.

[0036] Among them, the database on-shelf request carries metric configuration information and data processing rules. The above metric configuration information may but is not limited to include the basic identifier of the metric and the consumption logic of the metric. The above data processing rules may but is not limited to include offline storage resources and online storage resources. The offline metric database may but is not limited to be constructed from historical detail data stored in an offline data source. The offline metric database includes at least one offline table, and each offline table stores corresponding offline metric data.

[0037] Optionally, an operator sends a database on-shelf request carrying metric configuration information and data processing rules to the server 30 through the operation terminal 10. The server 30 receives the database on-shelf request. In response to the database on-shelf request, it counts the offline metric data stored in each offline table of the offline metric database to obtain the metric identifiers corresponding to each piece of offline metric data. Then, the server 30 generates offline metric metadata corresponding to the offline metric data according to the basic identifier of the metric and the consumption logic of the metric in the metric configuration information. The offline metric metadata includes metric identifier information of each metric, offline storage mapping information of each metric, status information of each metric, type information of each metric, offline consumption logic information of each metric, etc. Finally, the server 30 writes the offline metric metadata into the meta-information table for the unified data service to call, thereby registering the offline metric database to the unified data service.

[0038] Exemplarily, to meet the analysis and calculation requirements of massive data in the offline scenario (such as second-level analysis of hundreds of billions of data), the high-performance real-time analysis database Hologress can be selected as the offline metric database, which can support high-throughput and low-latency offline analysis and calculation tasks through its powerful OLAP (Online Analytical Processing) ability during the data application stage.

[0039] In this embodiment, the server 30, in response to the database on-shelf request, registers the offline metric database to the unified data service according to the metric configuration information carried in the database on-shelf request, and can write the offline metric metadata corresponding to the offline metric database into the meta-information table, facilitating the automated query of the offline metric data stored in the offline metric database through the unified data service during the data application stage, improving the efficiency and accuracy of offline analysis, and thus realizing reliable data processing.

[0040] S404: Process the data in the offline metric database through the unified data service to obtain the online metric database.

[0041] Optionally, after registering the offline metric database to the unified data service, the server 30 further automatically generates an online metric database based on the offline metric database through the unified data service. Specifically, the server 30 generates online metric metadata corresponding to the offline metric metadata in the meta-information table according to the metric configuration information carried in the database onboarding request. The online metric metadata includes the metric identification information of each metric, the online storage mapping information of each metric, the status information of each metric, the type information of each metric, the online consumption logic information of each metric, etc. Then, the server 30 processes the data in the offline metric database through the unified data service according to the online metric metadata and the data processing rules to generate the online metric database.

[0042] Exemplarily, to meet the online decision-making requirements of low latency and high availability in the online scenario, the database Lindorm with high performance, high throughput, and disaster tolerance capabilities can be selected as the main online metric database, and the high-performance and high-throughput Tbase can be configured as the disaster tolerance cache to achieve millisecond-level response and automatic fault recovery through a dual storage architecture in the data application stage.

[0043] In this embodiment, the server 30 generates online metric metadata corresponding to the offline metric metadata in the meta-information table through the unified data service according to the metric configuration information carried in the database onboarding request, and then processes the offline metric data according to the data processing rules and the online metric metadata carried in the database onboarding request to generate the online metric database. On the one hand, in the data processing stage, through the unified data service, a unified data management capability is provided, so that the data corresponding to the same set of metrics is stored in the offline database and the online database; on the other hand, it is convenient to provide an automated data query capability through the unified data service in the subsequent data application stage to ensure that the data caliber of the same metric is consistent, thereby improving the consistency, accuracy, and efficiency of data processing and realizing reliable data processing.

[0044] S406: In response to the metric data query request, query the corresponding metric data from the offline metric database and / or the online metric database through the unified data service.

[0045] Among them, the metric data query request carries the metric identification of at least one query metric.

[0046] Optionally, the operation staff can send an offline metric data query request for offline analysis to the server 30 through the operation terminal 10. The offline metric data query request carries the first metric identifier of the first query metric. At the same time, the client user can also send an online metric data query request for online decision-making to the server 30 through the user terminal 20. The online metric data query request carries the second metric identifier of the second query metric.

[0047] In response to the above offline metric data query request, the server 30 obtains the first metric metadata associated with the first metric identifier from the meta information table through the unified data service. The first metric metadata includes the offline storage mapping information of the first query metric. Then, the server 30 queries the corresponding offline metric data from the offline metric database through the unified data service according to the offline storage mapping information of the first query metric.

[0048] In response to the above online metric data query request, the server 30 obtains the second metric metadata associated with the second metric identifier from the meta information table through the unified data service. The second metric metadata includes the online storage mapping information of the second query metric. Then, the server 30 queries the corresponding online metric data from the online metric database through the unified data service according to the online storage mapping information of the second query metric.

[0049] In this embodiment, in response to the offline metric data query request and / or the online metric data query request, the server 30 calls the meta information table through the unified data service, and queries the corresponding metric data from the offline metric database and / or the online metric database, which can converge the metric query requests in the offline analysis scenario and the online decision-making scenario through the unified data service, improve the consistency, accuracy, and efficiency of data processing, and thus achieve reliable data processing.

[0050] Please refer to Figure 5 , which is a schematic architecture diagram of a data processing method provided by an exemplary embodiment of this specification in the data processing stage. As Figure 5 shown, in the data processing stage, the server providing the unified data service responds to the database on-shelf request sent by the operation terminal, registers the offline metric database to the unified data service, performs data processing on the offline metric database through the unified data service to obtain an online metric database including at least one online table, and writes the offline metric metadata corresponding to the offline metric database and the online metric metadata corresponding to the online metric database, that is, the metric metadata, into the meta information table during this process.

[0051] Please refer to Figure 6 , which is a schematic architecture diagram of a data processing method provided by an exemplary embodiment of this specification in the data application stage. As Figure 6As shown in the figure, in the data application stage, the operation personnel send the query request for the offline metric data used for offline analysis to the server that provides unified data services through the operation terminal; in response to the query request for the offline metric data, the server calls the meta-information table through the unified data service, and queries the corresponding offline metric data from the offline metric database for offline analysis. The client user sends the query request for the online metric data used for online analysis to the server through the user terminal; in response to the query request for the online metric data, the server calls the meta-information table through the unified data service, and queries the corresponding online metric data from the online metric database for online decision-making.

[0052] In the above data processing method, in the data processing stage, in response to the database on-shelf request, the server registers the offline metric database to the unified data service, and processes the data of the offline metric database through the unified data service to obtain the online metric database, which can provide unified data management capabilities, so that the offline metric database and the online metric database store the data corresponding to the same set of metrics; in the data application stage, in response to the metric data query request, the server queries the corresponding metric data from the offline metric database and / or the online metric database through the unified data service, which can provide unified data query capabilities, eliminate the difference problem in the application of the offline metric data and the online metric data, and converge the metric data query requests in two different scenarios of offline analysis and online decision-making. The entire data processing process can ensure the consistency of the data caliber of the same metric, improve the consistency and accuracy of the data processing results, and achieve reliable data processing.

[0053] In one embodiment, as Figure 7 shown, another data processing method is provided. Taking the server 30 shown in Figure 3 as an example, the method includes the following steps: S702: Batch process the historical detail data of the offline data source within a preset time period at a preset time interval to obtain at least one offline table.

[0054] Among them, the offline data source can be, but is not limited to, an offline data warehouse. Each offline table stores the corresponding offline metric data.

[0055] Optionally, the server 30 performs batch processing such as ETL (Extraction-Transformation-Loading) or aggregation calculation on the historical detail data of the offline data warehouse within the previous day on a daily basis to generate at least one offline metric table for use in the subsequent data processing stage.

[0056] S704: Build an offline metric database based on at least one offline table.

[0057] Optionally, the server 30 loads at least one offline table after batch processing into the high-performance real-time analysis database Hologress through an ETL tool or a data processing pipeline, forming a structured offline metric database.

[0058] In this embodiment, the server 30 performs batch processing on the historical detail data of the offline data source within a preset time period at a preset time interval, obtains at least one offline table, and constructs an offline metric database based on the at least one offline table, which can improve the efficiency and accuracy of subsequent data processing, thus realizing reliable data processing.

[0059] S706: In response to the database shelving request, count the offline metric data stored in the offline metric database.

[0060] Among them, the database shelving request carries metric configuration information. The metric configuration information may include, but is not limited to, at least one metric identifier and the metric consumption logic corresponding to each metric identifier. The above metric identifier may be, but is not limited to, a metric ID, and the above metric consumption logic includes an offline consumption logic and an online consumption logic.

[0061] Optionally, the server 30 counts the offline metric data stored in each offline table of the offline metric database in response to the database shelving request sent by the operation terminal 10, to obtain the metric identifier (such as a metric ID) of the metric corresponding to each offline metric data, which is convenient for subsequently generating the offline metric metadata corresponding to the offline metric data according to the metric identifier of each offline metric data and the preset metric configuration information.

[0062] S708: Generate the offline metric metadata corresponding to the offline metric data according to the metric configuration information.

[0063] Optionally, the server 30 generates the offline metric metadata corresponding to the offline metric data according to the metric identifier corresponding to each counted offline metric data, the metric identifier in the metric configuration information, and the offline consumption logic corresponding to each metric identifier.

[0064] Among them, the offline metric metadata includes the metric identification information of each metric, the storage mapping information of each metric in the offline metric database, the status information of each metric, the type information of each metric, the offline consumption logic information of each metric, etc. Specifically, the metric identification information of each metric can be but is not limited to the metric ID of each metric; the storage mapping information of each metric in the offline metric database, that is, the offline storage mapping information of each metric, which can be but is not limited to including the offline metric database identifier where each metric is located, the offline table identifier where each metric is located, the offline partition information of each metric, and the offline field identifier of each metric, etc.; the status information of each metric is used to determine the life cycle status of each metric, which can be but is not limited to at least one of the to-be-verified status, the on-shelf status, the discarded status, and the off-shelf status; the type information of each metric can be but is not limited to the field-level data type of each metric; the offline consumption logic information of each metric is used to define the query method of the metric and the usage scenario of the metric (in this case, the offline analysis scenario).

[0065] S710: Write the offline metric metadata into the meta information table for the unified data service to call.

[0066] Optionally, the server 30 writes the above-mentioned offline metric metadata into the meta information table for the unified data service to call, thereby registering the offline metric database to the unified data service, so that the server 30 can provide unified data management capabilities and unified data query capabilities through the unified data service and the metric metadata in the meta information table.

[0067] In this embodiment, in response to the database on-shelf request, the server 30 counts the offline metric data stored in the offline metric database, generates the offline metric metadata corresponding to the offline metric data according to the metric configuration information, and writes the offline metric metadata into the meta information table for the unified data service to call, which facilitates automatically querying the offline metric data stored in the offline metric database through the unified data service calling the meta information table during the data application stage, improving the accuracy and efficiency of offline analysis, and thus realizing reliable data processing.

[0068] S712: Generate the online metric metadata corresponding to the offline metric metadata in the meta information table according to the metric configuration information through the unified data service.

[0069] It can be understood that the offline metric metadata stored in the meta information table includes the metric identification information of each metric. Optionally, the server 30 generates the online metric metadata corresponding to the offline metric metadata in the meta information table through the unified data service according to the metric identification information of each metric in the meta information table, and at least one metric identification in the metric configuration information and the online consumption logic corresponding to each metric identification, so as to generate the online metric data that shares the same set of metrics with the offline metric data according to the online metric metadata in the future.

[0070] Among them, the online metric metadata includes the metric identification information of each metric, the storage mapping information of each metric in the online metric database, the status information of each metric, the type information of each metric, the online consumption logic information of each metric, etc. It can be understood that the metric identification information of each metric can be, but is not limited to, the metric ID of each metric; the storage mapping information of each metric in the online metric database is the offline storage mapping information of each metric, which can be, but is not limited to, including the online metric database identification where each metric is located, the online table identification where each metric is located, the online partition information of each metric, and the online field identification of each metric, etc.; the status information of each metric is used to determine the life cycle status of each metric, which can be, but is not limited to, at least one of the to-be-verified status, the on-shelf status, the discarded status, and the off-shelf status; the type information of each metric can be, but is not limited to, the field-level data type of each metric; the online consumption logic information of each metric is used to define the query method of the metric and the usage scenario of the metric (in this case, the online decision-making scenario).

[0071] S714: Through the unified data service, data processing is performed on the offline metric database according to the online metric metadata and data processing rules to generate an online metric database.

[0072] It can be understood that the database on-shelf request also carries data processing rules. The data processing rules can be, but are not limited to, including preset offline storage resources and online storage resources. For example, the data processing rules can include the type of the offline metric database, the offline table information in the offline metric database, the type of the online metric database, and the online table information of the online metric database, etc.

[0073] Optionally, the server 30 performs data processing on the offline metric database through the unified data service according to the type of the offline metric database, the offline table information in the offline metric database, the type of the online metric database, and the online table information of the online metric database, obtains online metric data sharing the same set of metrics as the offline metric data, and loads the online metric data into a specified online storage engine (such as Lindorm, Tbase) to generate an online metric database.

[0074] In this embodiment, the server 30 generates online metric metadata corresponding to offline metric metadata in the meta-information table according to the metric configuration information through the unified data service, and processes the data in the offline metric database according to the online metric metadata and the data processing rules to generate an online metric database. On the one hand, it can provide unified data management capabilities, enabling the offline database and the online metric database to store data corresponding to the same set of metrics, and enabling the meta-information table to store the metric metadata of the offline metric database and the online metric database respectively. On the other hand, it facilitates subsequent realization of unified data query capabilities through the unified data service to converge the metric query requests in two different scenarios of offline analysis and online decision-making. This embodiment helps to improve the consistency, accuracy, and efficiency of data processing, thereby realizing reliable data processing.

[0075] S716: In response to an offline metric data query request for offline analysis, query the corresponding offline metric data from the offline metric database through the unified data service.

[0076] Among them, the offline metric data query request carries the first metric identifier of the first query metric.

[0077] Optionally, in response to an offline metric data query request for offline analysis, the server 30 obtains the first metric metadata associated with the first metric identifier from the meta-information table through the unified data service. The first metric metadata includes the offline storage mapping information of the first query metric. Then, the server 30 queries the corresponding offline metric data from the offline metric database through the unified data service according to the offline storage mapping information of the first query metric.

[0078] S718: In response to an online metric data query request for online decision-making, query the corresponding online metric data from the online metric database through the unified data service.

[0079] Among them, the online metric data query request carries the second metric identifier of the second query metric.

[0080] Optionally, in response to an online metric data query request for online analysis, the server 30 obtains the second metric metadata associated with the second metric identifier from the meta-information table through the unified data service. The second metric metadata includes the online storage mapping information of the second query metric. Then, the server 30 queries the corresponding online metric data from the online metric database through the unified data service according to the online storage mapping information of the second query metric.

[0081] In this embodiment, the server 30 responds to the offline metric data query request and the online metric data query request, calls the meta-information table through the unified data service, and queries the corresponding metric data from the offline metric database and the online metric database, which can provide unified data query capabilities, ensuring the consistency, accuracy, and efficiency of data processing in two different scenarios of offline analysis and online decision-making, thereby achieving reliable data processing.

[0082] In the above data processing method, in the data processing stage, the server 30 batch processes the historical detail data of the offline data source within a preset time period at a preset time interval to construct an offline metric database. In response to the database online request, it generates offline metric metadata corresponding to the offline metric data, writes the offline metric metadata into the meta-information table for the unified data service to call, and then through the unified data service, generates online metric metadata corresponding to the offline metric metadata in the meta-information table according to the metric configuration information, and processes the offline metric database according to the online metric metadata and the data processing rules to generate an online metric database, which can provide unified data management capabilities, enabling the offline database and the online metric database to store data corresponding to the same set of metrics. In the data application stage, the server 30 responds to the offline metric data query request for offline analysis, queries the corresponding offline metric data from the offline metric database through the unified data service, and responds to the online metric data query request for online decision-making, queries the corresponding online metric data from the online metric database through the unified data service, which can provide unified data query capabilities, ensuring the consistency of the data caliber of the same metric and improving the data query efficiency. The entire process improves the consistency, accuracy, and efficiency of the data processing results, achieving reliable data processing.

[0083] In one embodiment, as Figure 8 shown, another data processing method is provided. Taking the method applied to the Figure 3 server 30 shown as an example, it includes the following steps: S802: Respond to the database online request and count the offline metric data stored in the offline metric database.

[0084] Specifically, S802 is the same as S706 and will not be elaborated here.

[0085] S804: Generate offline metric metadata corresponding to the offline metric data according to the metric configuration information.

[0086] Specifically, S804 is the same as S708 and will not be elaborated here.

[0087] S806: Write the offline metric metadata into the meta-information table for the unified data service to call.

[0088] Specifically, S806 is the same as S710, which will not be elaborated here.

[0089] S808: Through the unified data service, generate the online metric metadata corresponding to the offline metric metadata in the meta-information table according to the metric configuration information.

[0090] Specifically, S808 is the same as S712, which will not be elaborated here.

[0091] S810: Through the unified data service, perform data processing on the offline metric database according to the online metric metadata and data processing rules to generate an online metric database.

[0092] Specifically, S810 is the same as S714, which will not be elaborated here.

[0093] S812: In response to an offline metric data query request for offline analysis, obtain the first metric metadata associated with the first metric identifier from the meta-information table through the unified data service.

[0094] Optionally, the server 30 receives an offline metric data query request for offline analysis sent by the operation terminal. The offline metric data query request carries the first metric identifier (such as the metric ID of the first query metric) of the first query metric. In response to the offline metric data query request, the server 30 obtains the first metric metadata associated with the first metric identifier from the meta-information table through the unified data service according to the first metric identifier of the first query metric.

[0095] Among them, the first metric metadata includes the offline storage mapping information of the first query metric. It can be understood that the offline storage mapping information of the first query metric includes the offline metric database identifier where the first query metric is located, the offline table identifier where the first query metric is located, the offline partition information of the first query metric, and the offline field identifier of the first query metric, etc.

[0096] S814: Through the unified data service, query the corresponding offline metric data from the offline metric database according to the offline storage mapping information of the first query metric.

[0097] Optionally, the server 30 queries the corresponding offline metric data from the offline metric database through the unified data service according to the field information of the first query metric and the position index information of the field information of the first query metric in the offline table.

[0098] Optionally, the server 30 generates a structured query statement for the offline metric database through the unified data service according to the offline metric database identifier where the first query metric is located, the offline table identifier where the first query metric is located, the offline partition information of the first query metric, and the offline field identifier of the first query metric, and queries the corresponding offline metric data from the offline metric database through the unified data service according to the structured query statement (Structured Query Language, SQL) of the offline metric database.

[0099] In this embodiment, in response to the offline metric data query request, the server 30 obtains the first metric metadata associated with the first metric identifier from the meta information table through the unified data service, and queries the corresponding offline metric data from the offline metric database through the unified data service according to the offline storage mapping information of the first query metric in the first metric metadata, improving the accuracy and efficiency of data query and realizing reliable data processing.

[0100] S816: In response to the online metric data query request for online decision-making, obtain the second metric metadata associated with the second metric identifier from the meta information table through the unified data service.

[0101] Optionally, the server 30 receives an online metric data query request for online decision-making sent by the operation terminal, and the online metric data query request carries the second metric identifier of the second query metric (such as the metric ID of the second query metric). In response to the online metric data query request, the server 30 obtains the second metric metadata associated with the second metric identifier from the meta information table through the unified data service according to the second metric identifier of the second query metric.

[0102] Among them, the second metric metadata includes the online storage mapping information of the second query metric. It can be understood that the online storage mapping information of the second query metric includes the online metric database identifier where the second query metric is located, the online table identifier where the second query metric is located, the online partition information of the second query metric, and the online field identifier of the second query metric, etc.

[0103] S818: Query the corresponding online metric data from the online metric database through the unified data service according to the online storage mapping information of the second query metric.

[0104] Optionally, the server 30 queries the corresponding offline metric data from the offline metric database through the unified data service according to the field information of the first query metric and the position index information of the field information of the first query metric in the offline table.

[0105] Optionally, the server 30 generates a structured query statement for the offline metric database through the unified data service according to the offline metric database identifier where the first query metric is located, the offline table identifier where the first query metric is located, the offline partition information of the first query metric, and the offline field identifier of the first query metric, and queries the corresponding offline metric data from the offline metric database through the unified data service according to the structured query statement (Structured Query Language, SQL) of the offline metric database.

[0106] In this embodiment, in response to the offline metric data query request, the server 30 obtains the first metric metadata associated with the first metric identifier from the meta-information table through the unified data service, and queries the corresponding offline metric data from the offline metric database through the unified data service according to the offline storage mapping information of the first query metric in the first metric metadata, improving the accuracy and efficiency of data query and realizing reliable data processing.

[0107] S820: In response to the discard request for the first metric, update the status information of the first metric in the meta-information table to the discarded status applicable to the specified scenario.

[0108] Optionally, in response to the discard request for the first metric sent by the operation terminal, the server 30 updates the status of the first metric in the meta-information table to the discarded status only applicable to the specified scenario. Among them, the specified scenario can be specified in the whitelist in advance, and the specified scenario can be an old scenario.

[0109] It can be understood that the server 30 performs life cycle management on the metrics through the offline metric metadata and online metric metadata stored in the meta-information table to avoid the problem of continuously increasing complexity caused by metric inflation.

[0110] Exemplarily, please refer to Figure 9 , a complete metric life cycle includes a definition stage, a listing stage, a discard stage, and a delisting stage. Specifically, when the metric is in the definition stage, it means that the basic metadata of the metric has been stored in the database, and at this time, the status of the metric in the meta-information table is marked as the to-be-verified status. When the metric is in the listing stage, it means that the metric has passed the verification and is open for use in new scenarios, and at this time, the status of the metric in the meta-information table is marked as the listed status. When the metric is in the discard stage, it means that the metric can no longer be used in new scenarios and is only applicable to the specified old scenarios, and at this time, the status of the metric in the meta-information table is marked as the discarded status. When the metric is in the delisting stage, it means that the metric is no longer used in any scenario, and at this time, the status of the metric in the meta-information table is the delisted status.

[0111] S822: In response to the delisting request for the second metric, update the status information of the second metric in the meta-information table to the delisted status where it is prohibited from being used.

[0112] Optionally, in response to a delisting request for the second metric sent by the operation terminal, the server 30 updates the status of the second metric in the meta-information table to a delisted status that prohibits use in any scenario. It should be noted that after the second metric is updated to the delisted status, the server 30 does not immediately delete the metric metadata corresponding to the second metric and the metric data corresponding to the second metric from the meta-information table and the database. Instead, after the duration of the second metric being in the delisted status reaches a preset duration, the metric metadata corresponding to the second metric and the metric data corresponding to the second metric are deleted from the meta-information table and the database for traceability of the data processing result.

[0113] In this embodiment, the server 30 updates the status information of the first metric in the meta-information table to a discarded status applicable to a specified scenario in response to a discard request for the first metric sent by the operation terminal; in response to a delisting request for the second metric sent by the operation terminal, updates the status information of the second metric in the meta-information table to a delisted status prohibiting use, and after the duration of the second metric being in the delisted status reaches a preset duration, deletes the metric metadata corresponding to the second metric and the metric data corresponding to the second metric from the meta-information table and the database, which can effectively avoid the problem of continuously increasing complexity caused by metric inflation and achieve reliable data processing.

[0114] In the above data processing method, in the data application stage, the server, in response to an offline metric data query request, obtains the first metric metadata associated with the first metric identifier from the meta-information table through the unified data service, and queries the corresponding offline metric data from the offline metric database according to the offline storage mapping information of the first query metric in the first metric metadata, improving the accuracy and efficiency of data query; in response to a discard request for the first metric and a delisting request for the second metric sent by the operation terminal, updates the status information of the first metric in the meta-information table to a discarded status applicable to a specified scenario, updates the status information of the second metric to a delisted status prohibiting use, and after the duration of the second metric being in the delisted status reaches a preset duration, deletes the metric metadata corresponding to the second metric and the metric data corresponding to the second metric from the meta-information table and the database, which can effectively avoid the problem of continuously increasing complexity caused by metric inflation and achieve reliable data processing.

[0115] In one embodiment, as Figure 10 shown, another data processing method is provided. Taking the method applied to the Figure 3 server 30 shown as an example, it includes the following steps: S1002: In response to a database listing request, register the offline metric database to the unified data service.

[0116] Specifically, S1002 is the same as S402, which will not be elaborated here.

[0117] S1004: Process the data in the offline metric database through the unified data service to obtain an online metric database.

[0118] Specifically, S1004 is the same as S404, which will not be elaborated here.

[0119] Exemplarily, please refer to Figure 11 , which is a schematic diagram of the domain model in the data processing stage of a data processing method provided by an exemplary embodiment of this specification. As Figure 11 shown, the left half is the generation logic of metric metadata, and the right half is the consumption logic of metric metadata.

[0120] Please refer to Figure 11 the left half. After the offline metric database is registered with the unified data service, the unified data service abstracts the offline metric data stored in each offline table into data assets defined by the asset name and the ODPS (Open Data Processing Service) table name, and constructs a consumption view defined by the table fields and field types through the data assets. The consumption view integrates the decision view in the online scenario and the analysis view in the offline scenario. Among them, the decision view in the online scenario is defined by the online storage table configured by Lindorm, and the decision view in the online scenario is derived from the real-time metric decision view and the offline metric decision view. The real-time metric decision view is defined by the online storage table configured by Lindorm and the metric consistency offline table. This metric consistency offline table can ensure that the online storage table and the offline table share the same set of metrics. Through the decision view in the online scenario, online decision metric features defined by the feature name can be generated. In addition, the decision view in the online scenario can also query the metric data in the Lindorm database through SQL combined with the task type to complete real-time calculation tasks. The analysis view in the offline scenario is defined by the offline storage table configured by Holo, and the analysis view in the offline scenario is specifically constructed by the metric names of the pre-configured analysis metrics.

[0121] Please refer to Figure 11In the right half, the metrics supported by the unified data service are divided into multiple types, and each metric is defined by a metric name and a value range type. Specifically, the metrics supported by the unified data service can be either basic metrics defined by the original table fields and field types, or composite metrics defined by calculation logics and field types. The above composite metrics can be constructed based on derived metrics and derived metrics defined by combining different dimension values. The dimension system supported by the metric includes both basic dimensions from the original table field definitions and extended dimensions constructed by different dimension values in the derived metrics, and is also restricted by the consumption conditions supported by the metric. The consumption conditions supported by the metric depend on its metric type (such as basic metric, derived metric) and the consumption rules of the metric. The consumption rules of the metric are defined by the label name of the metric and the calculation logic of the metric, and are associated with the metric type (such as basic metric, derived metric) and the consumption population of the metric. The consumption population of the metric is identified by different population names and determines the available range and usage method of the metric. The consumption population of the metric can come from either physical populations defined based on population tables or logical populations defined based on population logics.

[0122] S1006: In response to an offline metric data query request for offline analysis, determine, through the unified data service, a first domain-specific language query statement corresponding to the offline metric data query request.

[0123] Among them, the offline metric data query request carries a first metric identifier (such as a first metric ID) of a first query metric, and the offline metric data query request is defined using a pre-configured service semantics.

[0124] It can be understood that the offline metric data and the online metric data are stored using different storage engines. Since writing two sets of SQLs for querying data from the offline metric database and the online metric database respectively may result in inconsistent metric data queried, it is necessary to use the unified data service to unify the data query caliber to ensure data consistency.

[0125] Optionally, the server 30, in response to an offline metric data query request sent by the operation terminal, determines, through the unified data service, a first domain-specific language query statement corresponding to the offline metric data query request. The above first domain-specific language query statement is implemented using Antlr4, and the above first domain-specific language query statement includes a first metric identifier of a first query metric.

[0126] S1008: Through the unified data service, obtain, from the meta-information table, first metric metadata associated with the first metric identifier according to the first domain-specific language query statement.

[0127] Optionally, the server 30 obtains, through the unified data service, the first metric metadata associated with the first metric identifier from the meta-information table according to the first domain-specific language query statement, so as to mask the SQL differences brought by different storage engines through the standardized domain-specific language.

[0128] S1010: Generate a structured query statement for the offline metric database through the unified data service according to the offline storage mapping information of the first query metric.

[0129] Among them, the offline storage mapping information of the first query metric is the storage mapping information of the first query metric in the offline metric database, which includes but is not limited to the offline metric database identifier where the first query metric is located, the offline table identifier where the first query metric is located, the offline partition information of the first query metric, and the offline field identifier of the first query metric, etc.

[0130] Optionally, the server 30 generates a structured query statement for the offline metric database through the unified data service according to the offline metric database identifier where the first query metric is located, the offline table identifier where the first query metric is located, the offline partition information of the first query metric, and the offline field identifier of the first query metric.

[0131] S1012: Query the corresponding offline metric data from the offline metric database through the unified data service according to the structured query statement of the offline metric database.

[0132] Optionally, the server 30 queries the corresponding offline metric data from the offline metric database (such as Hologress) through the unified data service according to the above-mentioned structured query statement of the offline metric database for offline analysis.

[0133] In this embodiment, the server 30 converts the offline metric data query request sent by the operation terminal into a standardized domain-specific language query statement through the unified data service, so as to obtain the corresponding offline storage mapping information from the meta-information table, and subsequently generate a structured query statement for the offline metric database according to the corresponding offline storage mapping information, so as to query the corresponding offline metric data from the offline metric database, which can provide a standardized data query ability, help to mask the differences in different structured query languages corresponding to the underlying online metric database and offline metric database, ensure the consistency of data in the offline analysis and online decision-making scenarios, and eliminate the difference problems in the data application stage between online data processing and offline data processing.

[0134] S1014: In response to an online metric data query request for online analysis, determine a second domain-specific language query statement corresponding to the online metric data query request through the unified data service.

[0135] Among them, the online indicator data query request carries the second indicator identifier of the second query indicator (such as the second indicator ID), and the online indicator data query request adopts the service semantics definition configured in advance.

[0136] Optionally, in response to the online indicator data query request sent by the operation terminal, the server 30 determines the second domain-specific language query statement corresponding to the online indicator data query request through the unified data service. The above-mentioned second domain-specific language query statement is implemented by Antlr4, and the second indicator identifier of the second query indicator is included in the above-mentioned second domain-specific language query statement.

[0137] S1016: Through the unified data service, obtain the second indicator metadata associated with the second indicator identifier from the meta-information table according to the second domain-specific language query statement.

[0138] Optionally, the server 30 obtains the second indicator metadata associated with the second indicator identifier from the meta-information table through the unified data service, so as to shield the SQL differences brought by different storage engines through the standardized domain-specific language.

[0139] S1018: Through the unified data service, generate a structured query statement for the online indicator database according to the online storage mapping information of the second query indicator.

[0140] Among them, the online storage mapping information of the second query indicator is the storage mapping information of the second query indicator in the online indicator database, which includes but is not limited to the online indicator database identifier where the second query indicator is located, the online table identifier where the second query indicator is located, the online partition information of the second query indicator, and the online field identifier of the second query indicator, etc.

[0141] Optionally, the server 30 generates a structured query statement for the online indicator database according to the online indicator database identifier where the second query indicator is located, the online table identifier where the second query indicator is located, the online partition information of the second query indicator, and the online field identifier of the second query indicator.

[0142] S1020: Through the unified data service, query the corresponding online indicator data from the online indicator database according to the structured query statement of the online indicator database.

[0143] Optionally, the server 30 queries the corresponding offline indicator data from the online indicator database (such as Lindorm) through the unified data service according to the above-mentioned structured query statement of the online indicator database for offline analysis.

[0144] In this embodiment, the server 30 converts the online metric data query request sent by the operation terminal into a standardized domain-specific language query statement through the unified data service, so as to obtain the corresponding online storage mapping information from the meta-information table, and subsequently generate a structured query statement for the online metric database according to the corresponding online storage mapping information, so as to query the corresponding online metric data from the online metric database. It can provide standardized data query capabilities, help to shield the differences in different structured query languages corresponding to the underlying online metric database and offline metric database, ensure the consistency of data in the offline analysis and online decision-making scenarios, and eliminate the difference problems in the data application stage between online data processing and offline data processing.

[0145] Exemplarily, please refer to Figure 12 , which is a schematic diagram of semantic conversion in the data application stage of a data processing method provided by an exemplary embodiment of this specification. As Figure 12 shown, the server 30 converts the service semantics into a unified domain-specific language (DSL) through the unified data service, and converts the unified domain-specific language into the structured query language Hologress SQL of the offline metric database and the structured query language Lindorm SQL of the online metric database through the conversion layer.

[0146] In the above data processing method, in the data processing stage, the server, through the unified data service, combines data assets, metric models, consumption views and actual scenarios to derive extensions of each basic model, so as to register and process the offline metric database including at least one offline table, and generate an online storage table configured with Lindorm and an offline storage table configured with Holo, which can ensure the consistency of the metric data used in the online decision-making and offline analysis processes and provide unified data management capabilities. In the data application stage, the server converts the service semantics into a standardized domain-specific language implemented through Antlr4 through the unified data service, and subsequently generates a structured query statement for the corresponding metric database according to the corresponding storage mapping information, which can provide standardized data query capabilities and eliminate the difference problems in the data application stage between online data processing and offline data processing. The whole process can ensure the consistency of the data caliber of the same metric, improve the consistency and accuracy of the data processing results, and realize reliable data processing.

[0147] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0148] Based on the inventive concept of the above data processing method, as Figure 13 shown, an embodiment of this specification further provides a data processing apparatus 1300 for implementing the above-mentioned data processing method. The data processing apparatus 1300 includes: A registration module 1301, configured to register an offline metric database to a unified data service in response to a database on-shelf request; A processing module 1302, configured to perform data processing on the offline metric database through the unified data service to obtain an online metric database; A query module 1303, configured to query corresponding metric data from the offline metric database and / or the online metric database through the unified data service in response to a metric data query request.

[0149] In a possible implementation manner, the database on-shelf request carries metric configuration information; the registration module 1301 is specifically configured to, in response to the database on-shelf request, count the offline metric data stored in the offline metric database; generate offline metric metadata corresponding to the offline metric data according to the metric configuration information; the offline metric metadata includes offline storage mapping information of each metric; write the offline metric metadata into a metadata table for the unified data service to call.

[0150] In a possible implementation manner, the database on-shelf request further carries a data processing rule; the processing module 1302 is specifically configured to, through the unified data service, generate online metric metadata corresponding to the offline metric metadata in the metadata table according to the metric configuration information; the online metric metadata includes online storage mapping information of each metric; through the unified data service, perform data processing on the offline metric database according to the online metric metadata and the data processing rule to generate an online metric database.

[0151] In a possible implementation, the offline metric metadata and the online metric metadata further include the status information of each metric; the data processing device 1300 further includes a status update module, configured to, in response to a discard request for a first metric, update the status information of the first metric in the meta-information table to a discarded status applicable to a specified scenario; and in response to a take-off shelf request for a second metric, update the status information of the second metric in the meta-information table to a take-off shelf status where it is prohibited from being used.

[0152] In a possible implementation, the data processing device 1300 further includes a database construction module, configured to perform batch processing on the historical detail data of the offline data source within a preset time period at a preset time interval to obtain at least one offline table; each offline table stores corresponding offline metric data; and based on the at least one offline table, construct an offline metric database.

[0153] In a possible implementation, the query module 1303 is specifically configured to, in response to an offline metric data query request for offline analysis, query corresponding offline metric data from the offline metric database through the unified data service; and in response to an online metric data query request for online decision-making, query corresponding online metric data from the online metric database through the unified data service.

[0154] In a possible implementation, the offline metric data query request carries a first metric identifier of a first query metric; the query module 1303 is specifically configured to, in response to an offline metric data query request for offline analysis, obtain first metric metadata associated with the first metric identifier from the meta-information table through the unified data service; the first metric metadata includes offline storage mapping information of the first query metric; and through the unified data service, query corresponding offline metric data from the offline metric database according to the offline storage mapping information of the first query metric.

[0155] In a possible implementation, the online metric data query request carries a second metric identifier of a second query metric; the query module 1303 is specifically configured to, in response to an online metric data query request for online decision-making, obtain second metric metadata associated with the second metric identifier from the meta-information table through the unified data service; the second metric metadata includes online storage mapping information of the second query metric; and through the unified data service, query corresponding online metric data from the online metric database according to the online storage mapping information of the second query metric.

[0156] In a possible implementation, the query module 1303 is specifically configured to, in response to an offline metric data query request for offline analysis, determine a first domain-specific language query statement corresponding to the offline metric data query request through the unified data service; obtain first metric metadata associated with a first metric identifier from the meta-information table according to the first domain-specific language query statement through the unified data service; generate a structured query statement for the offline metric database according to the offline storage mapping information of the first query metric through the unified data service; and query corresponding offline metric data from the offline metric database according to the structured query statement for the offline metric database through the unified data service.

[0157] In a possible implementation, the query module 1303 is specifically configured to, in response to an online metric data query request for online analysis, determine a second domain-specific language query statement corresponding to the online metric data query request through the unified data service; obtain second metric metadata associated with a second metric identifier from the meta-information table according to the second domain-specific language query statement through the unified data service; generate a structured query statement for the online metric database according to the online storage mapping information of the second query metric through the unified data service; and query corresponding online metric data from the online metric database according to the structured query statement for the online metric database through the unified data service.

[0158] Each module in the above data processing device 1300 can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or stored in a memory in the computer device in the form of software, so as to facilitate the processor to call and execute operations corresponding to each of the above modules.

[0159] The embodiments of this specification also provide an electronic device, which can be a server, and its internal structure diagram can be as Figure 14As shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The application database of the electronic device is used to store and process data. The input / output interface of the electronic device is used to exchange information between the processor and external devices. The communication interface of the electronic device is used to communicate with external terminals through a network connection. The processor of the electronic device executes the computer program to implement a data processing method.

[0160] Those skilled in the art can understand that Figure 14 the structure shown in is only a block diagram of some structures related to the solution of this specification, and does not constitute a limitation on the electronic device to which the solution of this specification is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0161] In a possible implementation manner, an electronic device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0162] This specification embodiment also provides a computer storage medium. Instructions are stored in the computer storage medium. When it runs on a computer or a processor, the computer or the processor is caused to execute one or more steps in the above embodiments. If the respective component modules of the above electronic device are implemented in the form of software function units and sold or used as independent products, they can be stored in the above computer storage medium.

[0163] In a possible implementation manner, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0164] This specification embodiment also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0165] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this specification are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a Digital Versatile Disc (DVD)), or a semiconductor medium (for example, a Solid State Disk (SSD)), etc.

[0166] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media that can store program codes, such as ROM, RAM, magnetic disks, or optical discs. Without conflict, the technical features in this embodiment and the implementation solutions can be combined arbitrarily.

[0167] The above-described embodiments are merely described in a preferred embodiment manner of this specification, and do not limit the scope of this specification. Without departing from the design spirit of this specification, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of this specification shall fall within the protection scope determined by the claims.

[0168] The above description has been made of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A data processing method, characterized in that, The method includes: In response to a database online request, registering an offline metric database to a unified data service; Performing data processing on the offline metric database through the unified data service to obtain an online metric database; In response to a metric data query request, querying corresponding metric data from the offline metric database and / or the online metric database through the unified data service.

2. The method according to claim 1, characterized in that, The database online request carries metric configuration information; The step of registering the offline metric database to the unified data service in response to the database online request includes: In response to the database online request, counting the offline metric data stored in the offline metric database; Generating offline metric metadata corresponding to the offline metric data according to the metric configuration information; the offline metric metadata includes offline storage mapping information of each metric; Writing the offline metric metadata into a metadata table for the unified data service to call.

3. The method according to claim 2, characterized in that The database online request also carries a data processing rule; The step of performing data processing on the offline metric database through the unified data service to obtain an online metric database includes: Through the unified data service, generating online metric metadata corresponding to the offline metric metadata in the metadata table according to the metric configuration information; the online metric metadata includes online storage mapping information of each metric; Through the unified data service, performing data processing on the offline metric database according to the online metric metadata and the data processing rule to generate the online metric database.

4. The method according to claim 3, wherein The offline metric metadata and the online metric metadata also include status information of each metric; The method further includes: In response to a discard request for a first metric, updating the status information of the first metric to a discarded state applicable to a specified scenario in the metadata table; In response to a take-off request for a second metric, updating the status information of the second metric to a taken-off state where it is prohibited from being used in the metadata table.

5. The method according to claim 1, wherein The method further includes: At a preset time interval, performing batch processing on historical detail data of an offline data source within a preset time period to obtain at least one offline table; each offline table stores corresponding offline metric data; Based on the at least one offline table, constructing the offline metric database.

6. The method according to claim 1, wherein The step of querying corresponding metric data from the offline metric database or the online metric database through the unified data service in response to a metric data query request includes: In response to an offline metric data query request for offline analysis, querying corresponding offline metric data from the offline metric database through the unified data service; In response to an online metric data query request for online decision-making, querying corresponding online metric data from the online metric database through the unified data service.

7. The method according to claim 6, wherein The offline metric data query request carries a first metric identifier of a first query metric; The step of querying corresponding offline metric data from the offline metric database through the unified data service in response to an offline metric data query request for offline analysis includes: In response to an offline metric data query request for offline analysis, obtain, via the unified data service, first metric metadata associated with the first metric identifier from the meta-information table; the first metric metadata includes offline storage mapping information for the first query metric. Via the unified data service, query corresponding offline metric data from the offline metric database according to the offline storage mapping information for the first query metric.

8. The method according to claim 6, wherein The online metric data query request carries a second metric identifier for a second query metric. The querying, in response to an online metric data query request for online decision-making, of corresponding online metric data from the online metric database via the unified data service includes: In response to an online metric data query request for online decision-making, obtain, via the unified data service, second metric metadata associated with the second metric identifier from the meta-information table; the second metric metadata includes online storage mapping information for the second query metric. Via the unified data service, query corresponding online metric data from the online metric database according to the online storage mapping information for the second query metric.

9. The method according to claim 7, wherein The obtaining, in response to an offline metric data query request for offline analysis, of first metric metadata associated with the first metric identifier from the meta-information table via the unified data service includes: In response to an offline metric data query request for offline analysis, determine, via the unified data service, a first domain-specific language query statement corresponding to the offline metric data query request. Via the unified data service, obtain first metric metadata associated with the first metric identifier from the meta-information table according to the first domain-specific language query statement. The querying, via the unified data service, of corresponding offline metric data from the offline metric database according to the offline storage mapping information for the first query metric includes: Via the unified data service, generate a structured query statement for the offline metric database according to the offline storage mapping information for the first query metric. Via the unified data service, query corresponding offline metric data from the offline metric database according to the structured query statement for the offline metric database.

10. The method according to claim 8, wherein The obtaining, in response to an online metric data query request for online decision-making, of second metric metadata associated with the second metric identifier from the meta-information table via the unified data service includes: In response to an online metric data query request for online analysis, determine, via the unified data service, a second domain-specific language query statement corresponding to the online metric data query request. Via the unified data service, obtain second metric metadata associated with the second metric identifier from the meta-information table according to the second domain-specific language query statement. The querying, via the unified data service, of corresponding online metric data from the online metric database according to the online storage mapping information for the second query metric includes: Generate a structured query statement for the online metric database through the unified data service according to the online storage mapping information of the second query metric; Query corresponding online metric data from the online metric database through the unified data service according to the structured query statement of the online metric database.

11. A data processing device, characterized in that, The device includes: A registration module, configured to register an offline metric database to the unified data service in response to a database on-shelf request; A processing module, configured to process the data of the offline metric database through the unified data service to obtain an online metric database; A query module, configured to query corresponding metric data from the offline metric database and / or the online metric database through the unified data service in response to a metric data query request.

12. An electronic device, characterized in that, Including: A processor and a memory; the memory stores a computer program, and when the processor executes the computer program, the method steps of any one of claims 1-10 are implemented.

13. A computer storage medium, characterized in that, The computer storage medium stores multiple instructions, and the instructions are suitable for being loaded and executed by the processor to perform the method steps of any one of claims 1-10.

14. A computer program product, characterized in that, Including a computer program, and when the computer program is executed by the processor, the steps of the method described in any one of claims 1-10 are implemented.

Images