Data processing method, apparatus and device

Abstract

This application provides a data processing method, apparatus, and device to acquire real-time scene information; determine factors corresponding to the real-time scene information based on the real-time scene information; determine first data corresponding to the factors based on the factors and preset rule information; and push the first data corresponding to the factors to the application system corresponding to the real-time scene information based on the real-time scene information. This application can provide different data services according to different real-time scenarios; based on the research and development framework of this method, it can connect with different development tasks, make full use of the data in the real-time data warehouse under this framework, and improve the utilization rate of data resources.

Description

Technical Field

[0001] This application relates to the field of big data computing technology, and in particular to a data processing method, apparatus and device. Background Technology

[0002] To meet the diverse needs of real-time scenarios, the real-time data warehouse needs to provide business data to the corresponding application systems in a timely manner for different real-time scenarios.

[0003] In existing technologies, the business data required for real-time scenarios is parsed into a custom format; then the processed business data is collected into a real-time data warehouse for real-time statistics, and the statistical data is then fed back to the application system.

[0004] However, in the above methods, the business requirements of real-time scenarios are different, and the data cached in the real-time data warehouse varies greatly, resulting in low data reuse and thus reducing the utilization rate of real-time data resources, causing waste of real-time resources. Summary of the Invention

[0005] This application provides a data processing method, apparatus, and device to solve the problem of reduced utilization of real-time data resources due to large differences in cached data in a real-time data warehouse.

[0006] Firstly, this application provides a data processing method, including:

[0007] Acquire real-time scene information; wherein, the real-time scene information represents the event corresponding to the real-time scene;

[0008] Based on the real-time scene information, factors corresponding to the real-time scene information are determined; wherein, the factors represent the associated tags of events corresponding to the real-time scene in a preset event factor library; the preset event factor library includes real-time scene information and factors corresponding to the real-time scene information;

[0009] Based on the factors and preset rule information, the first data corresponding to the factors is determined; wherein, the first data is the business data required by the real-time scenario in the real-time data warehouse; the preset rule information represents the business data processing rules corresponding to the real-time scenario; the real-time data warehouse includes business data corresponding to multiple real-time scenarios;

[0010] Based on the real-time scene information, the first data corresponding to the factor is pushed to the application system corresponding to the real-time scene information.

[0011] In one example, determining the factors corresponding to the real-time scene information based on the real-time scene information includes:

[0012] If it is determined that the real-time scene information represents a single event corresponding to the real-time scene, then the factor corresponding to the real-time scene information is determined according to the preset event factor library.

[0013] In one example, the factors corresponding to the real-time scene information include factors corresponding to the main event and factors corresponding to the auxiliary event; the method further includes:

[0014] If it is determined that the event corresponding to the real-time scene represented by the real-time scene information is a combined event, then the main event and the auxiliary event are determined from the combined event;

[0015] Based on the real-time scene information, determine the factor corresponding to the main event; and based on the real-time scene information and the preset waiting time, determine the factor corresponding to the auxiliary event.

[0016] In one example, determining the first data corresponding to the factor based on the factor and preset rule information includes:

[0017] Based on the factors, determine the factor values ​​and the corresponding second data; wherein, the factor values ​​represent the degree of correlation between real-time scene information and the factors; the second data is business data that has a preset correlation with the factors;

[0018] Based on the preset rule information and the factor value, the second data corresponding to the factor is processed to obtain the first data corresponding to the factor.

[0019] In one example, before determining the factor corresponding to the real-time scene information based on the real-time scene information, the method further includes:

[0020] If it is determined that there is no factor corresponding to the real-time scene information in the preset event factor library, the real-time scene information is processed to generate the factor corresponding to the real-time scene information.

[0021] The real-time scene information and the factors corresponding to the real-time scene information are stored in the preset event factor library; wherein, the preset event factor library includes the real-time scene information and the factors corresponding to the real-time scene information.

[0022] In one example, the method further includes:

[0023] Retrieve real-time data from various databases;

[0024] The real-time data is processed to obtain processed real-time data; and the processed real-time data is stored in layers.

[0025] In one example, after the processed real-time data is stored in layers, the process further includes:

[0026] In response to a data query request, and to determine third data; wherein the data query request is used to instruct the query of data; and the third data is processed real-time data in a preset external table;

[0027] The third data is parsed to obtain and return the query results; wherein the query results represent the business data requested by the data query request.

[0028] In one example, the method further includes:

[0029] The first data corresponding to the factor is processed to generate an analysis report; and the analysis report is sent to the application system.

[0030] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0031] The first acquisition unit is used to acquire real-time scene information; wherein, the real-time scene information represents the event corresponding to the real-time scene;

[0032] The first determining unit is configured to determine the factors corresponding to the real-time scene information based on the real-time scene information; wherein, the factors represent the associated tags of events corresponding to the real-time scene in a preset event factor library; the preset event factor library includes real-time scene information and the factors corresponding to the real-time scene information;

[0033] The second determining unit is used to determine the first data corresponding to the factor based on the factor and preset rule information; wherein, the first data is the business data required by the real-time scenario in the real-time data warehouse; the preset rule information represents the business data processing rules corresponding to the real-time scenario; the real-time data warehouse includes business data corresponding to multiple real-time scenarios;

[0034] The push unit is used to push the first data corresponding to the factor to the application system corresponding to the real-time scene information based on the real-time scene information.

[0035] In one example, the first determining unit includes:

[0036] The first determining module is used to determine the factor corresponding to the real-time scene information according to the preset event factor library if it is determined that the event corresponding to the real-time scene information is a single event.

[0037] In one example, the factors corresponding to the real-time scene information include factors corresponding to the main event and factors corresponding to the auxiliary event; the first determining unit further includes:

[0038] The second determining module is used to determine the main event and the auxiliary event from the combined events if it is determined that the event corresponding to the real-time scene represented by the real-time scene information is a combined event;

[0039] The third determining module is used to determine the factor corresponding to the main event based on the real-time scene information; and to determine the factor corresponding to the auxiliary event based on the real-time scene information and the preset waiting time.

[0040] In one example, the second determining unit includes:

[0041] The fourth determining module is used to determine the factor value and the second data corresponding to the factor based on the factor; wherein the factor value represents the degree of correlation between the real-time scene information and the factor; and the second data is business data that has a preset correlation with the factor.

[0042] The processing module is used to process the second data corresponding to the factor according to the preset rule information and the factor value to obtain the first data corresponding to the factor.

[0043] In one example, before the first determining unit determines the factor corresponding to the real-time scene information based on the real-time scene information, the method further includes:

[0044] The first generation unit is used to process the real-time scene information and generate the factor corresponding to the real-time scene information if it is determined that there is no factor corresponding to the real-time scene information in the preset event factor library.

[0045] The first storage unit is used to store the real-time scene information and the factors corresponding to the real-time scene information in the preset event factor library; wherein, the preset event factor library includes the real-time scene information and the factors corresponding to the real-time scene information.

[0046] In one example, the device further includes:

[0047] The second acquisition unit is used to acquire real-time data from various databases;

[0048] The processing unit is used to process the real-time data to obtain processed real-time data;

[0049] The second storage unit is used for hierarchical storage of the processed real-time data.

[0050] In one example, after the second storage unit is used to perform tiered storage of the processed real-time data, the method further includes:

[0051] The response unit is used to respond to data query requests;

[0052] The fourth determining unit is used to determine the third data; wherein the data query request is used to instruct data to be queried; the third data is processed real-time data in a preset external table;

[0053] The query unit is used to parse the third data, obtain and return the query result; wherein the query result represents the business data requested by the data query request.

[0054] In one example, the device further includes:

[0055] The second generation unit is used to process the first data corresponding to the factor and generate an analysis report;

[0056] The sending unit is used to send the analysis report to the application system.

[0057] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0058] The memory stores computer-executed instructions;

[0059] The processor executes computer execution instructions stored in the memory to perform the method described in the first aspect.

[0060] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0061] Fifthly, this application provides a computer program product comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the method described in the first aspect.

[0062] This application provides a data processing method, apparatus, and device for acquiring real-time scene information. The real-time scene information represents events corresponding to the real-time scene. Based on the real-time scene information, factors corresponding to the real-time scene information are determined. Each factor represents an association tag of an event corresponding to the real-time scene in a preset event factor library. The preset event factor library includes real-time scene information and factors corresponding to the real-time scene information. Based on the factors and preset rule information, first data corresponding to the factors is determined. The first data is business data required by the real-time scene in a real-time data warehouse. The preset rule information represents business data processing rules corresponding to the real-time scene. The real-time data warehouse includes business data corresponding to multiple real-time scenes. Based on the real-time scene information, the first data corresponding to the factors is pushed to the application system corresponding to the real-time scene information. By acquiring real-time application scenario information of the real-time data warehouse, including the events corresponding to those scenarios, and analyzing the events based on the acquired information, the system can find the corresponding events in a pre-defined event factor library. Then, it identifies the associated event factors. Based on these factors and the retrieved processing rules, the selected factors are processed within the real-time data warehouse. This process determines the necessary business data (the first data corresponding to the factor) from the business data of multiple real-time scenarios within the data warehouse. The calculated business data is then promptly pushed to the application system corresponding to the real-time scenario, providing real-time business data consumption and offering different data services based on different real-time scenarios. This research framework, based on this method, connects to different development tasks, fully utilizing the data in the real-time data warehouse within this framework to improve data resource utilization. Attached Figure Description

[0063] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0064] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0065] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0066] Figure 3 This is a schematic diagram illustrating a real-time data warehouse layered architecture according to an exemplary embodiment;

[0067] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0068] Figure 5 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application;

[0069] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0070] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0071] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0073] As the company's consolidation efforts advance and its real-time computing technology becomes increasingly sophisticated, users' demands for real-time scenarios are becoming more urgent. In order to meet the needs of real-time scenarios and achieve the free arrangement and combination of business event flows, it is necessary to provide business data in a timely manner.

[0074] In existing technologies, the business data required for real-time scenarios is parsed into a custom format; then the processed business data is collected into a real-time data warehouse for real-time statistics, and the statistical data is then fed back to the application system.

[0075] In one example, the process involves configuring the Flink SQL streaming computing framework, a connector for a custom query engine, setting a custom format, and adding a custom data table format to the Flink SQL. After acquiring the data, the custom format is set and the data is parsed into the custom format. The data is then collected into a unified real-time data warehouse for real-time statistics, and the custom data table format is added to the Flink SQL. Finally, the statistical data is fed back to the application.

[0076] However, in the above methods, the business requirements of real-time scenarios are different, and the data cached in the real-time data warehouse varies greatly, resulting in low data reuse and thus reducing the utilization rate of real-time data resources, causing waste of real-time resources.

[0077] This application provides a data processing method, apparatus, and device, which aims to solve the above-mentioned technical problems of the prior art.

[0078] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0079] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0080] S101. Obtain real-time scene information; wherein, real-time scene information represents the events corresponding to the real-time scene.

[0081] For example, the executing entity of this embodiment can be an electronic device, a server, a terminal device, or other apparatus or device capable of executing this embodiment. This embodiment uses an electronic device as an example for description.

[0082] Based on electronic devices, it is possible to obtain real-time application scenario information of the data warehouse or the application scenario it is connected to, including the events corresponding to the real-time application scenario.

[0083] In one example, based on electronic devices, users can deploy a decision center within a real-time data warehouse computing framework. This real-time data warehouse computing framework is connected to an application system, and the decision center is also connected to the application system. The application system can perceive external scene information in real time, that is, the events corresponding to the real-time scene, such as hotspot geofencing, internet behavior analysis, etc.

[0084] S102. Based on the real-time scene information, determine the factors corresponding to the real-time scene information; wherein, the factors represent the associated tags of the events corresponding to the real-time scene in the preset event factor library; the preset event factor library includes the real-time scene information and the factors corresponding to the real-time scene information.

[0085] For example, based on a preset event factor library, which includes real-time scene information and factors corresponding to the real-time scene information, i.e., events corresponding to the real-time scene and event factors that have a preset association relationship with the events corresponding to the real-time scene, i.e., association tags; after obtaining the real-time scene information, i.e. the events corresponding to the real-time scene, the event is analyzed and processed. The corresponding event corresponding to the real-time scene can be found in the preset event factor library, and then the event factors associated with the event can be found, i.e., the association tags of the events corresponding to the real-time scene.

[0086] In one example, a user can deploy a decision center within a real-time data warehouse computing framework. The decision center has a pre-set event factor library, including events corresponding to real-time scenarios and event factors (i.e., association tags) that have a pre-defined relationship with the events corresponding to the real-time scenarios. After obtaining the information of the real-time scenario, i.e., the event users corresponding to the real-time scenario, the user analyzes and processes the event based on the computing of the real-time data warehouse. For example, if the event is a business survey, the user will find the corresponding event in the pre-set event factor library, namely "business survey," and then find factors related to the business survey, such as service quality factors, in the pre-set event factor library.

[0087] S103. Based on the factors and preset rule information, determine the first data corresponding to the factors; wherein, the first data is the business data required by the real-time scenario in the real-time data warehouse; the preset rule information represents the business data processing rules corresponding to the real-time scenario; the real-time data warehouse includes business data corresponding to multiple real-time scenarios.

[0088] For example, in a real-time data warehouse, there is business data corresponding to multiple real-time scenarios. Based on the real-time scenario information, the business data processing rules corresponding to the real-time scenario can be retrieved, i.e., preset rule information. Based on the factors determined in the preset event factor library and the retrieved processing rules, the selected factors are processed in the real-time data warehouse. Then, among the business data corresponding to multiple real-time scenarios in the real-time data warehouse, the business data required for the real-time scenario, i.e., the first data corresponding to the factor, is determined.

[0089] In one example, the real-time data warehouse contains business data corresponding to multiple real-time scenarios. Each factor in the preset event factor library is associated with one-to-one business data in the real-time data warehouse. Based on the real-time scenario information, the business data processing rules corresponding to the real-time scenario, i.e., the preset rule information, can be retrieved. Based on the factors determined in the preset event factor library and the retrieved processing rules, the rule data, the factor, and the associated data are input into the remote dictionary service Redis. Real-time calculation is performed through the stream computing framework Flink deployed in the real-time data warehouse, and the business data required for the real-time scenario is calculated in the message processing system Kafka.

[0090] S104. Based on the real-time scene information, push the first data corresponding to the factor to the application system corresponding to the real-time scene information.

[0091] For example, based on a pre-deployed application system, it can sense external real-time scenarios and subscribe to the required scenario data. According to the real-time scenario information obtained by the application system, the calculated business data is pushed to the application system corresponding to the real-time scenario information in a timely manner to provide real-time business data consumption.

[0092] In one example, based on a pre-deployed application system, it can sense the external real-time scene and subscribe to the required scene data. According to the real-time scene information obtained by the application system, such as conducting enterprise surveys, the real-time computing and processing of the real-time data warehouse can obtain business data related to the enterprise survey, including detailed data tables or data summary tables such as service quality data, and push this data to the application system in a timely manner.

[0093] In this embodiment, real-time scene information is acquired; wherein, real-time scene information represents the event corresponding to the real-time scene; based on the real-time scene information, factors corresponding to the real-time scene information are determined; wherein, factors represent the associated tags of the events corresponding to the real-time scene in a preset event factor library; the preset event factor library includes real-time scene information and factors corresponding to the real-time scene information; based on the factors and preset rule information, first data corresponding to the factors is determined; wherein, the first data is the business data required by the real-time scene in the real-time data warehouse; the preset rule information represents the business data processing rules corresponding to the real-time scene; the real-time data warehouse includes business data corresponding to multiple real-time scenes; based on the real-time scene information, the first data corresponding to the factors is pushed to the application system corresponding to the real-time scene information. By acquiring real-time application scenario information of the real-time data warehouse, including the events corresponding to those scenarios, and analyzing the events based on the acquired information, the system can find the corresponding events in a pre-defined event factor library. Then, it identifies the associated event factors. Based on these factors and the retrieved processing rules, the selected factors are processed within the real-time data warehouse. This process determines the necessary business data (the first data corresponding to the factor) from the business data of multiple real-time scenarios within the data warehouse. The calculated business data is then promptly pushed to the application system corresponding to the real-time scenario, providing real-time business data consumption and offering different data services based on different real-time scenarios. This research framework, based on this method, connects to different development tasks, fully utilizing the data in the real-time data warehouse within this framework to improve data resource utilization.

[0094] Figure 2 A flowchart illustrating another data processing method provided in this application embodiment is shown below. Figure 2 As shown, the method includes:

[0095] S201. Obtain real-time data from various databases.

[0096] For example, the executing entity of this embodiment can be an electronic device, a server, a terminal device, or other apparatus or device capable of executing this embodiment. This embodiment uses an electronic device as an example for description.

[0097] For example, based on electronic devices, in order to meet the business needs of real-time scenarios, users first design a real-time data warehouse framework and connect to multiple different real-time databases. Then, they obtain data from the multiple connected real-time databases and perform the next step of computation and processing.

[0098] In one example, to meet the business needs of real-time scenarios, the user first designs a real-time data warehouse framework and connects to multiple different real-time databases, such as the open-source database HBase and the relational database MySQL. Then, the user obtains real-time data from the multiple connected real-time databases, such as real-time business data such as location, roaming, orders, and acceptance, for further calculation and processing.

[0099] S202. Process the real-time data to obtain processed real-time data; and store the processed real-time data in layers.

[0100] For example, based on the computing framework of the real-time data warehouse, after acquiring real-time data, the real-time data is processed, including statistical analysis, data summarization, and format conversion, to obtain processed real-time data; based on the layered architecture of the real-time data warehouse, the processed real-time data is stored in layers in the real-time data warehouse to realize the layered and domained nature of real-time data and the sharing of capabilities.

[0101] In one example, a real-time data warehouse-based computing framework can deploy a message processing system like Kafka or a log collection system like Flume, and introduce the Hudi component. After acquiring real-time data, it performs computational processing, including statistical analysis, data aggregation, and format conversion, resulting in processed real-time data stored in Hudi format. This is based on a layered architecture of the real-time data warehouse. Figure 3 This is a schematic diagram illustrating a real-time data warehouse layered architecture according to an exemplary embodiment, such as Figure 3As shown, for example, there are three data layers: Operational Data Store (ODS), Data Warehouse Detail (DWD), and Data Warehouse Appliance (DWA). The ODS layer includes location signaling data and roaming signaling data; the DWD layer includes user location details and user roaming details; and the DWA layer includes real-time user location data, user roaming data, user roaming out data, and real-time pedestrian traffic data from base stations. Data is imported from data sources through a message processing system like Kafka or a log collection system like Flume. Based on the Flink streaming computing framework, the processed real-time data is then stored layer by layer in the real-time data warehouse, following the data granularity from ODS to DWD to DWA. Each layer represents a different level of data granularity and information, providing data services such as reports, thus achieving layered and domain-specific real-time data and capability sharing.

[0102] S203, responding to a data query request; and determining third data; wherein the data query request is used to instruct the querying of data; and the third data is processed real-time data in a preset external table.

[0103] For example, if data needs to be extracted from the real-time data warehouse, after the acquired real-time data is stored in layers, an external table will be created in the data storage directory, i.e., a preset external table. When a data query request is received, the external table needs to be queried first. Based on the query engine, the data storage location that is connected to the external table is obtained, and the real-time data file corresponding to the external table is obtained from that location.

[0104] In one example, after tiered storage of the acquired real-time data, based on the introduction of the Hudi component, the acquired dataset is stored in the metadata directory of the base path. An external table (read-only database table) is created in the storage directory of the Hudi data lake on the distributed file system HDFS (Hadoop Distributed File System). When querying the external table, an SQL database statement is issued, and the query optimization engine calls the Hudi interface through JNI (Java Native Interface) to retrieve all valid Parquet file format data files at the current time point from the specified data directory.

[0105] S204. Parse the third data to obtain and return the query results; whereby the query results represent the business data requested by the data query request.

[0106] For example, after obtaining the real-time data file corresponding to the external table, the file is parsed based on the supporting large-scale parallel processing system MPP (Massively Parallel Processing) component and SQL database statements to obtain the business data requested by the data query request, and the business data is returned.

[0107] In one example, after obtaining all valid Parquet (file format) data files at the current time, load balancing is performed by HDFS block (the smallest unit of read / write data) to generate concurrent scan sub-plans. The execution engine scans the Parquet data files according to the sub-plans, performs SQL statement calculations, obtains the business data requested by the data query request, and returns the business data, realizing batch and stream integrated data calculation based on Hudi data lake.

[0108] S205. Obtain real-time scene information; wherein, real-time scene information represents the events corresponding to the real-time scene.

[0109] For example, it is possible to obtain real-time application scenario information of the real-time data warehouse, including the events corresponding to the real-time application scenario.

[0110] S206. If it is determined that there is no factor corresponding to the real-time scene information in the preset event factor library, the real-time scene information is processed to generate the factor corresponding to the real-time scene information.

[0111] For example, based on a preset event factor library, which includes real-time scene information and factors corresponding to the real-time scene information, i.e., events corresponding to the real-time scene and event factors that have a preset association with the events corresponding to the real-time scene, i.e., association tags; if, based on the obtained real-time scene information, it is determined that no event corresponding to the real-time scene is found in the preset event factor library, or no event factor associated with the event is found, then the event corresponding to the real-time scene is analyzed based on the factor analysis method to obtain the event factors that have a preset association with the events corresponding to the real-time scene.

[0112] S207. Store the real-time scene information and the factors corresponding to the real-time scene information in a preset event factor library; wherein, the preset event factor library includes the real-time scene information and the factors corresponding to the real-time scene information.

[0113] For example, after obtaining the event factors associated with the event corresponding to the real-time scene, the event corresponding to the real-time scene and the event factors associated with the event corresponding to the real-time scene are stored in a preset event factor library to expand the event factor library.

[0114] After step S207, either step S208 or step S209 is executed.

[0115] S208. If it is determined that the event corresponding to the real-time scene represented by the real-time scene information is a single event, then the factors corresponding to the real-time scene information are determined according to the preset event factor library.

[0116] For example, after step S207, based on the information of the real-time scene obtained, that is, the event corresponding to the real-time scene, the event is analyzed and processed. If it is determined that the event corresponding to the real-time scene is a single event, the event type corresponding to the real-time scene can be found in the preset event factor library, the event corresponding to the real-time scene can be determined, and then all event factors associated with the event can be found. Then, one or more factor combinations can be selected from all the factors, that is, the associated tags of the event corresponding to the real-time scene.

[0117] In one example, after obtaining information about the real-time scene, i.e. the event corresponding to the real-time scene, the event is analyzed and processed. If it is determined that the event corresponding to the real-time scene is a single event, such as obtaining a hotspot geofence, for this event, after selecting the event, all factors of the event in the event factor library are associated, including restaurant locations, car maintenance locations, etc. The decision center selects one factor or a combination of factors according to the actual scenario. For example, for car-related applications, only the user's car maintenance location data needs to be selected.

[0118] S209. If it is determined that the event corresponding to the real-time scene represented by the real-time scene information is a composite event, then the main event and the auxiliary event are determined from the composite event.

[0119] In one example, the factors corresponding to real-time scene information include factors corresponding to the main event and factors corresponding to the auxiliary event.

[0120] For example, after step S207, the acquired real-time scene information, i.e. the event corresponding to the real-time scene, is analyzed and processed. If it is determined that the event corresponding to the real-time scene is a combined event, the event is analyzed first to determine the main event and the auxiliary event from the combined event. The factors corresponding to the real-time scene information in the preset factor library include the factors corresponding to the main event and the factors corresponding to the auxiliary event.

[0121] In one example, after acquiring the real-time scene information, i.e. the event corresponding to the real-time scene, the event is analyzed and processed. If it is determined that the event corresponding to the real-time scene is a composite event, the event is analyzed first to determine the main event and auxiliary event from the composite event. For example, for composite events related to a precision advertising platform, the main event is a hotspot geofence tag, and the auxiliary event is a real-time internet application.

[0122] S210. Based on real-time scene information, determine the factors corresponding to the main event; and based on real-time scene information and preset waiting time, determine the factors corresponding to the auxiliary event.

[0123] For example, after determining the primary and secondary events in a combined event, the primary event is analyzed and processed. This involves finding the event type corresponding to the real-time scenario in a pre-defined event factor library, determining the event corresponding to the real-time scenario, finding all event factors associated with the event, and then selecting one or more factor combinations from all factors to obtain the associated tags for the primary event corresponding to the real-time scenario. For the secondary event, a pre-defined waiting time is set. Within this waiting time, the secondary event needs to be analyzed and processed. Again, the primary event type corresponding to the real-time scenario is found in the pre-defined event factor library, the event corresponding to the real-time scenario is determined, and then all event factors associated with the event are found. Finally, one or more factor combinations from all factors are selected to obtain the associated tags for the secondary event corresponding to the real-time scenario.

[0124] S211. Based on the factors, determine the factor values ​​and the corresponding second data; wherein, the factor value represents the degree of correlation between the real-time scene information and the factor; the second data is business data that has a preset correlation with the factor.

[0125] For example, based on the factors corresponding to the determined events, the correlation between the real-time scenario information and the factors is determined according to the factors and the needs of the real-time scenario, i.e., the factor value. Based on the selected factors or combinations of factors, business data with a preset correlation to the factors are determined in the real-time data warehouse.

[0126] In one example, based on the factors corresponding to the determined events, the values ​​of the selected factors are input according to the actual scenario, and the association type of the selected factors is set according to the factors corresponding to the determined events, including text box association and data table association. Then, based on the association information, the business data that has a preset association with the factors is determined.

[0127] S212. Based on the preset rule information and factor values, process the second data corresponding to the factor to obtain the first data corresponding to the factor.

[0128] For example, in a real-time data warehouse, there is business data corresponding to multiple real-time scenarios. Based on the real-time scenario information, the business data processing rules corresponding to the real-time scenario can be retrieved, i.e., preset rule information. Based on the factor values ​​determined in the preset event factor library and the retrieved processing rules, the selected factors are processed in the real-time data warehouse. Then, among the business data corresponding to multiple real-time scenarios in the real-time data warehouse, the business data required for the real-time scenario, i.e., the first data corresponding to the factor, is determined.

[0129] In one example, the real-time data warehouse includes business data corresponding to multiple real-time scenarios. Each factor in the preset event factor library is associated with one-to-one business data in the real-time data warehouse. Based on the real-time scenario information, the business data processing rules corresponding to the real-time scenario can be retrieved, i.e., the preset rule information. This configured rule information and factor value are pushed to Redis (remote dictionary service) to match and generate the result set of the event scenario, i.e., the business data required by the real-time scenario.

[0130] After step S212, step S213 or step S214 can be executed.

[0131] S213. Based on the real-time scene information, push the first data corresponding to the factor to the application system corresponding to the real-time scene information.

[0132] For example, after step S212, step S213 can refer to step S104, which will not be repeated here.

[0133] S214. Process the first data corresponding to the factor to generate an analysis report; and send the analysis report to the application system.

[0134] For example, after step S212, based on business needs, business data can be processed using a real-time data warehouse to generate an offline database. Then, based on the offline database, analysis reports or inspection reports can be generated. Subsequently, the generated reports are directly pushed to the connected application systems to quickly respond to business implementation and provide real-time data services.

[0135] In this embodiment, based on the above embodiments, a data query request is responded to, and third data is determined. The data query request indicates that data should be queried. The third data is processed real-time data from a preset external table. The third data is parsed to obtain and return the query result. The query result represents the business data requested by the data query request. If the real-time scene information represents a single event corresponding to the real-time scene, the factor corresponding to the real-time scene information is determined according to a preset event factor library. If the real-time scene information represents a combined event corresponding to the real-time scene, the main event and auxiliary event are determined from the combined events. The factor corresponding to the main event is determined according to the real-time scene information. The factor corresponding to the auxiliary event is determined according to the real-time scene information and a preset waiting time. Based on the real-time scene requirements, the required real-time data can be flexibly and dynamically combined and selected based on the decision center, achieving rapid response to real-time business support. Based on the deployment of the Hudi component in the real-time data warehouse, rapid data processing is achieved. Based on the research and development framework of this method, different development tasks are connected, fully utilizing the data in the real-time data warehouse under this framework to improve the utilization rate of data resources.

[0136] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application, as shown below. Figure 4 As shown, the device 300 includes:

[0137] The first acquisition unit 301 is used to acquire real-time scene information; wherein, the real-time scene information represents the events corresponding to the real-time scene.

[0138] The first determining unit 302 is used to determine the factors corresponding to the real-time scene information based on the real-time scene information; wherein, the factors represent the associated tags of the events corresponding to the real-time scene in the preset event factor library; the preset event factor library includes the real-time scene information and the factors corresponding to the real-time scene information.

[0139] The second determining unit 303 is used to determine the first data corresponding to the factor based on the factor and the preset rule information; wherein, the first data is the business data required by the real-time scenario in the real-time data warehouse; the preset rule information represents the business data processing rules corresponding to the real-time scenario; the real-time data warehouse includes business data corresponding to multiple real-time scenarios.

[0140] The push unit 304 is used to push the first data corresponding to the factor to the application system corresponding to the real-time scene information based on the real-time scene information.

[0141] The apparatus in this embodiment can execute the technical solutions in the above method. Its specific implementation process and technical principles are the same, and will not be repeated here.

[0142] Figure 5This is a schematic diagram of another data processing apparatus provided in an embodiment of this application, as shown below. Figure 5 As shown, the device 400 includes:

[0143] The first acquisition unit 401 is used to acquire real-time scene information; wherein, the real-time scene information represents the events corresponding to the real-time scene.

[0144] The first determining unit 402 is used to determine the factors corresponding to the real-time scene information based on the real-time scene information; wherein, the factors represent the associated tags of the events corresponding to the real-time scene in the preset event factor library; the preset event factor library includes the real-time scene information and the factors corresponding to the real-time scene information.

[0145] The second determining unit 403 is used to determine the first data corresponding to the factor based on the factor and the preset rule information; wherein, the first data is the business data required by the real-time scenario in the real-time data warehouse; the preset rule information represents the business data processing rules corresponding to the real-time scenario; the real-time data warehouse includes business data corresponding to multiple real-time scenarios.

[0146] The push unit 404 is used to push the first data corresponding to the factor to the application system corresponding to the real-time scene information based on the real-time scene information.

[0147] In one example, the first determining unit 402 includes:

[0148] The first determining module 4021 is used to determine the factors corresponding to the real-time scene information according to a preset event factor library if the event corresponding to the real-time scene information representing the real-time scene is a single event.

[0149] In one example, the factors corresponding to the real-time scene information include factors corresponding to the main event and factors corresponding to the auxiliary event; the first determining unit 402 further includes:

[0150] The second determining module 4022 is used to determine the main event and the auxiliary event from the combined events if the event corresponding to the real-time scene information representing the real-time scene is determined to be a combined event.

[0151] The third determining module 4023 is used to determine the factor corresponding to the main event based on real-time scene information; and to determine the factor corresponding to the auxiliary event based on real-time scene information and preset waiting time.

[0152] In one example, the second determining unit 403 includes:

[0153] The third determining module 4031 is used to determine the factor value and the corresponding second data based on the factor; wherein, the factor value represents the degree of correlation between the real-time scene information and the factor; and the second data is business data that has a preset correlation with the factor.

[0154] The processing module 4032 is used to process the second data corresponding to the factor according to the preset rule information and factor value to obtain the first data corresponding to the factor.

[0155] In one example, before the first determining unit 402 determines the factor corresponding to the real-time scene information based on the real-time scene information, the method further includes:

[0156] The first generation unit 405 is used to process the real-time scene information and generate the factor corresponding to the real-time scene information if it is determined that there is no factor corresponding to the real-time scene information in the preset event factor library.

[0157] The first storage unit 406 is used to store real-time scene information and the factors corresponding to the real-time scene information in a preset event factor library; wherein, the preset event factor library includes real-time scene information and the factors corresponding to the real-time scene information.

[0158] In one example, the device also includes:

[0159] The second acquisition unit 407 is used to acquire real-time data from various databases.

[0160] The processing unit 408 is used to process real-time data to obtain processed real-time data.

[0161] The second storage unit 409 is used for hierarchical storage of the processed real-time data.

[0162] In one example, after the second storage unit 409 is used for tiered storage of processed real-time data, it also includes:

[0163] Response unit 410 is used to respond to a data query request.

[0164] The fourth determining unit 411 is used to determine the third data; wherein, the data query request is used to instruct the query of data; the third data is the processed real-time data in a preset external table.

[0165] The query unit 412 is used to parse the third data, obtain and return the query results; wherein, the query results represent the business data requested by the data query request.

[0166] In one example, the device also includes:

[0167] The second generation unit 413 is used to process the first data corresponding to the factors and generate an analysis report.

[0168] The sending unit 414 is used to send the analysis report to the application system.

[0169] The apparatus in this embodiment can execute the technical solutions in the above method. Its specific implementation process and technical principles are the same, and will not be repeated here.

[0170] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the electronic device 50 includes: a memory 51 and a processor 52; the memory 51 is a memory for storing executable instructions of the processor 52.

[0171] The processor 52 is configured to perform the methods provided in the above embodiments.

[0172] The terminal device also includes a receiver 53 and a transmitter 54. The receiver 53 is used to receive instructions and data sent by other devices, and the transmitter 54 is used to send instructions and data to external devices.

[0173] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. The device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0174] Electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output interface 812, sensor component 814, and communication component 816.

[0175] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0176] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory, Programmable Read-Only Memory, Read-Only Memory, Magnetic Storage, Flash Memory, Disk, or Optical Disk.

[0177] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0178] Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel. If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When electronic device 800 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0179] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0180] The input / output interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

[0181] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0182] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth Bit Torrent (BT), and other technologies.

[0183] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0184] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0185] This application also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the above-described method.

[0186] According to an embodiment of this application, this application also provides a computer program product, which includes: a computer program stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the solution provided in any of the above embodiments.

[0187] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0188] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized by, The method comprises: acquiring real-time scene information; wherein the real-time scene information represents an event corresponding to the real-time scene; determining a factor corresponding to the real-time scene information according to the real-time scene information; wherein the factor represents an associated label of an event corresponding to the real-time scene in a preset event factor library; the preset event factor library comprises real-time scene information and a factor corresponding to the real-time scene information; determining a factor value and second data corresponding to the factor according to the factor; wherein the factor value represents a correlation degree between the real-time scene information and the factor; the second data is business data having a preset correlation with the factor; processing the second data corresponding to the factor according to preset rule information and the factor value to obtain first data corresponding to the factor; wherein the first data is business data required by the real-time scene in a real-time data warehouse; the preset rule information represents a business data processing rule corresponding to the real-time scene; the real-time data warehouse comprises business data corresponding to a plurality of real-time scenes; pushing the first data corresponding to the factor to an application system corresponding to the real-time scene information according to the real-time scene information.

2. The method of claim 1, wherein, The method further comprises: if it is determined that the real-time scene information represents a single event corresponding to the real-time scene, determining the factor corresponding to the real-time scene information according to the preset event factor library.

3. The method of claim 2, wherein, The factor corresponding to the real-time scene information comprises a factor corresponding to a main event and a factor corresponding to an auxiliary event; The method further comprises: if it is determined that the real-time scene information represents a combined event, determining a main event and an auxiliary event from the combined event; determining the factor corresponding to the main event according to the real-time scene information; and determining the factor corresponding to the auxiliary event according to the real-time scene information and a preset waiting time length.

4. The method of claim 1, wherein, Before determining the factor corresponding to the real-time scene information according to the real-time scene information, the method further comprises: if it is determined that the factor corresponding to the real-time scene information does not exist in the preset event factor library, processing the real-time scene information to generate the factor corresponding to the real-time scene information; storing the real-time scene information and the factor corresponding to the real-time scene information in the preset event factor library; wherein the preset event factor library comprises real-time scene information and a factor corresponding to the real-time scene information.

5. The method according to any one of claims 1-4, characterized in that, The method further comprises: acquiring real-time data from each database; processing the real-time data to obtain processed real-time data; and storing the processed real-time data in layers.

6. The method of claim 5, wherein, After storing the processed real-time data in layers, the method further comprises: responding to a data query request; and determining third data; wherein the data query request is used to indicate a query data; the third data is processed real-time data in a preset external table; analyzing the third data to obtain and return a query result; wherein the query result represents business data requested by the data query request.

7. The method according to any one of claims 1-4, characterized in that, The method further comprises: The first data corresponding to the factor is processed to generate an analysis report, and the analysis report is sent to the application system.

8. A data processing apparatus, characterized by, The device comprises: A first obtaining unit is configured to obtain real-time scene information, wherein the real-time scene information represents an event corresponding to the real-time scene. A first determining unit is configured to determine a factor corresponding to the real-time scene information according to the real-time scene information, wherein the factor represents an associated label of an event corresponding to the real-time scene in a preset event factor library, and the preset event factor library comprises real-time scene information and a factor corresponding to the real-time scene information. A second determining unit is configured to determine first data corresponding to the factor according to the factor and preset rule information, wherein the first data is business data required by the real-time scene in a real-time data warehouse, the preset rule information represents a business data processing rule corresponding to the real-time scene, and the real-time data warehouse comprises business data corresponding to a plurality of real-time scenes. A pushing unit is configured to push the first data corresponding to the factor to an application system corresponding to the real-time scene information according to the real-time scene information. The second determining unit comprises: A fourth determining module is configured to determine a factor value and second data corresponding to the factor according to the factor, wherein the factor value represents a correlation degree between the real-time scene information and the factor, and the second data is business data having a preset association with the factor. A processing module is configured to process the second data corresponding to the factor according to the preset rule information and the factor value to obtain the first data corresponding to the factor.

9. The apparatus of claim 8, wherein, The first determining unit comprises: A first determining module is configured to determine the factor corresponding to the real-time scene information according to the preset event factor library if it is determined that the event corresponding to the real-time scene information is a single event.

10. The apparatus of claim 9, wherein, The factor corresponding to the real-time scene information comprises a factor corresponding to a main event and a factor corresponding to an auxiliary event. The first determining unit further comprises: A second determining module is configured to determine a main event and an auxiliary event from the combined event if it is determined that the event corresponding to the real-time scene information is a combined event. A third determining module is configured to determine the factor corresponding to the main event according to the real-time scene information, and determine the factor corresponding to the auxiliary event according to the real-time scene information and a preset waiting time length.

11. The apparatus of claim 8, wherein, Before the first determining unit is configured to determine the factor corresponding to the real-time scene information according to the real-time scene information, the device further comprises: A first generating unit is configured to process the real-time scene information to generate the factor corresponding to the real-time scene information if it is determined that the factor corresponding to the real-time scene information does not exist in the preset event factor library. A first storage unit is configured to store the real-time scene information and the factor corresponding to the real-time scene information in the preset event factor library, wherein the preset event factor library comprises real-time scene information and a factor corresponding to the real-time scene information.

12. The apparatus of any one of claims 8-11, wherein, The device further comprises: A second obtaining unit is configured to obtain real-time data from each database. A processing unit is configured to process the real-time data to obtain processed real-time data. A second storage unit is configured to store the processed real-time data in layers.

13. The apparatus of claim 12, wherein, After the second storage unit stores the processed real-time data in layers, the method further includes: a response unit configured to respond to a data query request; a fourth determination unit configured to determine third data; wherein the data query request is used to indicate the data to be queried; and the third data is the processed real-time data in a preset external table; a query unit configured to analyze the third data to obtain and return a query result; wherein the query result represents the service data requested by the data query request.

14. The apparatus of any one of claims 8-11, wherein, The apparatus further includes: a second generation unit configured to process the first data corresponding to the factor to generate an analysis report; a sending unit configured to send the analysis report to the application system.

15. An electronic device, comprising: include: a processor, and a memory connected with the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1-7.

16. A computer readable storage medium characterized by: The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1-7.

Images