Data updating method and device based on stream computing, equipment and medium

Abstract

Embodiments of the present disclosure provide a data updating method and device based on stream computing, equipment and medium. To solve the problems of not timely updating, not meeting the data volume, time-consuming and laborious, and serious lag of data timeliness in related technologies, the data updating method based on stream computing comprises: acquiring initial data in real time, and acquiring a plurality of data tables; determining the data table in the plurality of data tables that meets the preset condition as first dimension table data; performing dimension table association on the initial data and the first dimension table data to obtain first result data and store it.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data update method, apparatus, device and medium based on stream computing. Background Technology

[0002] In related technologies, with the continuous updating and iteration of new technologies, the demand for back-end data updates in various industries such as banking and telecommunications has also increased significantly. On the one hand, the demand for timely updates is strong, and on the other hand, the demand for updating data is large. However, the current approach mostly combines manual updates with daily batch processing, which results in drawbacks such as untimely updates, insufficient updated data volume, time-consuming and labor-intensive processes, and serious lag in data timeliness. Summary of the Invention

[0003] This disclosure provides a data updating method, apparatus, device, and medium based on stream computing to solve the technical problems in related technologies, such as untimely result updates, insufficient updated data volume, time-consuming and labor-intensive processes, and severely lagging data timeliness. The technical solution of this disclosure is as follows:

[0004] In a first aspect, embodiments of this disclosure provide a data update method based on stream computing, including:

[0005] Acquire initial data in real time and retrieve multiple data tables;

[0006] The data table that meets the preset conditions from multiple data tables is selected as the first dimension table;

[0007] Join the initial data with the data in the first dimension table to obtain the first result data and store it.

[0008] In one possible implementation, the method provided in this embodiment of the invention involves performing a dimension table association between initial data and first dimension table data to obtain and store first result data, including:

[0009] Use the initial data as the input data for the first data table in the first dimension table;

[0010] Join the input data of the data table with the dimension table to obtain the result data of the data table;

[0011] Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table.

[0012] Obtain the first result data and store it.

[0013] In one possible implementation, after determining the data table that meets preset conditions from multiple data tables as the first dimension table data, the method further includes:

[0014] Data tables that do not meet the preset conditions from multiple data tables are identified as data in the second dimension table;

[0015] Join the initial data with the data in the second dimension table to obtain the second result data and store it.

[0016] In one possible implementation, the method provided in this embodiment of the invention involves performing a dimension table association between initial data and second dimension table data to obtain and store second result data, including:

[0017] Based on multiple data tables in the second dimension table, the target data table is obtained;

[0018] The initial data is joined with the target data table using dimension tables to obtain and store the second result data.

[0019] In one possible implementation, the method provided in this embodiment of the invention involves performing a dimension table association between initial data and first dimension table data to obtain and store first result data, including:

[0020] Add a second result data to the initial data;

[0021] The added initial data is joined with the data in the first dimension table to obtain the first result data, which is then stored.

[0022] In one possible implementation, the method provided in this embodiment of the invention involves performing a dimension table association between the increased initial data and the first dimension table data to obtain and store first result data, including:

[0023] The increased initial data is used as the input data for the first data table in the first dimension table;

[0024] Join the input data of the data table with the dimension table to obtain the result data of the data table;

[0025] Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table.

[0026] Obtain the first result data and store it.

[0027] In one possible implementation, the method provided in this embodiment of the invention includes one or more of messages, event logs, and backend service logs as initial data.

[0028] Secondly, embodiments of this disclosure also provide a data update apparatus based on stream computing, comprising:

[0029] The acquisition unit is used to acquire initial data in real time and retrieve multiple data tables;

[0030] The determining unit is used to determine the data table that meets the preset conditions from multiple data tables as the data of the first dimension table;

[0031] The processing unit is used to perform dimension table association between the initial data and the data in the first dimension table to obtain the first result data and store it.

[0032] In one possible implementation, the processing unit in the apparatus provided by the embodiments of the present invention is specifically used for:

[0033] Use the initial data as the input data for the first data table in the first dimension table;

[0034] Join the input data of the data table with the dimension table to obtain the result data of the data table;

[0035] Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table.

[0036] Obtain the first result data and store it.

[0037] In one possible implementation, the device provided in this embodiment of the invention further includes a determining unit configured to determine data tables in multiple data tables that do not meet preset conditions as second-dimensional table data.

[0038] The processing unit is also used to perform dimension table association between the initial data and the second dimension table data to obtain and store the second result data.

[0039] In one possible implementation, the processing unit in the apparatus provided by the embodiments of the present invention is specifically used for:

[0040] Based on multiple data tables in the second dimension table, the target data table is obtained;

[0041] The initial data is joined with the target data table using dimension tables to obtain and store the second result data.

[0042] In one possible implementation, the processing unit in the apparatus provided by the embodiments of the present invention is specifically used for:

[0043] Add a second result data to the initial data;

[0044] The added initial data is joined with the data in the first dimension table to obtain the first result data, which is then stored.

[0045] In one possible implementation, the processing unit in the apparatus provided by the embodiments of the present invention is further configured to:

[0046] The increased initial data is used as the input data for the first data table in the first dimension table;

[0047] Join the input data of the data table with the dimension table to obtain the result data of the data table;

[0048] Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table.

[0049] Obtain the first result data and store it.

[0050] In one possible implementation, the apparatus provided in this embodiment of the invention includes an acquisition unit for acquiring initial data in real time and acquiring multiple data tables, wherein the initial data includes one or more of messages, event logs, and backend service logs.

[0051] Thirdly, embodiments of this disclosure also provide an electronic device, including:

[0052] processor;

[0053] Memory used to store the processor's executable instructions;

[0054] The processor is configured to execute the instructions to implement the method described in the first aspect.

[0055] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the method described in the first aspect.

[0056] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the method described in the first aspect.

[0057] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0058] In the embodiments of this disclosure, initial data is first acquired in real time, and multiple data tables are obtained. Then, data tables that meet preset conditions are determined as the first dimension table data. Finally, the initial data and the first dimension table data are joined to obtain the first result data, which is then stored. This provides a data update scheme based on stream computing, which can ensure the real-time update of data with high timeliness requirements while maintaining the data volume, achieving stream-batch integration. This effectively reduces manpower consumption and makes the data more timely, comprehensive, and accurate.

[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0060] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0061] Figure 1 A flowchart illustrating a data update method based on stream computing provided in an embodiment of this disclosure;

[0062] Figure 2 A schematic diagram illustrating a specific process of a data update method based on stream computing provided in this embodiment of the disclosure;

[0063] Figure 3 A flowchart illustrating a data table acquisition method provided in an embodiment of this disclosure;

[0064] Figure 4 A flowchart illustrating an initial data acquisition method provided in an embodiment of this disclosure;

[0065] Figure 5 A flowchart illustrating an initial data processing method provided in an embodiment of this disclosure;

[0066] Figure 6 A schematic diagram illustrating a dimension table association process provided in an embodiment of this disclosure;

[0067] Figure 7 This is a schematic diagram illustrating the specific process of another data update method based on stream computing provided in an embodiment of this disclosure;

[0068] Figure 8 A schematic diagram illustrating the process of associating another dimension table as provided in an embodiment of this disclosure;

[0069] Figure 9 A schematic diagram of a data update device based on stream computing provided in an embodiment of this disclosure;

[0070] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0071] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0072] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0073] Before providing a detailed explanation of the stream computing-based data update method provided in this disclosure, let's first introduce the key terms involved, including:

[0074] Storm is a distributed real-time computing system. It is a free, open-source, distributed, and highly fault-tolerant real-time computing system that simplifies continuous stream computing.

[0075] Spark: A memory-based iterative computing framework.

[0076] Kafka: An open-source stream processing platform.

[0077] Flink is a distributed system that requires computing resources to execute applications.

[0078] HBase is a distributed, column-oriented open-source database.

[0079] Event-Driven Architecture (EDA): A way to achieve maximum loose coupling between components or services by using events as a medium.

[0080] OGG (Oracle Golden Gate) is a comprehensive software package used to retrieve data updates from a source database and send the changed data to the target database. Real-time data providers only need to grant read permissions to the corresponding database logs, resulting in minimal impact on application performance.

[0081] Change data capture (CDC) is a process used to identify and capture data created, deleted, modified, or queried in database tables and transmit it to downstream processes or systems in real time via message middleware according to the order of change records.

[0082] The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0083] Figure 1A flowchart of a data update method based on stream computing provided in the first embodiment of this disclosure is shown below. Figure 1 As shown, this stream computing-based data update method may include the following steps:

[0084] S101 acquires initial data in real time and retrieves multiple data tables.

[0085] In practice, multiple data tables from various sources are acquired, along with initial data obtained in real time. This initial data includes one or more types of data such as message logs, event tracking logs, and backend service logs, and is mostly time-sensitive data. This initial data is acquired and updated in real time. The acquired data tables can also be cleaned, transformed, processed, and integrated.

[0086] S102, the data table that meets the preset conditions in multiple data tables is determined as the first dimension table data.

[0087] In specific implementation, data tables that meet preset conditions are determined as the first dimension table data, and data tables that do not meet preset conditions are determined as the second dimension table data. The preset conditions are to judge the timeliness of the data, and can be set according to actual needs. This embodiment of the invention does not limit this.

[0088] S103, perform dimension table association between the initial data and the data in the first dimension table to obtain the first result data and store it.

[0089] In practice, the initial data is joined with the data in the first dimension table to obtain the first result data, which is then stored. The initial data is then joined with the data in the second dimension table to obtain the second result data, which is then stored.

[0090] The method for joining data in the first dimension table is as follows: Using the initial data as input to the first table in the first dimension table, join the input data of the first table with the first table in the first dimension table to obtain the result data of the first table. Then, using the result data of each table as input to the next table, join the first table in the first dimension table sequentially to obtain and store the first result data. In other words, the initial data is sequentially joined with each table in the first dimension table, and the result data is used as input for the next join, finally obtaining the final result data, which is the first result data.

[0091] The method for associating data in the second dimension table is as follows: Based on multiple data tables in the second dimension table, a target data table is obtained. The initial data is then joined with the target data table to obtain the second result data, which is then stored. In other words, all data tables are unified into a single data table, and then the initial data is joined with this single data table.

[0092] In another approach, a second result data is added to the initial data, and then the added initial data is joined with the first dimension table data to obtain the first result data, which is then stored. That is, the second result data, after joining the second dimension table data, is also joined with the first dimension table data.

[0093] Next, taking its application in the banking system as an example, combined with... Figure 2 , Figure 3 This disclosure provides a detailed description of the data update method based on stream computing.

[0094] Figure 2 This is a schematic diagram illustrating a specific process of a data update method based on stream computing provided in an embodiment of this disclosure, specifically including as follows: Figure 2 The following steps are shown:

[0095] S201, acquires initial data in real time and retrieves multiple data tables.

[0096] In practice, multiple data tables from various sources are acquired, along with initial data obtained in real time. This initial data includes one or more types of data such as message logs, event tracking logs, and backend service logs, and is mostly time-sensitive data. This initial data is acquired and updated in real time. The acquired data tables can also be cleaned, transformed, processed, and integrated.

[0097] In one example, taking a banking system as an example, such as Figure 3 As shown, the data tables from the warehouse posting area, result data tables involving other systems or projects, or external data tables are synchronized to the first database, i.e., the MPP database of this system, through authorized access or scheduled replication. Then, the data is cleaned, transformed, processed, and integrated. The processed data tables are stored in the MPP database of this system. For the initial data, it is as follows... Figure 4 As shown, data is collected in real time using three acquisition methods: EDA (Early Data Acquisition), OGG (Online Data Acquisition), and CDC (Data Collection and Distribution). This data primarily includes packets, event logs, and backend service logs. The initial collected data is then stored in real-time in the second database, the KAFKA database. KAFKA consists of three layers: the raw layer, the standard layer, and the access layer. This stage primarily involves the standard layer. Then, as... Figure 5 As shown, the collected initial data is cleaned, transformed, and standardized. The standardized data is then stored in KAFKA for data preprocessing in the subsequent stream computing development stage.

[0098] S202, determine the data table that meets the preset conditions from multiple data tables as the first dimension table data.

[0099] In practice, the timeliness of the data in the data table is used as the criterion for judgment. The data table is divided into a first-dimensional data table and a second-dimensional data table. Data tables that meet the preset conditions are determined as first-dimensional data tables, and data tables that do not meet the preset conditions are determined as second-dimensional data tables.

[0100] Using the same example as above, store the data of the first dimension table in the third database, namely the HBase database, and store the data of the second dimension table in the fourth database, namely the Oracle database.

[0101] S203, perform dimension table association between the initial data and the data in the first dimension table respectively, obtain the result data and store it.

[0102] In a specific implementation, one approach is to perform a dimension table association between the initial data and the data in the first dimension table to obtain and store the first result data, and then perform a dimension table association between the initial data and the data in the second dimension table to obtain and store the second result data.

[0103] In another approach, only the initial data is joined with the data in the first dimension table to obtain the first result data, which is then stored.

[0104] Continuing with the example above, such as Figure 6 As shown, for the first-dimensional table data with high timeliness requirements, a FLINK SQL operator is used to join the initial data with the first data table in the first-dimensional table data. Then, a second newly created FLINK SQL operator is used to join the resulting data with the second data table, and so on, until all data tables are joined. For the second-dimensional table data with lower timeliness requirements, two or more dimension tables are joined in batches and integrated into a single dimension table. The integrated dimension table is then joined with the initial data for stream computing development. Finally, the resulting data is stored in a fifth database (ES database) or an Oracle database. This effectively solves the defect of the cloud platform's FLINK SQL operator not supporting dimension table joins, realizes the stream computing development process, and meets business requirements.

[0105] In another example, only the data in the first dimension table, which has a high time sensitivity, is joined.

[0106] S204, the interface for accessing configuration result data.

[0107] In practice, an access interface is set up for the stored result data so that other applications can access the result data in real time.

[0108] Continuing with the previous example, configure the API interface of the big data cloud platform for the ES or Oracle database, and provide the results of stream computing development to various application systems through the data service interface. Each application component calls the cloud platform API interface to query and display the initial data, effectively meeting the business requirements for timeliness and accuracy.

[0109] Figure 7 This is a schematic diagram illustrating a specific process of another data update method based on stream computing provided in an embodiment of this disclosure, specifically including as follows: Figure 7 The following steps are shown:

[0110] S701 acquires initial data in real time and retrieves multiple data tables.

[0111] In practice, multiple data tables from various sources are acquired, along with initial data obtained in real time. This initial data includes one or more types of data such as message logs, event tracking logs, and backend service logs, and is mostly time-sensitive data. This initial data is acquired and updated in real time. The acquired data tables can also be cleaned, transformed, processed, and integrated.

[0112] In one example, taking a banking system as an example, such as Figure 7 As shown, the data tables from the warehouse posting area, result data tables involving other systems or projects, or external data tables are synchronized to the MPP database of this system through authorized access or scheduled replication. Then, the data is cleaned, transformed, processed, and integrated. The processed data tables are stored in the MPP database of this system. For the initial data, it is as follows... Figure 4 As shown, data is collected in real-time using three acquisition methods: EDA, OGG, and CDC. This primarily includes message collection, event logs, and backend service logs. The initial collected data is then stored in the KAFKA database in real-time. KAFKA consists of three layers: the raw layer, the standard layer, and the access layer. This stage primarily involves the standard layer. Then, as... Figure 5 As shown, the collected initial data is cleaned, transformed, and standardized. The standardized data is then stored in KAFKA for data preprocessing in the subsequent stream computing development stage.

[0113] S702, the data table that meets the preset conditions in multiple data tables is determined as the first dimension table data, and the data table that does not meet the preset conditions in multiple data tables is determined as the second dimension table data.

[0114] In practice, the timeliness of the data in the data table is used as the criterion for judgment, and the data table is divided into first-dimensional table data and second-dimensional table data.

[0115] Using the same example as above, store the data for the first dimension table in the HBase database and the data for the second dimension table in the Oracle database.

[0116] S703, perform dimension table associations between the initial data and the data in the first and second dimension tables respectively, obtain the result data and store it.

[0117] In practice, the initial data is joined with the data in the first dimension table to obtain the first result data, which is then stored. The initial data is then joined with the data in the second dimension table to obtain the second result data, which is then stored.

[0118] Continuing with the example above, such as Figure 8 As shown, for second-dimensional table data with lower timeliness requirements, two or more dimension tables are batch-linked and integrated into a single dimension table. The integrated dimension table is then linked with the initial data for stream computing development, yielding the second result data. For first-dimensional table data with higher timeliness requirements, the second result data is added to the initial data in real-time. Then, a Flink SQL operator is used to link the added initial data with the first data table in the first-dimensional table. A second, newly created Flink SQL operator is then used to link the result data with the second data table, continuing until all data tables are linked, yielding the first result data. Each result data is temporarily stored in a Kafka database. Finally, the first and second result data are stored in an Elasticsearch (ES) database or an Oracle database. This effectively solves the limitation of cloud platform Flink SQL operators not supporting dimension table linking, enabling stream computing development and meeting business requirements.

[0119] S704, the interface for accessing configuration result data.

[0120] In practice, an access interface is set up for the stored result data so that other applications can access the result data in real time.

[0121] Continuing with the previous example, configure the API interface of the big data cloud platform for the ES or Oracle database, and provide the results of stream computing development to various application systems through the data service interface. Each application component calls the cloud platform API interface to query and display the initial data, effectively meeting the business requirements for timeliness and accuracy.

[0122] Based on the same inventive concept, embodiments of this disclosure also provide a data update device based on stream computing. For example... Figure 9 As shown, the stream computing-based data update device 900 includes:

[0123] Acquisition unit 901 is used to acquire initial data in real time and acquire multiple data tables;

[0124] The determining unit 902 is used to determine the data table that meets the preset conditions from multiple data tables as the first dimension table data;

[0125] The processing unit 903 is used to perform dimension table association between the initial data and the data in the first dimension table to obtain and store the first result data.

[0126] In one possible implementation, the processing unit 903 in the apparatus provided by the embodiments of the present invention is specifically used for:

[0127] Use the initial data as the input data for the first data table in the first dimension table;

[0128] Join the input data of the data table with the dimension table to obtain the result data of the data table;

[0129] Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table.

[0130] Obtain the first result data and store it.

[0131] In one possible implementation, the determining unit 902 in the apparatus provided by the embodiments of the present invention is further configured to determine data tables in multiple data tables that do not meet preset conditions as second dimension table data;

[0132] Processing unit 903 is also used to perform dimension table association between the initial data and the second dimension table data to obtain and store the second result data.

[0133] In one possible implementation, the processing unit 903 in the apparatus provided by the embodiments of the present invention is specifically used for:

[0134] Based on multiple data tables in the second dimension table, the target data table is obtained;

[0135] The initial data is joined with the target data table using dimension tables to obtain and store the second result data.

[0136] In one possible implementation, the processing unit 903 in the apparatus provided by the embodiments of the present invention is specifically used for:

[0137] Add a second result data to the initial data;

[0138] The added initial data is joined with the data in the first dimension table to obtain the first result data, which is then stored.

[0139] In one possible implementation, the processing unit 903 in the apparatus provided by the embodiments of the present invention is further configured to:

[0140] The increased initial data is used as the input data for the first data table in the first dimension table;

[0141] Join the input data of the data table with the dimension table to obtain the result data of the data table;

[0142] Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table.

[0143] Obtain the first result data and store it.

[0144] In one possible implementation, the apparatus provided in this embodiment of the invention includes an acquisition unit 901 for acquiring initial data in real time and acquiring multiple data tables, wherein the initial data includes one or more of messages, event logs, and backend service logs.

[0145] The specific implementation and technical effects of the device provided in this disclosure are similar to those of the method embodiments described above, and will not be repeated here.

[0146] In addition, combined Figures 1-9 The data update method and apparatus based on stream computing described in the embodiments of this application can be implemented by an electronic device. Figure 10 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0147] like Figure 10 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003 to implement the stream computing-based data update method as described in the embodiments of this disclosure. Various programs and data required for the operation of the electronic device 1000 are also stored in the RAM 1003. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0148] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0149] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the voice control method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0150] This disclosure also provides a vehicle, including the data update device and electronic device based on stream computing described in the above embodiments.

[0151] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0152] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0153] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0154] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0155] Acquire initial data in real time and retrieve multiple data tables;

[0156] The data table that meets the preset conditions from multiple data tables is selected as the first dimension table;

[0157] Join the initial data with the data in the first dimension table to obtain the first result data and store it.

[0158] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0159] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0161] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0162] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0163] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0164] The data update method based on stream computing mentioned in the embodiments of the present invention includes:

[0165] First, initial data is acquired in real time, and multiple data tables are obtained. Then, data tables that meet preset conditions are selected as the first dimension table. Finally, the initial data and the first dimension table are joined to obtain the first result data, which is then stored. This provides a data update scheme based on stream computing, which can update time-sensitive data in real time while ensuring data volume, achieving stream-batch integration. This effectively reduces manpower consumption and makes data more timely, comprehensive, and accurate.

[0166] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0171] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A data update method based on stream computing, characterized in that, include: Initial data is acquired in real time, and multiple data tables are obtained. The initial data includes one or more of the following: messages, event logs, and backend service logs. The initial data is collected through the first collection method EDA, the second collection method OGG, and the third collection method CDC, and then stored in the KAFKA database. Data tables that meet preset conditions are identified as first-dimensional table data, and data tables that do not meet preset conditions are identified as second-dimensional table data. The preset condition is data timeliness. The data timeliness of the second-dimensional table data is lower than that of the first-dimensional table data. The first-dimensional table data is stored in the HBase database, and the second-dimensional table data is stored in the Oracle database. The process of associating the initial data with the second dimension table data to obtain and store the second result data includes: batch associating two or more dimension tables and integrating them into one dimension table. Adding the second result data to the initial data includes: associating a well-integrated dimension table with the initial data, performing stream computing development, and obtaining the second result data; The process of adding the initial data and associating it with the first dimension table data to obtain and store the first result data includes: using a FLINK SQL operator to associate the added initial data with the first data table in the first dimension table data to obtain the result data, and using a second newly opened FLINK SQL operator to associate the result data with the second data table, until all data tables are associated. Configure an access interface for the stored result data; The method further includes: during the association process, the result data for each step is stored in the KAFKA library, and after the association is completed, the first result data and the second result data are stored in the ES library.

2. The method according to claim 1, characterized in that, The method further includes: The process of performing a dimension table association between the initial data and the first dimension table data to obtain and store the first result data includes: The initial data is used as the input data for the first data table in the first dimension table data; The input data of the data table is joined with the data table through a dimension table to obtain the result data of the data table; Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table data. Obtain the first result data and store it.

3. The method according to claim 1, characterized in that, The step of performing a dimension table association between the initial data and the second dimension table data to obtain and store the second result data includes: Based on multiple data tables in the second dimension table, the target data table is obtained; The initial data is associated with the target data table using dimension tables to obtain and store the second result data.

4. The method according to claim 1, characterized in that, The step of performing a dimension table association between the increased initial data and the first dimension table data to obtain and store the first result data includes: The increased initial data is used as the input data for the first data table in the first dimension table; The input data of the data table is joined with the data table through a dimension table to obtain the result data of the data table; Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table data. Obtain the first result data and store it.

5. A data update device based on stream computing, characterized in that, include: The acquisition unit is used to acquire initial data in real time and acquire multiple data tables. The initial data includes one or more of messages, event logs, and backend service logs. The initial data is acquired through a first acquisition method (EDA), a second acquisition method (OGG), and a third acquisition method (CDC), and the initial data is stored in the KAFKA database. The determining unit is used to determine the data tables in the plurality of data tables that meet the preset conditions as the first dimension table data, and to determine the data tables in the plurality of data tables that do not meet the preset conditions as the second dimension table data. The preset conditions are data timeliness, the data timeliness of the second dimension table data is lower than that of the first dimension table data, the first dimension table data is stored in the HBase database, and the second dimension table data is stored in the Oracle database. Processing unit, used for: The process of associating the initial data with the second dimension table data to obtain and store the second result data includes: batch associating two or more dimension tables and integrating them into one dimension table. Adding the second result data to the initial data includes: associating a well-integrated dimension table with the initial data, performing stream computing development, and obtaining the second result data; The process of adding the initial data and associating it with the first dimension table data to obtain and store the first result data includes: using a FLINK SQL operator to associate the added initial data with the first data table in the first dimension table data to obtain the result data, and using a second newly opened FLINK SQL operator to associate the result data with the second data table, until all data tables are associated. Configure an access interface for the stored result data; The processing unit is further configured to: store the result data of each association in the KAFKA library during the association process, and store the first result data and the second result data in the ES library after the association is completed.

6. The apparatus according to claim 5, characterized in that, The processing unit is further configured to: The process of performing a dimension table association between the initial data and the first dimension table data to obtain and store the first result data includes: The initial data is used as the input data for the first data table in the first dimension table data; The input data of the data table is joined with the data table through a dimension table to obtain the result data of the data table; Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table data. Obtain the first result data and store it.

7. The apparatus according to claim 5, characterized in that, The processing unit is specifically used for: Based on multiple data tables in the second dimension table, the target data table is obtained; The initial data is associated with the target data table using dimension tables to obtain and store the second result data.

8. The apparatus according to claim 5, characterized in that, The processing unit is also used for: The increased initial data is used as the input data for the first data table in the first dimension table; The input data of the data table is joined with the data table through a dimension table to obtain the result data of the data table; Using the result data of each data table as the input data of the next data table, dimension table joins are performed sequentially on each data table in the first dimension table data. Obtain the first result data and store it.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data update method based on stream computing as described in any one of claims 1-4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data update method based on stream computing as described in any one of claims 1-4.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the data update method based on stream computing as described in any one of claims 1-4.

Images