Method and device for integrating real-time data, electronic equipment and storage medium

Abstract

The invention discloses a method and device for integrating real-time data, electronic equipment and a storage medium, and belongs to the technical field of data processing.The method comprises the steps that in response to an opening instruction of any real-time data integration task, real-time data of a data source is collected and read based on task configuration information of the real-time data integration task; the task configuration information comprises a mapping relation, monitoring the collected real-time data, and if it is monitored that any real-time data has metadata change information, pausing the real-time data integration task; sending a metadata information updating instruction to the write-in data source based on the metadata change information, so that the write-in data source modifies metadata information of the write-in data source according to the metadata change information; after a metadata information updating completion instruction sent by the write-in data source is received, the mapping relation is modified; and recovering to execute the real-time data integration task, and based on the modified mapping relationship, integrating the real-time data of the read data source into the write-in data source. And the timeliness of data real-time transmission is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for integrating real-time data. Background Art

[0002] With the continuous advancement of the city's digitalization process, various business departments have established many business systems. The use of offline data for business analysis can no longer meet the needs of fast-paced business activities. More and more users need to use real-time data for business analysis. However, due to the independence of the business of each business department, the data between business departments lacks relevance. In order to better conduct business analysis, the real-time data of each business department is integrated together, and business analysis is performed on the integrated data, which can provide better assistance for the business decision-making of each business department.

[0003] However, since the business of each business department may change, the metadata information of the data may be modified. In the prior art, during the real-time data integration process, if the metadata information of the data at the input end of the real-time data integration system changes, manual intervention is usually required to suspend the integration of the real-time data, and modify the metadata information of the data at the output end of the real-time data integration system, and then rebuild the integration task to continue integrating the real-time data. This method not only reduces the timeliness of real-time data transmission, but also increases the workload of operation and maintenance. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and storage medium for integrating real-time data, so as to improve the timeliness of real-time data transmission and reduce the workload of operation and maintenance.

[0005] In a first aspect, an embodiment of the present application provides a method for integrating real-time data, including:

[0006] In response to a start instruction of any real-time data integration task, real-time data of at least one read data source is collected based on task configuration information of the real-time data integration task; the task configuration information includes metadata information of the read data source, metadata information of the write data source, and a mapping relationship between the metadata information of the read data source and the metadata information of the write data source; wherein the mapping relationship is used to represent the position where the real-time data integration system integrates the collected real-time data into the write data source;

[0007] The collected real-time data is monitored, and if metadata change information is detected in any real-time data, the real-time data integration task is suspended;

[0008] Based on the metadata change information, sending a metadata information update instruction to the write data source, so that the write data source modifies the metadata information of the write data source according to the metadata change information;

[0009] After receiving the metadata information update completion instruction sent by the write data source, modifying the mapping relationship;

[0010] The real-time data integration task is resumed, and based on the modified mapping relationship, the real-time data of the read data source is integrated into the write data source.

[0011] In the embodiment of the present application, it is only necessary to build an integration task once to complete the integration of real-time data. If the metadata information of the data source read is changed during the real-time data integration process, no manual intervention is required, and the metadata of the data source written and the task configuration information of the integration task can be automatically modified synchronously, thereby improving the timeliness of real-time data transmission and reducing the operation and maintenance workload.

[0012] In some embodiments, the collecting real-time data of at least one reading data source based on the task configuration information of the real-time data integration task includes:

[0013] Based on the type of the read data source, determine a real-time data reading plug-in corresponding to the data source type of each read data source;

[0014] Based on the determined real-time data reading plug-ins, real-time data of each reading data source is collected.

[0015] In the embodiment of the present application, different reading plug-ins are used for different reading data sources, thereby enhancing the convenience of real-time data integration.

[0016] In some embodiments, after pausing the real-time data integration task, the method further includes:

[0017] Recording the location of the real-time data currently being collected;

[0018] The resuming and executing the real-time data integration task includes:

[0019] Based on the recorded positions of the real-time data currently being collected, the data positions in the read data source corresponding to the positions of each real-time data are located, and starting from the located data positions in each read data source, the real-time data integration task is resumed.

[0020] In an embodiment of the present application, the data positions of each data source read when the integration task is paused are recorded, and the integration task is resumed from the paused position without re-executing the integration task, thereby further improving the timeliness of real-time data transmission.

[0021] In some embodiments, the task configuration information further includes a conversion logic of a field, and the conversion logic of the field is used to represent a processing strategy for real-time data corresponding to the field during the execution of the real-time data integration task;

[0022] After modifying the mapping relationship and before resuming the execution of the real-time data integration task, the method further includes:

[0023] If the metadata change information indicates that a field is modified, a modification strategy of a predefined conversion logic of the field is determined, and based on the determined modification strategy, the conversion logic of the field in the task configuration information is modified.

[0024] In an embodiment of the present application, if the metadata change information represents a modification to a field, the conversion logic of the field in the task configuration information can be modified accordingly based on the modification strategy of the field conversion logic. There is no need for manual intervention to modify the conversion logic of the field in the task configuration information, thereby reducing the operation and maintenance workload and improving the timeliness of real-time data transmission.

[0025] In some embodiments, the modification strategy includes synchronously updating conversion logic and canceling conversion logic, and the modification of the conversion logic of the field in the task configuration information based on the modification strategy includes:

[0026] If the modification strategy is to synchronously update the conversion logic, then modify the conversion logic of the field in the task configuration information to the conversion logic before the field is changed;

[0027] If the modification strategy is to cancel the conversion logic, the conversion logic of the field in the task configuration information is deleted.

[0028] In the embodiment of the present application, two possible modification strategies, namely, synchronously updating the conversion logic and canceling the conversion logic, are provided to make real-time data integration more convenient.

[0029] In some embodiments, after pausing the real-time data integration task and before sending a metadata information update instruction to the write data source based on the metadata change information, the method further includes:

[0030] Sending an instruction to back up the data before the change to the write data source;

[0031] After receiving the pre-change data backup completion instruction sent by the write data source, a step of sending a metadata information update instruction to the write data source is performed.

[0032] In an embodiment of the present application, an instruction to back up the data before the change is first sent to the write data source, and then a step of sending a metadata information update instruction to the write data source is executed, which can ensure the accuracy of data during real-time data integration and prevent data loss due to task failure.

[0033] In a second aspect, an embodiment of the present application provides a device for integrating real-time data, including:

[0034] A collection module, used to respond to a start instruction of any real-time data integration task, and to collect real-time data of at least one read data source based on the task configuration information of the real-time data integration task; the task configuration information includes metadata information of the read data source, metadata information of the write data source, and a mapping relationship between the metadata information of the read data source and the metadata information of the write data source; wherein the mapping relationship is used to represent the position where the real-time data integration system integrates the collected real-time data into the write data source;

[0035] A monitoring module is used to monitor the collected real-time data, and if metadata change information is detected in any real-time data, the real-time data integration task is suspended;

[0036] A sending module, configured to send a metadata information update instruction to the write data source based on the metadata change information, so that the write data source modifies the metadata information of the write data source according to the metadata change information;

[0037] A modification module, configured to modify the mapping relationship after receiving a metadata information update completion instruction sent by the write data source;

[0038] The integration module is used to resume the execution of the real-time data integration task and integrate the real-time data of the read data source into the write data source based on the modified mapping relationship.

[0039] In some embodiments, the acquisition module is specifically used to:

[0040] Based on the type of the read data source, determine a real-time data reading plug-in corresponding to the data source type of each read data source;

[0041] Based on the determined real-time data reading plug-ins, real-time data of each reading data source is collected.

[0042] In some embodiments, it also includes:

[0043] A recording module, used for recording the location of the real-time data currently collected after the monitoring module suspends the real-time data integration task;

[0044] The integrated module is specifically used for:

[0045] Based on the recorded positions of the real-time data currently being collected, the data positions in the read data source corresponding to the positions of each real-time data are located, and starting from the located data positions in each read data source, the real-time data integration task is resumed.

[0046] In some embodiments, the task configuration information further includes field conversion logic, and the field conversion logic is used to characterize a processing strategy for real-time data corresponding to the field during the execution of the real-time data integration task; after the modification module modifies the mapping relationship and before the integration module resumes execution of the real-time data integration task, the modification module is further used to:

[0047] If the metadata change information indicates that a field is modified, a modification strategy of a predefined conversion logic of the field is determined, and based on the determined modification strategy, the conversion logic of the field in the task configuration information is modified.

[0048] In some embodiments, the modification strategy includes synchronously updating conversion logic and canceling conversion logic, and the modification module is specifically used to:

[0049] If the modification strategy is to synchronously update the conversion logic, then modify the conversion logic of the field in the task configuration information to the conversion logic before the field is changed;

[0050] If the modification strategy is to cancel the conversion logic, the conversion logic of the field in the task configuration information is deleted.

[0051] In some embodiments, after the monitoring module suspends the real-time data integration task, before the sending module sends the metadata information update instruction to the write data source based on the metadata change information, the sending module is further used to:

[0052] Sending an instruction to back up the data before the change to the write data source;

[0053] After receiving the pre-change data backup completion instruction sent by the write data source, a step of sending a metadata information update instruction to the write data source is performed.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0055] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the above-mentioned method for integrating real-time data.

[0056] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the above-mentioned method of integrating real-time data.

[0057] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0059] Figure 1 A schematic diagram of an application scenario of a method for integrating real-time data provided in an embodiment of the present application;

[0060] Figure 2 A flowchart of a method for integrating real-time data provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of a mapping relationship during initialization provided in an embodiment of the present application;

[0062] Figure 4 Another mapping relationship diagram provided for an embodiment of the present application;

[0063] Figure 5 Another mapping relationship diagram provided for an embodiment of the present application;

[0064] Figure 6 A flowchart of another method for integrating real-time data provided by an embodiment of the present application;

[0065] Figure 7 A schematic diagram of the structure of a real-time data integration system provided in an embodiment of the present application;

[0066] Figure 8 A schematic diagram of a real-time data integration process provided by an embodiment of the present application;

[0067] Fig. 9 A schematic diagram of the structure of a device for integrating real-time data provided in an embodiment of the present application;

[0068] Fig.10 A schematic diagram of the hardware structure of an electronic device for implementing a method for integrating real-time data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to improve the timeliness of real-time data transmission and reduce the workload of operation and maintenance, the embodiments of the present application provide a method, device, electronic device and storage medium for integrating real-time data.

[0070] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0071] It should be noted that in the description of the embodiments of the present application, "plurality" refers to two or more than two, and the terms "first" and "second" are used only for descriptive purposes and cannot be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0072] With the continuous advancement of the city's digitalization process, various business departments have established many business systems. Using offline data for business analysis can no longer meet the needs of fast-paced business activities. More and more users need to use real-time data for business analysis. However, due to the independence of the business of each business department, the data between the business departments lacks relevance. In order to better conduct business analysis, the real-time data of each business department is integrated and the integrated data is analyzed, which can provide better assistance for the business decision-making of each business department. However, due to the possible changes in the business of each business department, the metadata information of the data may be modified.

[0073] In the related art, during the process of integrating real-time data, if the metadata information of the data source read by the real-time data integration system changes, it takes a certain amount of time from task failure to manual intervention, and the integration of real-time data is manually suspended, and the metadata information of the data source written by the real-time data integration system is modified, and then the integration task is rebuilt to continue integrating real-time data. Not only is the process of real-time data transmission affected, but the operation and maintenance workload is also increased. Therefore, it is particularly important to solve the problem of metadata information changes during the real-time data integration process. To this end, the present application provides a method for integrating real-time data to solve the above problem.

[0074] Figure 1 An application scenario diagram of a method for integrating real-time data provided in an embodiment of the present application includes a reading data source 100, a real-time data integration system 200, and a writing data source 300.

[0075] The reading data source 100 can be understood as the source of real-time data, such as a relational database, a non-relational database, a file system, an API interface, and the like.

[0076] The real-time data integration system 200 is used to collect real-time data from the read data source, process the real-time data according to the preset task configuration information, and transmit the processed real-time data to the write data source. The real-time data integration system 200 manages the operation cycle of multiple tasks, such as starting, stopping, monitoring, etc. The real-time data integration system 200 realizes communication and data exchange with different data sources by establishing adapters or connectors.

[0077] The write data source 300 can be understood as the destination of real-time data, such as a relational database, message middleware, etc.

[0078] It is worth noting that the application scenario diagram in the embodiment of the present application is intended to more clearly illustrate the technical solution in the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application, nor is it limited to the integration of real-time data. For other application architectures and business applications, the technical solution provided in the embodiment of the present application is also applicable to similar problems.

[0079] It should be noted that in each embodiment of the present application, the data processing method is applied to Figure 1 The application architecture shown is used as an example for schematic description.

[0080] After introducing the application scenarios of the embodiments of the present application, the method for integrating real-time data proposed in the present application is described below with specific embodiments. Figure 2 A flowchart of a method for integrating real-time data provided in an embodiment of the present application, the method is applied to Figure 1 In the real-time data integration system 200, the method includes the following steps.

[0081] In step 201, in response to a start instruction of any real-time data integration task, real-time data of at least one read data source is collected based on the task configuration information of the real-time data integration task; the task configuration information includes metadata information of the read data source, metadata information of the write data source, and a mapping relationship between the metadata information of the read data source and the metadata information of the write data source.

[0082] During specific implementation, the real-time data integration system provides users with a visual operation interface, and users can configure any real-time data integration task based on the visual operation interface. The task configuration information includes metadata information of reading data sources, metadata information of writing data sources, and a mapping relationship between metadata information of reading data sources and metadata information of writing data sources. The metadata information includes at least data source name, data source type, table information, field information, etc.

[0083] The mapping relationship is used to represent the location where the real-time data integration system integrates the collected real-time data into the data source. For example, Figure 3 A schematic diagram of a mapping relationship during initialization is provided for an embodiment of the present application, wherein the read data source is data source A, the write data source is data source B, and there are 5 fields in a table in data source A, namely a, b, c, d and e. If the real-time data integration task requires that the prefix "T" be added to each field during real-time data transmission, that is, a becomes Ta, b becomes Tb, c becomes Tc, d becomes Td, and e becomes Te, then when the real-time data integration task is initialized, the mapping relationship between the metadata information of the read data source and the metadata information of the write data source is established as follows: Figure 3 As shown, thereafter, the real-time data integration system can determine the location of the write data source to integrate the collected real-time data according to the mapping relationship between the metadata information of the read data source and the metadata information of the write data source.

[0084] In step 202, the collected real-time data is monitored, and if metadata change information is detected in any real-time data, the real-time data integration task is suspended.

[0085] In specific implementation, based on the type of the read data source, a real-time data reading plug-in corresponding to the data source type of each read data source can be determined, and based on the determined real-time data reading plug-ins, real-time data of each read data source can be collected.

[0086] For example, if the read data source is the relational database MySQL, the corresponding read plug-in can be used to collect the binlog log of the MySQL database to collect real-time data of the MySQL database. Binlog is a binary log maintained by the MySQL database, which is mainly used to record SQL statements that update or potentially update MySQL data.

[0087] During specific implementation, the collected real-time data can be parsed to obtain metadata change information in the real-time data. For example, binlog log data can be parsed. If metadata change information is found in the binlog log, the real-time data integration task is suspended.

[0088] For example, the statements in the binlog log are as follows:

[0089] "ALTER TABLE`mydb`.`order_info`

[0090] ADD COLUMN`tel`varchar(255)NULL AFTER`addr`”

[0091] The above statement indicates that the `mydb`.`order_info` table is modified, and the field `tel` column is added after the field `addr`, and the data type is 'varchar'. When the above statement is listened to, it is determined that metadata change information is listened to in the real-time data, and the real-time data integration task is suspended.

[0092] During specific implementation, after pausing the real-time data integration task, the location of the currently collected real-time data can also be recorded, such as the log location of the database, the message offset of the message middleware, etc.

[0093] In step 203, based on the metadata change information, a metadata information update instruction is sent to the writing data source, so that the writing data source modifies the metadata information of the writing data source according to the metadata change information.

[0094] For example, metadata change information includes adding a new field, modifying a field, or deleting a field.

[0095] During specific implementation, if the metadata change information is to delete field a, the write data source will delete field a and the data corresponding to field a after executing the metadata information change. Therefore, in order to ensure the accuracy of data during the real-time data integration process and prevent data loss caused by task failure, after pausing the real-time data integration task, based on the metadata change information, before sending a metadata information update instruction to the write data source, an instruction to back up the data before the change is sent to the write data source is sent. After receiving the pre-change data backup completion instruction sent by the write data source, the step of sending the metadata information update instruction to the write data source is executed.

[0096] In step 204, after receiving the metadata information update completion instruction sent by the write data source, the mapping relationship is modified.

[0097] In specific implementation, the mapping relationship can be modified accordingly according to the metadata change information, such as adding a mapping relationship of a new field to the mapping relationship, modifying the field name corresponding to the mapping relationship of any field, and deleting the mapping relationship of a field.

[0098] For example, the above Figure 3 The mapping relationship in the example is the mapping relationship when the real-time data integration task is initialized. If the field a of data source A is changed to field aa, the corresponding mapping relationship is modified as follows Figure 4 As shown, Figure 4 Another mapping relationship diagram provided in the embodiment of the present application is composed of Figure 4 It can be seen that in the mapping relationship, field a of data source A is changed to field aa, and the corresponding field Ta of data source B is changed to field Taa, and the mapping relationship of other fields remains unchanged.

[0099] For example, the above Figure 3 The mapping relationship in the example is the mapping relationship when the real-time data integration task is initialized. If the field a of data source A is deleted, the corresponding mapping relationship is modified as follows Figure 5 As shown, Figure 5 Another mapping relationship diagram provided in the embodiment of the present application is composed of Figure 5 It can be seen that in the mapping relationship, the field a of the data source A and the corresponding field Ta of the data source B are deleted, and the mapping relationship of the other fields remains unchanged.

[0100] In a specific implementation, the task configuration information also includes field conversion logic, wherein the field conversion logic is used to represent a processing strategy for real-time data corresponding to the field during the execution of the real-time data integration task.

[0101] Therefore, after modifying the mapping relationship and before resuming the real-time data integration task, if the metadata change information indicates that the field is modified, a modification strategy for the conversion logic of the predefined field is determined, and based on the determined modification strategy, the conversion logic of the field in the task configuration information is modified.

[0102] In this way, there is no need for manual intervention to modify the conversion logic of the fields in the task configuration information, which reduces the workload of operation and maintenance and improves the timeliness of real-time data transmission.

[0103] In specific implementation, the modification strategy may include synchronously updating the conversion logic and canceling the conversion logic. If the modification strategy is synchronously updating the conversion logic, the conversion logic of the field in the task configuration information is modified to the conversion logic before the field is changed. If the modification strategy is canceling the conversion logic, the conversion logic of the field in the task configuration information is deleted.

[0104] For example, in the initialization phase of the real-time data integration task, the task configuration information configured by the user is to set field a, and during the real-time data transmission process, delete the blank characters in the data corresponding to field a. If the modification strategy of the pre-defined field conversion logic is to synchronously update the conversion logic, then when field a in the data table of the data source is modified to field aa, during the real-time data transmission process, the real-time data integration system can delete the blank characters in the data corresponding to field aa. Similarly, if the modification strategy of the pre-defined field conversion logic is to cancel the conversion logic, then when field a in the data table of the data source is modified to field aa, during the real-time data transmission process, the real-time data integration system deletes the conversion logic of field a in the task configuration information, that is, retains the blank characters in the data corresponding to field aa.

[0105] In this way, providing two possible modification strategies of synchronously updating the transformation logic and canceling the transformation logic can make real-time data integration more convenient.

[0106] In step 205, the real-time data integration task is resumed, and based on the modified mapping relationship, the real-time data of the read data source is integrated into the write data source.

[0107] In specific implementation, the data position in the read data source corresponding to the position of each real-time data can be located based on the recorded position of the currently collected real-time data, and the real-time data integration task can be resumed starting from the located data position in each read data source.

[0108] In this way, the integration task can be resumed from the paused position without having to re-execute the integration task, further improving the timeliness of real-time data transmission.

[0109] In the embodiment of the present application, it is only necessary to build an integration task once to complete the integration of real-time data. If the metadata information of the data source read is changed during the real-time data integration process, the metadata of the data source written and the task configuration information of the integration task can be automatically modified synchronously without manual intervention, thereby improving the timeliness of real-time data transmission and reducing the operation and maintenance workload.

[0110] Figure 6 A flowchart of another method for integrating real-time data provided in an embodiment of the present application, the method is applied to Figure 1 In the real-time data integration system 200, the method includes the following steps.

[0111] In step 601, in response to a start instruction of any real-time data integration task, real-time data of at least one read data source is collected based on the task configuration information of the real-time data integration task, and the task configuration information includes metadata information of the read data source, metadata information of the write data source, and a mapping relationship between the metadata information of the read data source and the metadata information of the write data source.

[0112] The mapping relationship is used to represent the location where the real-time data integration system integrates the collected real-time data into the write data source.

[0113] In step 602, the collected real-time data is monitored. If metadata change information is detected in any real-time data, the real-time data integration task is suspended and the location of the currently collected real-time data is recorded.

[0114] In specific implementation, based on the type of the read data source, the real-time data reading plug-in corresponding to the data source type of each read data source can be determined; based on the determined real-time data reading plug-in, the real-time data of each read data source is collected.

[0115] In step 603, an instruction to back up the data before the change is sent to the write data source.

[0116] In step 604, after receiving the pre-change data backup completion instruction sent by the write data source, a metadata information update instruction is sent to the write data source based on the metadata change information.

[0117] In step 605, after receiving the metadata information update completion instruction sent by the write data source, the mapping relationship is modified.

[0118] In specific implementation, if the metadata change information represents a modification to a field, a modification strategy for the conversion logic of a predefined field is determined, and based on the determined modification strategy, the conversion logic of the field in the task configuration information is modified. If the modification strategy is to synchronously update the conversion logic, the conversion logic of the field in the task configuration information is modified to the conversion logic before the field is changed; if the modification strategy is to cancel the conversion logic, the conversion logic of the field in the task configuration information is deleted.

[0119] In step 606, based on the recorded location of the currently collected real-time data, the data location in the read data source corresponding to each real-time data location is located, and starting from the located data location in each read data source, the real-time data integration task is resumed.

[0120] The method for integrating real-time data of the present application is described in detail below with reference to specific embodiments.

[0121] In an embodiment of the present application, the real-time data integration system is mainly used to respond to the start instruction of any real-time data integration task, collect real-time data from at least one read data source based on the task configuration information of the real-time data integration task, monitor the collected real-time data, and if metadata change information is detected in any real-time data, suspend the real-time data integration task, execute the metadata information change task based on the metadata change information, send a metadata information update instruction to the write data source, modify the mapping relationship in the task configuration information after receiving the metadata information update completion instruction sent by the write data source, and finally resume the execution of the real-time data integration task, and integrate the real-time data of the read data source into the write data source based on the modified mapping relationship.

[0122] Based on this, Figure 7A structural diagram of a real-time data integration system provided in an embodiment of the present application is provided, in which the real-time data integration system is abstracted into three units, namely an input unit, a transmission unit and an output unit, wherein the input unit is mainly used to respond to the start instruction of any real-time data integration task, collect real-time data of at least one read data source based on the task configuration information of the real-time data integration task, and monitor the collected real-time data; if metadata change information is detected in any real-time data, the real-time data integration task is suspended, and the output unit is notified of the metadata change information; the output unit is mainly used to execute the metadata information change task based on the metadata change information, send the metadata information update instruction to the write data source, modify the mapping relationship in the task configuration information after receiving the metadata information update completion instruction sent by the write data source, and notify the input unit after modifying the mapping relationship in the task configuration information; the input unit is also used to resume the execution of the real-time data integration task, and continue to read the real-time data of the data source based on the modified mapping relationship; the transmission unit is used to transmit the real-time data; and the output unit is also used to integrate the real-time data into the write data source.

[0123] Figure 8 A schematic diagram of a real-time data integration process provided for an embodiment of the present application includes a reading data source, an input unit of a real-time data integration system, a transmission unit, an output unit, and a writing data source, wherein both the reading data source and the writing data source have two fields, namely field 1 and field 2. When the metadata information of the reading data source changes, that is, step 1: adding a field 3, the input unit collects the real-time data of the reading data source through step 2, and enters step 3 to monitor the collected real-time data. At this time, if the metadata change information is detected in the real-time data, the real-time data transmission is suspended, and the metadata change information is notified to the output unit in step 5. The output unit executes the metadata based on the metadata change information. Information change task, then, enter step 6 to send a metadata information update instruction to the write data source, the write data source executes step 7 to update the metadata information based on the metadata information update instruction, that is, to add a new field 3, after the update is completed, enter step 8 to send an update completion instruction to the output unit, then the output unit modifies the mapping relationship in the task configuration information, and after modifying the mapping relationship in the task configuration information, enter step 9 to inform the input unit, the input unit executes step 10 to resume the execution of the real-time data integration task, and based on the modified mapping relationship, continue to read the real-time data of the data source, transmit the real-time data to the output unit via the transmission unit, and the output unit executes step 11 to integrate the real-time data into the write data source.

[0124] Based on the same technical concept, the embodiment of the present application also provides a device for integrating real-time data. The principle of solving the problem by the device for integrating real-time data is similar to the method for integrating real-time data mentioned above. Therefore, the implementation of the device for integrating real-time data can refer to the implementation of the method for integrating real-time data, and the repeated parts will not be repeated.

[0125] Fig. 9 A structural diagram of a device for integrating real-time data provided in an embodiment of the present application includes a collection module 901, a monitoring module 902, a sending module 903, a modification module 904, and an integration module 905.

[0126] The collection module 901 is used to respond to the start instruction of any real-time data integration task and collect the real-time data of at least one read data source based on the task configuration information of the real-time data integration task; the task configuration information includes the metadata information of the read data source, the metadata information of the write data source, and the mapping relationship between the metadata information of the read data source and the metadata information of the write data source; wherein the mapping relationship is used to represent the position where the real-time data integration system integrates the collected real-time data into the write data source;

[0127] A monitoring module 902 is used to monitor the collected real-time data, and if metadata change information is detected in any real-time data, the real-time data integration task is suspended;

[0128] A sending module 903 is used to send a metadata information update instruction to the write data source based on the metadata change information, so that the write data source modifies the metadata information of the write data source according to the metadata change information;

[0129] A modification module 904, configured to modify the mapping relationship after receiving a metadata information update completion instruction sent by the write data source;

[0130] The integration module 905 is used to resume the execution of the real-time data integration task, and integrate the real-time data of the read data source into the write data source based on the modified mapping relationship.

[0131] In some embodiments, the acquisition module 901 is specifically used to:

[0132] Based on the type of the read data source, determine a real-time data reading plug-in corresponding to the data source type of each read data source;

[0133] Based on the determined real-time data reading plug-ins, real-time data of each reading data source is collected.

[0134] In some embodiments, it also includes:

[0135] A recording module 906, configured to record the location of the real-time data currently collected after the monitoring module 902 suspends the real-time data integration task;

[0136] The integrated module 905 is specifically used for:

[0137] Based on the recorded positions of the real-time data currently being collected, the data positions in the read data source corresponding to the positions of each real-time data are located, and starting from the located data positions in each read data source, the real-time data integration task is resumed.

[0138] In some embodiments, the task configuration information further includes field conversion logic, and the field conversion logic is used to characterize a processing strategy for real-time data corresponding to the field during the execution of the real-time data integration task; after the modification module 904 modifies the mapping relationship and before the integration module 905 resumes execution of the real-time data integration task, the modification module 904 is further used to:

[0139] If the metadata change information indicates that a field is modified, a modification strategy of a predefined conversion logic of the field is determined, and based on the determined modification strategy, the conversion logic of the field in the task configuration information is modified.

[0140] In some embodiments, the modification strategy includes synchronously updating conversion logic and canceling conversion logic, and the modification module 904 is specifically used to:

[0141] If the modification strategy is to synchronously update the conversion logic, then modify the conversion logic of the field in the task configuration information to the conversion logic before the field is changed;

[0142] If the modification strategy is to cancel the conversion logic, the conversion logic of the field in the task configuration information is deleted.

[0143] In some embodiments, after the monitoring module 902 suspends the real-time data integration task, before the sending module 903 sends a metadata information update instruction to the write data source based on the metadata change information, the sending module 903 is further used to:

[0144] Sending an instruction to back up the data before the change to the write data source;

[0145] After receiving the pre-change data backup completion instruction sent by the write data source, a step of sending a metadata information update instruction to the write data source is performed.

[0146] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated into a processor, or may exist physically alone, or two or more modules may be integrated into one module. The coupling between the modules may be achieved through some interfaces, which are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0147] After introducing the method and device for integrating real-time data according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0148] Refer to the following Fig.10 The electronic device 130 implemented according to this embodiment of the present application is described. Fig.10 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0149] like Fig.10 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0150] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0151] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0152] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0153] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0154] In an exemplary embodiment, a storage medium is also provided, and when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned method of integrating real-time data. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0155] In an exemplary embodiment, the electronic device of the present application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor may execute the steps of any method of integrating real-time data provided in the embodiments of the present application.

[0156] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed by an electronic device, the electronic device can implement any exemplary method provided in the present application.

[0157] Furthermore, the computer program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0158] The program product for integrating real-time data in the embodiment of the present application may adopt a CD-ROM and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0159] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0160] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0161] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0162] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0163] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0164] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0165] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0168] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0169] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method to integrate real-time data, It is characterized in that include: In response to a start instruction of any real-time data integration task, real-time data of at least one read data source is collected based on task configuration information of the real-time data integration task; the task configuration information includes metadata information of the read data source, metadata information of the write data source, and a mapping relationship between the metadata information of the read data source and the metadata information of the write data source; wherein the mapping relationship is used to represent the position where the real-time data integration system integrates the collected real-time data into the write data source; The collected real-time data is monitored, and if metadata change information is detected in any real-time data, the real-time data integration task is suspended; Based on the metadata change information, sending a metadata information update instruction to the write data source, so that the write data source modifies the metadata information of the write data source according to the metadata change information; After receiving the metadata information update completion instruction sent by the write data source, modifying the mapping relationship; The real-time data integration task is resumed, and based on the modified mapping relationship, the real-time data of the read data source is integrated into the write data source.

2. The method according to claim 1, It is characterized in that The collecting of real-time data of at least one reading data source based on the task configuration information of the real-time data integration task includes: Based on the type of the read data source, determine a real-time data reading plug-in corresponding to the data source type of each read data source; Based on the determined real-time data reading plug-ins, real-time data of each reading data source is collected.

3. The method according to claim 1, It is characterized in that After pausing the real-time data integration task, the method further includes: Recording the location of the real-time data currently being collected; The resuming and executing the real-time data integration task includes: Based on the recorded positions of the real-time data currently being collected, the data positions in the read data source corresponding to the positions of each real-time data are located, and starting from the located data positions in each read data source, the real-time data integration task is resumed.

4. The method according to claim 1, It is characterized in that The task configuration information also includes field conversion logic, and the field conversion logic is used to represent the processing strategy of the real-time data corresponding to the field during the execution of the real-time data integration task; After modifying the mapping relationship and before resuming the execution of the real-time data integration task, the method further includes: If the metadata change information indicates that a field is modified, a modification strategy of a predefined conversion logic of the field is determined, and based on the determined modification strategy, the conversion logic of the field in the task configuration information is modified.

5. The method according to claim 4, It is characterized in that The modification strategy includes synchronously updating conversion logic and canceling conversion logic, and the conversion logic of modifying the field in the task configuration information based on the modification strategy includes: If the modification strategy is to synchronously update the conversion logic, then modify the conversion logic of the field in the task configuration information to the conversion logic before the field is changed; If the modification strategy is to cancel the conversion logic, the conversion logic of the field in the task configuration information is deleted.

6. The method according to claim 1, It is characterized in that After pausing the real-time data integration task, and before sending a metadata information update instruction to the write data source based on the metadata change information, the method further includes: Sending an instruction to back up the data before the change to the write data source; After receiving the pre-change data backup completion instruction sent by the write data source, a step of sending a metadata information update instruction to the write data source is performed.

7. A device for integrating real-time data, It is characterized in that include: A collection module, used to respond to a start instruction of any real-time data integration task, and to collect real-time data of at least one read data source based on the task configuration information of the real-time data integration task; the task configuration information includes metadata information of the read data source, metadata information of the write data source, and a mapping relationship between the metadata information of the read data source and the metadata information of the write data source; wherein the mapping relationship is used to represent the position where the real-time data integration system integrates the collected real-time data into the write data source; A monitoring module is used to monitor the collected real-time data, and if metadata change information is detected in any real-time data, the real-time data integration task is suspended; A sending module, configured to send a metadata information update instruction to the write data source based on the metadata change information, so that the write data source modifies the metadata information of the write data source according to the metadata change information; A modification module, configured to modify the mapping relationship after receiving a metadata information update completion instruction sent by the write data source; The integration module is used to resume the execution of the real-time data integration task and integrate the real-time data of the read data source into the write data source based on the modified mapping relationship.

8. The device according to claim 7, It is characterized in that The acquisition module is specifically used for: Based on the type of the read data source, determine a real-time data reading plug-in corresponding to the data source type of each read data source; Based on the determined real-time data reading plug-ins, real-time data of each reading data source is collected.

9. An electronic device, It is characterized in that include: at least one processor, and a memory communicatively connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 6.

10. A storage medium, It is characterized in that When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the method according to any one of claims 1 to 6.

Images