Real-time Data Integration Method, Device, Electronic Device and Storage Medium

Abstract

The present invention relates to the technical field of big data processing, and provides a real-time data integration method, device, electronic device, and storage medium. Obtain the data integration requirements submitted by the user through a real-time data integration system; it includes a source data storage engine, a target data storage engine, and a custom matching strategy; if it is determined according to the custom matching strategy that each started task in the real-time data integration system does not match the data integration requirements, then select a target task that matches the target resource amount required by the data integration requirements from all the started tasks according to the preset matching strategy; allocate the data integration requirements to the target task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the target task. Selecting tasks based on the user-defined matching strategy and the system's preset matching strategy can meet various business requirements, achieve adaptive resource allocation, and ensure the stability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular, to a real-time data integration method, device, electronic device and storage medium. Background Art

[0002] There are two types of data in big data processing, namely offline data and real-time data. Offline data generally refers to data with a relatively long time interval from generation to use, such as data at the hour level or day level; real-time data generally refers to data with a relatively short time interval from generation to use, such as data at the second level or minute level. Data integration refers to collecting big data from business system data sources or user terminals such as Web pages and APPs into a big data cluster. With the popularization and development of big data, traditional offline data analysis can gradually no longer meet the needs of fast-paced business activities. More and more users need to use real-time data for analysis. Therefore, real-time data integration is very important for big data processing. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a real-time data integration method, device, electronic device and storage medium.

[0004] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present invention provides a real-time data integration method, which is applied to an electronic device installed with a real-time data integration system. The method includes:

[0006] Obtain the data integration requirements submitted by the user through the real-time data integration system; the data integration requirements include a source data storage engine, a target data storage engine, and a custom matching strategy;

[0007] If it is determined according to the custom matching strategy that each started task in the real-time data integration system does not match the data integration requirements, then select a target task that matches the target resource amount required by the data integration requirements from all the started tasks according to a preset matching strategy;

[0008] Allocate the data integration requirements to the target task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the target task.

[0009] In an optional implementation manner, the step of selecting a target task that matches the target resource amount required by the data integration requirements from all the started tasks according to the preset matching strategy includes:

[0010] Estimate the target resource amount required by the data integration requirements and obtain the remaining resource amount of each started task;

[0011] Select each pending task from all the started tasks whose remaining resource amount is greater than or equal to the target resource amount;

[0012] For each of the pending tasks, evaluate based on each running parameter of the pending task to obtain an evaluation value of the pending task, and obtain the evaluation value of each of the pending tasks;

[0013] Based on the evaluation values of each of the pending tasks, select the target task from all the pending tasks.

[0014] In an alternative embodiment, all the running parameters include a data synchronization delay parameter, a data increase parameter, and a resource idle ratio parameter; the real-time data integration system stores preset conditions corresponding to the data synchronization delay parameter, preset conditions corresponding to the data increase parameter, and preset conditions corresponding to the resource idle ratio parameter;

[0015] The step of evaluating based on each running parameter of the pending task to obtain an evaluation value of the pending task includes:

[0016] If none of the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task reach the corresponding preset conditions, determine that the evaluation value of the pending task is a first value;

[0017] If only one of the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task reaches the corresponding preset condition, determine that the evaluation value of the pending task is a second value; the second value is greater than the first value;

[0018] If only two of the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task reach the corresponding preset conditions, determine that the evaluation value of the pending task is a third value; the third value is greater than the second value;

[0019] If all of the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task reach the corresponding preset conditions, determine that the evaluation value of the pending task is a fourth value; the fourth value is greater than the third value.

[0020] In an alternative embodiment, the step of selecting the target task from all the pending tasks based on the evaluation values of each of the pending tasks includes:

[0021] Obtain a candidate task with the largest evaluation value from all the pending tasks;

[0022] If there is only one candidate task, use the candidate task as the target task;

[0023] If there are multiple candidate tasks, the candidate task with the smallest remaining resource amount among all candidate tasks is used as the target task.

[0024] In an alternative embodiment, the method further includes:

[0025] If there is no started task in the real-time data integration system, estimate the target resource amount required for the data integration requirement, and start a new task based on the target resource amount;

[0026] Allocate the data integration requirement to the new task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the new task.

[0027] In an alternative embodiment, the method further includes:

[0028] Regularly monitor the current running status of each started task;

[0029] When it is monitored that there is a first task with an abnormal current running status, count the resource occupancy corresponding to the real-time data to be processed in the first task;

[0030] Judge whether there is a second task matching the resource occupancy among all started tasks according to a preset matching policy;

[0031] If there is, switch the real-time data to be processed to the second task, and integrate the real-time data to be processed by running the second task;

[0032] If not, start a third task based on the resource occupancy, and integrate the real-time data to be processed by running the third task.

[0033] In an alternative embodiment, after the step of obtaining the data integration requirement submitted by the user through the real-time data integration system, the method further includes:

[0034] When the source data storage engine and / or the target data storage engine do not match the real-time data integration system, provide the user with system opening rules; the system opening rules include the interaction interfaces of the real-time data integration system and the types of development languages matching them;

[0035] Obtain the program file submitted by the user that complies with the system opening rules, and obtain the task to be processed selected by the user;

[0036] Allocate the data integration requirement to the task to be processed, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the task to be processed.

[0037] In a second aspect, the present invention provides a real-time data integration device, which is applied to an electronic device. The electronic device is equipped with a real-time data integration system. The device includes:

[0038] An acquisition module, configured to acquire the data integration requirements submitted by the user through the real-time data integration system; the data integration requirements include a source data storage engine, a target data storage engine, and a custom matching strategy;

[0039] A selection module, configured to, if it is determined that each started task in the real-time data integration system does not match the data integration requirements according to the custom matching strategy, select a target task that matches the target resource amount required by the data integration requirements from all the started tasks according to a preset matching strategy;

[0040] An integration module, configured to allocate the data integration requirements to the target task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the target task.

[0041] In a third aspect, the present invention provides an electronic device, including a processor and a memory. The memory stores a computer program. When the processor executes the computer program, the method described in any one of the foregoing embodiments is implemented.

[0042] In a fourth aspect, the present invention provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any one of the foregoing embodiments is implemented.

[0043] The real-time data integration method, device, electronic device, and storage medium provided by the present invention acquire the data integration requirements submitted by the user through the real-time data integration system; the data integration requirements include a source data storage engine, a target data storage engine, and a custom matching strategy; if it is determined that each started task in the real-time data integration system does not match the data integration requirements according to the custom matching strategy, then a target task that matches the target resource amount required by the data integration requirements is selected from all the started tasks according to a preset matching strategy; the data integration requirements are allocated to the target task, and the real-time data obtained from the source data storage engine is integrated into the target data storage engine by running the target task. According to the user-defined matching strategy and the system preset matching strategy, that is, based on multiple strategies to select tasks to process the data integration requirements, it can not only meet various business requirements proposed by the user to improve the universality of data integration, but also achieve adaptive resource allocation to ensure the stability of the system, and at the same time improve the real-time performance and accuracy of data integration.

[0044] To make the above objects, features, and advantages of the present invention more apparent and understandable, the following provides preferred embodiments in conjunction with the accompanying drawings and detailed descriptions are as follows. Description of the Drawings

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0046] Figure 1 Shows a block diagram of an electronic device provided by an embodiment of the present invention;

[0047] Figure 2 Shows a schematic flowchart of a real-time data integration method provided by an embodiment of the present invention;

[0048] Figure 3 Shows another schematic flowchart of a real-time data integration method provided by an embodiment of the present invention;

[0049] Figure 4 Shows another schematic flowchart of a real-time data integration method provided by an embodiment of the present invention;

[0050] Figure 5 Shows another schematic flowchart of a real-time data integration method provided by an embodiment of the present invention;

[0051] Figure 6 Shows another schematic flowchart of a real-time data integration method provided by an embodiment of the present invention;

[0052] Figure 7 Shows a functional module diagram of a real-time data integration device provided by an embodiment of the present invention.

[0053] Icons: 100 - Electronic device; 110 - Bus; 120 - Processor; 130 - Memory; 150 - I / O module; 170 - Communication interface; 300 - Real-time data integration device; 310 - Acquisition module; 330 - Selection module; 350 - Integration module; 370 - Monitoring module; 390 - Open module. Detailed Embodiments

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Usually, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] Accordingly, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0056] It should be noted that relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0057] With the popularization and development of big data, traditional offline data analysis can gradually no longer meet the needs of fast-paced business activities. More and more users need to use real-time data for analysis. Therefore, real-time data integration is very important for big data processing. At present, although dedicated hardware devices are used, such as adapters in power information systems to collect and integrate power data in real time, their application scenarios are single and their scalability is low. There are also third-party open-source components such as big data computing engines like Flink, Storm, and Spark Streaming, but their technical thresholds are high, resources cannot be dynamically expanded, and users cannot perform independent operation and maintenance. Moreover, due to the continuous change of business requirements, professional developers often need to re-develop and deploy, resulting in a large amount of time and labor costs. Therefore, the embodiments of the present invention provide a real-time data integration method to solve the above problems.

[0058] Please refer to Figure 1 , which is a block diagram of an electronic device 100 provided by an embodiment of the present invention. The electronic device 100 includes a bus 110, a processor 120, a memory 130, an I / O module 150, and a communication interface 170.

[0059] The bus 110 can be a circuit that interconnects the above elements and transmits communication (such as control messages) between the above elements.

[0060] The processor 120 can receive commands from the other components described above (such as the memory 130, the I / O module 150, the communication interface 170, etc.) via the bus 110, can interpret the received commands, and can perform calculations or data processing according to the interpreted commands.

[0061] The processor 120 can be an integrated circuit chip with signal processing capabilities. The processor 120 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0062] The memory 130 can store commands or data received from the processor 120 or other components (such as the I / O module 150, the communication interface 170, etc.) or commands or data generated by the processor 120 or other components.

[0063] The memory 130 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM).

[0064] The I / O module 150 can receive commands or data input by the user via input-output means (such as sensors, keyboards, touchscreens, etc.) and can transmit the received commands or data to the processor 120 or the memory 130 via the bus 110. And it is used to display various information received, stored, and processed from the above components (such as multimedia data, text data), and can display videos, images, data, etc. to the user.

[0065] The communication interface 170 can be used for signaling or data communication with other node devices.

[0066] It can be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100, and the electronic device 100 may also include more or fewer components than Figure 1 shown therein, or have the same asFigure 1 The different configurations shown. Figure 1 Each component shown in can be implemented using hardware, software, or a combination thereof.

[0067] It should be understood that the electronic device 100 is installed with a real-time data integration system, and the real-time data integration method provided by the embodiments of the present invention is implemented through the real-time data integration system.

[0068] Next, taking the above-mentioned electronic device 100 as the execution subject, each step in each method provided by the embodiments of the present invention will be executed, and the corresponding technical effects will be achieved.

[0069] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a real-time data integration method provided by an embodiment of the present invention.

[0070] Step S202, obtaining the data integration requirements submitted by the user through the real-time data integration system; the data integration requirements include a source data storage engine, a target data storage engine, and a custom matching policy;

[0071] Among them, the data storage engine refers to the component used to store big data in a big data cluster, such as Hive, Doris, ClickHouse, etc. The source data storage engine can be understood as the source of real-time data, and the target data storage engine can be understood as the destination of real-time data.

[0072] In this embodiment, the real-time data integration system supports providing services to users in ways such as WEB pages, Http Restful APIs, etc. Users can submit data integration requirements and provide relevant information about the real-time data to be integrated, such as the source data storage engine, the target data storage engine, and the custom matching policy, through interactive operations in the real-time data integration system. The custom matching policy can be understood as the policy set by the user himself to select tasks to process the data integration requirements. It should be understood that in one implementation, the user can also not set the custom matching policy, and then the subsequent process will use the default matching policy of the system, that is, the preset matching policy, to select tasks.

[0073] Optionally, the data integration requirements may further include the connection information of the source data storage engine and the account with readable permissions, the connection information of the target data storage engine and the account with writable permissions, the format of the data source, the format of the data written into the target data storage engine, whether the data needs to be converted or parsed into column information or encrypted and decrypted, etc. Through these information, it can be ensured that real-time data can be normally read from the source data storage engine and written into the target data storage engine subsequently.

[0074] Step S204, if it is determined that each started task in the real-time data integration system does not match the data integration requirement according to the custom matching policy, then select a target task that matches the target resource amount required by the data integration requirement from all the started tasks according to the preset matching policy;

[0075] It can be understood that the real-time data integration system manages the running cycles of multiple tasks, such as start, stop, monitoring, etc. Tasks are used to achieve the integration and synchronization of real-time data, which can be understood as long-running computer programs, and their specific implementations vary with the different computer programming languages used. The real-time data integration system may also include various functional components, such as a data management component, a data engine management component, a work order management component, a user permission component, a monitoring and alarm component, etc.

[0076] In this embodiment, after the real-time data integration system obtains the data integration requirement submitted by the user, it can first detect whether there are any started tasks currently. If there are, it determines whether there is a started task that matches the data integration requirement according to the custom matching policy.

[0077] For example, the custom matching policy may be to specify a certain task to handle the data integration requirement. If the specified task has been started and its remaining resource amount is sufficient to handle the data integration requirement, it is determined that the matching is successful. Then, the real-time data obtained from the source data storage engine is integrated into the target data storage engine by running the started task that matches the data integration requirement.

[0078] If the specified task has not been started or its remaining resource amount is insufficient to handle the data integration requirement, it is determined that the matching fails. Then, a target task that matches the target resource amount required by the data integration requirement is selected from all the started tasks according to the preset matching policy.

[0079] Step S206, allocate the data integration requirement to the target task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the target task;

[0080] In this embodiment, after selecting the target task, the relevant information of the data integration requirement can be allocated to the target task. The target task is run to obtain real-time data from the source data storage engine, and the real-time data is integrated into the target data storage engine to achieve real-time data integration.

[0081] It can be understood that the embodiments of the present invention can select tasks based on user-defined matching policies, which can meet various business requirements and support various application scenarios, improving the universality of data integration. And when no matching tasks can be obtained based on the user-defined matching policy, matching target tasks can also be selected according to the preset matching policy and the amount of resources required for data integration requirements, realizing adaptive resource allocation, ensuring the stability of system operation, and at the same time improving the real-time performance and accuracy of data integration.

[0082] It can be seen that based on the above steps, the data integration requirements submitted by the user are obtained through the real-time data integration system; the data integration requirements include the source data storage engine, the target data storage engine, and the user-defined matching policy; if it is determined that each started task in the real-time data integration system does not match the data integration requirements according to the user-defined matching policy, then the target task that matches the amount of target resources required for the data integration requirements is selected from all the started tasks according to the preset matching policy; the data integration requirements are assigned to the target task, and the real-time data obtained from the source data storage engine is integrated into the target data storage engine by running the target task. According to the user-defined matching policy and the system preset matching policy of the present invention, that is, selecting tasks to process data integration requirements based on multiple policies, it can meet various business requirements proposed by the user to improve the universality of data integration, and can also realize adaptive resource allocation to ensure the stability of the system, and at the same time improve the real-time performance and accuracy of data integration.

[0083] Optionally, for the above step S204, an embodiment of the present invention provides a possible implementation manner. Please refer to Figure 3 .

[0084] Step S204-1: Estimate the amount of target resources required for the data integration requirements and obtain the remaining resources of each started task;

[0085] In this embodiment, the real-time data integration system estimates the amount of resources required for the data integration requirements to obtain the target resources. For example, the business type of the real-time data to be integrated can be determined according to the data integration requirements submitted by the user, and then the data volume in the past period of time, such as within 7 days, of this business type is obtained, and based on reference values such as the average of the peak and trough of the data volume, the trend of the real-time data to be integrated is predicted, so as to estimate the amount of resources required to process the data integration requirements, that is, the target resources are obtained.

[0086] The real-time data integration system also obtains the remaining resources of each started task. For example, for each started task, the real-time data information being processed in the started task, the remaining memory space size, the current position of the data being read, the amount of data processed in the past period of time, and the number of threads currently occupied, etc. can be monitored regularly, so as to count the remaining resources of the started task, that is, obtain the remaining resources of each started task.

[0087] Step S204-3, select each pending task from all the started tasks whose remaining resources are greater than or equal to the target resources;

[0088] In this embodiment, based on the remaining resources of each started task, each started task sufficient to process the data integration requirements can be selected from all the started tasks, that is, each started task whose remaining resources are greater than or equal to the target resources is used as a pending task, that is, each pending task is obtained.

[0089] Step S204-5, for each pending task, evaluate based on each running parameter of the pending task to obtain the evaluation value of the pending task, and obtain the evaluation value of each pending task;

[0090] Step S204-7, based on the evaluation values of each pending task, select the target task from all the pending tasks;

[0091] In this embodiment, after each pending task is selected, it can be evaluated based on all the running parameters of each pending task to obtain the evaluation value of each pending task. It can be understood as evaluating the remaining data processing capabilities of each pending task, and then based on the evaluation values of each pending task, the target task is selected from them.

[0092] It can be understood that by selecting the target task that matches the capabilities required for data integration processing based on the remaining capabilities of the pending tasks, it is possible to avoid waste of resources caused by excessive remaining capabilities of some tasks and also relieve the running pressure of some tasks. That is, through intelligent control of the resources of the tasks, reasonable resource allocation is achieved and the stable operation of the system is ensured.

[0093] It should be understood that in some scenarios, it may not be possible to select the target task from all the started tasks according to the preset matching strategy. Then, new tasks can be started based on the target resources required for the data integration requirements, and the data integration requirements are assigned to the new tasks, so as to integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the new tasks.

[0094] Optionally, for the process of evaluating based on each running parameter of the pending task to obtain the evaluation value of the pending task in step S204-5 above, the embodiment of the present invention provides a possible implementation manner.

[0095] In this embodiment, all operating parameters may include a data synchronization delay parameter, a data increase parameter, and a resource idle ratio parameter, and each of them has a corresponding preset condition.

[0096] The data synchronization delay parameter refers to the delay duration for a task to synchronize real-time data from the source data storage engine to the target data storage engine. The corresponding preset condition may be less than a preset delay threshold, such as 3 minutes. The data increase parameter refers to the increase in the amount of data in a task within a preset period in the future. The corresponding preset condition may be less than a preset increase threshold, such as 30%. The resource idle ratio parameter refers to the ratio of the time length during which the remaining resources of a task exceed the target resources within the past N days. The corresponding preset condition may be greater than a preset ratio threshold, such as 95%.

[0097] It should be noted that the preset conditions corresponding to the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter can be set according to actual applications, and the embodiments of the present invention do not make limitations.

[0098] It can be understood that the method for evaluating each pending task is similar. For the sake of brevity, a pending task will be used as an example for illustration below.

[0099] Step S204-5A, if the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task do not reach their corresponding preset conditions, then determine that the evaluation value of the pending task is the first value;

[0100] In the first case, if the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task do not reach their respective corresponding preset conditions, such as the data synchronization delay parameter is greater than or equal to 3 minutes, the data increase parameter is greater than or equal to 30%, and the resource idle ratio parameter is less than or equal to 95%, then use the first value as the evaluation value of the pending task. Assuming the first value is 0, the evaluation value of the pending task is 0.

[0101] Step S204-5B, if only one of the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task reaches the corresponding preset condition, then determine that the evaluation value of the pending task is the second value; the second value is greater than the first value;

[0102] In the second case, if only one of the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task reaches the corresponding preset condition, such as the data synchronization delay parameter is less than 3 minutes, or the data increase parameter is less than 30%, or the resource idle ratio parameter is greater than 95%, then use the second value as the evaluation value of the pending task. Assuming the second value is 1, the evaluation value of the pending task is 1.

[0103] Step S204-5C, if only two of the data synchronization delay parameter, data increase parameter, and resource idle ratio parameter of the to-be-determined task reach their corresponding preset conditions, then determine that the evaluation value of the to-be-determined task is the third value; the third value is greater than the second value;

[0104] For the third case, if only two of the data synchronization delay parameter, data increase parameter, and resource idle ratio parameter of the to-be-determined task reach their corresponding preset conditions, such as the data synchronization delay parameter is greater than or equal to 3 min and the data increase parameter is less than 30% and the resource idle ratio parameter is greater than 95%, or the data synchronization delay parameter is less than 3 min and the data increase parameter is greater than or equal to 30% and the resource idle ratio parameter is greater than 95%, or the data synchronization delay parameter is less than 3 min and the data increase parameter is less than 30% and the resource idle ratio parameter is less than or equal to 95%, then use the third value as the evaluation value of the to-be-determined task. Assume the third value is 2, then the evaluation value of the to-be-determined task is 2.

[0105] Step S204-5D, if the data synchronization delay parameter, data increase parameter, and resource idle ratio parameter of the to-be-determined task all reach their corresponding preset conditions, then determine that the evaluation value of the to-be-determined task is the fourth value; the fourth value is greater than the third value;

[0106] For the fourth case, if the data synchronization delay parameter, data increase parameter, and resource idle ratio parameter of the to-be-determined task all reach their respective corresponding preset conditions, such as the data synchronization delay parameter is less than 3 min and the data increase parameter is less than 30% and the resource idle ratio parameter is greater than 95%, then use the fourth value as the evaluation value of the to-be-determined task. Assume the fourth value is 3, then the evaluation value of the to-be-determined task is 3.

[0107] It should be noted that the first value, the second value, the third value, and the fourth value can be set according to actual applications, and the embodiments of the present invention do not make limitations.

[0108] Optionally, for the above step S204-7, the embodiments of the present invention provide a possible implementation manner.

[0109] Step S204-7-1, obtain the candidate task with the largest evaluation value from all the to-be-determined tasks;

[0110] Step S204-7-3A, if there is one candidate task, then use the candidate task as the target task;

[0111] Step S204-7-3B, if there are multiple candidate tasks, then use the candidate task with the smallest remaining resource amount among all the candidate tasks as the target task.

[0112] In this embodiment, the larger the evaluation value is, the more the remaining capacity of the to-be-determined task matches the capacity required to process the data integration requirement. Based on the evaluation value of each to-be-determined task, the to-be-determined task with the largest evaluation value among all to-be-determined tasks can be used as the candidate task.

[0113] If there is only one candidate task, then use this candidate task as the target task. If there are multiple candidate tasks, then obtain the remaining resource amount of each candidate task, and use the candidate task with the smallest remaining resource amount as the target task.

[0114] Optionally, after the above step S202, the embodiments of the present invention also provide a possible implementation manner. Please refer to Figure 4 .

[0115] Step S208, if there is no started task in the real-time data integration system, then estimate the target resource amount required for the data integration requirement, and start a new task based on the target resource amount;

[0116] Step S210, allocate the data integration requirement to the new task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the new task.

[0117] In this embodiment, after the real-time data integration system obtains the data integration requirement submitted by the user, it can first detect whether there is a started task currently. If there is no started task in the real-time data integration system, then estimate the resource amount required for the data integration requirement to obtain the target resource amount, and start a new task and allocate resources to the new task according to the target resource amount. The method for estimating the target resource amount can adopt the implementation manner introduced in the above step S204-1.

[0118] After starting the new task, allocate the data integration requirement to the new task, and then run the new task to obtain real-time data from the source data storage engine, and integrate the real-time data into the target data storage engine, that is, realize real-time data integration.

[0119] Optionally, in order to further ensure the stable operation of the system, the embodiments of the present invention will also monitor each started task, and then provide a possible implementation manner. Please refer to Figure 5 , after the above step S206, the following steps may also be included:

[0120] Step S212, regularly monitor the current running status of each started task;

[0121] Step S214, when it is monitored that there is a first task with an abnormal current running status, count the resource occupancy corresponding to the real-time data to be processed in the first task;

[0122] Step S216: Determine whether there is a second task that matches the resource occupancy among all the started tasks according to a preset matching strategy.

[0123] Step S218A: If there is, switch the real-time data to be processed to the second task, and perform data integration on the real-time data to be processed by running the second task.

[0124] Step S218B: If not, start a third task based on the resource occupancy, and perform data integration on the real-time data to be processed by running the third task.

[0125] In this embodiment, the real-time data integration system can regularly monitor the current running status of each started task at a preset period. When it is detected that there is a started task with an abnormal current running status, that is, the first task, it indicates that the growth rate of the data volume in the first task is too fast, resulting in the resources of the first task being insufficient. Then, the real-time data to be processed in the first task needs to be allocated to other tasks to relieve the running pressure of the first task.

[0126] The resource occupancy required for the real-time data to be processed in the first task can be counted, and then it is determined whether there is a started task that matches the resource occupancy, that is, the second task, among all the started tasks according to a preset matching strategy; if there is, then switch the real-time data to be processed to the second task, and perform data integration on the real-time data to be processed by running the second task; if not, then start a third task based on the resource occupancy, and perform data integration on the real-time data to be processed by running the third task.

[0127] It can be seen that the embodiment of the present invention automatically monitors and predicts tasks, can automatically switch when a task is found to be abnormal, and the whole process does not require manual participation, thereby improving the stability of real-time data integration and reducing the operation and maintenance costs.

[0128] Optionally, in order to improve the scalability of the system, the embodiment of the present invention can also support users to customize and implement the data integration function, and further provides a possible implementation method. Please refer to Figure 6 , after the above step S202, the following steps can also be included:

[0129] Step S220: When the source data storage engine and / or the target data storage engine do not match the real-time data integration system, provide the system opening rules to the user; the system opening rules include the interaction interfaces of the real-time data integration system and the types of development languages they match.

[0130] Step S222: Obtain the program file submitted by the user that conforms to the system opening rules, and obtain the task to be processed selected by the user.

[0131] Step S223: Allocate the data integration requirements to the task to be processed, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the task to be processed.

[0132] In some scenarios, when the user submits data integration requirements and the real-time data integration system detects that it cannot support the source data storage engine and / or the target data storage engine, that is, the source data storage engine and / or the target data storage engine does not match the real-time data integration system, or cannot support the data conversion logic, the system open rules can be provided to the user to facilitate the user to independently implement the data integration function.

[0133] An open interface document can be provided to the user, which includes the interaction interfaces of the real-time data integration system, such as the data source engine interface, the data destination engine interface, the data processing UDF (User-Defined-Function) interface, and the data monitoring information interface. Among them, the data source engine interface refers to the relevant interfaces of the source data storage engine; the data destination engine interface refers to the relevant interfaces of the target data storage engine; the data processing UDF interface means that the system defaults to providing some UDF functions for the user to select for processing some columns. If the user needs to support other UDFs, they can be custom-implemented through the interface. The data monitoring information interface refers to the open interfaces of the task information, thread execution information, and data information provided by the system. The user can control through the system page or use the interface to dock with other monitoring or alarm systems to achieve real-time data monitoring.

[0134] The user can first complete the executable program file, such as a Jar package, a Python file, etc., based on the development language type matching the real-time data integration system and the open interface document, and submit it to the real-time data integration system. At the same time, the user can select the task, that is, the task to be processed, that is expected to process the data integration requirements. Then, the real-time data integration system allocates the data integration requirements to the task to be processed, and integrates the real-time data obtained from the source data storage engine into the target data storage engine by running the task to be processed.

[0135] If the task selected by the user is a started task, there is no need to perform a restart operation. After receiving the relevant information of the data integration requirements, the started task will dynamically load the program file submitted by the user. If the task selected by the user is an unstarted task, the program file submitted by the user will be loaded during the startup phase of the task. If an error occurs when the real-time data integration system executes the program file, it will feedback to the user through the error log for the user to modify and adjust.

[0136] It can be seen that in the embodiment of the present invention, the open tool can support users to upload custom program files according to the system open rules, enabling users to independently develop data integration functions based on their own needs without integrating into the basic code of the system, improving the scalability and development efficiency of the system, and achieving the effect of self-operation and self-upgrade iteration. At the same time, there is no need to restart the task, which can ensure the accuracy and stability of the real-time data being processed.

[0137] To execute the corresponding steps in the above embodiments and each possible manner, an implementation manner of a real-time data integration device is given below. Please refer to Figure 7 , Figure 7 FIG. 7 is a functional module diagram of a real-time data integration device 300 provided by an embodiment of the present invention. It should be noted that the real-time data integration device 300 provided in this embodiment has the same basic principle and technical effects as those in the above embodiments. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding content in the above embodiments. The real-time data integration device 300 includes:

[0138] An acquisition module 310, configured to acquire the data integration requirements submitted by the user through the real-time data integration system; the data integration requirements include a source data storage engine, a target data storage engine, and a custom matching policy;

[0139] A selection module 330, configured to, if it is determined that each started task in the real-time data integration system does not match the data integration requirements according to the custom matching policy, select a target task that matches the target resource amount required by the data integration requirements from all the started tasks according to a preset matching policy;

[0140] An integration module 350, configured to allocate the data integration requirements to the target task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the target task.

[0141] Optionally, the selection module 330 is further configured to: estimate the target resource amount required by the data integration requirements and obtain the remaining resource amount of each started task; select each pending task with a remaining resource amount greater than or equal to the target resource amount from all the started tasks; for each pending task, evaluate the pending task based on each running parameter of the pending task to obtain an evaluation value of the pending task, and obtain an evaluation value of each pending task; based on the evaluation value of each pending task, select a target task from all the pending tasks.

[0142] Optionally, the selection module 330 is further configured to: if the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task do not reach the corresponding preset conditions, determine that the evaluation value of the pending task is a first value;

[0143] If only one of the data synchronization delay parameter, data increase parameter, and resource idle ratio parameter of the to-be-determined task reaches the corresponding preset condition, determine that the evaluation value of the to-be-determined task is the second value; the second value is greater than the first value;

[0144] If only two of the data synchronization delay parameter, data increase parameter, and resource idle ratio parameter of the to-be-determined task reach the corresponding preset condition, determine that the evaluation value of the to-be-determined task is the third value; the third value is greater than the second value;

[0145] If all of the data synchronization delay parameter, data increase parameter, and resource idle ratio parameter of the to-be-determined task reach the corresponding preset condition, determine that the evaluation value of the to-be-determined task is the fourth value; the fourth value is greater than the third value.

[0146] Optionally, the selection module 330 is further configured to: obtain a candidate task with the largest evaluation value from all to-be-determined tasks; if there is one candidate task, use the candidate task as the target task; if there are multiple candidate tasks, use the candidate task with the smallest remaining resource amount among all candidate tasks as the target task.

[0147] Optionally, the selection module 330 is further configured to: if there is no started task in the real-time data integration system, estimate the target resource amount required for the data integration requirement, and start a new task based on the target resource amount; allocate the data integration requirement to the new task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the new task.

[0148] The monitoring module 370 is configured to regularly monitor the current running status of each started task; when it is monitored that there is a first task with an abnormal current running status, count the resource occupancy corresponding to the to-be-processed real-time data in the first task; determine whether there is a second task that matches the resource occupancy among all started tasks according to a preset matching strategy; if there is, switch the to-be-processed real-time data to the second task, and perform data integration on the to-be-processed real-time data by running the second task; if not, start a third task based on the resource occupancy, and perform data integration on the to-be-processed real-time data by running the third task.

[0149] The open module 390 is configured to provide system open rules to the user when the source data storage engine and / or the target data storage engine do not match the real-time data integration system; the system open rules include the interaction interfaces of the real-time data integration system and the types of development languages they match; obtain the program files submitted by the user that conform to the system open rules, and obtain the to-be-processed tasks selected by the user; allocate the data integration requirement to the to-be-processed tasks, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the to-be-processed tasks.

[0150] An embodiment of the present invention further provides an electronic device, including a processor and a memory. The memory stores a computer program. When the processor executes the computer program, the real-time data integration method disclosed in the embodiments of the present invention is implemented.

[0151] An embodiment of the present invention further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the real-time data integration method disclosed in the embodiments of the present invention is implemented.

[0152] In several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of apparatuses, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0153] In addition, in each embodiment of the present invention, the functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0154] If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0155] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A real-time data integration method, characterized in that, Applied to an electronic device, the electronic device is installed with a real-time data integration system, and the method includes: Obtaining the data integration requirements submitted by the user through the real-time data integration system; the data integration requirements include a source data storage engine, a target data storage engine, and a custom matching policy; If it is determined according to the custom matching policy that each started task in the real-time data integration system does not match the data integration requirements, then estimate the target resource amount required for the data integration requirements and obtain the remaining resource amount of each started task; select each pending task from all started tasks whose remaining resource amount is greater than or equal to the target resource amount; for each pending task, obtain the evaluation value of the pending task by evaluating based on each running parameter of the pending task, and obtain the evaluation value of each pending task; based on the evaluation value of each pending task, select a target task from all pending tasks; Allocate the data integration requirements to the target task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the target task.

2. The method according to claim 1, wherein All running parameters include a data synchronization delay parameter, a data increase parameter, and a resource idle ratio parameter; the real-time data integration system stores the preset conditions corresponding to the data synchronization delay parameter, the preset conditions corresponding to the data increase parameter, and the preset conditions corresponding to the resource idle ratio parameter; The step of obtaining the evaluation value of the pending task by evaluating based on each running parameter of the pending task includes: If the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task do not reach the corresponding preset conditions, then determine that the evaluation value of the pending task is a first value; If only one running parameter of the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task reaches the corresponding preset condition, then determine that the evaluation value of the pending task is a second value; the second value is greater than the first value; If only two running parameters of the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task reach the corresponding preset condition, then determine that the evaluation value of the pending task is a third value; the third value is greater than the second value; If the data synchronization delay parameter, the data increase parameter, and the resource idle ratio parameter of the pending task all reach the corresponding preset conditions, then determine that the evaluation value of the pending task is a fourth value; the fourth value is greater than the third value.

3. The method according to claim 1, wherein The step of selecting the target task from all pending tasks based on the evaluation value of each pending task includes: Obtain a candidate task with the largest evaluation value from all pending tasks; If there is one candidate task, then use the candidate task as the target task; If there are multiple candidate tasks, then use the candidate task with the smallest remaining resource amount among all candidate tasks as the target task.

4. The method according to claim 1, wherein The method further includes: If there is no such started task in the real-time data integration system, estimate the target resource amount required for the data integration requirement, and start a new task based on the target resource amount; Allocate the data integration requirement to the new task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the new task.

5. The method according to claim 1, wherein The method further includes: Regularly monitor the current running status of each started task; When it is monitored that there is a first task with an abnormal current running status, count the resource occupancy corresponding to the real-time data to be processed in the first task; Judge whether there is a second task that matches the resource occupancy among all the started tasks according to a preset matching strategy; If there is, switch the real-time data to be processed to the second task, and integrate the real-time data to be processed by running the second task; If not, start a third task based on the resource occupancy, and integrate the real-time data to be processed by running the third task.

6. The method according to claim 1, wherein After the step of obtaining the data integration requirement submitted by the user through the real-time data integration system, the method further includes: When the source data storage engine and / or the target data storage engine do not match the real-time data integration system, provide the user with system opening rules; the system opening rules include the interaction interfaces of the real-time data integration system and the types of development languages they match; Obtain the program file submitted by the user that conforms to the system opening rules, and obtain the task to be processed selected by the user; Allocate the data integration requirement to the task to be processed, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the task to be processed.

7. A real-time data integration device, characterized in that, Applied to an electronic device, the electronic device is installed with a real-time data integration system, and the device includes: An obtaining module, configured to obtain the data integration requirement submitted by the user through the real-time data integration system; the data integration requirement includes a source data storage engine, a target data storage engine, and a custom matching strategy; A selecting module, configured to estimate the target resource amount required for the data integration requirement and obtain the remaining resource amount of each started task if it is determined according to the custom matching strategy that each started task in the real-time data integration system does not match the data integration requirement; select each pending task whose remaining resource amount is greater than or equal to the target resource amount from all the started tasks; for each pending task, obtain the evaluation value of the pending task by evaluating based on each running parameter of the pending task, and obtain the evaluation value of each pending task; based on the evaluation value of each pending task, select a target task from all the pending tasks; An integrating module, configured to allocate the data integration requirement to the target task, and integrate the real-time data obtained from the source data storage engine into the target data storage engine by running the target task.

8. An electronic device, characterized in that, It includes a processor and a memory. The memory stores a computer program. When the processor executes the computer program, the method described in any one of claims 1 to 6 is implemented.

9. A storage medium, characterized in that, A computer program is stored on the storage medium. When the computer program is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images