Secure data transfer method, device and equipment of data element in heterogeneous data source

Abstract

The invention provides a secure data transfer method, device and equipment for data elements in heterogeneous data sources, and the method comprises the steps: obtaining connection information and super management user information between the data sources, and registering the connection information and the super management user information to a management center; the connection process between the data sources and the generation process of the data elements are managed through the management center; receiving a data query request of a client through the task scheduling node, and sending the request to the management center; the management center queries and analyzes metadata information of the source data source and related information corresponding to the target data source, and transmits analyzed data to the task scheduling node; the task scheduling node registers a node task, and executes the node task according to the analyzed data to load the data; and the client obtains a data loading result of the task scheduling node through the data query request, executes a corresponding user task, and obtains and writes out a calculation result after execution is completed. According to the method, safe and efficient data transfer between heterogeneous data sources is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for secure data transfer of data elements in heterogeneous data sources. Background Art

[0002] With the development of information technology, data-driven business environment has become an important part of enterprises. Due to the heterogeneity of various data sources, the security, reliability and efficiency of data exchange and sharing between different data sources are crucial.

[0003] In related technologies, data transfer between heterogeneous data sources is usually managed through permissions, and the service provider implements permission control based on the permissions embodied by the user during use. This not only increases management costs, but also may cause data leakage risks due to improper permission settings. Summary of the invention

[0004] The present invention provides a method, device and medium for secure data transfer of data elements in heterogeneous data sources, which solves the problem of how to achieve secure and efficient data transfer between heterogeneous data sources in the related art.

[0005] In order to achieve the above purpose, this application adopts the following technical solutions:

[0006] In a first aspect, a method for securely transferring data from a data element to a heterogeneous data source is provided, comprising:

[0007] Based on the established connection relationship between data sources and the super management account, the connection information between data sources and the super management user information are obtained, and the data sources are registered with the management center through the JDBC corresponding to the data source;

[0008] Managing the connection process between data sources through the management center;

[0009] Managing the generation process of data elements through the management center;

[0010] Receive a data query request initiated by a client through a pre-configured task scheduling node, and send the data query request to the management center;

[0011] The management center queries and parses the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and transmits the parsed data to the task scheduling node;

[0012] The task scheduling node registers a node task according to the data query request, and executes the node task to load data according to the parsed data;

[0013] The client obtains the data loading result of the task scheduling node through a data query request, and executes the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out.

[0014] In a first possible implementation manner of the first aspect, the step of managing the connection process between data sources by the management center includes:

[0015] Determine and obtain the user's role according to the data element development, production, and distribution process;

[0016] Automatically authorize or deny connection to the data source according to the role of the user;

[0017] According to the health status of the data source, the data source is automatically online and offline; when online, the access request of the user during the offline period is notified to restore the online access;

[0018] Detect the number of JDBC connections of the data source; issue an early warning when the number of connections exceeds 80% of a preset threshold; sort the current connected users by activity, and send a message notification of possible closure to the TOP20% of the inactive connected users; when the number of connections exceeds the threshold, close the current TOP20% of the inactive connected users; wherein the standard for the inactive connected users is set as: no data transmission is detected within 10 minutes.

[0019] In a second possible implementation manner of the first aspect, the step of managing the generation process of the data element by the management center includes:

[0020] Regularly scan and maintain metadata information of each of the data sources, and record the scanning time;

[0021] When the data source data increases, the metadata information of the data source is automatically and synchronously updated;

[0022] The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available;

[0023] Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time;

[0024] Based on the virtual JDBC provided by the management center, the data resources required in the development, production and circulation of the data element and the metadata information of the data source are automatically routed to the JDBC corresponding to the data source.

[0025] In a third possible implementation manner of the first aspect, the step of transmitting the parsed data to the task scheduling node includes:

[0026] The data is passed through reflection or through a common cache layer in conjunction with fine-grained permissions.

[0027] In a second aspect, a secure data transfer device for a data element in a heterogeneous data source is provided, comprising:

[0028] A data source registration module is used to obtain the connection information between data sources and the super management user information based on the established connection relationship between data sources and the super management account, and register with the management center through the JDBC corresponding to the data source;

[0029] A data source connection management module, used to manage the connection process between data sources through the management center;

[0030] A data element generation management module, used to manage the generation process of data elements through the management center;

[0031] A task scheduling node module, used to receive a data query request initiated by a client through a pre-configured task scheduling node, and send the data query request to the management center;

[0032] A query and analysis module, which is used by the management center to query and analyze the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and transmit the analyzed data to the task scheduling node;

[0033] A node data loading module, used for the task scheduling node to register a node task according to the data query request, and to execute the node task to load data according to the parsed data;

[0034] The user task execution module is used for the client to obtain the data loading result of the task scheduling node through a data query request, and execute the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out.

[0035] In a first possible implementation manner of the second aspect, the data source connection management module is specifically configured to:

[0036] Determine and obtain the user's role according to the data element development, production, and distribution process;

[0037] Automatically authorize or deny connection to the data source according to the role of the user;

[0038] According to the health status of the data source, the data source is automatically online and offline; when online, the access request of the user during the offline period is notified to restore the online access;

[0039] Detect the number of JDBC connections of the data source; issue an early warning when the number of connections exceeds 80% of a preset threshold; sort the current connected users by activity, and send a message notification of possible closure to the TOP20% of the inactive connected users; when the number of connections exceeds the threshold, close the current TOP20% of the inactive connected users; wherein the standard for the inactive connected users is set as: no data transmission is detected within 10 minutes.

[0040] In a second possible implementation manner of the second aspect, the data element generation management module is specifically configured to:

[0041] Regularly scan and maintain metadata information of each of the data sources, and record the scanning time;

[0042] When the data source data increases, the metadata information of the data source is automatically and synchronously updated;

[0043] The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available;

[0044] Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time;

[0045] Based on the virtual JDBC provided by the management center, the data resources required in the development, production and circulation of the data element and the metadata information of the data source are automatically routed to the JDBC corresponding to the data source.

[0046] In a third possible implementation manner of the second aspect, the query and parsing module is specifically configured to:

[0047] The data is passed through reflection or through a common cache layer in conjunction with fine-grained permissions.

[0048] According to a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for securely transferring data elements in heterogeneous data sources as described in the first aspect.

[0049] In a fourth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for secure data transfer of data elements in heterogeneous data sources as described in the first aspect are implemented.

[0050] Beneficial effects:

[0051] This application avoids the limitation of database's cumbersome configuration information relying on personnel management, realizes centralized and automated management of configuration information, makes database configuration more consistent and standardized, facilitates monitoring and maintenance, reduces human errors and improves efficiency.

[0052] The present application provides a flexible data abstraction form in which data is available but not visible. By avoiding the generation of intermediate process tables, the risk of improper access or leakage is reduced, and the security of data is protected. At the same time, the demand for data storage can be reduced and the efficiency of data processing can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic flow chart of a method for secure data transfer of a data element in a heterogeneous data source provided in an embodiment of the present application;

[0054] Figure 2 A schematic flow chart of another method for secure data transfer of a data element in a heterogeneous data source provided in an embodiment of the present application;

[0055] Figure 3 A structural block diagram of a secure data transfer device for a data element in a heterogeneous data source provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the technical solutions in the embodiments of the present application are clearly described. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present application.

[0058] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0059] The description of the method flow in the specification of the present application and the steps of the flowchart in the drawings of the present specification do not have to be strictly executed according to the step numbers, and the method steps can be executed in a different order. Moreover, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0060] The following is a detailed description of the secure data transfer method, device, equipment and medium of the data element in the heterogeneous data source provided by the embodiment of the present application in combination with the accompanying drawings and preferred embodiments.

[0061] First, the application scenario of the method for secure data transfer of data elements in heterogeneous data sources according to the embodiment of the present application is described in detail.

[0062] Different database management systems, file formats, network protocols, etc., lead to the heterogeneity of different data sources.

[0063] In the process of data transfer between heterogeneous data sources, intermediate process tables are often generated, which not only increases the complexity of data management, but also may lead to the risk of data leakage. Existing data transfer methods often ignore the security issues caused by the intermediate process tables during the data migration process, including the form of data components and the security of data reading and writing when heterogeneous data is connected. The existence of these problems makes data vulnerable to attacks during the transfer process, increasing the risk of data leakage and tampering.

[0064] In response to the above technical problems, the present application provides a method for secure data transfer of data elements in heterogeneous data sources, which can effectively manage the connection information and metadata of multiple data sources, and can also ensure the security and integrity of data during the data transfer process to meet the high requirements of modern enterprises for data security and reliability.

[0065] See also Figure 1-2 , the embodiment of the present application provides a method for secure data transfer of a data element in a heterogeneous data source, such as Figure 1-2As shown, the secure data transfer method of the embodiment of the present application includes the following steps:

[0066] Step S1, based on the established connection relationship between data sources and the super management account, the connection information between data sources and the super management user information are obtained, and the corresponding data source is registered with the management center through the JDBC owned by the data source.

[0067] Step S2: managing the connection process between data sources through the management center.

[0068] In some possible implementations, the above-mentioned management of the connection process includes:

[0069] Step S201, determining and obtaining the user's role according to the data component development, production, and distribution process.

[0070] Step S202: automatically authorize the data source or reject the connection according to the user's role.

[0071] Step S203, according to the health status of the data source, the data source is automatically online and offline; when online, an online recovery notification is issued for the access request of the user during the offline period.

[0072] Step S204, detect the number of JDBC connections of the data source; issue an early warning when the number of connections exceeds 80% of the preset threshold; sort the current connected users by activity, and send a message notification of possible closure to the top 20% inactive connected users; when the number of connections exceeds the threshold, close the top 20% inactive connected users. The standard for inactive connected users is set as: no data transmission is detected within 10 minutes.

[0073] Step S3: managing the generation process of the data element through the management center.

[0074] In the process of generating data components, unified super account management realizes unified management of multiple data sources, and the metadata information of data sources can be automatically scanned and updated through the management center; virtual jdbc direct authorization access to source data is realized through the management center. In this way, the data timeliness and effectiveness of the component data generation process and the data source data elastic routing authorization are achieved.

[0075] In some possible implementations, managing the generation process of data elements includes:

[0076] Step S301 : regularly scan and maintain metadata information of each data source, and record the scanning time.

[0077] Step S302: When the data source data increases, the metadata information of the data source is automatically and synchronously updated.

[0078] Step S303: The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available.

[0079] Step S304, regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time.

[0080] Step S305, based on the virtual JDBC provided by the management center, the JDBC is automatically routed to the JDBC of the corresponding data source according to the data resources required in the process of data component development, production and circulation and the metadata information of the data source.

[0081] Step S4: receiving the data query request initiated by the client through the pre-configured task scheduling node, and sending the data query request to the management center.

[0082] Step S5: The management center queries and parses the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and transmits the parsed data to the task scheduling node.

[0083] The above data transmission process may be achieved by transmitting the above data through reflection; it may also be achieved by using a public cache layer in combination with fine-grained permissions.

[0084] Step S6, the task scheduling node registers the node task according to the data query request, and executes the node task to load data according to the parsed data.

[0085] Step S7: The client obtains the data loading result of the task scheduling node through a data query request, and executes the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out.

[0086] In the above steps, during the data transfer process of heterogeneous data sources, the metadata management platform can be understood as being isolated from the task scheduling and execution platform. The data component receives the data query request initiated by the query client through the task scheduling node set in the heterogeneous database system; through the scheduling node, the management center is requested to parse the metadata information of the source data source and the relevant information of the target data source, so as to ensure that the connection information of the data component is not saved when the data component transfers data on the scheduling platform, and it can be checked immediately after use. In the entire task scheduling platform, only the program knows the metadata information of the data component.

[0087] Therefore, when converting data between heterogeneous data sources, the existence of data components can be understood as elastic data abstraction, which represents an immutable, partitionable set of elements that can be calculated in parallel. Elasticity is reflected in:

[0088] (1) Fault-tolerant elasticity: data loss can be automatically recovered;

[0089] (2) Storage flexibility: automatic switching between memory and disk;

[0090] (3) Computational flexibility: retry mechanism for computational errors;

[0091] (4) Sharding flexibility: Sharding can be done again as needed.

[0092] The data processing process of the data component does not generate an intermediate process table. The data has been loaded before the user uses it, and the user does not touch the data source, thus ensuring the reliability and security of the data component during operation.

[0093] Based on the above technical solution, this application avoids the limitation of the database's cumbersome configuration information relying on personnel management, realizes centralized and automated management of configuration information, makes the database configuration more consistent and standardized, facilitates monitoring and maintenance, reduces human errors, and improves efficiency.

[0094] The present application provides a flexible data abstraction form in which data is available but not visible. By avoiding the generation of intermediate process tables, the risk of improper access or leakage is reduced, and the security of data is protected. At the same time, the demand for data storage can be reduced and the efficiency of data processing can be improved.

[0095] See also Figure 3 Corresponding to the above-mentioned data element secure data transfer method embodiment in heterogeneous data sources, the present application embodiment provides a data element secure data transfer device in heterogeneous data sources, the secure data transfer device comprising:

[0096] The data source registration module 1001 is used to obtain the connection information and super management user information between the data sources based on the established connection relationship between the data sources and the super management account, and register with the management center through the JDBC corresponding to the data source;

[0097] The data source connection management module 1002 is used to manage the connection process between data sources through the management center;

[0098] A data element generation management module 1003, used to manage the generation process of data elements through the management center;

[0099] The task scheduling node module 1004 is used to receive a data query request initiated by a client through a pre-configured task scheduling node, and send the data query request to the management center;

[0100] The query and analysis module 1005 is used for the management center to query and analyze the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and transmit the analyzed data to the task scheduling node;

[0101] A node data loading module 1006, configured for the task scheduling node to register a node task according to the data query request, and to execute the node task to load data according to the parsed data;

[0102] The user task execution module 1007 is used for the client to obtain the data loading result of the task scheduling node through a data query request, and execute the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out.

[0103] Furthermore, the data source connection management module is specifically used to:

[0104] Determine and obtain the user's role according to the data element development, production, and distribution process;

[0105] Automatically authorize or deny connection to the data source according to the role of the user;

[0106] According to the health status of the data source, the data source is automatically online and offline; when online, the access request of the user during the offline period is notified to restore the online access;

[0107] Detect the number of JDBC connections of the data source; issue an early warning when the number of connections exceeds 80% of a preset threshold; sort the current connected users by activity, and send a message notification of possible closure to the TOP20% of the inactive connected users; when the number of connections exceeds the threshold, close the current TOP20% of the inactive connected users; wherein the standard for the inactive connected users is set as: no data transmission is detected within 10 minutes.

[0108] Furthermore, the data element generation management module is specifically used to:

[0109] Regularly scan and maintain metadata information of each of the data sources, and record the scanning time;

[0110] When the data source data increases, the metadata information of the data source is automatically and synchronously updated;

[0111] The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available;

[0112] Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time;

[0113] Based on the virtual JDBC provided by the management center, the data resources required in the development, production and circulation of the data element and the metadata information of the data source are automatically routed to the JDBC corresponding to the data source.

[0114] Furthermore, the query and analysis module is specifically used to:

[0115] The data is passed through reflection or through a common cache layer in conjunction with fine-grained permissions.

[0116] The above-mentioned data element secure data transfer device in heterogeneous data sources implements the steps and various processes of the above-mentioned data element secure data transfer method embodiment in heterogeneous data sources, and can achieve the same technical effect. To avoid repetition, they are not repeated here.

[0117] See also Figure 4 Corresponding to the above-mentioned data element secure data transfer method embodiment in heterogeneous data sources, the present application embodiment provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the above-mentioned data element secure data transfer method embodiment in heterogeneous data sources and the various processes of the embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0118] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0119] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0120] Corresponding to the above-mentioned data element in the embodiment of the method for securely transferring data in heterogeneous data sources, the embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps of the above-mentioned data element in the embodiment of the method for securely transferring data in heterogeneous data sources and the various processes of the embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0121] The processor is the processor in the electronic device described in the above embodiment of the present application. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0122] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0124] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for secure data transfer of data elements in heterogeneous data sources, characterized in that: include: Based on the established connection relationship between data sources and the super management account, the connection information between data sources and the super management user information are obtained, and the data sources are registered with the management center through the JDBC corresponding to the data source; Managing the connection process between data sources through the management center; Managing the generation process of data elements through the management center; Receive a data query request initiated by a client through a pre-configured task scheduling node, and send the data query request to the management center; The management center queries and parses the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and transmits the parsed data to the task scheduling node; The task scheduling node registers a node task according to the data query request, and executes the node task to load data according to the parsed data; The client obtains the data loading result of the task scheduling node through a data query request, and executes the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out.

2. The method for secure data transfer of data elements in heterogeneous data sources according to claim 1, characterized in that: The step of managing the connection process between data sources through the management center includes: Determine and obtain the user's role according to the data element development, production, and distribution process; Automatically authorize or deny connection to the data source according to the role of the user; According to the health status of the data source, the data source is automatically online and offline; when online, the access request of the user during the offline period is notified to restore the online access; Detect the number of JDBC connections of the data source; issue an early warning when the number of connections exceeds 80% of a preset threshold; sort the current connected users by activity, and send a message notification of possible closure to the TOP20% of the inactive connected users; when the number of connections exceeds the threshold, close the current TOP20% of the inactive connected users; wherein the standard for the inactive connected users is set as: no data transmission is detected within 10 minutes.

3. The method for secure data transfer of data elements in heterogeneous data sources according to claim 1, characterized in that: The step of managing the generation process of the data element by the management center includes: Regularly scan and maintain metadata information of each of the data sources, and record the scanning time; When the data source data increases, the metadata information of the data source is automatically and synchronously updated; The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available; Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time; Based on the virtual JDBC provided by the management center, the data resources required in the development, production and circulation of the data element and the metadata information of the data source are automatically routed to the JDBC corresponding to the data source.

4. The method for secure data transfer of data elements in heterogeneous data sources according to claim 1, characterized in that: The step of transferring the parsed data to the task scheduling node includes: The data is passed through reflection or through a common cache layer in conjunction with fine-grained permissions.

5. A secure data transfer device for data elements in heterogeneous data sources, characterized in that: include: A data source registration module is used to obtain the connection information between data sources and the super management user information based on the established connection relationship between data sources and the super management account, and register with the management center through the JDBC corresponding to the data source; A data source connection management module, used to manage the connection process between data sources through the management center; A data element generation management module, used to manage the generation process of data elements through the management center; A task scheduling node module, used to receive a data query request initiated by a client through a pre-configured task scheduling node, and send the data query request to the management center; A query and analysis module, which is used by the management center to query and analyze the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and transmit the analyzed data to the task scheduling node; A node data loading module, used for the task scheduling node to register a node task according to the data query request, and to execute the node task to load data according to the parsed data; The user task execution module is used for the client to obtain the data loading result of the task scheduling node through a data query request, and execute the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out.

6. The secure data transfer device for data elements in heterogeneous data sources according to claim 5, characterized in that: The data source connection management module is specifically used to: Determine and obtain the user's role according to the data element development, production, and distribution process; Automatically authorize or deny connection to the data source according to the role of the user; According to the health status of the data source, the data source is automatically online and offline; when online, the access request of the user during the offline period is notified to restore the online access; Detecting the number of JDBC connections of the data source; issuing an early warning when the number of connections exceeds 80% of a preset threshold; The current connected users are sorted by activity, and a message notification of possible closure is sent to the TOP20% inactive connected users; when the number of connections exceeds the threshold, the TOP20% inactive connected users are closed; wherein the standard for the inactive connected users is set as: no data transmission is detected within 10 minutes.

7. The secure data transfer device for data elements in heterogeneous data sources according to claim 5, characterized in that: The data element generation management module is specifically used for: Regularly scan and maintain metadata information of each of the data sources, and record the scanning time; When the data source data increases, the metadata information of the data source is automatically and synchronously updated; The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available; Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time; Based on the virtual JDBC provided by the management center, the data resources required in the development, production and circulation of the data element and the metadata information of the data source are automatically routed to the JDBC corresponding to the data source.

8. The secure data transfer device for data elements in heterogeneous data sources according to claim 5, characterized in that: The query and analysis module is specifically used for: The data is passed through reflection or through a common cache layer in conjunction with fine-grained permissions.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for secure data transfer of data elements in heterogeneous data sources as claimed in any one of claims 1 to 4.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method for secure data transfer of a data element in a heterogeneous data source as claimed in any one of claims 1 to 4 are implemented.

Images