Data element in heterogeneous data source security data transfer method, device and equipment

Abstract

The application provides a secure data transfer method, device and equipment of data elements among heterogeneous data sources. The method comprises the following steps: acquiring connection information and super-user information among data sources, and registering to a management center; managing a connection process among the data sources and a generation process of the data elements by the management center; receiving a data query request of a client by a task scheduling node, and sending the request to the management center; inquiring and analyzing metadata information of a source data source and related information corresponding to a target data source by the management center, and delivering the analyzed data to the task scheduling node; registering a node task by the task scheduling node, and executing the node task to load data according to the analyzed data; obtaining a data loading result of the task scheduling node by the client through the data query request, and executing a corresponding user task, and obtaining a calculation result and writing out after the execution is completed. The application realizes the secure and efficient data transfer among the heterogeneous data sources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a secure data transfer method, device and equipment of data elements in heterogeneous data sources and a medium. BACKGROUND

[0002] With the development of information technology, data-driven business environment has become an important part of enterprises. Due to the heterogeneity of various data sources, the security, reliability and efficiency of data exchange and sharing between different data sources are crucial.

[0003] In related technologies, data transfer between heterogeneous data sources is usually managed by permissions. In use, the service party implements permission control according to the permissions embodied by the user. Not only does this increase management costs, but it may also cause data leakage risks due to improper permission settings. SUMMARY

[0004] The present application provides a secure data transfer method, device and equipment of data elements in heterogeneous data sources, which solves the problem of how to achieve secure and efficient data transfer between heterogeneous data sources in related technologies.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, a secure data transfer method of data elements in heterogeneous data sources is provided, comprising:

[0007] Based on the established connection relationship between data sources and the super management account, the connection information between data sources and the super user information are obtained, and the corresponding JDBC of the data source is registered to the management center;

[0008] The management center manages the connection process between data sources;

[0009] The management center manages the generation process of data elements;

[0010] The task scheduling node receives a data query request initiated by a client, and sends the data query request to the management center;

[0011] The management center queries and analyzes the metadata information of the source data source and the related information corresponding to the target data source according to the data query request, and transmits the analyzed data to the task scheduling node;

[0012] The task scheduling node registers a node task according to the data query request, and executes the node task to load data according to the analyzed data;

[0013] The client obtains the data loading result of the task scheduling node through a data query request, and executes a corresponding user task according to the data loading result, and obtains a calculation result after the user task is executed and writes out.

[0014] In a first possible implementation manner of the first aspect, the step of managing, by the management center, a connection process between data sources comprises:

[0015] According to the development, production and flow process of the data element, the role of the user is determined and obtained;

[0016] According to the role of the user, the automatic authorization or rejection of connection of the data source is performed;

[0017] According to the health state of the data source, the automatic online and offline of the data source are realized; when online, the access request of the user during the offline period is notified of online recovery;

[0018] The number of JDBC connections of the data source is detected; when the number of connections exceeds 80% of a preset threshold, a warning is given; the active degree of the current connection user is sorted, and a possible closing message is sent to the TOP 20% of the connection users; when the number of connections exceeds the threshold, the TOP 20% of the connection users are executed to close; wherein, the standard of the inactive connection user is that no data transmission is detected within 10 minutes.

[0019] In a second possible implementation manner of the first aspect, the step of managing, by the management center, the generation process of the data element comprises:

[0020] Periodically scan and maintain the metadata information of each data source, and record the scanning time;

[0021] When the data of the data source increases, the metadata information of the data source is automatically updated;

[0022] The management center periodically scans the heartbeat information of the data source to ensure that the data source is maintained in an available state;

[0023] Periodically scan the registered data source list to ensure that the new data source can be discovered and used at any time;

[0024] Based on the virtual JDBC provided by the management center, the data resource required in the development, production and flow process of the data element and the metadata information of the data source are automatically routed to the JDBC of the corresponding data source.

[0025] In a third possible implementation manner of the first aspect, the step of delivering the parsed data to the task scheduling node comprises:

[0026] The data is delivered in a reflective manner or in a manner of using fine-grained permissions in cooperation with a common cache layer.

[0027] In a second aspect, a device for secure data transfer of data elements in a heterogeneous data source is provided, comprising:

[0028] A data source registration module is configured to acquire connection information between data sources and super-user information based on an established connection relationship between the data sources and a super management account, and register with a management center through a corresponding JDBC owned by the data source;

[0029] A data source connection management module is configured to manage a connection process between data sources through the management center;

[0030] A data element generation management module is configured to manage a generation process of data elements through the management center;

[0031] A task scheduling node module is configured to receive a data query request initiated by a client through a pre-configured task scheduling node, and send the data query request to the management center;

[0032] A query and analysis module is configured to query and analyze metadata information of a source data source and related information corresponding to a target data source according to the data query request, and deliver parsed data to the task scheduling node;

[0033] A node data loading module is configured to register a node task according to the data query request, and perform the node task according to the parsed data to load data;

[0034] A user task execution module is configured to obtain a data loading result of the task scheduling node through the data query request, and execute a corresponding user task according to the data loading result, and obtain a calculation result and write out after the user task is executed.

[0035] In a first possible implementation manner of the second aspect, the data source connection management module is specifically configured to:

[0036] Determine and acquire a role of a user according to a development, production and flow process of the data element;

[0037] Automatically authorize or reject connection of the data source according to the role of the user;

[0038] Based on the health status of the data source, the system automatically brings the data source online and offline; when the data source is brought online, a notification is sent to users who requested access during the offline period to restore the online status.

[0039] The system detects the number of JDBC connections to the data source; issues an alert when the number of connections exceeds 80% of a preset threshold; sorts the current connected users by activity level and sends a notification message to the top 20% of inactive connected users that they may be closed; when the number of connections exceeds the threshold, it performs a closure action on the top 20% of inactive connected users; wherein, the standard setting for inactive connected users is: no data transmission detected within 10 minutes.

[0040] In a second possible implementation of the second aspect, the data element generation management module is specifically used for:

[0041] Regularly scan and maintain the metadata information of each of the data sources, and record the scanning time;

[0042] When the data source data increases, the metadata information of the data source is automatically synchronized and updated;

[0043] The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available.

[0044] Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time;

[0045] Based on the virtual JDBC provided by the management center, the system automatically routes data to the corresponding JDBC of the data source according to the data resources required during the development, production, and circulation of the data element and the metadata information of the data source.

[0046] In the third possible implementation of the second aspect, the query and parsing module is specifically used for:

[0047] The data can be transmitted via reflection or by using a public caching layer in conjunction with fine-grained permissions.

[0048] Thirdly, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the secure data transfer method for data elements in heterogeneous data sources as described in the first aspect.

[0049] Fourthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the secure data transfer method for data elements in a heterogeneous data source as described in the first aspect.

[0050] Beneficial effects:

[0051] This application avoids the limitations of relying on personnel to manage cumbersome database configuration information, and realizes centralized and automated management of configuration information, making the database configuration more consistent and standardized, facilitating monitoring and maintenance, reducing human error, and improving efficiency.

[0052] This application provides a flexible form of data abstraction where data is available but not visible. By avoiding the generation of intermediate process tables, it reduces the risk of improper access or leakage, protects data security, and at the same time reduces the need for data storage and improves the efficiency of data processing. Attached Figure Description

[0053] Figure 1 A schematic flowchart illustrating a secure data transfer method for data elements in a heterogeneous data source, provided in an embodiment of this application;

[0054] Figure 2 A schematic flowchart illustrating another secure data transfer method for data elements in heterogeneous data sources provided in this application embodiment;

[0055] Figure 3 A structural block diagram of a secure data transfer device for data elements in heterogeneous data sources provided in this application embodiment;

[0056] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0057] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions in the embodiments of this application are clearly described. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0058] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0059] The steps described in the specification and the flowcharts in the accompanying drawings of this invention are not necessarily strictly executed according to the step numbers; the execution order of the method steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.

[0060] The following detailed description, in conjunction with the accompanying drawings and preferred embodiments, describes the secure data transfer method, apparatus, device, and medium for data elements in heterogeneous data sources provided in this application.

[0061] First, the application scenarios of the data elements in the secure data transfer method of heterogeneous data sources according to the embodiments of this application will be described in detail.

[0062] Different database management systems, file formats, and network protocols lead to heterogeneity in different data sources.

[0063] During data transfer between heterogeneous data sources, intermediate process tables are often generated, which not only increases the complexity of data management but also may lead to the risk of data leakage. Existing data transfer methods often overlook the security issues caused by intermediate process tables during data migration, as well as the format of data elements and the security of data reading and writing when data elements are connected in heterogeneous data environments. The existence of these issues makes data vulnerable to attacks during the transfer process, increasing the risk of data leakage and tampering.

[0064] To address the aforementioned technical issues, this application provides a secure data transfer method for data elements across heterogeneous data sources. This method effectively manages connection information and metadata from multiple data sources and ensures data security and integrity during the data transfer process, thereby meeting the high requirements of modern enterprises for data security and reliability.

[0065] Please see Figures 1-2 This application provides a method for secure data transfer of data elements across heterogeneous data sources, such as... Figures 1-2As shown, the secure data transfer method of this application embodiment includes the following steps:

[0066] Step S1: Based on the established connection relationship between data sources and the super administrator account, obtain the connection information between data sources and the super administrator user information, and register them with the management center through the JDBC of the corresponding data source.

[0067] Step S2: Manage the connection process between data sources through the management center.

[0068] In some possible implementations, the above-described management of the connection process includes:

[0069] Step S201: Determine and obtain the user's role based on the data element development, production, and distribution process.

[0070] Step S202: Based on the user's role, automatically authorize or deny the connection to the data source.

[0071] Step S203: Based on the health status of the data source, automatically bring the data source online and offline; when it comes online, send an online recovery notification to users who accessed the data source during the offline period.

[0072] Step S204: Detect the number of JDBC connections to the data source; issue an alert when the number of connections exceeds 80% of a preset threshold; sort the current connected users by activity level and send a notification message to the top 20% of inactive connected users that they may be closed; when the number of connections exceeds the threshold, perform a shutdown action on the top 20% of inactive connected users. The standard for inactive connected users is defined as: no data transmission detected within 10 minutes.

[0073] Step S3: The data element generation process is managed through the management center.

[0074] During the data element generation process, a unified super account management system enables unified management of multiple data sources. The management center can automatically scan and update metadata information of the data sources and directly authorize access to source data via virtual JDBC. This ensures data timeliness, validity, and flexible routing authorization of data sources during the element data generation process.

[0075] In some possible implementations, the generation process of data elements is managed, including:

[0076] Step S301: Regularly scan and maintain the metadata information of each data source, and record the scanning time.

[0077] Step S302: When data from the data source increases, the metadata information of the data source is automatically synchronized and updated.

[0078] Step S303: The management center periodically scans the heartbeat information of the data source to ensure that the data source remains in an available state.

[0079] Step S304: Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time.

[0080] Step S305: Based on the virtual JDBC provided by the management center, the system automatically routes to the JDBC of the corresponding data source according to the data resources required during the development, production, and circulation of data elements and the metadata information of the data source.

[0081] Step S4: Receive the data query request initiated by the client through the pre-configured task scheduling node, and send the data query request to the management center.

[0082] In step S5, the management center queries and parses the metadata information of the source data source and the relevant information of the target data source according to the data query request, and then transmits the parsed data to the task scheduling node.

[0083] The process of transmitting data described above can be achieved through reflection; alternatively, it can be implemented using a public caching layer in conjunction with fine-grained permissions.

[0084] Step S6: The task scheduling node registers node tasks based on the data query request, and executes the node tasks to load data based on the parsed data.

[0085] In step S7, the client obtains the data loading result of the task scheduling node through a data query request, executes the corresponding user task according to the data loading result, and obtains and writes the calculation result after the user task is completed.

[0086] In the above steps, during the data transfer process from heterogeneous data sources, the data element can be understood as being isolated from the metadata management platform and the task scheduling and execution platform. This allows the data element in the heterogeneous database system to receive data query requests initiated by query clients through a designated task scheduling node; it then requests the management center through the scheduling node to parse the metadata information of the source data source and the relevant information of the target data source. This ensures that the data element does not retain its connection information during data transfer on the scheduling platform, allowing for immediate querying. Within the entire task scheduling platform, only the program knows the metadata information of the data element.

[0087] Therefore, when data elements are used for data transformation between heterogeneous data sources, their existence can be understood as a flexible data abstraction. It represents an immutable, partitionable set whose elements can be computed in parallel. This flexibility is reflected in:

[0088] (1) Fault tolerance elasticity: Data loss can be automatically recovered;

[0089] (2) Storage flexibility: automatic switching between memory and disk;

[0090] (3) Flexibility of computation: retry mechanism for computational errors;

[0091] (4) Flexibility of fragmentation: fragmentation can be re-fractionated as needed.

[0092] The data processing of data elements does not generate intermediate process tables. The data is loaded before the user uses it, and the user does not contact the data source, thus ensuring the reliability and security of the data element during operation.

[0093] Based on the above technical solution, this application avoids the limitation of relying on personnel to manage the cumbersome configuration information of the database, realizes centralized and automated management of configuration information, makes the database configuration more consistent and standardized, facilitates monitoring and maintenance, reduces human error, and improves efficiency.

[0094] This application provides a flexible form of data abstraction where data is available but not visible. By avoiding the generation of intermediate process tables, it reduces the risk of improper access or leakage, protects data security, and at the same time reduces the need for data storage and improves the efficiency of data processing.

[0095] See Figure 3 Corresponding to the above-described embodiments of the secure data transfer method for data elements in heterogeneous data sources, this application provides a secure data transfer device for data elements in heterogeneous data sources, the secure data transfer device comprising:

[0096] The data source registration module 1001 is used to obtain connection information between data sources and super administrator user information based on the established connection relationship between data sources and the super administrator account, and register the data source with the management center through the JDBC of the data source.

[0097] The data source connection management module 1002 is used to manage the connection process between data sources through the management center;

[0098] The data element generation management module 1003 is used to manage the generation process of data elements through the management center;

[0099] The task scheduling node module 1004 is used to receive data query requests initiated by clients through a pre-configured task scheduling node, and send the data query requests to the management center.

[0100] The query and parsing module 1005 is used by the management center to query and parse the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and to transmit the parsed data to the task scheduling node.

[0101] The node data loading module 1006 is used for the task scheduling node to register a node task according to the data query request, and to execute the node task to load data according to the parsed data.

[0102] The user task execution module 1007 is used by the client to obtain the data loading result of the task scheduling node through a data query request, and to execute the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out.

[0103] Furthermore, the data source connection management module is specifically used for:

[0104] Based on the development, production, and distribution process of the data elements, the user's role is determined and obtained;

[0105] Based on the user's role, automatically authorize or deny the connection to the data source;

[0106] Based on the health status of the data source, the system automatically brings the data source online and offline; when the data source is brought online, a notification is sent to users who requested access during the offline period to restore the online status.

[0107] The system detects the number of JDBC connections to the data source; issues an alert when the number of connections exceeds 80% of a preset threshold; sorts the current connected users by activity level and sends a notification message to the top 20% of inactive connected users that they may be closed; when the number of connections exceeds the threshold, it performs a closure action on the top 20% of inactive connected users; wherein, the standard setting for inactive connected users is: no data transmission detected within 10 minutes.

[0108] Furthermore, the data element generation and management module is specifically used for:

[0109] Regularly scan and maintain the metadata information of each of the data sources, and record the scanning time;

[0110] When the data source data increases, the metadata information of the data source is automatically synchronized and updated;

[0111] The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available.

[0112] Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time;

[0113] Based on the virtual JDBC provided by the management center, the system automatically routes data to the corresponding JDBC of the data source according to the data resources required during the development, production, and circulation of the data element and the metadata information of the data source.

[0114] Furthermore, the query and parsing module is specifically used for:

[0115] The data can be transmitted via reflection or by using a public caching layer in conjunction with fine-grained permissions.

[0116] The aforementioned data element in the secure data transfer device for heterogeneous data sources implements the steps and processes of the aforementioned data element in the secure data transfer method embodiment for heterogeneous data sources, and can achieve the same technical effect. To avoid repetition, these will not be repeated here.

[0117] See Figure 4 Corresponding to the above-described embodiments of the secure data transfer method for data elements in heterogeneous data sources, this application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps and processes of the above-described embodiments of the secure data transfer method for data elements in heterogeneous data sources, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0118] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback function, image playback function, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0119] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0120] Corresponding to the above-described embodiments of the secure data transfer method for data elements in heterogeneous data sources, this application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps and processes of the above-described embodiments of the secure data transfer method for data elements in heterogeneous data sources, and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0121] The processor is the processor in the electronic device described in the above embodiments of this application. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0122] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0124] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for secure data transfer of data elements in heterogeneous data sources, characterized in that, include: Based on the established connection relationships between data sources and the super administrator account, obtain the connection information between data sources and the super administrator user information, and register them with the management center through the JDBC of the corresponding data source. The management center manages the connection process between data sources; The data element generation process is managed through the management center; The system receives data query requests initiated by clients through a pre-configured task scheduling node and sends the data query requests to the management center. The management center queries and parses the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and transmits the parsed data to the task scheduling node; The task scheduling node registers node tasks according to the data query request, and executes the node tasks to load data according to the parsed data. The client obtains the data loading result of the task scheduling node through a data query request, and executes the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out. The steps for managing the connection process between data sources through the management center include: Based on the development, production, and distribution process of the data elements, the user's role is determined and obtained; Based on the user's role, automatically authorize or deny the connection to the data source; Based on the health status of the data source, the data source can be automatically brought online and taken offline. Detect the number of JDBC connections to the data source; The steps for managing the generation process of data elements through the management center include: Regularly scan and maintain the metadata information of each of the data sources, and record the scanning time; When the data source data increases, the metadata information of the data source is automatically synchronized and updated; The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available. Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time; Based on the virtual JDBC provided by the management center, the system automatically routes data to the corresponding JDBC of the data source according to the data resources required during the development, production, and circulation of the data element and the metadata information of the data source.

2. The secure data transfer method for data elements in heterogeneous data sources according to claim 1, characterized in that, When the system goes live, it will send a notification to users who requested access during the offline period to restore the system. An alert is issued when the number of connections exceeds 80% of a preset threshold. The system sorts the current connected users by activity level and sends a notification message to the top 20% of inactive connected users that they may be closed. When the number of connections exceeds the threshold, the system closes the connections for the top 20% of inactive connected users. The standard for inactive connected users is defined as: no data transmission has been detected within 10 minutes.

3. The secure data transfer method for data elements in heterogeneous data sources according to claim 1, characterized in that, The step of transmitting the parsed data to the task scheduling node includes: The data can be transmitted via reflection or by using a public caching layer in conjunction with fine-grained permissions.

4. A secure data transfer device for data elements in heterogeneous data sources, characterized in that, include: The data source registration module is used to obtain connection information between data sources and super administrator user information based on the established connection relationship between data sources and the super administrator account, and register the data source with the management center through the JDBC of the data source. The data source connection management module is used to manage the connection process between data sources through the management center; The data element generation management module is used to manage the generation process of data elements through the management center; The task scheduling node module is used to receive data query requests initiated by clients through pre-configured task scheduling nodes and send the data query requests to the management center. The query and parsing module is used by the management center to query and parse the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and to transmit the parsed data to the task scheduling node; A node data loading module is used for the task scheduling node to register a node task according to the data query request, and to execute the node task to load data according to the parsed data. The user task execution module is used by the client to obtain the data loading result of the task scheduling node through a data query request, and to execute the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out. The data source connection management module is specifically used for: Based on the development, production, and distribution process of the data elements, the user's role is determined and obtained; Based on the user's role, automatically authorize or deny the connection to the data source; Based on the health status of the data source, the data source can be automatically brought online and taken offline. Detect the number of JDBC connections to the data source; The data element generation and management module is specifically used for: Regularly scan and maintain the metadata information of each of the data sources, and record the scanning time; When the data source data increases, the metadata information of the data source is automatically synchronized and updated; The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available. Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time; Based on the virtual JDBC provided by the management center, the system automatically routes data to the corresponding JDBC of the data source according to the data resources required during the development, production, and circulation of the data element and the metadata information of the data source.

5. The secure data transfer device for data elements in heterogeneous data sources according to claim 4, characterized in that, The data source connection management module is further used for: When the system goes live, it will send a notification to users who requested access during the offline period to restore the system. An alert is issued when the number of connections exceeds 80% of a preset threshold. The system sorts the current connected users by activity level and sends a notification message to the top 20% of inactive connected users that they may be closed. When the number of connections exceeds the threshold, the system closes the connections for the top 20% of inactive connected users. The standard for inactive connected users is defined as: no data transmission has been detected within 10 minutes.

6. The secure data transfer device for data elements in heterogeneous data sources according to claim 4, characterized in that, The query and parsing module is specifically used for: The data can be transmitted via reflection or by using a public caching layer in conjunction with fine-grained permissions.

7. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the secure data transfer method for data elements in heterogeneous data sources as described in any one of claims 1 to 3.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the secure data transfer method for data elements in a heterogeneous data source as described in any one of claims 1 to 3.

Images