Data access method, apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a data access method, apparatus, electronic device, and storage medium. The method includes: sending a data connection request to a data source system, where the data connection request includes connection configuration information, and the connection configuration information is for representing configuration information necessary to complete a data connection with the data source system; loading access target data determined by the data source system in response to the data connection request; analyzing the loaded access target data and detecting the analyzed access target data based on the connection configuration information; and accessing the detected access target data to a target database.

Description

Technical Field

[0001] (Related Application) This disclosure claims the priority of a Chinese patent application with the application number CN202210555604.7 and the application title "Data Access Method, Apparatus, Electronic Device, and Storage Medium", which was filed on May 20, 2022, and all of its contents are incorporated herein by reference.

[0002] This disclosure relates to the field of data processing, and particularly to data access methods, apparatuses, electronic devices, and storage media.

Background Art

[0003] A Data Middle Platform has profound significance in the digital transformation process of governments and enterprises. It constructs a system for building, managing, and using data, including data technology, data management, data operation, etc., for the business and data precipitation of each business unit to achieve data empowerment. To ensure the reliability of the data assets and data services of the Data Middle Platform, the data accessed from the data source to the Data Middle Platform should: 1) Data integrity: All data should be collected without omission. 2) Data real-time performance: The time delay until the data reaches the Data Middle Platform should not affect the timeliness requirements of the service applications that depend on the data. 3) Data accuracy: It should meet the requirement that the data is not modified during the collection process.

Summary of the Invention

Problems to be Solved by the Invention

[0004] This disclosure provides a data access method, apparatus, electronic device, and storage medium.

Means for Solving the Problems

[0005] According to a first aspect of the present disclosure, there is provided a data access method including: sending a data connection request to a data source system, where the data connection request includes connection configuration information, and the connection configuration information is for representing configuration information necessary to complete a data connection with the data source system; loading access target data determined by the data source system in response to the data connection request; analyzing the successfully loaded access target data and detecting the analyzed access target data based on the connection configuration information; and accessing the successfully detected access target data to a target database.

[0006] According to a second aspect of the present disclosure, there is further provided a data access apparatus including: a first transmission module configured to send a data connection request to a data source system, where the data connection request includes connection configuration information, and the connection configuration information is for representing configuration information necessary to complete a data connection with the data source system; a load module configured to load access target data determined by the data source system in response to the data connection request; an analysis module configured to analyze the successfully loaded access target data and detect the analyzed access target data based on the connection configuration information; and an access module configured to access the successfully detected access target data to a target database.

[0007] According to a third aspect of the present disclosure, there is provided an electronic device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication via the communication bus, the memory is for storing a computer program, and the processor is for implementing the data access method described in the first aspect when executing the program stored in the memory.

[0008] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium storing a computer program, wherein the computer program realizes the data access method described in the first aspect when executed by a processor.

Brief Description of the Drawings

[0009] The drawings here are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure and for interpreting the principles of the present disclosure together with the specification.

[0010] Hereinafter, in order to more clearly explain the embodiments of the present disclosure or the technical solutions in the related art, the drawings necessary for the description of the embodiments or the related art will be briefly described. However, for those skilled in the art, it goes without saying that other drawings can be obtained based on these drawings without any inventive work.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Best Mode for Carrying Out the Invention

[0011] In order to make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. However, the embodiments described below are only some embodiments of the present disclosure, not all embodiments. All other embodiments obtained by those skilled in the art without creative work based on the embodiments of the present disclosure are also included in the protection scope of the present disclosure.

[0012] Currently, the management control over data quality (e.g., data integrity, real-time nature, accuracy) during the implementation of a data middleware platform is usually realized at a stage after data access. When the network state is stable and the data access method is single, the data quality of the data middleware platform can be guaranteed. However, when the network situation is complex and multiple types of data source-side systems evolve independently, the data quality of the data middleware platform cannot be effectively guaranteed, and the operation and maintenance costs will increase.

[0013] FIG. 1 is a flowchart of a data access method according to an embodiment of the present disclosure. As shown in FIG. 1, the data access method may include the following steps 101 to 104.

[0014] In step 101, a data connection request is sent to a data source system, and the data connection request includes connection configuration information. The connection configuration information is for representing the configuration information necessary to complete a data connection with the data source system.

[0015] The above connection configuration information is artificially preset information, and the connection configuration information may include, but is not limited to, an IP address corresponding to a data source system, a connection protocol type, a port number of the connection protocol, authentication information, and identification information of data to be accessed. Here, the connection protocol type may include a File Transfer Protocol (abbreviated as FTP), an SSH File Transfer Protocol (abbreviated as SFTP), a Java Database Connectivity (abbreviated as JDBC), a message queue, Kafka (an open-source stream processing platform developed by the Apache Software Foundation), etc.

[0016] The identification information of the data to be accessed may be catalog information, database table information, theme information, etc. When the type of the data to be accessed is different, the corresponding identification information is also different. For example, when the data source of the data source system is an FTP or SFTP type data source, the identification information of the data to be accessed may be catalog information; when the data source of the data source system is a Kafka or message queue type data source, the identification information of the data to be accessed may be theme information; when the data source of the data source system is a JDBC type data source, the identification information of the data to be accessed may be database table information.

[0017] The above data connection request is for requesting a data connection with a data source system. The connection configuration information is included in the data connection request, and based on the connection configuration information, the data source system participating in the data connection can be quickly identified, and a link for establishing a data connection with the data source system can be established.

[0018] Note that, as shown in FIG. 2, the data access method can be applied to a digital information system (also called a data middleware platform), and the digital information system can establish a data connection with a plurality of data source systems. The digital information system can realize a data connection with a data source system based on an application layer protocol (i.e., a connection protocol) above the network layer, form data assets, and provide data services.

[0019] Here, the data access procedure depends on the conditions that (1) the route of the network layer from the digital information system to the data source system is reachable, (2) the application layer (corresponding to the application layer protocol port) is in communication, (3) the authentication of the data source system is successful, and (4) the information such as the catalog (SFTP / FTP class), theme information (Kafka), and database table (JDBC) of the data access configuration is accurate.

[0020] In step 102, the data source system loads the access target data determined in response to the data connection request.

[0021] In this step, after receiving the data connection request, the data source system determines the access target data in response to the data connection request and pushes the access target data to the digital information system, so that the digital information system can load the pushed access target data.

[0022] In the loading process, the access target data can be detected based on the pre-set connection configuration information. For example, it can be detected whether the access target data is reachable, whether the authentication is successful, whether the catalog / database table name / theme is correct, etc.

[0023] If all detections pass, it indicates that the access target data can be successfully loaded, and subsequently step 103 is executed. If the detections do not pass, it indicates that the access target data cannot be successfully loaded, and it is necessary to store the access target data in the abnormal data catalog and record and store the abnormal detailed information so as to facilitate the subsequent identification of abnormalities and cause analysis.

[0024] In step 103, analyze the access target data that has been successfully loaded, and detect the analyzed access target data based on the connection configuration information.

[0025] The connection configuration information can also ensure the stability of the accessed data format (i.e., the data source version) by standardizing the data content of the access target data, such as the field content and field order of the data. Even when the data source system is upgraded or changed, the data format of the access target data in the digital information system does not need to be modified.

[0026] In this step, the digital information system can obtain the content and order of each field of the access target data by analyzing the access target data that has been successfully loaded, and further detect the content and order of each field of the access target data based on the connection configuration information. For example, it can detect whether the file content compilation of the data file conforms to the standard, whether the number and order of fields conform to the standard, whether the number and order of the table fields in the database table conform to the standard, whether the message content (fields and order) in the message queue conforms to the standard, etc.

[0027] In this way, it can be ensured that the data format of the accessed data is stable.

[0028] In step 104, access the detected access target data to the target database.

[0029] The target database is a database or a data warehouse for storing normal access data in a digital information system. In this step, the digital information system can access the detected access target data to the target database, form a data asset based on the access data in the target database, and provide a data service. When accessing the target database, a warehousing timestamp can be recorded, and the warehousing time delay of the access target data can be fully calculated based on the warehousing timestamp.

[0030] In this embodiment, in the process of accessing the access target data to the target database, the access target data is detected based on the connection configuration information, and by accessing the detected access target data, the quality of the accessed data can be effectively guaranteed, and the operation and maintenance cost can be reduced.

[0031] Furthermore, the connection configuration information includes an IP address corresponding to the data source system, a connection protocol type, a port number of the connection protocol, authentication information, and identification information of the access target data, and the identification information of the access target data includes catalog information, database table information, or theme information.

[0032] The above step of the data source system loading the access target data in response to a data connection request includes: establishing a link for data interaction with the data source system based on the IP address, connection protocol type, and port number of the connection protocol corresponding to the data source system; performing identity authentication on the data source system using the link and authentication information; when the identity authentication of the data source system passes, determining the access target data from the data source system based on the identification information of the access target data, and loading the access target data.

[0033] In some embodiments, the connection configuration information may include the IP address corresponding to the data source system, the connection protocol type, the port number of the connection protocol, the authentication information, and the identification information of the access target data. Here, the connection protocol type may include the File Transfer Protocol (abbreviated as FTP), the SSH File Transfer Protocol (abbreviated as SFTP), Java Database Connectivity (abbreviated as JDBC), the message queue, Kafka (an open-source stream processing platform developed by the Apache Software Foundation), and the like.

[0034] The identification information of the data to be accessed may be catalog information, database table information, theme information, etc. When the types of the data to be accessed are different, the corresponding identification information is also different. For example, when the data source of the data source system is an FTP or SFTP type data source, the identification information of the data to be accessed may be catalog information. When the data source of the data source system is a Kafka or message queue type data source, the identification information of the data to be accessed may be theme information. When the data source of the data source system is a JDBC type data source, the identification information of the data to be accessed may be database table information.

[0035] The process of the digital information system loading the data to be accessed may include stages such as link establishment, authentication, and download / reception of the data to be accessed. In the link establishment stage, the digital information system and the data source system need to establish a link for data interaction based on the IP address, connection protocol type, and port number of the connection protocol corresponding to the data source system. In the authentication stage, the digital information system and the data source system need to perform identity authentication on the data source system based on the established link and authentication information. In the download / reception stage of the data to be accessed, the data to be accessed is determined from the data source system based on the identification information of the data to be accessed, the data to be accessed is downloaded / received, and the downloaded / received data to be accessed can be loaded.

[0036] In this embodiment, in stages such as link establishment, authentication, and download / reception of the data to be accessed, all need to be completed based on the connection configuration information in the digital information system. Thereby, the detection of the data to be accessed in the data loading procedure is realized, and the quality of the accessed data is improved.

[0037] Furthermore, when the method successfully loads the data to be accessed, it includes the step of recording and storing the access detailed information of the data to be accessed, where the access detailed information includes at least one of a file name list, a file size, and a file generation time; and when the method fails to load the data to be accessed, it includes the step of accessing the data to be accessed to an abnormal data catalog and recording and storing the abnormal detailed information of the data to be accessed, where the abnormal detailed information includes at least one of a load time, an IP address corresponding to a data source system, a connection protocol type, a port number of a connection protocol, authentication information, and identification information of the data to be accessed.

[0038] In some embodiments, when the method successfully loads the data to be accessed, it can record and store the access detailed information of the data to be accessed that has been successfully loaded. The access detailed information here may vary depending on the data type of the data to be accessed. For example, when the data type of the data to be accessed is a data file, a message queue, or Kafka, the data access details may include one or more of a file name list, a file size, and a file generation time; when the data type of the data to be accessed is a database table, the data access details may include the acquisition time of the set of records, etc. By recording and storing the data access details, a detailed snapshot of the data access situation can be taken. When the operational data store layer data file is deleted by the life cycle management logic in the digital information system, related problems can be backtracked through the historical detailed snapshot.

[0039] If the loading of the data to be accessed fails, the data to be accessed that has failed to load can be accessed in the abnormal data catalog, and the detailed information about the abnormality of the data to be accessed that has failed to load can be recorded and stored. The detailed information about the abnormality here may include the loading time and information about the data to be accessed, such as one or more of the IP address corresponding to the data source system, the connection protocol type, the port number of the connection protocol, the authentication information, and the identification information of the data to be accessed. By recording and storing the data that has failed to load and the detailed information about the abnormality, the cause of the abnormality can be quickly identified, and the operation and maintenance costs can be reduced.

[0040] Furthermore, in the process of loading the data to be accessed, the method includes the step of periodically sending a handshake request to the data source system at preset time intervals, where the handshake request includes connection configuration information, and the step of determining the link state between the data source system and the access status of the data to be accessed based on the response status of the data source system in response to the handshake request, where the response status includes whether the response message of the data source system has been received and whether the response message indicates success in detecting the connection configuration information if the response message of the data source system has been received, and the step of recording and storing the link state and the access status.

[0041] In some embodiments, during the access process of the digital information system to the data to be accessed, the digital information system can periodically send handshake requests to the data source system at preset time intervals. Based on the response status of the data source system in response to the handshake requests, the link status between the digital information system and the data source system and the access status of the data to be accessed within each period can be determined, and the link status and access status can be recorded and stored. Furthermore, the link status and access status across the entire time dimension can be completely recorded. Here, the link status includes link non-communication / link communication, port unreachable / port reachable, authentication success / authentication failure, etc., and the access status includes no file catalog / file catalog available, no database table / database table available, theme match / theme mismatch, etc.

[0042] In addition, the digital information system can independently initiate periodic handshake requests for each data source system. The period corresponding to each data source system can be independently set, and the handshake period can be set based on the link status of each data source system. For example, for a data source system with a relatively stable link, the period can be set longer to reduce the overhead of network resources, and for a data source system with a relatively unstable link, the period can be set shorter to increase the intensity of link status monitoring.

[0043] Furthermore, the method further includes the steps of recording and storing a detection log when the link status is determined to be abnormal, where the detection log is for recording the cause of the link abnormality, and outputting warning information, where the output method of the warning information includes at least one of display on the user interaction interface, pushing a message to the user's mobile terminal, and sending an email to the user.

[0044] In some embodiments, when it is determined that the link state is abnormal, the digital information system can record and store a detection log. Here, the detection log is for recording the cause of the link abnormality, such as the inability to reach the application layer protocol, incorrect authentication information, incorrect connection catalog or database table information, etc. And the digital information system can also output warning information, and the output methods of the warning information can include displaying it on the user interaction interface, pushing a message to the user's mobile terminal, sending an email to the user, etc. In this way, when the data access is abnormal, the abnormality can be quickly discovered and identified.

[0045] Furthermore, the connection configuration information further includes data feature information, and the data feature information is for standardizing the field content and field order of the data to be accessed.

[0046] The steps of analyzing the access target data that has been successfully loaded and detecting the analyzed access target data based on the connection configuration information include: analyzing the access target data that has been successfully loaded to obtain the field content and field order of the access target data; detecting whether the field content and field order satisfy the requirements of the data feature information; when it is detected that the field content and field order satisfy the requirements of the data feature information, accessing the detected access target data to the target database; when it is detected that the field content and field order do not satisfy the requirements of the data feature information, accessing the access target data to the abnormal data catalog and recording and storing the cause of the abnormality of the access target data, where the cause of the abnormality of the access target data includes abnormal field content or abnormal field order of the access target data.

[0047] By standardizing the data content of the data to be accessed, such as the field content and field order of the data, the connection configuration information can ensure the stability of the data format to be accessed (i.e., the data source version). Even when the data source system is upgraded or changed, the data format of the data to be accessed in the digital information system will not be modified.

[0048] In this step, the digital information system can obtain the content and order of each field of the data to be accessed by analyzing the successfully loaded data to be accessed. Furthermore, based on the connection configuration information, the system can detect the content and order of each field of the data to be accessed. For example, it can detect whether the file content compilation of the data file conforms to the standard, whether the number and order of fields conform to the standard, whether the number and order of table fields in the database table conform to the standard, and whether the message content (fields and order) in the message queue conforms to the standard.

[0049] In this way, it can be ensured that the data format of the accessed data is stable.

[0050] Furthermore, there are multiple types of data source systems, and the connection protocol types corresponding to different types of data source systems are different.

[0051] In some embodiments, the data source system may include multiple types such as, for example, FTP / SFTP data sources, JDBC data sources, message queue data sources, Kafka data sources, etc.

[0052] Here, different types of data source systems correspond to different connection protocol types, for example, the connection protocol type adopted by an FTP data source may be FTP, the connection protocol type adopted by an SFTP data source may be SFTP, the connection protocol type adopted by a JDBC data source may be JDBC, etc. The digital information system in this embodiment has good compatibility with multiple types of connection protocol types, which improves the stability of the link.

[0053] In some embodiments, the procedure for a digital information system to complete an interactive data connection with a data source system includes the following steps 301 to 307, as shown in FIG.

[0054] In step 301, the digital information system initiates a data access request to a data source system based on connection configuration information.

[0055] The digital information system needs to complete the configuration of connection configuration information in advance, where the connection configuration information includes an IP address corresponding to the data source system, a connection protocol type, a port number of the connection protocol, authentication information, catalog information, database table information, theme information, data feature information, etc.

[0056] In step 302, the data source system establishes a link with the digital information system in response to a data access request, completes an authentication procedure, and downloads / receives the accessed data.

[0057] In step 303, the digital information system loads and detects the data to be accessed.

[0058] For example, detailed information such as abnormality details and access details are written to a specified database, and warning information is reported in the event of consecutive handshake failures or failure to verify connection configuration information.

[0059] In step 304, the digital information system analyzes the accessed data, and detects and warehouses the accessed data after the analysis.

[0060] The successfully loaded access target data is analyzed to obtain the field contents and field order of the access target data, and it is detected whether the field contents and field order satisfy the requirements of the data feature information. If it is detected that the field contents and field order satisfy the requirements of the data feature information, the successfully detected access target data is accessed to the target database. If it is detected that the field contents and field order do not satisfy the requirements of the data feature information, the access target data is accessed to an abnormal data catalog, and the cause of the abnormality of the access target data is recorded and stored, and the cause of the abnormality of the access target data includes an abnormality in the field content or an abnormality in the field order of the access target data.

[0061] In step 305, the digital information system initiates periodic handshake requests to the data source system based on the connection configuration information.

[0062] In step 306, the data source system periodically monitors the link state and data access status in response to the handshake request.

[0063] When it is determined that the data to be accessed is unavailable, the warehousing of the detection log is completed and warning information is issued. The handshake and verification content includes, but is not limited to, the scenarios of inaccessibility of the application layer protocol, incorrect authentication information, and incorrect connection catalog or database table information. The warning method may be display on the user interaction interface, pushing of messages to the user's mobile terminal, sending of emails to the user, etc.

[0064] In step 307, when the data service is abnormal, trace the data quality.

[0065] When the data service is abnormal, based on the time corresponding to the abnormality of the data service, match the detailed record, log record, and data to be accessed corresponding to the abnormal time, and identify the cause of the abnormality based on the detailed record, log record, and data to be accessed.

[0066] It should be noted that the procedures from step 301 to step 304 and the procedures from step 305 to step 306 are two independent procedures, and they may be executed one after another or simultaneously, but are not specifically limited in the present disclosure.

[0067] The present disclosure provides a management mechanism corresponding according to sub - dimensions for the problem of data access quality in the construction of a digital information system, thereby realizing multi - dimensional detection of the data access scenario, standardizing the data access procedure, improving the stability of data access, reducing the investigation and identification cost in the operation and maintenance procedure, and improving the quality of the data service.

[0068] FIG. 4 is a diagram showing the configuration of a data access device according to an embodiment of the present disclosure. As shown in FIG. 4, the data access device 400 includes a first transmission module 401, a load module 402, an analysis module 403, and an access module 404.

[0069] The first transmission module 401 is configured to send a data connection request to the data source system, the data connection request includes connection configuration information, and the connection configuration information is for representing the configuration information necessary to complete the data connection between the data source system and the data connection.

[0070] The load module 402 is configured to load the access target data determined by the data source system in response to the data connection request.

[0071] The analysis module 403 is configured to analyze the access target data that has been successfully loaded and detect the analyzed access target data based on the connection configuration information.

[0072] The access module 404 is configured to access the detected access target data to the target database.

[0073] Furthermore, the connection configuration information includes an IP address corresponding to the data source system, a connection protocol type, a port number of the connection protocol, authentication information, and identification information of the access target data, and the identification information of the access target data includes catalog information, database table information, or theme information.

[0074] The load module 402 includes an establishment sub-module configured to establish a link for data interaction with the data source system based on the IP address corresponding to the data source system, the connection protocol type, and the port number of the connection protocol, an identity authentication sub-module configured to perform identity authentication on the data source system using the link and authentication information, and a load sub-module configured to determine the access target data from the data source system and load the access target data based on the identification information of the access target data when the identity authentication of the data source system passes.

[0075] Furthermore, when the data access device 400 successfully loads the data to be accessed, it is a first processing module configured to record and store access detailed information of the data to be accessed, where the access detailed information includes at least one of a file name list, a file size, and a file generation time; and when the data access device 400 fails to load the data to be accessed, it is a second processing module configured to access the data to be accessed to an abnormal data catalog and record and store abnormal detailed information of the data to be accessed, where the abnormal detailed information includes at least one of a load time, an IP address corresponding to a data source system, a connection protocol type, a port number of a connection protocol, authentication information, and identification information of the data to be accessed.

[0076] Furthermore, in the process of loading the data to be accessed, the data access device 400 is a second transmission module configured to periodically transmit a handshake request to the data source system at preset time intervals, where the handshake request includes connection configuration information; a determination module configured to determine a link state with the data source system and an access state of the data to be accessed based on a response status of the data source system to the handshake request, where the response status includes whether a response message of the data source system is received and whether the response message indicates success in detecting connection configuration information when the response message of the data source system is received; and a recording and storage module configured to record and store the link state and the access state.

[0077] Furthermore, when it is determined that the link state is abnormal, the data access device 400 is a third processing module configured to record and store a detection log, where the detection log is for recording the cause of the link abnormality, and an output module configured to output warning information, where the output method of the warning information includes at least one of display on a user interaction interface, pushing a message to the user's mobile terminal, and sending an email to the user.

[0078] Furthermore, the analysis module 403 includes an analysis sub-module configured to analyze the accessed target data that has been successfully loaded to obtain the field content and field order of the accessed target data, a detection sub-module configured to detect whether the field content and field order satisfy the requirements of the data feature information, an execution sub-module configured to access the detected accessed target data that satisfies the requirements of the data feature information to the target database when it is detected that the field content and field order satisfy the requirements of the data feature information, and an access sub-module configured to access the accessed target data to the abnormal data catalog and record and store the cause of the abnormality of the accessed target data when it is detected that the field content and field order do not satisfy the requirements of the data feature information, where the cause of the abnormality of the accessed target data includes an abnormal field content or an abnormal field order of the accessed target data.

[0079] Furthermore, there are multiple types of data source systems, and the connection protocol types corresponding to different types of data source systems are different.

[0080] When the data access device 400 is executed, it can implement the steps of any one of the data access methods in the foregoing method embodiments and achieve similar technical effects, so the repeated description is omitted here.

[0081] As shown in FIG. 5, an embodiment of the present disclosure includes a processor 511, a communication interface 512, a memory 513, and a communication bus 514. Here, the processor 511, the communication interface 512, and the memory 513 provide an electronic device that completes communication with each other via the communication bus 514. The memory 513 is for storing a computer program.

[0082] In one embodiment of the present disclosure, when the processor 511 executes a program stored in the memory 513, it is for implementing any one of the data access methods of the method embodiments. The steps include: sending a data connection request to a data source system, where the data connection request includes connection configuration information, and the connection configuration information represents the configuration information required to complete a data connection with the data source system; loading the access target data determined by the data source system in response to the data connection request; analyzing the loaded access target data and detecting the analyzed access target data based on the connection configuration information; and accessing the detected access target data to a target database.

[0083] The embodiment of the present disclosure further provides a computer-readable medium storing a computer program that, when executed by a processor, implements the steps of any one of the data access methods of the method embodiments.

[0084] In an embodiment of the present disclosure, a data connection request is sent to a data source system, and the data connection request includes connection configuration information. The connection configuration information represents configuration information necessary to complete a data connection between the data source system and the data connection. The data source system loads the access target data determined in response to the data connection request, analyzes the access target data that has been successfully loaded, detects the analyzed access target data based on the connection configuration information, and accesses the detected access target data that has been successfully detected to a target database.

[0085] In this way, in the process of accessing the access target data to the target database, by detecting the access target data based on the connection configuration information and accessing the detected access target data that has been successfully detected, the quality of the accessed data can be effectively guaranteed, and the operation and maintenance cost can be reduced.

[0086] Note that in this specification, terms representing relationships such as "first" and "second" are used to distinguish one entity or operation from another entity or operation, and do not require or imply the existence of such an actual relationship or order between these entities or operations. Also, "including", "comprising" or any other modification thereof covers non-exclusive inclusion, so that a procedure, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements specific to such a procedure, method, article or device.

[0087] Unless otherwise restricted, elements limited by the phrase "including one..." do not exclude the existence of other similar elements in the procedure, method, article or device including the said elements.

[0088] The above are specific embodiments of the present disclosure, which are only for those skilled in the art to understand or implement the present disclosure. Multiple modifications to these examples will be obvious to those skilled in the art, and the general principle defined in this specification can also be realized in other examples without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments described in this specification, but includes the broadest scope consistent with the principles and novel features disclosed in this specification.

Claims

1. Sending a data connection request to a data source system, wherein the data connection request includes connection configuration information, and the connection configuration information is for representing configuration information necessary to complete a data connection with the data source system; Loading access target data determined by the data source system in response to the data connection request; Analyzing the access target data successfully loaded and detecting the analyzed access target data based on the connection configuration information; Accessing the detected access target data to a target database, comprising: A data access method.

2. The connection configuration information includes an IP address corresponding to the data source system, a connection protocol type, a port number of the connection protocol, authentication information, and identification information of the access target data, and the identification information of the access target data includes catalog information, database table information, or theme information. The step of loading the access target data determined by the data source system in response to the data connection request is: Establishing a link for data interaction with the data source system based on the IP address corresponding to the data source system, the connection protocol type, and the port number of the connection protocol; Performing identity authentication on the data source system by using the link and the authentication information; When the identity authentication of the data source system passes, determining the access target data from the data source system based on the identification information of the access target data and loading the access target data, comprising: The method according to claim 1.

3. When the loading of the data to be accessed is successful, it is a step of recording and storing the access detailed information of the data to be accessed, where the access detailed information includes at least one of a file name list, a file size, and a file generation time. When the loading of the data to be accessed fails, it is a step of accessing the data to be accessed to an abnormal data catalog and recording and storing the abnormal detailed information of the data to be accessed, where the abnormal detailed information includes at least one of a loading time, an IP address corresponding to the data source system, the connection protocol type, a port number of the connection protocol, the authentication information, and identification information of the data to be accessed. The method according to claim 2.

4. In the process of loading the data to be accessed, it is a step of periodically sending a handshake request to the data source system at preset time intervals, where the connection configuration information is included in the handshake request. Based on the response status made by the data source system in response to the handshake request, it is a step of determining the link state between the data source system and the access status of the data to be accessed, where the response status includes whether a response message of the data source system is received, and whether the response message represents a successful detection of the connection configuration information when the response message of the data source system is received. It further includes a step of recording and storing the link state and the access status. The method according to claim 1.

5. When it is determined that the link state is abnormal, it is a step of recording and storing a detection log, where the detection log is for recording the abnormal cause of the link. A step of outputting alarm information, wherein the output method of the alarm information includes at least one of display on a user interaction interface, pushing of a message to the user's mobile terminal, and sending of an email to the user, and further includes: The method according to claim 4.

6. The connection configuration information further includes data feature information for standardizing the field content and field order of the data to be accessed. The step of analyzing the data to be accessed that has been successfully loaded and detecting the analyzed data to be accessed based on the connection configuration information includes: Analyzing the data to be accessed that has been successfully loaded to obtain the field content and field order of the data to be accessed; Detecting whether the field content and the field order satisfy the requirements of the data feature information; When it is detected that the field content and the field order satisfy the requirements of the data feature information, accessing the successfully detected data to be accessed to a target database; When it is detected that the field content and the field order do not satisfy the requirements of the data feature information, accessing the data to be accessed to an abnormal data catalog and recording and storing the cause of the abnormality of the data to be accessed, wherein the cause of the abnormality of the data to be accessed includes an abnormality in the field content or an abnormality in the field order of the data to be accessed, and includes: The method according to claim 2.

7. The types of the data source systems are multiple, and the connection protocol types corresponding to different types of the data source systems are different. The method according to claim 1.

8. A first transmission module configured to send a data connection request to a data source system, wherein connection configuration information is included in the data connection request, and the connection configuration information is for representing configuration information necessary to complete a data connection between the data source system and the data connection. A load module configured to load access target data determined by the data source system in response to the data connection request. An analysis module configured to analyze the access target data that has been successfully loaded and detect the analyzed access target data based on the connection configuration information. An access module configured to access the detected access target data to a target database, including. Data access device.

9. An electronic device including a processor, a communication interface, a memory, and a communication bus, The processor, communication interface, and memory complete mutual communication via the communication bus, The memory is for storing a computer program, When the processor executes the program stored in the memory, it is for realizing the data access method according to any one of claims 1 to 7. Electronic device.

10. A computer-readable storage medium storing a computer program, When the computer program is executed by a processor, it realizes the data access method according to any one of claims 1 to 7. Computer-readable storage medium.

Images