Data access method and device, electronic equipment and storage medium

Abstract

The application relates to a data access method and device, electronic equipment and a storage medium. The method comprises the following steps: sending a data docking request to a data source system, wherein the data docking request carries docking configuration information, the docking configuration information is used for representing configuration information required for completing data docking with the data source system; based on the docking configuration information, loading and detecting to-be-accessed data determined by the data source system in response to the data docking request; periodically sending a handshake request to the data source system to determine the link state between the data source system; and in the case that the link state between the data source system is normal and the to-be-accessed data loading and detection are successful, accessing the to-be-accessed data to a target database. In this way, only the to-be-accessed data with a normal link state between the data source system and successful loading and detection can be accessed, the quality of the accessed data is effectively guaranteed, and the operation and maintenance cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a data access method and device, electronic equipment and storage medium. BACKGROUND

[0002] A data middle platform is a data construction, management and use system including data technology, data governance and data operation, which is built by sinking the data of each business unit in the process of digital transformation of government and enterprises, and has far-reaching significance. In order to ensure the reliability of data assets and data services of the data middle platform, the data accessed from the data source to the data middle platform should meet the following requirements:

[0003] 1) Data integrity: data should be collected without omission;

[0004] 2) Data real-time: the time delay of data reaching the data middle platform does not affect the timeliness requirements of business applications relying on the data;

[0005] 3) Data accuracy: data is not modified during the collection process.

[0006] At present, the control of data quality (such as data integrity, real-time and accuracy) of the data middle platform is usually implemented after data access. This can guarantee the data quality of the data middle platform in the case of stable network state and single data access mode, but in the case of complex network and independent evolution of multiple data source side systems, it cannot effectively guarantee the data quality of the data middle platform, resulting in high operation and maintenance cost. SUMMARY

[0007] The present application provides a data access method, device, electronic equipment and storage medium to solve the technical problem that the existing control of data quality is implemented after data access, which cannot effectively guarantee the data quality of the data middle platform, resulting in high operation and maintenance cost.

[0008] In a first aspect, the present application provides a data access method, which comprises:

[0009] sending a data docking request to a data source system, wherein the data docking request carries docking configuration information, and the docking configuration information is used to represent the configuration information required for completing data docking with the data source system;

[0010] based on the docking configuration information, loading and detecting the data to be accessed determined by the data source system in response to the data docking request;

[0011] periodically sending a handshake request to the data source system to determine the link state between the data source system and the data source system;

[0012] In a case that a link state between the data source system is normal and the to-be-accessed data loading and detection is successful, the to-be-accessed data is accessed to the target database.

[0013] Optionally, the docking configuration information comprises data feature information, and the data feature information is used for specifying field content and field order of the to-be-accessed data.

[0014] The loading and detection of the to-be-accessed data determined by the data source system in response to the data docking request based on the docking configuration information comprises:

[0015] Loading the to-be-accessed data based on the docking configuration information.

[0016] Parsing the to-be-accessed data loaded successfully to obtain field content and field order of the to-be-accessed data.

[0017] Detecting whether the field content and the field order meet requirements of the data feature information.

[0018] In a case that the field content and the field order meet the requirements of the data feature information, determining that the to-be-accessed data loading and detection is successful.

[0019] Optionally, the docking configuration information further comprises an IP address corresponding to the data source system, a docking protocol type, a port number of the docking protocol, authentication information and identification information of the to-be-accessed data, and the identification information of the to-be-accessed data comprises directory information, database table information or topic information.

[0020] The loading of the to-be-accessed data based on the docking configuration information comprises:

[0021] Establishing a link for data interaction with the data source system based on the IP address corresponding to the data source system, the docking protocol type and the port number of the docking protocol.

[0022] Performing identity authentication on the data source system by using the link and the authentication information.

[0023] In a case that the identity authentication on the data source system is passed, determining the to-be-accessed data from the data source system based on the identification information of the to-be-accessed data and loading the to-be-accessed data.

[0024] In the case that the data to be accessed is successfully loaded, access details of the data to be accessed are recorded and stored, the access details including at least one of a file name list, a file size, and a file generation time;

[0025] In the case that the data to be accessed fails to be loaded, the data to be accessed is accessed to an abnormal data directory, and abnormal details of the data to be accessed are recorded and stored, the abnormal details including at least one of a loading time, an IP address corresponding to the data source system, a type of the interfacing protocol, a port number of the interfacing protocol, the authentication information, and identification information of the data to be accessed.

[0026] Optionally, after detecting whether the field content and the field sequence meet the requirements of the data feature information, the method further includes:

[0027] In the case that the field content and the field sequence do not meet the requirements of the data feature information, the data to be accessed is accessed to an abnormal data directory, and an abnormal reason of the data to be accessed is recorded and stored, the abnormal reason of the data to be accessed including field content abnormality or field sequence abnormality of the data to be accessed.

[0028] Optionally, periodically sending a handshake request to the data source system to determine a link state between the data source system includes:

[0029] Periodically sending a handshake request to the data source system every preset time length, the handshake request carrying the interfacing configuration information;

[0030] Determining the link state between the data source system according to a response made by the data source system based on the handshake request, the response including whether a response message of the data source system is received, and whether the response message represents that the interfacing configuration information is detected successfully in the case that the response message of the data source system is received.

[0031] Optionally, after periodically sending the handshake request to the data source system every preset time length, the method further includes:

[0032] Determining an access condition of the data to be accessed according to the response made by the data source system based on the handshake request;

[0033] Recording and storing the link state and the access condition.

[0034] Optionally, after determining the link state between the data source system according to the response made by the data source system based on the handshake request, the method further includes:

[0035] In a case where it is determined that the link state is abnormal, a detection log is recorded and stored, the detection log being used to record an abnormal reason of the link;

[0036] Output alarm information, the output mode of the alarm information including at least one of displaying on a user interactive interface, pushing a message to a mobile terminal of the user, and sending an email to the user.

[0037] Optionally, the data source systems are of multiple types, and different types of the data source systems correspond to different types of docking protocols. In a second aspect, the present application further provides a data access device, the device comprising:

[0038] A first sending module, configured to send a data docking request to a data source system, the data docking request carrying docking configuration information, the docking configuration information being used to represent configuration information required for completing data docking with the data source system;

[0039] A loading and detecting module, configured to load and detect to-be-accessed data determined by the data source system in response to the data docking request;

[0040] A determining module, configured to periodically send a handshake request to the data source system, and determine a link state between the data source system and the device;

[0041] An accessing module, configured to access the to-be-accessed data to a target database in a case where the link state between the data source system and the device is normal, and the to-be-accessed data is successfully loaded and detected.

[0042] In a third aspect, the present application further provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0043] The memory is used to store a computer program;

[0044] The processor is used to execute the program stored on the memory, and realize steps of the data access method according to any one of the first aspect.

[0045] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, the computer program being executed by a processor to realize steps of the data access method according to any one of the first aspect.

[0046] In the embodiment of the present application, a data docking request is sent to a data source system, the data docking request carrying docking configuration information, the docking configuration information being used to represent configuration information required for completing data docking with the data source system; based on the docking configuration information, loaded and detected are to-be-accessed data determined by the data source system in response to the data docking request; a handshake request is periodically sent to the data source system to determine a link state between the data source system; in the case that the link state between the data source system is normal and the loaded and detected to-be-accessed data is successful, the to-be-accessed data is accessed to a target database. In this way, during the process of accessing the to-be-accessed data to the target database, the to-be-accessed data is loaded and detected based on the docking configuration information to ensure the integrity and accuracy of the to-be-accessed data, and at the same time, the handshake request is periodically sent to the data source system to determine the link state between the data source system to ensure the real-time performance of the to-be-accessed data, so that the to-be-accessed data with a normal link state between the data source system and successful loading and detection can be accessed, thereby effectively ensuring the quality of the accessed data and reducing the operation and maintenance cost. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0048] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced hereinafter. Obviously, for those of ordinary skill in the art, the other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A flowchart of a data access method provided by the embodiment of the present application;

[0050] Figure 2 A data access scenario diagram provided by the embodiment of the present application;

[0051] Figure 3 A flowchart of a process of interaction between a digital information system and a data source system for completing data docking provided by the embodiment of the present application;

[0052] Figure 4 A structural diagram of a data access device provided by the embodiment of the present application;

[0053] Figure 5 A structural diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0055] Referring to Figure 1 , Figure 1 A flowchart of a data access method provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the data access method can include the following steps: Figure 1

[0056] Step 101, a data docking request is sent to a data source system, and the data docking request carries docking configuration information, which is used to represent the configuration information required for completing data docking with the data source system.

[0057] Specifically, the docking configuration information is artificially pre-configured information, which can include but is not limited to the IP address corresponding to the data source system, the docking protocol type, the port number of the docking protocol, the authentication information, and the identification information of the data to be accessed, etc. The docking protocol type can include File Transfer Protocol (FTP), Secret File Transfer Protocol (SFTP), Java Database Connectivity (JDBC), message queue, Kafka (an open source stream processing platform developed by Apache Software Foundation), etc. The identification information of the data to be accessed can be directory information, database table information or topic information, etc. The types of the data to be accessed are different, and the corresponding identification information is also different. For example, when the data source of the data source system is an FTP or SFTP type data source, the identification information of the data to be accessed can be directory information; when the data source of the data source system is a Kafka or message queue type data source, the identification information of the data to be accessed can be topic information; and when the data source of the data source system is a JDBC type data source, the identification information of the data to be accessed can be database table information. The data docking request is used to request data docking with the data source system, and the data docking request carries the docking configuration information, based on which the data source system participating in data docking can be quickly located, and a link for data docking with the data source system can be established.

[0058] ​It should be noted that the data access method can be applied to a digital information system (also referred to as a data middle platform), which can be connected with a plurality of data source systems, as shown in Figure 2 The digital information system can realize data connection with the data source system based on an application layer protocol (i.e., a connection protocol) above a network layer, form data assets, and provide data services. The data access process needs to rely on the following conditions:

[0059] (1) The network layer route of the digital information system to the data source system is accessible;

[0060] (2) The application layer (corresponding to the application layer protocol port) is connected;

[0061] (3) The data source system authentication is passed;

[0062] (4) The directory (SFTP / FTP type), topic information (Kafka), database table (JDBC) and other information of the data access configuration are correct.

[0063] Step 102, based on the connection configuration information, loading and detecting the data to be accessed determined by the data source system in response to the data connection request.

[0064] In this step, after receiving the data connection request, the data source system determines the data to be accessed in response to the data connection request, and pushes the data to be accessed to the digital information system. In this way, the digital information system can load the data to be accessed pushed, and after loading successfully, analyze and detect the data to be accessed loaded successfully. Specifically, the digital information system can analyze the data to be accessed loaded successfully to obtain the field content and field order of the data to be accessed, and then can detect the field content and field order of the data to be accessed based on the connection configuration information, such as detecting whether the file content organization of the data file is compliant, whether the field quantity and field order are compliant, detecting whether the table field quantity and order of the database table are compliant, detecting whether the message content (field and order) in the message queue is compliant, etc. In this way, it can be ensured that the data format of the accessed data is stable.

[0065] Step 103, periodically sending a handshake request to the data source system to determine the link state between the digital information system and the data source system.

[0066] The digital information system can also periodically send a handshake request to the data source system every preset time period during the process of loading and detecting the data to be accessed. Thus, according to the response of the data source system based on the handshake request, the link state between the digital information system and the data source system in each period can be determined.

[0067] It should be noted that the digital information system can independently initiate a periodic handshake request to each data source system, and the period corresponding to each data source system can be independently set. Specifically, the handshake period can be set according to the link state of each data source system. For example, for a data source system with a relatively stable link, the period can be set to be relatively long to reduce network resource consumption; for a data source system with a relatively unstable link, the period can be set to be relatively short to increase the granularity of link state monitoring.

[0068] Step 104: In the case that the link state between the data source system is normal and the data to be accessed is successfully loaded and detected, the data to be accessed is accessed to the target database.

[0069] Specifically, the target database refers to a database or data warehouse in the digital information system for storing normally accessed data. In this step, the digital information system can access the data to be accessed to the target database when the link state between the data source system is normal and the data to be accessed is successfully loaded and detected, and form data assets and provide data services based on the accessed data in the target database. When accessing the target database, the time stamp of entering the warehouse can also be recorded to calculate the delay of entering the warehouse of the data to be accessed based on the time stamp of entering the warehouse.

[0070] In this embodiment, the data to be accessed can be loaded and detected based on the docking configuration information during the process of accessing the data to be accessed to the target database, to ensure the integrity and accuracy of the data to be accessed, and the link state between the data source system is determined by periodically sending a handshake request to the data source system to ensure the real-time performance of the data to be accessed, so that the data to be accessed with a normal link state between the data source system and successful loading and detection can be accessed, thereby effectively ensuring the quality of the accessed data and reducing the operation and maintenance cost.

[0071] Further, the docking configuration information includes data feature information, and the data feature information is used to standardize the field content and field order of the data to be accessed;

[0072] The above step 102 of loading and detecting the data to be accessed determined by the data source system in response to the data docking request based on the docking configuration information includes:

[0073] Loading the data to be accessed based on the docking configuration information;

[0074] The loaded data to be accessed is parsed to obtain the field content and field order of the data to be accessed;

[0075] Detecting whether the field content and field order meet the requirements of the data feature information;

[0076] In the case that the field content and the field order meet the requirements of the data feature information, it is determined that the data loading and detection are successful.

[0077] Specifically, the docking configuration information can specify the data content of the data to be accessed, such as the field content and the field order of the data, so as to ensure the stability of the data format (i.e., the data source version) of the accessed data. Even if the data source system is upgraded or changed, the data format of the data to be accessed in the digital information system will not be modified.

[0078] In this step, the digital information system can parse the successfully loaded data to be accessed, so as to obtain the field content and the field order of the data to be accessed. Then, based on the docking configuration information, the field content and the field order of the data to be accessed can be detected, such as whether the file content organization of the data file is compliant, whether the field quantity and the field order are compliant, whether the table field quantity and the order of the database table are compliant, whether the message content (field and order) in the message queue is compliant, and the like. In this way, it can be ensured that the data format of the accessed data is stable.

[0079] Further, the docking configuration information further includes an IP address corresponding to the data source system, a docking protocol type, a port number of the docking protocol, authentication information, and identification information of the data to be accessed, the identification information of the data to be accessed including directory information, database table information, or topic information.

[0080] The above steps of loading the data to be accessed based on the docking configuration information include:

[0081] Based on the IP address corresponding to the data source system, the docking protocol type, and the port number of the docking protocol, a link for data interaction with the data source system is established.

[0082] The data source system is authenticated based on the link and the authentication information.

[0083] In the case that the data source system is authenticated, the data to be accessed is determined from the data source system based on the identification information of the data to be accessed, and the data to be accessed is loaded.

[0084] In an embodiment, the docking configuration information can further include an IP address corresponding to the data source system, a docking protocol type, a port number of the docking protocol, authentication information, and identification information of the data to be accessed. The docking protocol type can include a File Transfer Protocol (FTP), a Secret File Transfer Protocol (SFTP), a Java Database Connectivity (JDBC), a message queue, Kafka (an open source stream processing platform developed by the Apache Software Foundation), and the like. The identification information of the data to be accessed can be directory information, database table information, or topic information, and the like. Different types of data to be accessed correspond to different identification information. For example, when the data source of the data source system is an FTP or SFTP type data source, the identification information of the data to be accessed can be directory information; when the data source of the data source system is a Kafka or message queue type data source, the identification information of the data to be accessed can be topic information; and when the data source of the data source system is a JDBC type data source, the identification information of the data to be accessed can be database table information.

[0085] The process of loading the data to be accessed by the digital information system can include chain building, authentication, downloading / receiving the data to be accessed, and the like. Specifically, in the chain building process, the digital information system and the data source system need to establish a link for data interaction based on the IP address corresponding to the data source system, the docking protocol type, and the port number of the docking protocol. In the authentication process, the digital information system and the data source system need to perform identity authentication on the data source system based on the established link and the authentication information. In the downloading / receiving the data to be accessed process, the data to be accessed can be determined from the data source system based on the identification information of the data to be accessed, and the data to be accessed can be downloaded / received and loaded.

[0086] In the loading process, the data to be accessed can be detected based on the preset docking configuration information, such as whether the data to be accessed is reachable, whether the authentication is passed, whether the directory / database table name / topic is correct, and the like. If all the detections are passed, it indicates that the data to be accessed can be successfully loaded, and the subsequent steps are continued to be executed; if the detection is not passed, it indicates that the data to be accessed cannot be successfully loaded, and the data to be accessed needs to be stored in an exception data directory, and exception detail information is recorded and stored, which facilitates subsequent exception positioning and cause analysis.

[0087] In the embodiment, the connection establishment, authentication, download / receiving of the to-be-accessed data and other links are completed based on the docking configuration information in the digital information system, so that the detection of the to-be-accessed data is realized in the data loading process, and the quality of the accessed data is improved.

[0088] Further, after the to-be-accessed data is loaded based on the docking configuration information in the step 102, the method further includes:

[0089] In the case that the to-be-accessed data is loaded successfully, the access detail information of the to-be-accessed data is recorded and stored, and the access detail information includes at least one of the file name list, the file size and the file generation time.

[0090] In the case that the to-be-accessed data is loaded successfully, the access detail information of the to-be-accessed data is recorded and stored, and the access detail information includes at least one of the file name list, the file size and the file generation time.

[0091] In an embodiment, in the case that the to-be-accessed data is loaded successfully, the access detail information of the to-be-accessed data loaded successfully can be recorded and stored. The access detail information herein can be different according to different data types of the to-be-accessed data. For example, when the data type of the to-be-accessed data is a data file, a message queue or Kafka, the data access detail can include one or more of the file name list, the file size and the file generation time; when the data type of the to-be-accessed data is a database table, the data access detail can include the acquisition time of the group of records. By recording and storing the data access detail, a detail snapshot of the data access situation can be obtained, and when the source layer data file is deleted by the life cycle management logic in the digital information system, the related problems can be traced back based on the historical detail snapshot.

[0092] In the case that the to-be-accessed data is loaded successfully, the access detail information of the to-be-accessed data loaded successfully can be recorded and stored. The access detail information herein can be different according to different data types of the to-be-accessed data. For example, when the data type of the to-be-accessed data is a data file, a message queue or Kafka, the data access detail can include one or more of the file name list, the file size and the file generation time; when the data type of the to-be-accessed data is a database table, the data access detail can include the acquisition time of the group of records. By recording and storing the data access detail, a detail snapshot of the data access situation can be obtained, and when the source layer data file is deleted by the life cycle management logic in the digital information system, the related problems can be traced back based on the historical detail snapshot.

[0093] Further, after the to-be-accessed data is loaded based on the docking configuration information in the step 102, the method further includes:

[0094] In a case where it is detected that the field content and the field sequence do not meet the requirements of the data feature information, the to-be-accessed data is accessed to an abnormal data directory, and an abnormal reason of the to-be-accessed data is recorded and stored, the abnormal reason of the to-be-accessed data including a field content abnormality or a field sequence abnormality of the to-be-accessed data.

[0095] In an embodiment, in a case where it is detected that the field content and the field sequence do not meet the requirements of the data feature information, the to-be-accessed data is accessed to an abnormal data directory, and an abnormal reason of the to-be-accessed data is recorded and stored, by recording and storing the data information that does not meet the requirements of the data feature information, the abnormal reason can be quickly located, and the operation and maintenance cost is reduced.

[0096] Further, a handshake request is periodically sent to the data source system to determine a link state between the data source system, including:

[0097] The handshake request is periodically sent to the data source system every preset time length, and the handshake request carries the docking configuration information;

[0098] According to a response of the data source system based on the handshake request, a link state between the data source system is determined, and the response includes whether the response message of the data source system is received, and whether the response message represents that the docking configuration information is detected successfully in a case where the response message of the data source system is received.

[0099] In an embodiment, the digital information system can periodically send a handshake request to the data source system every preset time length, so that the link state between the digital information system and the data source system in each period can be determined according to the response of the data source system based on the handshake request, thereby ensuring the real-time performance of the to-be-accessed data.

[0100] It should be noted that the digital information system can independently initiate a periodic handshake request to each data source system, and the period corresponding to each data source system can be independently set. Specifically, the handshake period can be set according to the link state of each data source system. For example, for a data source system with a relatively stable link, the period can be set to be relatively long to reduce network resource consumption; for a data source system with a relatively unstable link, the period can be set to be relatively short to increase the granularity of link state monitoring.

[0101] Further, after periodically sending the handshake request to the data source system every preset time length, the method further includes:

[0102] According to a response of the data source system based on the handshake request, an access state of the to-be-accessed data is determined;

[0103] The link state and the access state are recorded and stored.

[0104] In an embodiment, the digital information system can further determine the access condition of the data to be accessed according to the response made by the data source system, record and store the link state and the access condition, and thus record the link state and the access condition in the entire time dimension. The link state herein includes link disconnection / link connection, port unreachable / port reachable, authentication success / authentication rejection, etc., and the access condition includes file directory nonexistence / file directory existence, database table nonexistence / database table existence, theme consistency / theme inconsistency, etc.

[0105] Further, after determining the link state with the data source system according to the response made by the data source system based on the handshake request, the method further includes:

[0106] In the case of determining the abnormal link state, the detection log is recorded and stored, and the detection log is used to record the abnormal reason of the link;

[0107] The alarm information is outputted, and the output mode of the alarm information includes at least one of displaying on the user interactive interface, pushing a message to the mobile terminal of the user, and sending an email to the user.

[0108] In an embodiment, the digital information system can record and store the detection log when determining the abnormal link state. The detection log is used to record the abnormal reason of the link, such as unreachable application layer protocol, authentication information error, and interface directory or library table information error, etc. The digital information system can also output the alarm information, and the output mode of the alarm information can be the display mode on the user interactive interface, the message pushing mode to the mobile terminal of the user, the email sending mode to the user, etc. In this way, the abnormality can be quickly found and located when the data access is abnormal.

[0109] Further, the types of the data source systems are multiple, and the interface protocol types corresponding to the data source systems of different types are different.

[0110] In an embodiment, the data source systems can include multiple types, such as FTP / SFTP data source, JDBC data source, message queue data source, Kafka data source, etc. The interface protocol types corresponding to the data source systems of different types are different, such as the interface protocol type of the FTP data source is FTP, the interface protocol type of the SFTP data source is SFTP, the interface protocol type of the JDBC data source is JDBC, etc. The digital information system in this embodiment can be compatible with multiple interface protocol types, thereby improving the stability of the link.

[0111] In an embodiment, the process of the digital information system interacting with the data source system to complete the data interface includes the following steps as shown in Figure 3 ​

[0112] Step 301, the digital information system initiates a data access request to the data source system based on the docking configuration information;

[0113] The digital information system needs to complete the configuration of the docking configuration information in advance; wherein the docking configuration information includes the IP address corresponding to the data source system, the docking protocol type, the port number of the docking protocol, the authentication information, the directory information, the database table information, the topic information, the data feature information, etc.

[0114] Step 302, the data source system completes the chain building and authentication process with the digital information system based on the data access request, and downloads / receives the data to be accessed;

[0115] Step 303, the digital information system loads and detects the data to be accessed;

[0116] Write details such as abnormal details and access details into the specified database, and report alarm information when continuous handshake fails or docking configuration information verification fails.

[0117] Step 304, the digital information system parses the data to be accessed, and detects and stores the parsed data to be accessed;

[0118] The loaded data to be accessed is parsed to obtain the field content and field order of the data to be accessed; whether the field content and field order meet the requirements of the data feature information is detected; in the case where the field content and field order meet the requirements of the data feature information, the detected data to be accessed is accessed to the target database; in the case where the field content and field order do not meet the requirements of the data feature information, the data to be accessed is accessed to the abnormal data directory, and the abnormal reason of the data to be accessed is recorded and stored, the abnormal reason of the data to be accessed including field content abnormality or field order abnormality of the data to be accessed.

[0119] Step 305, the digital information system initiates a periodic handshake request to the data source system based on the docking configuration information;

[0120] Step 306, the data source system periodically monitors the link state and data access based on the handshake request;

[0121] When it is determined that the data to be accessed is unavailable, the detection log is stored and alarm information is initiated. The handshake and verification content includes but is not limited to the following scenarios: application layer protocol unreachable, authentication information error, docking directory or database table information error. The alarm mode can be displayed on the user interaction interface, push message to the user's mobile terminal, send email to the user, etc.

[0122] Step 307, trace the data quality when the data service is abnormal;

[0123] When the data service is abnormal, the detail record, the log record and the to-be-accessed data corresponding to the abnormal time are matched according to the time corresponding to the data service abnormality, and the abnormal reason is located based on the detail record, the log record and the to-be-accessed data.

[0124] It should be noted that the processes of steps 301 to 304 and the processes of steps 305 to 306 are two independent processes, which can be executed in sequence or simultaneously, and the application does not make specific limitations.

[0125] The application provides a corresponding management mechanism according to the subdivision dimension, realizes multi-dimensional detection of a data access scene, standardizes a data access process, improves data access stability, reduces troubleshooting and positioning costs in an operation and maintenance process, and improves data service quality.

[0126] Referring to Figure 4 , Figure 4 A structural schematic diagram of a data access device provided by an embodiment of the application is shown in FIG. 4. Figure 4 As shown in FIG. 4, the data access device 400 includes:

[0127] A first sending module 401 is configured to send a data docking request to a data source system, wherein the data docking request carries docking configuration information, and the docking configuration information is used to represent configuration information required for completing data docking with the data source system;

[0128] A loading and detecting module 402 is configured to load and detect to-be-accessed data determined by the data source system in response to the data docking request;

[0129] A determining module 403 is configured to periodically send a handshake request to the data source system and determine a link state between the data source system and the data access device;

[0130] An accessing module 404 is configured to access the to-be-accessed data to a target database when the link state between the data source system and the data access device is normal and the to-be-accessed data is successfully loaded and detected.

[0131] Further, the docking configuration information includes data feature information, and the data feature information is used to standardize field content and field order of the to-be-accessed data;

[0132] The loading and detecting module 402 includes:

[0133] A loading submodule is configured to load the to-be-accessed data based on the docking configuration information;

[0134] An analyzing submodule is configured to analyze the to-be-accessed data loaded successfully to obtain the field content and the field order of the to-be-accessed data;

[0135] detecting whether the field content and the field sequence meet the requirement of the data characteristic information;

[0136] The first determining sub-module is configured to determine that the data loading and detection are successful if it is detected that the field content and the field sequence meet the requirement of the data characteristic information.

[0137] Further, the interfacing configuration information further includes an IP address corresponding to the data source system, a type of interfacing protocol, a port number of the interfacing protocol, authentication information, and identification information of the data to be accessed, the identification information of the data to be accessed including directory information, database table information, or topic information.

[0138] The loading sub-module includes:

[0139] The establishing unit is configured to establish a link for data interaction with the data source system based on the IP address corresponding to the data source system, the type of interfacing protocol, and the port number of the interfacing protocol.

[0140] The identity authentication unit is configured to perform identity authentication on the data source system by using the link and the authentication information.

[0141] The loading unit is configured to determine the data to be accessed from the data source system based on the identification information of the data to be accessed and load the data to be accessed if the identity authentication on the data source system is passed.

[0142] Further, the loading and detection module 402 further includes:

[0143] The first recording and storing sub-module is configured to record and store access detail information of the data to be accessed if the data to be accessed is successfully loaded, the access detail information including at least one of a file name list, a file size, and a file generation time.

[0144] The second recording and storing sub-module is configured to access the data to be accessed to an abnormal data directory and record and store abnormal detail information of the data to be accessed if the data to be accessed fails to be loaded, the abnormal detail information including at least one of a loading time, an IP address corresponding to the data source system, a type of interfacing protocol, a port number of the interfacing protocol, authentication information, and identification information of the data to be accessed.

[0145] Further, the loading and detection module 402 further includes:

[0146] The third recording and storing sub-module is configured to access the data to be accessed to an abnormal data directory and record and store an abnormal reason of the data to be accessed if it is detected that the field content and the field sequence do not meet the requirement of the data characteristic information, the abnormal reason of the data to be accessed including field content abnormality or field sequence abnormality of the data to be accessed.

[0147] Further, the determining module 403 comprises:

[0148] a sending sub-module, configured to periodically send a handshake request to the data source system every preset time length, the handshake request carrying the docking configuration information;

[0149] a second determining sub-module, configured to determine a link state between the data source system according to a response made by the data source system based on the handshake request, the response including whether an acknowledgement message of the data source system is received, and whether the acknowledgement message represents that the docking configuration information is detected successfully in the case that the acknowledgement message of the data source system is received.

[0150] Further, the determining module 403 further comprises:

[0151] a third determining sub-module, configured to determine an access condition of the data to be accessed according to the response made by the data source system based on the handshake request;

[0152] a fourth recording and storing sub-module, configured to record and store the link state and the access condition.

[0153] Further, the determining module 403 further comprises:

[0154] a fifth recording and storing sub-module, configured to record and store a detection log in the case that the link state is determined to be abnormal, the detection log being used to record an abnormal reason of the link;

[0155] an outputting sub-module, configured to output an alarm information, the outputting mode of the alarm information including at least one of displaying on a user interactive interface, pushing a message to a mobile terminal of the user, and sending an email to the user.

[0156] Further, the types of the data source systems are multiple, and the docking protocol types corresponding to different types of the data source systems are different.

[0157] It should be noted that the data access apparatus 400 can execute the steps of the data access method provided by any one of the foregoing method embodiments, and achieve the same technical effects, and thus will not be described here in detail.

[0158] As shown in Figure 5 the embodiment of the present application provides an electronic device, which comprises a processor 511, a communication interface 512, a memory 513 and a communication bus 514, wherein the processor 511, the communication interface 512 and the memory 513 complete mutual communication through the communication bus 514,

[0159] the memory 513 is used for storing a computer program;

[0160] In an embodiment of the present application, the processor 511, when executing the program stored in the memory 513, implements the data access method provided by any one of the foregoing method embodiments, including:

[0161] sending a data docking request to the data source system, the data docking request carrying docking configuration information, the docking configuration information being used to represent configuration information required for completing data docking with the data source system;

[0162] based on the docking configuration information, loading and detecting the to-be-accessed data determined by the data source system in response to the data docking request;

[0163] periodically sending a handshake request to the data source system to determine the link state between the data source system and the data source system;

[0164] in the case that the link state between the data source system and the data source system is normal and the loading and detection of the to-be-accessed data are successful, accessing the to-be-accessed data to the target database.

[0165] The embodiment of the present application further provides a computer readable storage medium, which has a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the data access method provided by any one of the foregoing method embodiments.

[0166] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0167] The above description is merely one specific implementation of the present application, which enables a person skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features set forth herein.

Claims

1. A data access method, characterized by, The method comprises: sending a data docking request to a data source system, the data docking request carrying docking configuration information, the docking configuration information comprising: an IP address corresponding to the data source system, a docking protocol type, a port number of the docking protocol, authentication information, and identification information of data to be accessed; based on the IP address corresponding to the data source system, the docking protocol type, and the port number of the docking protocol, establishing a link for data interaction with the data source system; using the link and the authentication information to perform identity authentication on the data source system; in the case where the identity authentication of the data source system is passed, determining the data to be accessed from the data source system based on the identification information of the data to be accessed, and loading the data to be accessed; parsing the data to be accessed loaded successfully, and detecting the parsed data to be accessed based on the docking configuration information; accessing the data to be accessed detected successfully to a target database.

2. The method of claim 1, wherein, The identification information of the data to be accessed comprises directory information, database table information, or topic information.

3. The method of claim 1, wherein, The method further comprises: in the case where the data to be accessed is loaded successfully, recording and storing access detail information of the data to be accessed, the access detail information comprising at least one of a file name list, a file size, and a file generation time; in the case where the data to be accessed fails to be loaded, accessing the data to be accessed to an abnormal data directory, and recording and storing abnormal detail information of the data to be accessed, the abnormal detail information comprising at least one of a loading time, the IP address corresponding to the data source system, the docking protocol type, the docking protocol port number, the authentication information, and the identification information of the data to be accessed.

4. The method of claim 1, wherein, The method further comprises: in the process of loading the data to be accessed, periodically sending a handshake request to the data source system every preset time length, the handshake request carrying the docking configuration information; determining a link state between the data source system and an access state of the data to be accessed according to a response of the data source system based on the handshake request, the response comprising whether a response message of the data source system is received, and whether the response message represents that the docking configuration information is detected successfully in the case where the response message of the data source system is received; recording and storing the link state and the access state.

5. The method of claim 4, wherein, The method further comprises: in the case where the link state is determined to be abnormal, recording and storing a detection log, the detection log being used to record an abnormal reason of the link; outputting alarm information, the output mode of the alarm information comprising at least one of displaying on a user interaction interface, pushing a message to a mobile terminal of a user, and sending an email to a user.

6. The method of claim 1, wherein, The docking configuration information further comprises data feature information, the data feature information being used to standardize field content and field order of the data to be accessed; the method of parsing the data to be accessed loaded successfully, and detecting the parsed data to be accessed based on the docking configuration information, comprises: parsing the to-be-accessed data successfully loaded, to obtain field content and field order of the to-be-accessed data; detecting whether the field content and the field order meet requirements of the data feature information; in a case where it is detected that the field content and the field order meet the requirements of the data feature information, performing a step of accessing the to-be-accessed data successfully detected to a target database; in a case where it is detected that the field content and the field order do not meet the requirements of the data feature information, accessing the to-be-accessed data to an abnormal data directory, and recording and storing an abnormal reason of the to-be-accessed data, the abnormal reason of the to-be-accessed data including field content abnormality or field order abnormality of the to-be-accessed data.

7. The method of claim 1, wherein, The types of the data source systems are multiple, and the types of the data source systems of different types correspond to different types of docking protocols.

8. A data access device, characterized by The apparatus comprises: a first sending module configured to send a data docking request to a data source system, the data docking request carrying docking configuration information, the docking configuration information including an IP address corresponding to the data source system, a docking protocol type, a port number of the docking protocol, authentication information, and identification information of to-be-accessed data; a loading module configured to establish a link for data interaction with the data source system based on the IP address corresponding to the data source system, the docking protocol type, and the port number of the docking protocol; perform identity authentication on the data source system by using the link and the authentication information; in a case where the identity authentication on the data source system is passed, determine the to-be-accessed data from the data source system based on the identification information of the to-be-accessed data, and load the to-be-accessed data; a parsing module configured to parse the to-be-accessed data successfully loaded, and detect the to-be-accessed data parsed based on the docking configuration information; an accessing module configured to access the to-be-accessed data successfully detected to a target database.

9. An electronic device, comprising: The apparatus comprises a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication by the communication bus; the memory is configured to store a computer program; the processor is configured to execute the program stored on the memory, to implement steps of the data access method in any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement steps of the data access method in any one of claims 1-7.

Images