External data access platform and external data access method

Abstract

The invention discloses an external data access platform and an external data access method, and relates to the technical field of information transmission, and the external data access platform comprises an external docking module which is used for packaging target external data sent by an external data source and generating a data access file; the middleware module is used for forwarding the data access file to the data storage module; the data storage module is used for receiving, processing and storing a data access file; the calling statistics module is used for generating log records and storing the log records into a preset log database according to burying point data obtained by deploying a monitoring and log recording mechanism on each preset key point; and the data quality analysis module is used for performing data quality analysis on the data access file according to the log record and a preset data analysis strategy to generate a data quality analysis result. According to the invention, the external data access process can be standardized, the access efficiency is greatly improved, and the compliance and necessity of data external query are ensured.

Description

Technical Field

[0001] The present application relates to the field of information transmission technology, and in particular to an external data access platform and an external data access method. Background Art

[0002] In the context of the rapid development of information technology, enterprises are facing increasing demands for external data access management. These demands cover multiple aspects such as technological innovation, strategy optimization, and sharing and cooperation, aiming to improve data management efficiency and maximize the value of external access data. However, the existing external data access and management platforms have many defects, such as the difficulty of centralized access support for various data access modes through a unified platform, the complex data access development process, the failure of traditional coding methods to meet standardization requirements, the inability to establish effective external authorization monitoring and verification, and the proneness to unauthorized retrieval, triggering compliance risks, etc., which limits the efficiency of data management and the realization of data value. Summary of the invention

[0003] The main purpose of this application is to provide an external data access platform and an external data access method, aiming to solve the problem that traditional external data access and management platforms limit data management efficiency and the realization of data value.

[0004] To achieve the above-mentioned purpose, the present application proposes an external data access platform, wherein the external docking module is used to encapsulate the target external data sent by the external data source and generate a data access file;

[0005] The calling statistics module is used to generate log records corresponding to the data access file according to the buried point data obtained by deploying the monitoring and log recording mechanism at each preset key point, and store the log records in a preset log database;

[0006] The data quality analysis module is used to perform data quality analysis on the data access file according to the log record and the preset data analysis strategy, and generate a data quality analysis result;

[0007] The middleware module is used to forward the data access file and the data quality analysis result to the data storage module to achieve external data access.

[0008] In one embodiment, the external data access platform also includes a low-code access configuration module, and the low-code access configuration module is used to support the target user to complete data access configuration through the low-code configuration mode.

[0009] In one embodiment, the external data access platform further includes a service guarantee module, and the service guarantee module is used to ensure that the data access file can be stably transmitted between modules.

[0010] In one embodiment, the external connection module is further used to perform external data query on the external data source, and analyze and process the response file returned by the external data source.

[0011] In one embodiment, the data storage module is used to receive, process and store data access files and data quality analysis results.

[0012] The present application proposes an external data access method, the method comprising:

[0013] Encapsulate the target external data sent by the external data source and generate a data access file;

[0014] Generate log records corresponding to the data access file based on the embedded point data obtained by deploying monitoring and log recording mechanisms at each preset key point, and store the log records in a preset log database;

[0015] According to the log records and the preset data analysis strategy, a data quality analysis is performed on the data access file to generate a data quality analysis result;

[0016] The data access file and the data quality analysis result are stored in a local database to achieve external data access.

[0017] In one embodiment, the method further comprises:

[0018] Get the data access type corresponding to the target user;

[0019] If the data access type is interface type docking, obtaining a request field parameter design and an interface parameter design, and determining a first data access component and first workflow information based on the request field parameter design and the interface parameter design;

[0020] If the data access type is data file docking, the source file information is obtained, and the data processing flow is obtained through a preset canvas tool;

[0021] Based on the source file information and the data processing flow, a second data access component and second workflow information are determined.

[0022] In one embodiment, the method further comprises:

[0023] Obtain the data importance and data volume of each data item in the data access file;

[0024] Based on the importance of the data and the amount of data, each of the data items is split into several distributed clusters for data transmission;

[0025] Monitor key indicator values ​​of thread pool in data transmission;

[0026] Based on the key indicator values ​​and the preset current limiting strategy, the data transmission processing capability is balanced to ensure stable data transmission.

[0027] In one embodiment, the method further comprises:

[0028] Obtaining a first data external query request, parsing the first data external query request and extracting parameters to obtain a request permission parameter;

[0029] If the request permission parameter passes the permission check, the first data external query request is encrypted, and based on the first data external query request, data in the preset relational database is deduplicated to obtain a data deduplication result to avoid repeated queries;

[0030] If the data deduplication result is a non-duplicate request, sending the encrypted first data external query request to the external data source;

[0031] Decrypting and parsing the external query encrypted result message returned by the external data source to generate a data external query result;

[0032] The data integrity, data structure, data file field validity, and data consistency of the data external search results are checked to complete the data external search.

[0033] In one embodiment, the method further comprises:

[0034] Obtaining a second external data query request, and initiating a query on the external data source based on a data file uniform resource locator URL in the second external data query request;

[0035] Receiving an external query response file returned by the external data source;

[0036] If the external inquiry response file is sent back in the secure file transfer protocol SFTP mode, a scheduled task is set to perform the data transmission and verification task of the external inquiry response file to ensure the integrity and consistency of the data;

[0037] If the external inquiry response file is transmitted back in the form of a data stream, data reception and data processing are performed simultaneously, and the external inquiry response file is processed and verified and then stored in a message queue and a preset relational database.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes an external data access device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the external data access method described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the external data access method described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the external data access method described above are implemented.

[0041] The present application provides an external data access platform and an external data access method, wherein the external data access platform includes: an external docking module, which is used to encapsulate the target external data sent by the external data source and generate a data access file; a middleware module, which is used to forward the data access file to the data storage module; a data storage module, which is used to receive, process and store the data access file; a calling statistics module, which is used to generate log records and store the log records in a preset log database according to the buried point data obtained by deploying a monitoring and log recording mechanism at each preset key point; a data quality analysis module, which is used to perform data quality analysis on the data access file according to the log record and the preset data analysis strategy, and generate data quality analysis results, so as to standardize the external data access process, greatly improve the access efficiency, and ensure the compliance and necessity of external data inquiry. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of the platform architecture provided for the first embodiment of the external data access platform of this application;

[0045] Figure 2 A schematic diagram of a data processing flow for an interface-type docking mode provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a data processing flow for a data file docking mode provided in an embodiment of the present application;

[0047] Figure 4A schematic diagram of a data processing flow for authorization verification provided in an embodiment of the present application;

[0048] Figure 5 A flowchart of the first embodiment of the external data access method of the present application is provided;

[0049] Figure 6 An example diagram of the platform architecture provided for the external data access platform of this application;

[0050] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the external data access method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a big data service platform, an external data access system, etc. The following takes the external data access system as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the embodiment of the present application provides an external data access platform, referring to Figure 1 , Figure 1 A schematic diagram of the platform architecture provided for the first embodiment of the external data access platform of this application.

[0056] In this embodiment, Figure 1 As shown, an embodiment of the present invention provides a data access platform, which includes: an external docking module 11, a middleware module 12, a data storage module 13, a call statistics module 14 and a data quality analysis module 15, wherein the external docking module 11 is used to encapsulate the target external data sent by the external data source and generate a data access file;

[0057] It should be noted that the external docking module 11 is used to encapsulate the target external data sent by the external data source and generate a data access file. The encapsulation of the target external data sent by the external data source includes request parameter parsing, request message encryption, data deduplication, authorization verification, result message decryption, message parsing, result data processing and integration, file verification and other services, as shown in Table 1. Table 1 is a service description corresponding to the functions included in the external docking module.

[0058] In one embodiment, the external docking module 11 supports API interface docking and data file docking. The data processing procedures of the two docking modes are different. Figure 2 as well as Figure 3 , Figure 2 A schematic diagram of a data processing flow for an interface-type docking mode provided in an embodiment of the present application; Figure 3 A schematic diagram of a data processing flow for a data file docking mode provided in accordance with an embodiment of the present application.

[0059] Specifically, for the authorization verification service, in the authorization verification process within the bank, the channel-side application first initiates an authorization verification request, which is sent to the bank's authorization verification system, where the bank's authorization verification system is responsible for processing the authorization verification and interacting with the electronic signature verification center to verify the legitimacy of the electronic signature. Once the authorization verification is completed, the bank's authorization verification system will synchronize the results of the authorization verification to the authorization record database, which is a relational database used to store all authorization information. The business processing application will then perform external data queries as needed, that is, on-demand external queries. Before conducting an external query, the business processing application will submit the request to the external docking module. The external docking module is responsible for communicating with external data suppliers, but before initiating an external query, it will query the authorization record database to confirm whether there is valid authorization information.

[0060] Furthermore, if the authorization record database confirms the authorization information, the external docking module will initiate a data query request to the external data supplier based on this information. After the authorization is completed, the external data supplier will return the required data to complete the entire external query process, thereby ensuring the compliance of data access while improving the security and efficiency of data access. In this way, the system can control and monitor access to external data while ensuring that all operations are performed within the scope of authorization. For reference, Figure 4 , Figure 4 A schematic diagram of a data processing flow for authorization verification provided in accordance with an embodiment of the present application.

[0061] Table 1

[0062]

[0063]

[0064] The calling statistics module 14 is used to generate log records corresponding to the data access file according to the embedded point data obtained by deploying the monitoring and log recording mechanism at each preset key point, and store the log records in a preset log database;

[0065] It should be noted that the call statistics module 14 relies on the deployment of monitoring and logging mechanisms at various key points on the platform. In view of the high timeliness requirements of call statistical data, a time series database TSDB (TimeSeries Database) is used to achieve high-performance reading, writing, storage and real-time analysis of buried point data, generate log records corresponding to the data access files, and store the log records in the preset log database.

[0066] It should be further explained that, since the platform has packaged the external query logic into an API interface before initiating the external query, each external query interface is defined as a data item and assigned an ID, and each call to the business system is also assigned an ID. The business system generates a log record for each external query initiated and stores it in the database. The call statistics module can aggregate various statistical indicators based on the data item ID or the business system ID. The statistical indicators can be referred to in Table 2, where the data is refreshed every 10 seconds and the statistical frequency is every minute to every day. There is no restriction here and it can be set according to actual conditions.

[0067] Furthermore, the indicator statistical results are displayed through the Web page after the conditional query is set by the backend. The specific display logic can be customized by the user and is not restricted here. Therefore, by calling the statistical module, the external data external query system performance monitoring can be realized to measure whether the external query response time and throughput meet the business needs, and statistical charts and exception code information can be combined to quickly locate abnormal problems and guide the direction of troubleshooting. The monitoring and alarm function can also be realized in the platform through statistics. When the abnormal rate exceeds the threshold, the development and operation personnel are quickly reminded to handle the fault to avoid delays caused by the fault.

[0068] Table 2

[0069]

[0070] The data quality analysis module 15 is used to perform data quality analysis on the data access file according to the log record and the preset data analysis strategy, and generate a data quality analysis result;

[0071] It should be noted that the data quality analysis module 15 supports the target user to configure the key field statistical strategy through the Web interface, which can be referred to in Table 3, and then the strategy configuration result is transmitted to the database configuration table, and different data is buried according to the configuration strategy. When the subsequent external query result message is returned, the stream calculation is realized through the streaming database, and the message is searched through a regular expression, and different indicators are counted. The statistical results are consumed and stored. The statistical frequency supports every minute to every day, and the conditional query is initiated through the backend through the Web page and then displayed. At the same time, the data quality analysis module 15 also supports monitoring and early warning functions. The backend can make real-time judgments and early warnings based on consumption data. If the threshold is triggered, the user can be notified. Therefore, the data quality analysis module can meet the user's needs for external query data quality reliability analysis, and provide analysis result information through a visual interface, which is simple and efficient. In addition, data quality analysis helps to eliminate dirty data and outliers, ensure that business decisions are based on real and reliable information, and improve customer satisfaction.

[0072] Table 3

[0073]

[0074] The middleware module 12 is used to forward the data access file and the data quality analysis result to the data storage module 13 to achieve external data access.

[0075] It should be noted that the middleware module 12 is divided into cache middleware and message middleware (message queue), wherein the cache middleware is used to cache the response message results (i.e., data access files) of the query processing to relieve database pressure; the message middleware is used for asynchronous decoupling of client query applications to avoid process blocking.

[0076] In one embodiment, the middleware module mainly includes two types of message queue systems, Apache Kafka and MQ (Message Queue). KAFKA (Apache Kafka is a distributed stream processing platform) is used for result callback in asynchronous query, and MQ is used for application decoupling when asynchronous query is initiated. Among them, KAFKA message queue is mainly used to process a large number of callback data streams, thereby ensuring that the high data throughput and low latency requirements in the enterprise are met.

[0077] This embodiment uses an external docking module to encapsulate the target external data sent by the external data source and generate a data access file; a middleware module to forward the data access file to the data storage module; a data storage module to receive, process and store the data access file; a calling statistics module to generate log records and store the log records into a preset log database based on the embedded point data obtained by deploying the monitoring and logging mechanisms at each preset key point; a data quality analysis module to perform data quality analysis on the data access file according to the log records and preset data analysis strategies, and generate data quality analysis results, thereby standardizing the external data access process, greatly improving the access efficiency, and ensuring the compliance and necessity of external data inquiries.

[0078] In a feasible implementation, the external data access platform also includes a low-code access configuration module 16, and the low-code access configuration module 16 is used to support the target user to complete data access configuration through a low-code configuration mode.

[0079] It should be noted that the low-code access configuration module 16 is used to support users to complete data access development through a low-code, configured mode. In one embodiment, the low-code access configuration mode includes two technical docking methods: API interface type and data file type. Among them, the components, workflows, data connectors, etc. required in the data access process are pre-encapsulated through custom scripts. The components can be called according to user needs in the platform's external query and storage processes. The workflow is used to support complex query process orchestration and clarify interface-related information before data query, which is used for the platform to subsequently execute predefined query processes.

[0080] In a feasible implementation manner, the external data access platform further includes a service guarantee module 17, and the service guarantee module 17 is used to ensure that the data access file can be stably transmitted between modules.

[0081] It should be noted that in order to ensure that the middleware module can stably, reliably and efficiently collect, process and transmit various required transmission information to the target system or service, the service guarantee module 17 mainly includes high availability guarantee, thread control, process control, and flow limiting and degradation services. Please refer to Table 4.

[0082] Table 4

[0083]

[0084]

[0085] In a feasible implementation manner, the external docking module is further used to perform external data query on the external data source, and analyze and process the response file returned by the external data source.

[0086] It should be noted that the external data source includes interface data and data file data.

[0087] In a feasible implementation manner, the data storage module 13 is used to receive, process and store data access files and data quality analysis results.

[0088] It should be noted that the data storage module 13 is responsible for receiving, processing and storing data from different sources. In one embodiment, the data storage module may include a relational database to store structured message result data, so as to realize fast retrieval of text data through search engines such as Elasticsearch (ES for short). The data storage module may include a time series database to store time series data and realize real-time data analysis through stream computing.

[0089] Based on this, the present application embodiment provides an external data access method, referring to Figure 5 , Figure 5 A flowchart of the first embodiment of the external data access method of the present application is provided.

[0090] Step S61, encapsulating the target external data sent by the external data source to generate a data access file;

[0091] Step S62, generating a log record corresponding to the data access file according to the embedded point data obtained by deploying the monitoring and log recording mechanism at each preset key point, and storing the log record in a preset log database;

[0092] Step S63, performing data quality analysis on the data access file according to the log record and the preset data analysis strategy, and generating a data quality analysis result;

[0093] Step S64, storing the data access file and the data quality analysis result in a local database to achieve external data access.

[0094] It should be noted that the external data source refers to a database, service, application or other data provider outside the organization or system. The target external data refers to data that is specifically required to be obtained from the external data source and is valuable for business processing or analysis. The data access file is a file formed by encapsulating the target external data obtained from the external data source. These files meet the data format and structure requirements of the internal system, which is convenient for further processing and storage.

[0095] Furthermore, the preset key points refer to key links or nodes pre-defined in the data access and processing process, such as data reception, processing, conversion, storage, etc., and monitoring and logging mechanisms are deployed at the key points. The buried point data refers to the data collected at the preset key points, which records the status and information of the data flowing through each key point, such as processing time, data volume, error rate, etc. The log record is a record generated based on the buried point data, which records in detail the relevant information in the generation, processing and transmission process of the data access file, such as timestamp, data source, processing status, error information, etc.

[0096] Furthermore, the preset data analysis strategy refers to pre-defined data analysis rules and standards for evaluating the quality of data, such as completeness, accuracy, consistency, timeliness, etc. The data quality analysis results are the results obtained after analyzing the data access files according to the preset data analysis strategy. These results reflect the quality status of the data, such as whether it meets the quality standards, the existence of errors or anomalies, etc.

[0097] Specifically, in the process of implementing external data access, the target data received from the external data source is first encapsulated. For example, if business data is obtained from an external data source, the system will encapsulate the business data into a data access file with a timestamp, location, and specific business items to ensure that the format and structure of the data meet the internal processing requirements of the platform. The monitoring and logging mechanisms deployed at the key points of data transmission and processing are then used to collect embedded data, thereby generating log records corresponding to the data access files, and storing these records in the preset log database.

[0098] Furthermore, the system will perform data quality analysis on the data access file based on the log records and the preset data analysis strategy, including checking the accuracy, completeness and timeliness of the data. After the analysis is completed, the system will generate data quality analysis results, which record the quality status of the data in detail. Furthermore, the system stores the encapsulated data access file and the data quality analysis results together in the local database, thereby achieving effective access to external data, and through log records and data quality analysis, it provides support for data management and maintenance. For example, the local database may store a record of business data, including raw data, log information and quality analysis results, to facilitate future queries and audits.

[0099] This embodiment encapsulates the target external data sent by the external data source to generate a data access file, and then generates log records corresponding to the data access file based on the embedded point data obtained by deploying the monitoring and logging mechanism at each preset key point, and stores the log records in a preset log database, so as to perform data quality analysis on the data access file according to the log records and the preset data analysis strategy, generate data quality analysis results, and then store the data access file and the data quality analysis results in the local database to realize external data access, thereby forming a complete external data access and management process together, thereby improving the efficiency and quality of data access, and ensuring the reliability and availability of data.

[0100] In a feasible implementation manner, the method further includes:

[0101] Step S71, obtaining the data access type corresponding to the target user;

[0102] It should be noted that the data access type includes interface type docking and data file type docking, among which the interface type docking refers to the method of exchanging data with an external system through an API (application programming interface), for example, the interface type docking is to implement the POST / GET request under the HTTP transmission protocol using the API. The data file type docking refers to acquiring data with an external system through file transfer, such as downloading data files through SFTP, FTP, etc.

[0103] Step S72: if the data access type is interface docking, obtain a request field parameter design and an interface parameter design, and determine a first data access component and first workflow information based on the request field parameter design and the interface parameter design;

[0104] It should be noted that the request field parameter design refers to the design and definition of the fields and processing methods that need to be included in the API request body parameters, such as field codes, whether field duplication is required, verification rules, etc. The interface parameter design includes defining the specific parameters and processing methods of the API interface, such as path parameters, communication protocols, request methods, encryption methods, response message processing methods, etc.

[0105] It should be further explained that the first data access component refers to a component designed and configured according to request field parameters and interface parameters, and is used to construct and send API requests, and receive and process responses. The first workflow information refers to the workflow configuration related to the first data access component, which defines the steps and logic of data access, such as data conversion, error handling, data storage, etc.

[0106] Specifically, if the data access type is an interface-type docking, the user only needs to fill in the request field parameter design and interface parameter design on the provided Web page to implement the data docking parameter configuration development, wherein the request field parameter design is used for subsequent external query request message format verification, user-defined request field standard format, whether duplicate elimination needs to be performed, whether it can be empty at the same time and other logic, and if the verification fails, the external query is stopped. The API interface parameter design is used to define the interface query process and post-query data processing method, including the design and configuration of external query request and response message parsing. Preferably, for historical common query needs, users can choose to use the template provided by the platform without the need to customize the configuration again.

[0107] Step S73: if the data access type is data file type docking, then the source file information is obtained, and the data processing flow is obtained through a preset canvas tool;

[0108] Step S74: Determine a second data access component and second workflow information based on the source file information and the data processing flow.

[0109] It should be noted that the source file information refers to the source, format, storage location, etc. of the data file. The data processing flow refers to the steps and logic for processing the data file, such as data cleaning, conversion, verification, etc., which are usually configured through visualization tools (canvas tools).

[0110] It should be further explained that the second data access component refers to a component based on source file information and data processing flow, which is used to perform data file transmission and processing. The second workflow information refers to the workflow configuration related to the second data access component, which defines the steps and logic of data file class docking, such as file upload, download, analysis, storage, etc.

[0111] Specifically, the user pre-configures the source file information and completes the processing flow arrangement through the preset canvas tool. In one embodiment, the required information includes the file source path, metadata information, data encryption and decryption methods, data verification and quality inspection, etc., so as to match the second data access component and generate the second workflow information according to the source file information and the data processing flow.

[0112] This embodiment obtains the data access type corresponding to the target user, and if the data access type is interface type docking, obtains the request field parameter design and the interface parameter design, and determines the first data access component and the first workflow information based on the request field parameter design and the interface parameter design. Additionally, if the data access type is data file type docking, the source file information is obtained, and the data processing flow is obtained through a preset canvas tool, and then based on the source file information and the data processing flow, the second data access component and the second workflow information are determined, thereby improving the flexibility, efficiency and maintainability of data access through modular and customized design, while reducing the complexity and cost of operations.

[0113] In a feasible implementation manner, the method further includes:

[0114] Step S81, obtaining the data importance and data size of each data item in the data access file;

[0115] Step S82, based on the importance of the data and the amount of the data, split each of the data items into a number of distributed clusters for data transmission;

[0116] Step S83, monitoring key indicator values ​​of the thread pool in data transmission;

[0117] Step S84, based on the key indicator value and the preset current limiting strategy, balance the data transmission processing capacity to ensure stable data transmission.

[0118] It should be noted that the data importance refers to the criticality of the data to business operations, decision-making, compliance requirements, etc. In one embodiment, high-importance data may include customer personal information, transaction records, account balances, etc. These data are critical to the bank's business and need to be processed and protected with priority. The data volume refers to the scale or quantity of the data, including the number of data records, the frequency of data transmission, and the size of a single data record, which may involve a large amount of transaction data, log files, or customer behavior data.

[0119] It should be further explained that the distributed cluster refers to a system consisting of multiple network-connected computers (nodes) that work together to provide higher computing power, storage capacity and reliability. In banking, distributed clusters can be used to process a large number of concurrent requests, store large-scale data, or provide highly available services. The thread pool is a mechanism for executing concurrent tasks, which allows multiple tasks (threads) to be executed in parallel in the background without creating and destroying threads for each task, so as to improve the performance of the application because it reduces the overhead of thread creation and destruction and can control the number of threads running simultaneously.

[0120] Furthermore, the key indicator value refers to a quantitative indicator used to measure system performance and health during data transmission and processing, including thread pool utilization, task queue length, average task response time, error rate, system resource utilization (such as CPU, memory, disk I / O), etc.

[0121] Specifically, the system evaluates the importance and volume of each data item in the data access file. For example, a bank may evaluate customer transaction data and account balance data, where customer transaction data may be rated as high importance due to its importance to real-time monitoring and fraud detection, while account balance data may be rated as high volume due to its large volume. Based on the importance and volume of the data, the system assigns these data items to different distributed clusters for processing. For example, high-importance transaction data will be assigned to a high-performance, low-latency cluster, while large-capacity account balance data will be assigned to a high-storage capacity cluster.

[0122] Furthermore, during the data transmission process, the system will continuously monitor the key indicator values ​​of the thread pool, such as the utilization rate of the thread pool, the length of the task queue, the task completion time, etc., so as to help understand the real-time performance and potential bottlenecks of data transmission, and then the system will adjust the processing capacity of data transmission according to the monitored key indicator values ​​and the preset current limiting strategy. For example, if the utilization rate of the thread pool is detected to be too high, the system will start the current limiting strategy, such as increasing the size of the thread pool, optimizing task allocation, or temporarily reducing the data transmission rate, to ensure the stability of data transmission and the responsiveness of the system.

[0123] This embodiment obtains the data importance and data size of each data item in the data access file, and then splits each data item into several distributed clusters for data transmission based on the data importance and the data size, thereby monitoring the key indicator value of the thread pool in the data transmission, and then based on the key indicator value and the preset current limiting strategy, balances the data transmission processing capability to ensure stable data transmission, thereby improving the efficiency, security and reliability of data processing, while reducing costs and risks.

[0124] In a feasible implementation manner, the method further includes:

[0125] Step S91, obtaining a first data external query request, parsing the first data external query request and extracting parameters to obtain request permission parameters;

[0126] It should be noted that various business systems initiate external query requests through the client, and the platform side pre-sets a whitelist in the gateway for the IP address of the system to be called. The gateway controls whether the first data external query request is made based on the whitelist.

[0127] Specifically, after the first data external query request passes through the gateway, the first data external query request is obtained, and the external docking module receives the first data external query request, parses the first data external query request and extracts parameters to obtain the request authority parameters.

[0128] Step S92: if the request permission parameter passes the permission check, encrypt the first data external query request, and perform data deduplication on the data in the preset relational database based on the first data external query request to obtain a data deduplication result to avoid repeated queries;

[0129] Specifically, the request parameters are parsed and the user permissions are verified through the relevant applications. After the permissions are passed, the message field content involved in the first data external query request needs to be further verified. The verification logic is executed according to the configuration information of the request field parameter design. If the field verification fails, the external query process is terminated. Then, the external query request message is constructed according to the preset API parameter design configuration information. Some data external queries need to encrypt the first data external query request because the request message involves sensitive information. The encryption method can be pre-configured. After the platform reads the configuration parameter information, it calls the encryption component library to implement encryption.

[0130] Furthermore, the frequency of user or IP pre-query and the response time are used to determine whether a current limiting strategy needs to be implemented. If the pre-query continuously reports errors or continuously times out for more than a specified number of times, it is determined that the index has a fault and needs to be isolated or fused. Subsequent requests will no longer access the faulty cluster, and the requests will be blocked or transferred. The platform will send them to a specific thread pool based on different current limiting strategies. At the same time, query log records are added and saved to the platform library.

[0131] Additionally, before initiating the external query, a deduplication query is initiated on the preset relational database, for example, a search is performed based on the data in the preset relational database to achieve data deduplication.

[0132] Step S93: if the data deduplication result is a non-duplicate request, the encrypted first data external query request is sent to the external data source;

[0133] Specifically, if the data deduplication result is a non-duplicate request, then after determining that there is no duplicate query and no authorization compliance risk, the encrypted first data external query request is sent to the external data source. In addition, if there is a duplicate query record, the external query is terminated. At the same time, some external query requests need to determine the customer authorization information by querying the internal library. For strong authorization scenarios, if there is a lack of data authorization information after database comparison, an error code is returned.

[0134] Step S94, decrypting and parsing the external query encryption result message returned by the external data source to generate a data external query result;

[0135] Specifically, the external docking module provides message decryption, message data processing and integration, and message parsing services according to the preset response message parsing and orchestration process, and encapsulates and generates data external query results. The data in the external query encryption result message can be processed and parsed by the intermediate node, or the original message can be returned to the business system without any operation. Message data processing and integration includes functions such as data cleaning, data integration, and field mapping. In addition, the code processing logic of the corresponding processing steps can be encapsulated into a plug-in, and the original message after parsing is processed and reorganized according to the previous user configuration requirements to generate data external query results and speed up the parsing rate.

[0136] Step S95, performing data integrity, data structure, data file field validity, and data consistency verification on the data external search result to complete the data external search.

[0137] Specifically, according to the preset integrity rules, check whether the data external query results contain all the necessary information, for example, ensure that the bank transaction records contain the transaction date, amount, account information of both parties to the transaction, etc., and then verify whether the data structure conforms to the expected format, such as JSON, XML or other custom formats, and ensure that the organization and nesting relationship of the data fields are correct. The system will then perform a validity check on each field in the data file, which involves checking whether the field value is within the permitted range, such as whether the date field is a valid date, whether the numeric field is within a reasonable range, and perform data consistency checks to ensure that the data external query results are consistent with the information in the internal database, such as by comparing the account balance provided by the external data source with the balance recorded internally by the bank. Furthermore, use hash algorithms or other verification methods to verify that the data has not been tampered with during transmission, ensure the integrity and accuracy of the data, and thus ensure the quality and reliability of the data external query results, providing a solid foundation for subsequent data processing and analysis.

[0138] In addition, the relational database and ES in the storage module are responsible for consuming data from the message queue. For a single query, the query result is encapsulated into a format suitable for insertion and the insertion command is executed. For batch queries, a scheduled task scheduling tool is used to regularly insert data in the message queue into the database in batches. For asynchronous queries, the parsed message data is called back to the message queue, and then the KAFKA topic is consumed by the callback application. For synchronous queries, the result message directly responds to the business system.

[0139] The embodiment of the present invention obtains a first data external query request, parses the first data external query request and extracts parameters to obtain request permission parameters, and then if the request permission parameters pass the permission check, the first data external query request is encrypted, and based on the first data external query request, data in a preset relational database is deduplicated to obtain a data deduplication result to avoid repeated queries, and then if the data deduplication result is a non-duplicate request, the encrypted first data external query request is sent to the external data source, and then the external query encryption result message returned by the external data source is decrypted and parsed to generate a data external query result, and then the data integrity, data structure, data file field validity, and data consistency verification are performed on the data external query result to complete the data external query to implement a series of security, efficiency and quality control measures, thereby improving the reliability and effectiveness of the data external query, which is crucial for business operations that rely on external data.

[0140] In a feasible implementation manner, the method further includes:

[0141] Step S101, obtaining a second data external query request, and initiating a query on the external data source based on the data file uniform resource locator URL in the second data external query request;

[0142] Step S102, receiving the external query response file returned by the external data source;

[0143] Step S103, if the external query response file is transmitted back in a secure file transfer protocol (SFTP) manner, a scheduled task is set to perform data transmission and verification tasks for the external query response file to ensure data integrity and consistency;

[0144] Specifically, the business system initiates a second data external query request, the platform obtains the second data external query request and parses the data file URL, and initiates a query to the external data source, and then receives the external query response file returned by the external data source, sets a scheduled task, and automatically executes the data file transmission, processing or update task at a predetermined time point through the task scheduler, downloads the SFTP file to the database in the platform through the external server, and executes the data transmission and verification tasks of the external query response file according to the preset processing flow arrangement information, so that the data stored in the message queue can be consumed and used by the downstream business system, and the data stored in the relational database is to be extracted by the in-line data warehouse.

[0145] Step S104, if the external query response file is returned in the form of a data stream, data reception and data processing are performed synchronously, and the external query response file is processed and verified and then stored in a message queue and a preset relational database.

[0146] Specifically, the business system initiates a second data external query request, the platform obtains the second data external query request and parses the data file URL, and initiates a query to the external data source, and then receives the external query response file returned by the external data source, downloads it through the client and saves it to the local data cache. If the external query response file is transmitted back in the form of a data stream, data reception and data processing are performed synchronously, and the processing and verification of the data file are completed according to the preset configuration of the processing flow orchestration information, so that the data stored in the message queue can be consumed and used by the downstream business system, and the data stored in the preset relational database is to be extracted by the in-line data warehouse.

[0147] The embodiment of the present invention obtains a second data external query request, and based on the data file uniform resource locator URL in the second data external query request, initiates a query to the external data source, and then receives an external query response file returned by the external data source. If the external query response file is transmitted back in the secure file transfer protocol SFTP mode, a scheduled task is set to perform the data transmission and verification task of the external query response file to ensure the integrity and consistency of the data. In addition, if the external query response file is transmitted back in the form of a data stream, data reception and data processing are performed synchronously, and the external query response file is processed and verified and then stored in a message queue and a preset relational database. Therefore, through the design of automation, security and real-time performance, the efficiency and reliability of data external query are improved, and the data quality and system stability are ensured.

[0148] It can be understood that based on the above external data access platform and external data access method, a layered architecture design is adopted for the external data access platform. Please refer to Figure 6 , Figure 6 This is an example diagram of the platform architecture provided for the external data access platform of this application.

[0149] Specifically, the platform architecture includes a display layer (functional interface interaction is realized through front-end web pages, including data access low-code development configuration, data quality configuration and display, call statistics configuration and display), a data processing layer (using a microservice architecture to divide multi-dimensional functions such as external data docking development, external query, storage, and operation into different applications, including gateway applications, external docking applications, service assurance applications, call statistics applications, data quality analysis applications, data synchronization applications, data storage applications, and data cache applications. Among them, the external docking and service assurance applications are the core of the platform, with multiple detailed services built in, each providing different functions), a data layer (the data synchronization application is responsible for batch synchronization of external retrieved data to the in-line data warehouse platform for centralized storage, which is convenient for subsequent sharing and use. The data storage application is responsible for storing the retrieved data in the platform's built-in relational database. The data cache application is responsible for asynchronous decoupling of external query applications to improve the platform's external query efficiency and performance), a basic component library (pre-defines multiple basic building components to encapsulate specific communication protocols, data processing and integration processing logic, allowing users to request dynamic loading and use of components through configuration) and a database (including relational databases, historical databases, cache databases, message queues and other basic databases).

[0150] It should be noted that the examples in the figure are only used to understand the present application and do not constitute a limitation on the external data access method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0151] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0152] The present application provides an external data access device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the external data access method in the above-mentioned embodiment one.

[0153] Reference below Figure 7, which shows a schematic diagram of the structure of an external data access device suitable for implementing the embodiment of the present application. The external data access device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The external data access device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0154] like Figure 7 As shown, the external data access device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the external data access device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the external data access device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an external data access device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0155] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0156] The external data access device provided by the present application adopts the external data access method in the above embodiment, which can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the external data access device provided by the present application are the same as the beneficial effects of the external data access method provided by the above embodiment, and other technical features in the external data access device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0157] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0158] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0159] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the external data access method in the above-mentioned embodiment.

[0160] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0161] The computer-readable storage medium may be included in the external data access device; or may exist independently without being assembled into the external data access device.

[0162] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the external data access device, the external data access device:

[0163] Encapsulate the target external data sent by the external data source and generate a data access file;

[0164] Generate log records corresponding to the data access file based on the embedded point data obtained by deploying monitoring and log recording mechanisms at each preset key point, and store the log records in a preset log database;

[0165] According to the log records and the preset data analysis strategy, a data quality analysis is performed on the data access file to generate a data quality analysis result;

[0166] The data access file and the data quality analysis result are stored in a local database to achieve external data access.

[0167] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0170] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned external data access method, and can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the external data access method provided in the above-mentioned embodiment, and will not be repeated here.

[0171] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the external data access method as described above when executed by a processor.

[0172] The computer program product provided in this application can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the external data access method provided in the above embodiment, which will not be repeated here.

[0173] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An external data access platform, comprising an external docking module, a middleware module, a data storage module, a call statistics module and a data quality analysis module, characterized in that: The external docking module is used to encapsulate the target external data sent by the external data source and generate a data access file; The calling statistics module is used to generate log records corresponding to the data access file according to the buried point data obtained by deploying the monitoring and log recording mechanism at each preset key point, and store the log records in a preset log database; The data quality analysis module is used to perform data quality analysis on the data access file according to the log record and the preset data analysis strategy, and generate a data quality analysis result; The middleware module is used to forward the data access file and the data quality analysis result to the data storage module to achieve external data access.

2. The external data access platform according to claim 1, characterized in that: It also includes a low-code access configuration module, which is used to support target users to complete data access configuration through a low-code configuration mode.

3. The external data access platform according to claim 1, characterized in that: It also includes a service guarantee module, which is used to ensure that the data access file can be stably transmitted between modules.

4. The external data access platform according to claim 1, characterized in that: The external docking module is also used to perform data query on the external data source and analyze and process the response file returned by the external data source.

5. The external data access platform according to claim 1, characterized in that: The data storage module is used to receive, process and store data access files and data quality analysis results.

6. An external data access method, characterized in that: The method comprises: Encapsulate the target external data sent by the external data source and generate a data access file; Generate log records corresponding to the data access file based on the embedded point data obtained by deploying monitoring and log recording mechanisms at each preset key point, and store the log records in a preset log database; According to the log records and the preset data analysis strategy, a data quality analysis is performed on the data access file to generate a data quality analysis result; The data access file and the data quality analysis result are stored in a local database to achieve external data access.

7. The external data access method according to claim 6, characterized in that: The method further comprises: Get the data access type corresponding to the target user; If the data access type is interface type docking, obtaining a request field parameter design and an interface parameter design, and determining a first data access component and first workflow information based on the request field parameter design and the interface parameter design; If the data access type is data file docking, the source file information is obtained, and the data processing flow is obtained through a preset canvas tool; Based on the source file information and the data processing flow, a second data access component and second workflow information are determined.

8. The external data access method according to claim 6, characterized in that: The method further comprises: Obtain the data importance and data volume of each data item in the data access file; Based on the importance of the data and the amount of data, each of the data items is split into several distributed clusters for data transmission; Monitor key indicator values ​​of thread pool in data transmission; Based on the key indicator values ​​and the preset current limiting strategy, the data transmission processing capability is balanced to ensure stable data transmission.

9. The external data access method according to claim 6, characterized in that: The method further comprises: Obtaining a first data external query request, parsing the first data external query request and extracting parameters to obtain a request permission parameter; If the request permission parameter passes the permission check, the first data external query request is encrypted, and based on the first data external query request, data in the preset relational database is deduplicated to obtain a data deduplication result to avoid repeated queries; If the data deduplication result is a non-duplicate request, sending the encrypted first data external query request to the external data source; Decrypting and parsing the external query encrypted result message returned by the external data source to generate a data external query result; The data integrity, data structure, data file field validity, and data consistency of the data external search results are checked to complete the data external search.

10. The external data access method according to claim 6, characterized in that: The method further comprises: Obtaining a second external data query request, and initiating a query on the external data source based on a data file uniform resource locator URL in the second external data query request; Receiving an external query response file returned by the external data source; If the external inquiry response file is sent back in a secure file transfer protocol (SFTP) manner, a scheduled task is set to perform data transmission and verification tasks for the external inquiry response file to ensure data integrity and consistency; If the external inquiry response file is transmitted back in the form of a data stream, data reception and data processing are performed simultaneously, and the external inquiry response file is processed and verified and then stored in a message queue and a preset relational database.

Images