Data docking method, system and device and storage medium

Abstract

The embodiment of the invention discloses a data connection method, system and device and a storage medium, and the method comprises the steps: receiving a configuration parameter and a storage model identifier, the configuration parameter comprises a request address and request content, assembling the request address and the request content to generate a data access request, and sending the data access request to a server, sending the request content to the server to enable the server to call and send corresponding system data, receiving the system data sent by the server, carrying out data analysis on the system data based on the type of the request content to obtain structured system data, carrying out coding verification on the structured system data, and sending the structured system data to the server under the condition that the coding verification is passed. And binding the storage model identifier with the structured system data, storing the storage model identifier and the structured system data in a consumption queue, and writing the structured system data in the consumption queue into a data warehouse based on the storage model identifier, so that the data docking cost is reduced, and the data docking efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data docking method, system, device and storage medium. Background Art

[0002] With the continuous development of network technology, more and more enterprises are adopting diversified information systems to meet their different business needs. However, these information systems are usually deployed independently, and their development methods and data storage formats are different. This leads to barriers between data and between system data and data warehouses, making it difficult to store data from various platforms in the data warehouse, which hinders the effective transmission and sharing of data.

[0003] In related technologies, when connecting data between most information systems, developers are required to adapt and develop system interfaces. This requires a certain amount of manpower costs and affects the efficiency of data connection, and cannot guarantee the validity of the connected data. Summary of the invention

[0004] The embodiments of the present application provide a data docking method, system, device and storage medium, which solve the problems of low docking efficiency and high labor cost in the data docking process, reduce the data docking cost and improve the data docking efficiency.

[0005] In a first aspect, an embodiment of the present application provides a data docking method, including:

[0006] receiving configuration parameters and a storage model identifier, wherein the configuration parameters include a request address and a request content, assembling the request address and the request content to generate a data access request, and sending the data access request to a server so that the server retrieves and sends corresponding system data;

[0007] receiving the system data sent by the server, parsing the system data based on the type of the request content to obtain structured system data, and performing coding verification on the structured system data;

[0008] When the coding verification is passed, the storage model identifier is bound to the structured system data and stored in the consumption queue, and the structured system data in the consumption queue is written into the data warehouse based on the storage model identifier.

[0009] Optionally, the configuration parameter further includes a time window, and the step of assembling the request address and the request content to generate a data access request includes:

[0010] The request address and the request content are cyclically assembled according to the number of the time windows to generate a data access request.

[0011] Optionally, the performing coding verification on the structured system data includes:

[0012] The encoding format of the structured system data is compared with the encoding format of the data warehouse. If the encoding formats are the same, the encoding verification is passed.

[0013] Optionally, after obtaining the structured system data, the method further includes:

[0014] A data storage table is created according to the field information and table construction index provided by the data warehouse storage model, and the structured system data is stored in the data storage table, so as to feedback the query result when a query request is received.

[0015] Optionally, after receiving the configuration parameters, the method further includes:

[0016] Creating a request template, filling the request template with data according to the configuration parameters, and performing authority verification on the filled request template;

[0017] Accordingly, assembling the request address and the request content to generate a data access request includes:

[0018] When the authority verification is passed, the request address and the request content are assembled to generate a data access request.

[0019] Optionally, writing the structured system data in the consumption queue into a data warehouse based on the storage model identifier includes:

[0020] Classifying the structured system data in the consumption queue according to the storage model identifier bound thereto, assembling multiple groups of structured system data based on the classification results and performing character escape processing on each of them;

[0021] The processed groups of structured system data are written into the data warehouse respectively.

[0022] Optionally, the method further includes:

[0023] Determining whether the processed structured system data is written successfully;

[0024] In case of a write failure, the structured system data is written into the dead letter queue one by one, and the structured system data in the dead letter queue is rewritten into the data warehouse;

[0025] The structured system data that failed to be rewritten is written into the exception queue, and an alarm message is sent.

[0026] In a second aspect, an embodiment of the present application provides a data docking system, including:

[0027] A receiving module, used to receive configuration parameters and a storage model identifier, wherein the configuration parameters include a request address and a request content;

[0028] A data access request generating module, used for assembling the request address and the request content to generate a data access request, and sending the data access request to the server, so that the server retrieves and sends the corresponding system data;

[0029] A data parsing module, configured to receive the system data sent by the server, and parse the system data based on the type of the request content to obtain structured system data;

[0030] A coding verification module, used for performing coding verification on the structured system data;

[0031] A storage module, used for binding the storage model identifier with the structured system data and storing the data in a consumption queue when the coding verification is passed;

[0032] A data writing module is used to write the structured system data in the consumption queue into a data warehouse based on the storage model identifier.

[0033] In a third aspect, an embodiment of the present application provides a data docking device, comprising: one or more processors; a storage device configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data docking method described in the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the data docking method as described in the first aspect.

[0035] The embodiment of the present application receives configuration parameters and storage model identifiers, wherein the configuration parameters include a request address and a request content, assembles the request address and the request content to generate a data access request, and sends the data access request to the server so that the server retrieves and sends the corresponding system data, receives the system data sent by the server, performs data parsing on the system data based on the type of the request content to obtain structured system data, and performs coding verification on the structured system data. When the coding verification passes, the storage model identifier is bound to the structured system data and stored in a consumption queue, and the structured system data in the consumption queue is written into a data warehouse based on the storage model identifier. Data access requests can be automatically sent through configuration parameters to achieve automatic data access, thereby avoiding the increase in labor costs for developers to adapt and develop system interfaces, and improving the efficiency of data access. By dynamically parsing the accessed data, structured system data is obtained to meet the storage specifications of the system, and through coding consistency verification, data processing and analysis are facilitated, and the transmission efficiency of structured system data can also be improved. By storing structured system data in corresponding partitions through storage model identification classification, the load of each partition can be balanced and the parallel execution of data access and data consumption can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a data docking method provided in an embodiment of the present application;

[0037] Figure 2 is a flow chart of a structured data consumption method provided in an embodiment of the present application;

[0038] Figure 3 It is a flowchart of a method for processing structured system data writing exception provided by an embodiment of the present application;

[0039] Figure 4 It is a structural schematic diagram of a data docking system provided in an embodiment of the present application;

[0040] Figure 5 It is a structural schematic diagram of a data docking device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0043] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0044] The data docking method, device, equipment and storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0045] The embodiment of the present application can be used in scenarios where various background data are automatically connected to a data warehouse through configuration parameters. Based on the above usage scenarios, it can be understood that the executor of this solution can be a data center system of a financial data warehouse. The data docking method provided in this embodiment can be executed by a data docking method device, which can be implemented by software and / or hardware. The data docking device can be composed of two or more physical entities, or it can be composed of one physical entity. Generally speaking, the data docking device can be a computer or a tablet, etc.

[0046] The following describes a data connection method as an example. Figure 1 , Figure 1 This is a flow chart of a data connection method provided by an embodiment of the present application, which specifically includes:

[0047] Step S101, receiving configuration parameters and storage model identifiers, wherein the configuration parameters include a request address and a request content, assembling the request address and the request content to generate a data access request, and sending the data access request to a server so that the server retrieves and sends corresponding system data.

[0048] In one embodiment, the configuration parameter may be a parameter setting involved in the process of automatic data docking. The configuration parameter may be a parameter input by a user through a visual acquisition interface provided by the system. The configuration parameter may include parameters such as parameter description, parameter type, value range, and default value. The parameter description may be "**Client program name for connecting to the database", and the parameter type may be "**POSTMASTER" or "**USERSET", etc. The value range may be "**Integer" or "**String", etc. The default value may be "**gsql" or "**3", etc. Optionally, the configuration parameter also includes a request address and a request content, wherein the request address may be used to indicate the access location and access method of the data. The request content includes the request address, request header, request body, and the type of the request content, etc. The request header may include a value range, the domain name and port number of the server, etc. The request body can be the part of the http protocol used to transmit the data sent by the client to the server, and is used to pass the parameters, content or data described in the request to the server. The request body can contain various types of data, such as form data, JSON data, files, etc. The specific type of the request content is specified by the Content-Type field in the request header, which can include text / xml format or json format. The storage model identifier can be a unique identifier for each message in the message queue. Multiple structured system data form an ordered queue. The identifier can also refer to the offset of the queue. The data access request can be an http request, which generates a data access request by combining the request address and request parameters. The system data can be specific data determined according to the configuration parameters in the data access request, that is, the required data accessed from other platforms.

[0049] In one embodiment, the data center system automatically triggers the data access process according to a preset time, receives the configuration parameters input by the user, generates a data access request that meets the user's needs based on the configuration parameters, and sends the data access request to the server. The server retrieves the required type of parameters from the corresponding location based on the received data access request, and returns the retrieved specific parameters to the data center system. Whether the request is successful is determined by receiving a response code from the server. If the response code is 200, it means that the server responds successfully, that is, the server finds the requested content in the data access request and returns the found content to the system. If the status code is 404, it means that the requested content in the data access request cannot be found on the server.

[0050] In another embodiment, after receiving the configuration parameters, a request template is created, the request template is filled with data according to the configuration parameters, and permission verification is performed on the filled request template.

[0051] If the permission verification is passed, the request address and request content are assembled to generate a data access request.

[0052] In order to enable an http request to be processed by a specific request template, a separate request template is created, which includes a request header and a request body. The request template can be a dynamic parameter expression, which is executed by a mysql function. The received request address and request content are filled into the request body and request header of the request template respectively, wherein paging parameters can also be filled in the request body, such as (pageIndes=1, pageSize=10000), and the filled request template is authenticated by the formula. The system information can be obtained while receiving the configuration parameters, and the filled request template is authenticated according to the obtained system information to determine whether the data center system has the authority to send data requests. The system information can be the identity information of the system administrator, such as the account and password of the system administrator. After the authority verification is passed, the request address and request content are assembled to generate a data access request. It can prevent an http request from being processed by multiple configuration items at the same time, and can be processed through specific configuration parameters, thereby improving the efficiency of data access. By verifying the authority of the request template, it can prevent illegal users from stealing data from other platforms through the data center system, thereby ensuring the security of data access.

[0053] In one embodiment, the configuration parameters also include a time window, and the request address and the request content are assembled to generate a data access request, including:

[0054] The request address and request content are cyclically assembled according to the number of time windows to generate a data access request.

[0055] A time window can be used to describe a time period. It can be used to describe the completion time of a given task, or it can be used to describe the requirement that a specific task be completed within a given time period, where there can be multiple given time periods. The system can pull data in a rolling manner according to the time window, such as 2023-01-01 to 2023-01-31, which will be split into 30 days for data pulling, thereby reducing the query pressure on the data provider and improving the efficiency of data docking.

[0056] In one embodiment, data access requests are assembled cyclically according to the number of pre-set time windows. Exemplarily, 2023-01-01 to 2023-01-31 is split into 30 days for data pulling. 2023-01-01 to 2023-01-02 can be used as a time window to assemble data access requests to pull data, and 2023-01-02 to 2023-01-03 can be used as a time window to assemble data access requests to pull data, and so on, pulling data 30 times from 2023-01-01 to 2023-01-31. Avoiding pulling all data at once, increasing the data query pressure on the data provider, and affecting the efficiency of data docking, it also helps to manage data pulling tasks, reduce the risk of data pulling failure, and avoid data pulling delays or failures.

[0057] Step S102: receiving the system data sent by the server, parsing the system data based on the type of the request content to obtain structured system data, and performing coding verification on the structured system data.

[0058] Data parsing can include JSON data parsing and XML data parsing. JSON parsing is a lightweight data exchange format. Parsing JSON can pass data from one application to another, and the parsed data can be read and used by other applications. XML data parsing can be reading XML documents into memory and converting them into data that can be processed by the program. The program can process these data to complete various operations, such as data editing, conversion, import and export, etc. Structured system data is a data type analysis of the storage form. Structured data is also a database that can store data in a standardized or unified format. Unstructured data includes data in the form of video, audio, image, document or text. Data parsing is to parse unstructured data into structured data. Coding verification can be to compare the parsed structured data with the encoding format of the database to determine whether the structured data meets the storage specifications of the system.

[0059] When the status code returned by the server is 200, the server receives the system data sent by the server and parses the system data according to the type of the request content in the data access request. Exemplarily, if the type of the request content is in XML format, the received system data is parsed in XML format; if the type of the request content is in JSON format, the received system data is parsed in JSON format. The system data is parsed into structured data, the encoding method of the consumption queue is queried through query instructions, and the encoding format of the data warehouse is compared with the encoding format of the structured data to determine whether the encoding verification is passed. In one embodiment, the encoding format of the structured system data is compared with the encoding format of the data warehouse. If the encoding formats are the same, the encoding verification passes, otherwise the encoding verification fails.

[0060] In one embodiment, after obtaining the structured system data, a data storage table is created according to the field information and table construction index provided by the data warehouse storage model, and the structured system data is stored in the data storage table for feeding back the query result when a query request is received.

[0061] A data warehouse model can be a model used to represent the way data is stored in a data warehouse. The storage model can be a data table, which includes field information and table indexes. The field information can also include field name, data type, length, unique constraint, and foreign key. A unique constraint can be a constraint that limits the values ​​of one or more columns in a database table, ensuring that the values ​​in these columns are different. On columns with unique constraints defined, the data warehouse system will automatically check insert or update operations to ensure the uniqueness of the data. If duplicate values ​​are to be inserted, or values ​​that violate the unique constraint are to be updated, the data warehouse will reject these operations and return an error. A foreign key is a field that points to the primary key in another table. Through foreign keys, associations can be established between different tables to achieve data consistency and integrity. Query requests can include the time when data is stored and the type of data storage.

[0062] In one embodiment, field information in structured data is extracted according to field information in the warehouse storage model, and a table index of a corresponding structured data table is determined according to the table index in the warehouse storage model. A data storage table is established based on the extracted field information and the determined table index. The data storage table may be a structured data storage table. The structured data is stored in the data storage table. After the data storage table is established, the data storage table may be divided into tables according to time. Exemplarily, the tables may be divided into tables by month or by year. When the system receives a query request, it may query the corresponding data table according to the time of the stored data contained in the query request, and feedback the query result. By storing structured data in a data table and performing table division, the query speed of the data can be improved, and the amount of data in each table is small, and the maintenance cost is low.

[0063] Step S103: When the coding verification is passed, the storage model identifier is bound to the structured system data and stored in the consumption queue, and the structured system data in the consumption queue is written into the data warehouse based on the storage model identifier.

[0064] The consumer queue can be a distributed event storage and stream processing platform that can provide a unified, high-throughput, low-latency platform for processing real-time data streams. The consumer queue can be a kafka cluster, and the messages stored in the kafka cluster are records classified by topic, and each message corresponds to an identifier. The data warehouse can be a database that partitions and stores structured system data. There are multiple partitions in the data warehouse, and different partitions are used to store different structured system data. When the coding verification passes, the received storage model identifier is bound to the structured system data and stored in the message queue, and the amount of data in the consumer queue is recorded. The consumer queue can store multiple sets of binding relationships, and the structured system data bound to it can be written into the data warehouse based on the storage model identifier according to the set rules.

[0065] The embodiment of the present application receives configuration parameters and storage model identifiers, the configuration parameters include request addresses and request contents, assembles the request addresses and request contents to generate data access requests, and sends the data access request to the server so that the server retrieves and sends the corresponding system data, receives the system data sent by the server, performs data parsing on the system data based on the type of the request content, obtains structured system data, and performs coding verification on the structured system data. When the coding verification passes, the storage model identifier is bound to the structured system data and stored in the consumption queue, and the structured system data in the consumption queue is written into the data warehouse based on the storage model identifier. Data access requests can be automatically sent through configuration parameters to achieve automatic data access, avoiding the increase of labor costs for developers to adapt and develop system interfaces, and improving the efficiency of data access. By dynamically parsing the accessed data, structured system data is obtained to meet the storage specifications of the system, and through coding consistency verification, data processing and analysis are facilitated, and the transmission efficiency of structured system data can also be improved. By storing structured system data in corresponding partitions in a storage model identifier classification manner, the load of each partition can be balanced, and data access and data consumption can be executed in parallel.

[0066] Figure 2 is a flow chart of a structured data consumption method provided in an embodiment of the present application, such as Figure 2 As shown, specifically including:

[0067] Step S1031: classify the structured system data in the consumption queue according to the storage model identifier bound thereto, assemble multiple groups of structured system data based on the classification results, and perform character escape processing on each of them.

[0068] Step S1032: Write the processed groups of structured system data into the data warehouse respectively.

[0069] In one embodiment, there are multiple sets of binding relationships in the consumption queue, and the structured system data in the consumption queue needs to be written into the data warehouse. The data warehouse contains multiple different partitions, each partition is composed of multiple ordered storage model identifiers, and new structured data can be continuously appended to the tail of the corresponding partition. The partition can be regarded as an appendable log file at the storage level for storing data. It can be classified based on the topics created in the consumption queue, the storage model identifiers of the same topic are classified into one category, and the structured data corresponding to the storage model identifiers of the same category are assembled and written into different partitions, which can be assembled based on pre-set partitioning rules. The partitioning rules can be evenly distributed according to the number of storage model identifiers in the consumption queue and the total number of partitions in the consumption queue. Exemplarily, if the number of storage model identifiers in the consumption queue is 30, there are 5 partitions, and every 6 structured data are assembled and written into a partition in order, and it can also be selected according to specific partitioning rules to write to which partition. Character escape processing can be to avoid conflicts with quotation marks in the string itself, using the escape character "\" to escape single quotation marks. For example, if you want to insert a single quotation mark in a string, you can use \' to represent it. In one embodiment, classification is performed according to the storage model identifier, and structured system data of the same type are assembled. The assembled structured system data are subjected to character escape processing so that special characters can be processed as ordinary characters. The structured system data after escape processing are written to each partition for storage, so that consumers can consume the structured system data in each partition.

[0070] The embodiment of the present application classifies the structured system data in the consumption queue according to the storage model identifier bound thereto, assembles multiple groups of structured system data based on the classification results, performs character escape processing respectively, and writes the processed multiple groups of structured system data into the data warehouse respectively. By partitioning and storing the data in the consumption queue and reasonably controlling the tasks of the partitions, the load of each partition can be balanced, the system can receive data in partitions, and consumers can consume data in partitions, thereby achieving the degree of parallelism between data access and data consumption. By performing character escape processing on the structured system data, a specific text format or layout can be achieved, avoiding errors in the compilation process.

[0071] Figure 3 is a flowchart of a method for processing structured system data writing exceptions provided by an embodiment of the present application, such as Figure 3 As shown, including:

[0072] Step S301: Determine whether the processed structured system data is written successfully.

[0073] In one embodiment, when writing structured system data, the writing of structured system data may fail due to network jitter, abnormality, etc., so it is necessary to determine whether the writing of structured system data is successful when writing structured system data to the data warehouse. Whether the writing of structured system data is successful can be determined by comparing the amount of written data with the amount of data stored in the data warehouse, and the specific data that failed to be written can be determined.

[0074] Step S302: In case of a write failure, the structured system data is written into a dead letter queue one by one, and the structured system data in the dead letter queue is rewritten into the data warehouse.

[0075] The dead letter queue can be used to store data that cannot be processed or consumed normally. In order to prevent the unprocessed structured system data from causing program exceptions and affecting the system, the abnormal structured data can be written to the dead letter queue one by one, and the data that has timed out and not been consumed in the consumption queue can also be transferred to the dead letter queue. Avoiding program exceptions caused by unprocessed dead letters can improve the reliability of the system. Rewrite the structured system data in the dead letter queue into the data warehouse.

[0076] Step S303: Write the structured system data that failed to be rewritten into the abnormal queue, and send an alarm message.

[0077] The exception queue can be used to store data that has failed to be written to the data warehouse multiple times. The alarm information can include abnormal structured system data, the number of abnormal data, and the failure details of the abnormal data. When the structured system data in the dead letter queue fails to be written to the data warehouse again, the structured system data that failed to be written is written to the exception queue, and an alarm message is sent to the system administrator at a preset interval.

[0078] The embodiment of the present application determines whether the processed structured system data is written successfully. In the case of a write failure, the structured system data is written into the dead letter queue one by one, and the structured system data in the dead letter queue is rewritten into the data warehouse, and the structured system data that failed to be rewritten is written into the exception queue, and an alarm message is sent. By writing the abnormally written data into the dead letter queue and rewriting it into the data warehouse, program exceptions caused by unprocessed or failed dead letters can be avoided, and the reliability of the system can be improved. By sending data that failed to be written multiple times to the system user, it is convenient for the user to understand and process the abnormal data, which improves the efficiency of abnormal data processing and improves the user experience.

[0079] Figure 4 is a structural diagram of a data docking system provided in an embodiment of the present application, such as Figure 4 As shown, including:

[0080] A receiving module 41, configured to receive configuration parameters and a storage model identifier, wherein the configuration parameters include a request address and a request content;

[0081] A data access request generating module 42, configured to generate a data access request by assembling the request address and the request content, and send the data access request to the server, so that the server retrieves and sends the corresponding system data;

[0082] A data parsing module 43, configured to receive the system data sent by the server, and parse the system data based on the type of the request content to obtain structured system data;

[0083] A coding verification module 44, used to perform coding verification on the structured system data;

[0084] The storage module 45 is used to bind the storage model identifier with the structured system data and store them in the consumption queue when the coding verification is passed;

[0085] The data writing module 46 is used to write the structured system data in the consumption queue into the data warehouse based on the storage model identifier.

[0086] The embodiment of the present application receives configuration parameters and storage model identifiers, the configuration parameters include request addresses and request contents, assembles the request addresses and request contents to generate data access requests, and sends the data access request to the server so that the server retrieves and sends the corresponding system data, receives the system data sent by the server, performs data parsing on the system data based on the type of the request content, obtains structured system data, and performs coding verification on the structured system data. When the coding verification passes, the storage model identifier is bound to the structured system data and stored in the consumption queue, and the structured system data in the consumption queue is written into the data warehouse based on the storage model identifier. Data access requests can be automatically sent through configuration parameters to achieve automatic data access, avoiding the increase of labor costs for developers to adapt and develop system interfaces, and improving the efficiency of data access. By dynamically parsing the accessed data, structured system data is obtained to meet the storage specifications of the system, and through coding consistency verification, data processing and analysis are facilitated, and the transmission efficiency of structured system data can also be improved. By storing structured system data in corresponding partitions in a storage model identifier classification manner, the load of each partition can be balanced, and data access and data consumption can be executed in parallel.

[0087] In a possible embodiment, the configuration parameters also include a time window, and the data access request generation module 42 is specifically configured to cyclically assemble the request address and the request content according to the number of time windows to generate a data access request.

[0088] In a possible embodiment, the coding verification module 44 is specifically used to compare the coding format of the structured system data with the coding format of the data warehouse. If the coding formats are the same, the coding verification is passed.

[0089] In a possible embodiment, the data docking system also includes a table creation module, which is used to create a data storage table based on the field information and table creation index provided by the data warehouse storage model, store the structured system data in the data storage table, and feedback the query results when a query request is received.

[0090] In a possible embodiment, the data docking system further includes a template creation module, which is used to create a request template, fill the request template with data according to the configuration parameters, and perform authority verification on the filled request template.

[0091] In a possible embodiment, the data access request generating module 42 is specifically configured to assemble the request address and the request content to generate a data access request when the authority verification is passed.

[0092] In a possible embodiment, the data writing module 46 is specifically used to classify the structured system data in the consumption queue according to the storage model identifier bound thereto, assemble multiple groups of structured system data based on the classification results, and perform character escape processing on each of them;

[0093] The processed groups of structured system data are written into the data warehouse respectively.

[0094] In a possible embodiment, the data docking system further includes an abnormal data processing module, and the abnormal data processing module is specifically used to determine whether the processed structured system data is written successfully;

[0095] In case of a write failure, the structured system data is written into the dead letter queue one by one, and the structured system data in the dead letter queue is rewritten into the data warehouse;

[0096] The structured system data that failed to be rewritten is written into the exception queue, and an alarm message is sent.

[0097] The embodiment of the present application also provides a data docking device, which can integrate a data docking system provided by the embodiment of the present application. Figure 5 Schematic diagram of a data connection device provided by an embodiment of the present application. Figure 5The data docking device includes: an input device 53, an output device 54, a memory 52 and one or more processors 51; the memory 52 is used to store one or more programs; when one or more programs are executed by one or more processors 51, the one or more processors 51 implement the data docking method provided in the above embodiment. The input device 53, the output device 54, the memory 52 and the processor 51 can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0098] The memory 52, as a computing device readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the data docking method provided in any embodiment of the present application. The memory 52 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 52 may further include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] The input device 53 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 54 may include a display device such as a display screen.

[0100] The processor 51 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 52, that is, realizes the above-mentioned data docking method.

[0101] The data docking system, device and computer provided above can be used to execute the data docking method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0102] The present application also provides a storage medium storing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to execute the data docking method provided in the above embodiment. The data docking method includes:

[0103] Receive configuration parameters and storage model identifiers, the configuration parameters include a request address and a request content, assemble the request address and the request content to generate a data access request, and send the data access request to the server so that the server retrieves and sends corresponding system data;

[0104] Receive system data sent by the server, parse the system data based on the type of the request content, obtain structured system data, and perform coding verification on the structured system data;

[0105] When the coding verification is passed, the storage model identifier is bound to the structured system data and stored in the consumption queue. Based on the storage model identifier, the structured system data in the consumption queue is written into the data warehouse.

[0106] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0107] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the data docking method described above, can also execute related operations in the data docking method provided in any embodiment of the present application.

[0108] The data docking system, device and storage medium provided in the above embodiments can execute the data docking method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the data docking method provided in any embodiment of the present application.

[0109] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A data docking method, characterized in that: include: receiving configuration parameters and a storage model identifier, wherein the configuration parameters include a request address and a request content, assembling the request address and the request content to generate a data access request, and sending the data access request to a server so that the server retrieves and sends corresponding system data; receiving the system data sent by the server, parsing the system data based on the type of the request content to obtain structured system data, and performing coding verification on the structured system data; When the coding verification is passed, the storage model identifier is bound to the structured system data and stored in the consumption queue, and the structured system data in the consumption queue is written into the data warehouse based on the storage model identifier.

2. The data docking method according to claim 1, characterized in that: The configuration parameters also include a time window, and the step of assembling the request address and the request content to generate a data access request includes: The request address and the request content are cyclically assembled according to the number of the time windows to generate a data access request.

3. The data connection method according to claim 1, characterized in that: The coding verification of the structured system data includes: The encoding format of the structured system data is compared with the encoding format of the data warehouse. If the encoding formats are the same, the encoding verification is passed.

4. The data docking method according to claim 1, characterized in that: After obtaining the structured system data, the method further includes: A data storage table is created according to the field information and table construction index provided by the data warehouse storage model, and the structured system data is stored in the data storage table, so as to feedback the query result when a query request is received.

5. The data docking method according to claim 1, characterized in that: After receiving the configuration parameters, the method further includes: Creating a request template, filling the request template with data according to the configuration parameters, and performing authority verification on the filled request template; Accordingly, assembling the request address and the request content to generate a data access request includes: When the authority verification is passed, the request address and the request content are assembled to generate a data access request.

6. The data docking method according to claim 1, characterized in that: The step of writing the structured system data in the consumption queue into the data warehouse based on the storage model identifier includes: Classifying the structured system data in the consumption queue according to the storage model identifier bound thereto, assembling multiple groups of structured system data based on the classification results and performing character escape processing on each of them; The processed groups of structured system data are written into the data warehouse respectively.

7. The data connection method according to claim 6, characterized in that: The method further comprises: Determining whether the processed structured system data is written successfully; In case of a write failure, the structured system data is written into the dead letter queue one by one, and the structured system data in the dead letter queue is rewritten into the data warehouse; The structured system data that failed to be rewritten is written into the exception queue, and an alarm message is sent.

8. A data docking system, characterized in that: include: A receiving module, used to receive configuration parameters and a storage model identifier, wherein the configuration parameters include a request address and a request content; A data access request generating module, used for assembling the request address and the request content to generate a data access request, and sending the data access request to the server, so that the server retrieves and sends the corresponding system data; A data parsing module, configured to receive the system data sent by the server, and parse the system data based on the type of the request content to obtain structured system data; A coding verification module, used for performing coding verification on the structured system data; A storage module, used for binding the storage model identifier with the structured system data and storing the data in a consumption queue when the coding verification is passed; A data writing module is used to write the structured system data in the consumption queue into a data warehouse based on the storage model identifier.

9. A data connection device, comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data docking method as described in any one of claims 1-7.

10. A storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the data docking method according to any one of claims 1 to 7 when executed by a computer processor.

Images