A data processing method, device and data processing system for concurrent connectors

Concurrent channels are built between presto and GreenPlum databases through concurrent connectors, which solves the problem of inefficient reading and writing efficiency, realizes efficient data processing and deletion operations, and improves the stability and user experience of the system.

CN117312373BActive Publication Date: 2025-08-19DUXIAOMAN TECH (BEIJING) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311266125.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-27
Publication Date
2025-08-19
Estimated Expiration
2043-09-27

AI Technical Summary

Technical Problem

When interacting with GreenPlum database, the existing presto has low read and write efficiency, the master node is prone to crash, and it is unable to perform real-time data processing and deletion operations, resulting in poor user experience.

Method used

Concurrent channels are built through concurrent connectors, and the segmented nodes of the data source are connected with the working nodes of the computing engine to realize concurrent reading and writing of data, and embed deletion operators in the connector to support the deletion operation of the data source.

Benefits of technology

It greatly improves data reading and writing efficiency, alleviates the resource pressure of the master node, ensures the stability and real-time nature of data processing, expands the data processing scenario, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117312373B_ABST
    Figure CN117312373B_ABST
Patent Text Reader

Abstract

The present disclosure provides a data processing method, device and data processing system for a concurrent connector, which relates to the field of big data technology. The specific implementation of the method includes: receiving one or more data read requests; searching for a data source address corresponding to a data source identifier and an engine address corresponding to an engine identifier from a local concurrent configuration table, and accessing a data source master node corresponding to the data source address; receiving the first node information assigned by the data source master node to the data to be read, and accessing the engine master node corresponding to the engine address to obtain the second node information of the computing engine; using the first node information and the second node information, docking the data source and the computing engine, so that the computing engine concurrently collects the data to be read from the data source. This implementation can utilize the concurrent channel of the constructed concurrent connector to dock the segmented nodes in the data source with the working nodes of the computing engine, thereby realizing concurrent reading and writing of data, and improving reading and writing efficiency and writing stability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a data processing method, device, and system for a concurrent connector. Background Art

[0002] Presto is a commonly used real-time query and computing engine for big data. It offers excellent computing performance, enhanced usability, and enhanced scalability. It can support processing of various data sources, even cross-data source computing scenarios. Common data sources include GP, Hive, MySQL, HBase, etc., with large amounts of data and diverse distribution formats.

[0003] When existing Presto obtains data from GP, it usually uses JDBC database connection to build a link between Presto and GP, and then reads data from or writes data to the GP master node.

[0004] However, with the rapid development of big data technology, Presto often needs to perform correlation calculations on data in multiple tables distributed in different locations in GP. On the one hand, relying solely on the read and write efficiency of the master node is low, and due to the huge amount of data, it will cause great read and write pressure on the master node, which is very likely to cause data read and write failures, master node crashes, and thus cause GP blockage and unstable operation. On the other hand, the data written to GP is not static and is often accompanied by operations such as data deletion and modification. The existing JDBC database connection cannot perform deletion and other operations. It can only delete all the original data and then re-import the full amount of data after the changes, resulting in low data processing efficiency and inability to meet real-time query requests. The user usage scenarios are limited and the user experience is poor. Summary of the Invention

[0005] In view of this, the embodiments of the present disclosure provide a data processing method, device and data processing system for a concurrent connector, which can solve the problems of low reading and writing efficiency between presto and GP, high reading and writing pressure on the master node, easy occurrence of data reading and writing failures, master node crashes, and thus GP blockage and unstable operation; inability to perform operations such as deletion, low data processing efficiency, inability to meet real-time query requests, limited user usage scenarios, and poor user experience.

[0006] To achieve the above objectives, according to one aspect of the present disclosure, a data processing method for a concurrent connector is provided, comprising:

[0007] Receive one or more data read requests; wherein the data read request includes the data to be read, the data source identifier of the data source, and the engine identifier of the computing engine;

[0008] Searching the local concurrency configuration table for a data source address corresponding to the data source identifier and an engine address corresponding to the engine identifier, and accessing a data source master node corresponding to the data source address;

[0009] receiving first node information allocated by the data source master node to the data to be read, and accessing the engine master node corresponding to the engine address to obtain second node information of the computing engine;

[0010] The data source and the computing engine are connected by using the first node information and the second node information, so that the computing engine concurrently collects the data to be read from the data source.

[0011] According to another aspect of the present disclosure, there is provided a data processing device for a concurrent connector, comprising:

[0012] A receiving module, configured to receive one or more data read requests; wherein the data read request includes the data to be read, a data source identifier of a data source, and an engine identifier of a computing engine;

[0013] An access module, configured to search a local concurrent configuration table for a data source address corresponding to the data source identifier and an engine address corresponding to the engine identifier, and access a data source master node corresponding to the data source address;

[0014] an acquisition module, configured to receive first node information allocated by the data source master node to the data to be read, and access the engine master node corresponding to the engine address to acquire second node information of the computing engine;

[0015] A docking module is used to dock the data source and the computing engine using the first node information and the second node information, so that the computing engine concurrently collects the data to be read from the data source.

[0016] According to another aspect of the present disclosure, a data processing system is provided, comprising: a computing engine, a data source, and a concurrent connector, wherein the concurrent connector comprises a data processing device of the concurrent connector, wherein:

[0017] The engine master node of the computing engine receives the data query request sent by the terminal, parses the data query request, generates the data read request and sends it to the concurrent connector, wherein the data read request includes the data to be read, the data source identifier of the data source and the engine identifier;

[0018] In response to the data read request, the concurrent connector searches a local concurrent configuration table for a data source address corresponding to the data source identifier and an engine address corresponding to the engine identifier, and accesses a data source master node corresponding to the data source address;

[0019] The data source master node of the data source allocates first node information to the data to be read;

[0020] The concurrent connector receives the first node information, accesses the engine master node corresponding to the engine address, obtains the second node information of the computing engine, and uses the first node information and the second node information to connect the data source and the computing engine, so that the computing engine concurrently collects the data to be read from the data source.

[0021] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0022] processor; and

[0023] Memory for storing programs,

[0024] The program includes instructions, which, when executed by the processor, cause the processor to execute the data processing method of the concurrent connector.

[0025] According to another aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the data processing method of the concurrent connector.

[0026] One or more technical solutions provided in the embodiments of the present application connect the working nodes of the computing engine and the segmented nodes of the data source through the concurrent channel of the concurrent connector, and embed the modified deletion operator in the concurrent connector, so that the computing engine can concurrently read data from the data source, concurrently write data to the data source, and delete data in the data source, greatly improving the data reading and writing efficiency, expanding the data computing and processing scenarios, and meeting the technical effects of diverse business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A flowchart of a data processing method of a concurrent connector according to an exemplary embodiment of the present disclosure is shown;

[0029] Figure 2 A schematic diagram illustrating a data processing system according to an exemplary embodiment of the present disclosure is shown;

[0030] Figure 3 A flowchart illustrating a method for determining a data read request according to an exemplary embodiment of the present disclosure is shown;

[0031] Figure 4A flowchart of a method for accessing a data source master node according to an exemplary embodiment of the present disclosure is shown;

[0032] Figure 5 A schematic diagram illustrating the connection between a data source, a computing engine, and a concurrent connector according to an exemplary embodiment of the present disclosure is shown;

[0033] Figure 6 A flowchart showing a method for generating first node information according to an exemplary embodiment of the present disclosure is shown;

[0034] Figure 7 A flowchart of a method for concurrently reading data to be read according to an exemplary embodiment of the present disclosure is shown;

[0035] Figure 8 A flowchart of a concurrent writing method for data to be written according to a first exemplary embodiment of the present disclosure is shown;

[0036] Figure 9 A flowchart of a concurrent writing method for data to be written according to a second exemplary embodiment of the present disclosure is shown;

[0037] Figure 10 A flowchart of a method for deleting data to be deleted according to an exemplary embodiment of the present disclosure is shown;

[0038] Figure 11 A schematic block diagram of a data processing device of a concurrent connector according to an exemplary embodiment of the present disclosure is shown;

[0039] Figure 12 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0040] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0041] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0042] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "in the embodiments of the present disclosure" means "at least one embodiment". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0044] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0045] Presto: is an open source big data distributed SQL query engine suitable for interactive analytical queries. It supports a wide range of data sources and provides interfaces for developing data source connectors.

[0046] GreenPlum: GP for short, is a distributed columnar database management system that uses a massively parallel processing architecture to achieve high-performance data query and analysis of large-scale data sets, supporting columnar storage and batch operations.

[0047] PostgreSQL: A free software object-relational database management system.

[0048] JDBC: Java Database Connectivity, also known as Java database connection, is an application program interface in the Java language used to standardize the rules for client programs to access the database, providing methods such as querying and updating data in the database.

[0049] When Presto queries require GP data, Presto connects to the GP master node through PostgreSQL-JDBC to collect the accumulated data of the GP master node. The accumulated data of the master node comes from multiple data tables in different storage locations in GP. When the data volume is large, reading data from the master node one by one is very slow and the data reading efficiency is relatively low.

[0050] When Presto writes data to GP, Presto connects to the GP master node through PostgreSQL-JDBC and writes data to the GP master node one by one. When the amount of data is large, writing data to the master node one by one is not only slow and inefficient, but also puts great pressure on the computing resources of the GP master node, resulting in excessive fluctuations in the master node connection and a high risk of crash. This in turn leads to unstable GP operation, causing write failures and forced interruption of the connection between GP and other engines or data sources, and sharply increasing the risk of big data processing.

[0051] Through the data processing method of the concurrent connector disclosed in the present invention, the constructed concurrent connector is used to connect each segment node in the data source with each working node of the computing engine through multiple concurrent channels in the concurrent connector, thereby realizing concurrent reading and writing of data, improving reading and writing efficiency and writing stability.

[0052] Furthermore, the delete operator is added to the concurrent connector to directly delete the data in the GP partition, which expands the application scenarios of the connector, can meet various actual usage needs, and improve the user experience.

[0053] Aspects of the present disclosure are described below with reference to the accompanying drawings.

[0054] Figure 1 A flowchart of a data processing method of a concurrent connector according to an exemplary embodiment of the present disclosure is shown. Figure 1 As shown, the data processing method of the concurrent connector disclosed in the present invention includes the following steps:

[0055] In the embodiment of the present disclosure, the data processing method of the concurrent connector of the present disclosure is applied to the concurrent connector, such as Figure 2 As shown, the data processing system of the present invention includes a computing engine, a data source and a concurrent connector. The concurrent connector is used to establish connectivity between the computing engine and the data source, realize data transmission and processing between the two, so that the computing engine can perform concurrent reading, writing, deletion and other operations on the data of the data source, including concurrently reading data from the data source, concurrently writing data to the data source, deleting data in the data source, etc., so as to improve the data transmission efficiency between the two, thereby improving the computing and query efficiency of the computing engine, alleviating the resource pressure and data processing pressure of the data source master node, ensuring the stable operation of the data source, and improving the user experience.

[0056] Step S101: receiving one or more data reading requests; wherein the data reading request includes data to be read, a data source identifier of a data source, and an engine identifier of a computing engine.

[0057] In the embodiment of the present disclosure, the computing engine receives a data query request initiated by a user through a terminal, and the engine master node of the computing engine parses the query statement of the data query request, determines the data that needs to be read from the data source, generates a data read request including the data source identifier of the data source, the engine identifier of the computing engine, and the data to be read, and sends it to the concurrent connector, such as Figure 3 Alternatively, after the computing engine generates a data read request, it calls the concurrent connector for processing, as shown in Figure 3 shown.

[0058] Furthermore, the computing engine is presto, the data source is GP, presto receives and parses the data query request, generates a data read request and sends it to the concurrent connector; or, after presto generates the data read request, if the data source identifier of the data read request is GP, the concurrent connector is called for processing.

[0059] Step S102: searching a local concurrent configuration table for a data source address corresponding to the data source identifier and an engine address corresponding to the engine identifier, and accessing a data source master node corresponding to the data source address.

[0060] In an embodiment of the present disclosure, a concurrent connector pre-stores a concurrent configuration table containing configuration information for each data source and computing engine. This configuration information includes the data source identifier of each data source, the data source address of the access link to the data source master node, the data source account number, the data source password, etc. It also includes the engine identifier of each engine, the engine address of the access link to the engine master node, the engine account number, the engine password, etc. The concurrent connector accesses the data source master node of the data source based on the data source address corresponding to the data source identifier in the concurrent configuration table. The data source address and engine address may be IP addresses.

[0061] Further, if Figure 4 As shown, the method for accessing the data source master node of the present disclosure includes the following steps:

[0062] Step S401: linking the data source address corresponding to the data source master node identified by the data source.

[0063] In the embodiment of the present disclosure, for example, the concurrent configuration table of the concurrent connector stores the configuration information of the GP, including the data source identifier GP01 of the GP, the IP address of the data source master node of the GP01, the GP01 account and the GP01 password, and the IP address of the concurrent connector link and the data source master node.

[0064] Step S402: Log in to the console of the data source master node using the data source account and the data source password.

[0065] In the embodiment of the present disclosure, for example, the concurrent connector uses the GP01 account and GP01 password of GP01 in the configuration table to log in to the console of the data source master node.

[0066] Step S403, determining whether the response result of the console is verification success, if yes, go to step S404; if not, go to step S405.

[0067] Step S404: sending the data to be read to the data source master node.

[0068] In the disclosed embodiments, if the data source master node's console responds with a successful verification result, indicating that the data source account and password are correct, the concurrent server can communicate with the data source master node and send the data to be read to the data source master node. For example, if the GP01 data source master node successfully verifies the GP01 account and password, the concurrent connector is successfully logged in and can communicate with the GP01 data source master node and send the data to be read to the data source master node.

[0069] Step S405: Return the login result of failed verification to the computing engine.

[0070] In an embodiment of the present disclosure, when the response result of the console of the data source master node is verification failure, it indicates that at least one of the data source account and the data source password is incorrect and login is denied, that is, the concurrent server does not have access rights to the data source, and then the login result of verification failure is returned to the computing engine, so that the computing engine returns the login result to the terminal, and the developer locates and corrects the problem in the local concurrent configuration table of the concurrent server.

[0071] In the embodiment of the present disclosure, through the access method of the data source master node disclosed in the present disclosure, the concurrent server obtains the address, account, password and other related information of the data source from the locally stored concurrent configuration table, and uses the account and password to communicate with the data source master node to send the data to be read, so as to facilitate subsequent concurrent reading and writing according to the corresponding relationship between the nodes, which can ensure the security of data reading and prevent data leakage.

[0072] Step S103: receiving the first node information allocated by the data source master node to the data to be read, and accessing the engine master node corresponding to the engine address to obtain the second node information of the computing engine.

[0073] In the embodiment of the present disclosure, Figure 5As shown, the data source includes a data source master node and multiple segmentation nodes, and the computing engine includes an engine master node and multiple slave nodes. After the data source master node receives the data to be read, it allocates a matching number of segmentation nodes to the data to be read based on the distribution location and data volume of the data to be read, generates the first node information and sends it to the concurrent connector.

[0074] Furthermore, the engine master node's local engine configuration table stores second node information, which includes worker node information for each worker node of the computing engine, including worker node identifiers, worker node addresses, and the like. The concurrent connector utilizes the disclosed method for accessing the data source master node to link to the engine address corresponding to the engine master node identified by the engine identifier, logs into the console of the engine master node using the engine account and engine password, and, if the console of the engine master node responds with a successful authentication result, communicates with the computing engine master node and obtains the second node information from the engine configuration table of the engine master node.

[0075] Furthermore, when the response result of the console of the engine master node is verification failure, it indicates that at least one of the engine account and engine password is incorrect and login is denied, that is, the concurrent server does not have access rights to the computing engine. The computing engine returns the login result of verification failure to the terminal, which is located and corrected by the developer.

[0076] In the embodiment of the present disclosure, Figure 6 As shown, the method for generating the first node information disclosed in the present invention includes the following steps:

[0077] In an embodiment of the present disclosure, the method for generating the first node information of the present disclosure is executed by a data source master node.

[0078] Step S601: receiving the data to be read.

[0079] Step S602: Locate, according to the file identifier of the data to be read, a plurality of first target segment nodes to which the data to be read belongs in the data source.

[0080] Step S603: Generate the first node information using the segment node identifier and segment node address of the first target segment node.

[0081] In an embodiment of the present disclosure, through the method for generating the first node information of the present disclosure, the data source master node determines the first target segment node where the data to be read is located, and generates the first node information, which can facilitate the subsequent concurrent connector to build a concurrent channel between the data source and the computing engine to alleviate the pressure on the data source master node and improve data reading and writing efficiency.

[0082] Step S104: using the first node information and the second node information, connect the data source and the computing engine, so that the computing engine concurrently collects the data to be read from the data source.

[0083] In the embodiment of the present disclosure, Figure 5 As shown, the concurrent server includes multiple concurrent channels. According to the first node information and the second node information, the concurrent channel is used to connect the first target segment node of the data source and the working node of the computing engine, so that the computing engine concurrently collects the data to be read from the data source, as shown in FIG. Figure 7 As shown, the concurrent reading method of the data to be read disclosed in the present invention includes the following steps:

[0084] In an embodiment of the present disclosure, the concurrent reading method of the to-be-read data disclosed herein is executed by a concurrent connector.

[0085] Step S701: Determine the segment node identifier, segment node address, and first quantity of each first target segment node according to the first node information.

[0086] Step S702: Determine the working node identifiers, working node addresses, and second quantity of all working nodes according to the second node information.

[0087] Step S703: Select first target working nodes whose number is greater than or equal to the first number from the second number of working nodes.

[0088] In an embodiment of the present disclosure, the concurrent connector selects a first number of working nodes greater than or equal to the first target segment node from the working nodes of the computing engine as the first target working nodes.

[0089] Step S704, assign a first target working node to each of the first target segment nodes to obtain a first corresponding relationship; wherein the first corresponding relationship is the correspondence between the segment node identifier and segment node address of the first target segment node and the working node identifier and working node address of the first target working node.

[0090] In the embodiment of the present disclosure, alternatively, multiple first target working nodes may be allocated to a first target segment node to further improve data reading efficiency. Accordingly, in the first corresponding relationship, one segment node identifier corresponds to multiple working node identifiers.

[0091] Step S705: According to the first corresponding relationship, each of the first target segmentation nodes and the first target working node is connected through the concurrent channel, so that each of the target working nodes stores the first target data concurrently collected from the first target segmentation node to the computing engine through the corresponding concurrent channel.

[0092] In an embodiment of the present disclosure, a concurrent channel links the first target segmentation node and the first target working node according to the segmentation node address of a first target segmentation node and the working node address of a first target working node, thereby constructing a connecting channel between the first target segmentation node and a first target working node. Each first target working node concurrently collects the data to be read from the corresponding first target segmentation node and sends it to the engine master node, which is then aggregated into the first target data by the engine master node and stored in the computing engine.

[0093] Furthermore, or alternatively, each first target working node calculates the collected data to be read according to the data query request and sends it to the engine master node, which aggregates and calculates the data into the first target data and stores it in the calculation engine.

[0094] In the embodiment of the present disclosure, through the concurrent reading method of the data to be read disclosed in the present disclosure, the first target working node and the first target segmentation node are connected through a concurrent channel, so as to concurrently collect the data to be read, which can improve the data reading efficiency of the computing engine, thereby improving the data calculation and query efficiency, expanding the data processing scenarios, and meeting diverse business needs.

[0095] In the embodiments of the present disclosure, the production environments of computing engines are diverse, and the data processing requests are also diverse. For example, the computing engine writes data to the data source, or the computing engine deletes data from the data source. The concurrent connector of the present disclosure is based on native secondary development and can also realize concurrent writing or deletion of data, highly integrating actual data processing scenarios.

[0096] Furthermore, if Figure 8 As shown, the concurrent writing method for data to be written in the first embodiment of the present disclosure includes the following steps:

[0097] In an embodiment of the present disclosure, the concurrent writing method for data to be written of the first embodiment of the present disclosure is executed by a concurrent connector.

[0098] Step S801, receiving one or more data write requests; wherein the data write request includes the data to be written, the engine identifier of the computing engine, the data source identifier of the data source, and third node information; wherein the third node information is generated by the engine master node of the computing engine using the second target working node allocated to the data to be written.

[0099] In the disclosed embodiment, the engine master node of the computing engine allocates a second target work node for the data to be written based on the amount of data to be written, and obtains third node information, which includes the work node identifiers and work node addresses of the third number of allocated second target work nodes. The computing engine generates a data write request using the engine identifier, the data to be written, the data source identifier of the data source being written, and the third node information, and sends or calls a concurrent connector for processing.

[0100] Furthermore, the computing engine is Presto, the data source is GP, and Presto sends the data write request to the concurrent connector; or, when the data source identifier of the data write request is GP, Presto calls the concurrent connector for processing.

[0101] Step S802: search the local concurrent configuration table for the data source address corresponding to the data source identifier, access the data source master node corresponding to the data source address, and obtain the fourth node information in the data source configuration table of the data source master node.

[0102] In an embodiment of the present disclosure, the concurrent connector utilizes the access method of the data source main node disclosed in the present disclosure to communicate with the data source main node and obtain the fourth node information in the data source configuration table of the data source main node. The fourth node information includes the segment node information of each segment node of the data source, including the segment node identifier, segment node address, etc.

[0103] Step S803: Connect the data source and the computing engine according to the third node information and the fourth node information, so that the computing engine concurrently writes the data to be written into the data source.

[0104] Step S8031: Determine the working node identifier, working node address, and third quantity of each second target working node based on the third node information.

[0105] Step S8032: Determine the segment node identifiers, segment node addresses, and fourth quantities of all segment nodes according to the fourth node information.

[0106] Step S8033: Select second target segment nodes equal to the third number from all segment nodes of the fourth number.

[0107] Step S8034, assign a second target segmentation node to each of the second target working nodes to obtain a second corresponding relationship; wherein, the second corresponding relationship is the correspondence between the working node identifier and working node address of the second target working node and the segmentation node identifier and segmentation node address of the second target segmentation node.

[0108] In the embodiment of the present disclosure, a concurrent channel connects a second target segment node and a second target working node.

[0109] Step S8035: According to the second corresponding relationship, each of the second target working nodes and the second target segmentation node is connected through the concurrent channel, so that each of the target working nodes concurrently writes the data to be written into the data source through the corresponding concurrent channel.

[0110] In an embodiment of the present disclosure, a concurrent channel links a second target working node and a second target segmentation node based on a working node address of a second target working node and a segmentation node address of a second target segmentation node, and then constructs a connecting channel between a second target working node and a second target segmentation node, so that each second target working node concurrently writes the data to be written into the data source.

[0111] Further, if Figure 9 As shown, the concurrent writing method for data to be written in the second embodiment of the present disclosure includes the following steps:

[0112] In an embodiment of the present disclosure, the concurrent writing method of the data to be written in the second embodiment of the present disclosure is executed by the data source and the computing engine.

[0113] Step S901: The data source master node creates a temporary table and a target write table, and synchronizes the temporary table to each of the second target segment nodes; wherein the primary key of the temporary table is a random distribution key.

[0114] Step S902: Each of the second target working nodes concurrently writes the data to be written into the temporary table of the corresponding second target segment node through each of the concurrent channels.

[0115] In the embodiment of the present disclosure, for example, the concurrent channel may implement the insert into function to write the second target working node to the temporary table.

[0116] Step S903: the data source master node aggregates the temporary tables of the second target segment nodes into the target write table to obtain the second target data.

[0117] In an embodiment of the present disclosure, through the concurrent writing method of the data to be written of the second embodiment of the present disclosure, the working node of the computing engine concurrently writes data to the temporary table of the segmentation node of the data source through the concurrent channel of the concurrent connection, and then the data source main node summarizes the second target data, which can greatly improve the data writing efficiency, alleviate the resource pressure and computing principles of the data source main node, and meet diverse computing needs and computing scenarios.

[0118] In the embodiments of the present disclosure, the concurrent writing method of the data to be written of the first embodiment of the present disclosure avoids the traditional behavior of centrally writing data to the main node of the data source, and realizes the concurrent writing of multiple segmented nodes of the data source. In addition, a random distribution key temporary table is used to prevent data writing conflicts, ensure data writing integrity, improve writing efficiency and improve writing accuracy, expand data processing scenarios, and meet various business needs.

[0119] In the embodiment of the present disclosure, Figure 10 As shown, the method for deleting data to be deleted disclosed in the present invention includes the following steps:

[0120] In an embodiment of the present disclosure, the method for deleting data to be deleted of the present disclosure is executed by a concurrent connector. By adding a deletion operator in the concurrent connector, the predicate push-down logic of the deletion operator is accurate to the data source identifier of the data source, so that the concurrent connector can identify various data sources, locate the position of the data to be deleted in the data source and delete it, thereby realizing the computing engine's deletion logic for the data in the data source.

[0121] Step S1001: Receive one or more data deletion requests; wherein the data deletion request includes the data to be deleted and the data source identifier of the data source.

[0122] Step S1002: access the data source master node corresponding to the data source identifier, send the data to be deleted to the data source master node, and receive the deletion verification result returned by the data source master node.

[0123] In the disclosed embodiments, to ensure the integrity and reliability of the data source, only restricted deletion of the data source by concurrent connectors is supported. Specifically, because deleting full table data is a dangerous behavior that threatens data integrity, full table deletion by concurrent connectors is restricted. Upon receiving data to be deleted from the concurrent connector, the data source master node determines whether the data to be deleted is full table data and generates a deletion verification result.

[0124] Furthermore, the deletion verification result includes allowing deletion and refusing deletion. The deletion verification result of allowing deletion also includes the segment node identifiers of one or more third target segment nodes to which the data to be deleted belongs and the target primary key segments of each third target segment node.

[0125] Step S1003, determine whether the deletion verification result is deletion allowed, if yes, go to step S1004; if not, go to step S1005.

[0126] Step S1004 : executing a deletion operation on the data to be deleted according to the segment node identifier and the target primary key segment of the third target segment node in the deletion verification result.

[0127] In an embodiment of the present disclosure, when the deletion verification result is that deletion is allowed, the concurrent connector locates the data to be deleted based on the node identifier and target primary key segment of the third target segment node, and performs a deletion operation on the data to be deleted.

[0128] Step S1005: reject the data deletion request.

[0129] In an embodiment of the present disclosure, when the deletion verification result is deletion rejection, the deletion operation of the concurrent connector is intercepted by the data source master node to ensure the integrity and reliability of the data source.

[0130] In the embodiments of the present disclosure, through the deletion method of the data to be deleted disclosed in the present disclosure, the improved deletion operator in the concurrent connector is utilized to implement the deletion operation of the computing engine on the data source, which greatly expands the application scenarios of data processing and can meet diverse business processing needs.

[0131] In the embodiments of the present disclosure, through the data processing method of the concurrent connector of the present disclosure, a newly added concurrent connector is used to connect the computing engine and the data source, and concurrent reading, writing and deletion between the computing engine and the data source are realized. This can greatly improve the reading and writing efficiency of the computing engine in reading data from the data source and writing data to the data source, and make up for the deletion defect of the computing engine on the data source. It has been verified that the reading and writing efficiency can be improved by more than 10 times, which greatly alleviates the resource pressure and computing pressure of the main node of the data source, improves the stability of data reading, writing, deletion and storage, expands the actual data processing application scenarios, and improves the user experience.

[0132] Figure 11 is a schematic diagram of the main modules of the data processing device of the concurrent connector according to an embodiment of the present disclosure, such as Figure 11 As shown, the data processing device 1100 of the concurrent connector of the present disclosure includes:

[0133] The receiving module 1101 is configured to receive one or more data reading requests, wherein the data reading request includes the data to be read, the data source identifier of the data source, and the engine identifier of the computing engine.

[0134] Alternatively, the receiving module 1101 is used to receive one or more data write requests; wherein the data write request includes the data to be written, the data source identifier of the data source and the third node information; wherein the third node information is generated by the engine master node of the computing engine using the second target working node allocated to the data to be written.

[0135] Alternatively, the receiving module 1101 is configured to receive one or more data deletion requests, wherein the data deletion request includes the data to be deleted and a data source identifier of the data source.

[0136] The access module 1102 is configured to search the local concurrency configuration table for the data source address corresponding to the data source identifier and the engine address corresponding to the engine identifier, and access the data source master node corresponding to the data source address.

[0137] Alternatively, the access module 1102 is used to search the data source address corresponding to the data source identifier from the local concurrent configuration table, access the data source master node corresponding to the data source address, and obtain the fourth node information in the data source configuration table of the data source master node.

[0138] Alternatively, the access module 1102 is configured to send the data to be deleted to the data source master node corresponding to the data source identifier, and receive a deletion verification result returned by the data source master node.

[0139] The acquisition module 1103 is configured to receive the first node information allocated by the data source master node to the data to be read, and access the engine master node corresponding to the engine address to acquire the second node information of the computing engine.

[0140] The docking module 1104 is configured to dock the data source and the computing engine using the first node information and the second node information, so that the computing engine concurrently collects the data to be read from the data source.

[0141] Alternatively, the docking module 1104 is configured to dock the data source and the computing engine according to the third node information and the fourth node information, so that the computing engine concurrently writes the data to be written into the data source.

[0142] Alternatively, the docking module 1104 is configured to, when the deletion verification result indicates that deletion is allowed, perform a deletion operation on the data to be deleted according to the segment node identifier and target primary key segment of the third target segment node of the deletion verification result.

[0143] In the embodiment of the present disclosure, the interaction process of the data processing system includes:

[0144] The engine master node of the computing engine receives the data query request sent by the terminal, parses the data query request, generates the data read request and sends it to the concurrent connector, wherein the data read request includes the data to be read, the data source identifier of the data source and the engine identifier;

[0145] In response to the data read request, the concurrent connector searches a local concurrent configuration table for a data source address corresponding to the data source identifier and an engine address corresponding to the engine identifier, and accesses a data source master node corresponding to the data source address;

[0146] The data source master node of the data source allocates first node information to the data to be read;

[0147] The concurrent connector receives the first node information, accesses the engine master node corresponding to the engine address, obtains the second node information of the computing engine, and uses the first node information and the second node information to connect the data source and the computing engine, so that the computing engine concurrently collects the data to be read from the data source.

[0148] Furthermore, the data source master node of the data source allocates first node information to the data to be read, including:

[0149] The data source master node receives the data to be read, and locates a plurality of first target segment nodes to which the data to be read belongs in the data source according to a file identifier of the data to be read.

[0150] The data source master node generates the first node information using the segment node identifier and segment node address of the first target segment node.

[0151] In an embodiment of the present disclosure, alternatively, the interaction process of the data processing system includes:

[0152] The engine master node of the computing engine allocates a second target working node for the data to be written, obtains third node information, generates a data write request using the data to be written, the data source identifier of the data source, and the third node information, and sends the data write request to the concurrent connector;

[0153] In response to the data write request, the concurrent connector searches for a data source address corresponding to the data source identifier from a local concurrent configuration table, accesses a data source master node corresponding to the data source address, and obtains fourth node information from the data source configuration table of the data source master node;

[0154] The concurrent connector connects the data source and the computing engine according to the third node information and the fourth node information, so that the computing engine concurrently writes the to-be-written data into the data source.

[0155] Furthermore, the concurrent connector connects the data source and the computing engine according to the third node information and the fourth node information, so that the computing engine concurrently writes the to-be-written data into the data source, including:

[0156] The concurrent connector determines the working node identifier, working node address and third quantity of each second target working node according to the third node information;

[0157] The concurrent connector determines the segment node identifiers, segment node addresses, and fourth quantities of all segment nodes according to the fourth node information;

[0158] The concurrent connector selects second target segment nodes equal to the third number from the fourth number of all segment nodes;

[0159] The concurrent connector assigns a second target segment node to each of the second target working nodes to obtain a second corresponding relationship; wherein the second corresponding relationship is a corresponding relationship between the working node identifier and working node address of the second target working node and the segment node identifier and segment node address of the second target segment node;

[0160] According to the second corresponding relationship, the concurrent connector connects each of the second target working nodes and the second target segment nodes through the concurrent channel, so that each of the target working nodes concurrently writes the data to be written into the data source through the corresponding concurrent channel.

[0161] Furthermore, each of the target working nodes concurrently writes the to-be-written data into the data source through a corresponding concurrent channel, including:

[0162] The data source master node creates a temporary table and a target write table, and synchronizes the temporary table to each of the second target segment nodes; wherein the primary key of the temporary table is a random distribution key;

[0163] Each of the second target working nodes concurrently writes the data to be written into a temporary table of the corresponding second target segment node through each of the concurrent channels;

[0164] The data source master node aggregates the temporary tables of each second target segment node into the target write table to obtain second target data.

[0165] In an embodiment of the present disclosure, alternatively, the interaction process of the data processing system includes:

[0166] The concurrent connector receives one or more data deletion requests; wherein the data deletion request includes the data to be deleted and the data source identifier of the data source;

[0167] The concurrent connector accesses the data source master node corresponding to the data source identifier, and sends the data to be deleted to the data source master node;

[0168] The data source master node determines whether the data to be deleted is full-table data, generates a deletion verification result, and sends it to the concurrent connector;

[0169] When the deletion verification result indicates that deletion is allowed, the concurrent connector executes a deletion operation on the data to be deleted according to the segment node identifier and the target primary key segment of the third target segment node of the deletion verification result.

[0170] The exemplary embodiments of the present disclosure further provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being configured to cause the electronic device to perform a method according to an exemplary embodiment of the present disclosure when executed by the at least one processor.

[0171] Exemplary embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present disclosure.

[0172] Exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to cause the computer to perform the method according to the embodiment of the present disclosure.

[0173] refer to Figure 12 , a block diagram of an electronic device 1200 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0174] like Figure 12 As shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0175] Multiple components within electronic device 1200 are connected to I / O interface 1205, including an input unit 1206, an output unit 1207, a storage unit 1208, and a communication unit 1209. Input unit 1206 can be any type of device capable of inputting information into electronic device 1200. Input unit 1206 can receive input numeric or character information and generate key input signals related to user settings and / or function control of the electronic device. Output unit 1207 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 1204 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 1209 allows electronic device 1200 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a Bluetooth™ device, a WiFi device, a Wi-Fi device, a cellular communication device, and / or the like.

[0176] The computing unit 1201 may be a variety of general and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above. For example, in some embodiments, Figure 1 、 Figures 3 to 4 、 Figures 6 to 10 The method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1200 via the ROM 1202 and / or the communication unit 1209. In some embodiments, the computing unit 1201 can be configured to execute the computer program by any other suitable means (for example, by means of firmware). Figure 1 、 Figures 3 to 4 、 Figures 6 to 10 method.

[0177] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0178] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0179] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0180] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0181] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0182] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A data processing method for a concurrent connector, characterized in that: include: Receive one or more data read requests; wherein the data read request includes the data to be read, the data source identifier of the data source, and the engine identifier of the computing engine; Searching the local concurrency configuration table for a data source address corresponding to the data source identifier and an engine address corresponding to the engine identifier, and accessing a data source master node corresponding to the data source address; receiving first node information allocated by the data source master node to the data to be read, and accessing the engine master node corresponding to the engine address to obtain second node information of the computing engine; Using the first node information and the second node information, the data source and the computing engine are connected, so that the computing engine concurrently collects the data to be read from the data source; Receive one or more data write requests; wherein the data write request includes the data to be written, a data source identifier of the data source, and third node information; wherein the third node information is generated by the engine master node of the computing engine using the second target working node allocated to the data to be written; Searching the local concurrent configuration table for a data source address corresponding to the data source identifier, accessing the data source master node corresponding to the data source address, and obtaining fourth node information from the data source configuration table of the data source master node; The data source and the computing engine are connected according to the third node information and the fourth node information, so that the computing engine concurrently writes the data to be written into the data source.

2. The data processing method according to claim 1, wherein: The utilizing the first node information and the second node information to connect the data source and the computing engine so that the computing engine concurrently collects the to-be-read data from the data source includes: Determine a first target segmentation node of the data source and a first target working node of the computing engine according to the first node information and the second node information; Allocating a first target working node to each of the first target segment nodes to obtain a first corresponding relationship; wherein the first corresponding relationship is a corresponding relationship between the segment node identifier and segment node address of the first target segment node and the working node identifier and working node address of the first target working node; According to the first corresponding relationship, each of the first target segmentation nodes and the first target working node is connected through a concurrent channel, so that each of the target working nodes stores the first target data concurrently collected from the first target segmentation node to the computing engine through the corresponding concurrent channel.

3. The data processing method according to claim 2, wherein: The determining, according to the first node information and the second node information, a first target segmentation node of the data source and a first target working node of the computing engine includes: Determine, according to the first node information, a segment node identifier, a segment node address, and a first quantity of each first target segment node; Determine the working node identifiers, working node addresses, and second quantity of all working nodes according to the second node information; Select first target working nodes that are greater than or equal to the first number from the second number of working nodes.

4. The data processing method according to claim 1, wherein: The concurrency configuration table also includes a data source account and a data source password; accessing the data source master node corresponding to the data source address includes: Linking the data source address corresponding to the data source master node identified by the data source; Log in to the console of the data source master node using the data source account and the data source password; When the response result of the console is that the verification is successful, the data to be read is sent to the data source master node.

5. The data processing method according to claim 1, wherein: The step of connecting the data source and the computing engine according to the third node information and the fourth node information so that the computing engine concurrently writes the to-be-written data into the data source includes: Determine, according to the third node information, the working node identifier, the working node address, and the third quantity of each second target working node; Determine the segment node identifiers, segment node addresses, and fourth quantities of all segment nodes according to the fourth node information; Selecting second target segmentation nodes equal to the third number from all segmentation nodes of the fourth number, and assigning a second target segmentation node to each of the second target working nodes, to obtain a second corresponding relationship; wherein the second corresponding relationship is a corresponding relationship between the working node identifier and working node address of the second target working node and the segmentation node identifier and segmentation node address of the second target segmentation node; According to the second corresponding relationship, each of the second target working nodes and the second target segmentation node is connected through a concurrent channel, so that each of the target working nodes concurrently writes the data to be written into the data source through the corresponding concurrent channel.

6. The data processing method according to claim 1, wherein: include: Receiving one or more data deletion requests; wherein the data deletion request includes the data to be deleted and the data source identifier of the data source; Accessing the data source master node corresponding to the data source identifier, sending the data to be deleted to the data source master node, and receiving a deletion verification result returned by the data source master node; In the case that the deletion verification result is that deletion is allowed, a deletion operation is performed on the data to be deleted according to the segment node identifier and the target primary key segment of the third target segment node of the deletion verification result.

7. The data processing method according to any one of claims 1 to 6, characterized in that: The computing engine is Presto, and the data source is GreenPlum.

8. A data processing device for a concurrent connector, characterized in that: include: A receiving module, configured to receive one or more data read requests; wherein the data read request includes the data to be read, a data source identifier of a data source, and an engine identifier of a computing engine; An access module, configured to search a local concurrent configuration table for a data source address corresponding to the data source identifier and an engine address corresponding to the engine identifier, and access a data source master node corresponding to the data source address; an acquisition module, configured to receive first node information allocated by the data source master node to the data to be read, and access the engine master node corresponding to the engine address to acquire second node information of the computing engine; a docking module, configured to dock the data source and the computing engine using the first node information and the second node information, so that the computing engine concurrently collects the data to be read from the data source; The receiving module is further configured to receive one or more data write requests; wherein the data write request includes the data to be written, a data source identifier of a data source, and third node information; wherein the third node information is generated by the engine master node of the computing engine using the second target working node allocated to the data to be written; The access module is further configured to search a local concurrent configuration table for a data source address corresponding to the data source identifier, and access a data source master node corresponding to the data source address; The acquisition module is further configured to acquire the fourth node information in the data source configuration table of the data source master node; The docking module is further configured to dock the data source and the computing engine according to the third node information and the fourth node information, so that the computing engine concurrently writes the data to be written into the data source.

9. A data processing system, characterized in that: include: A computing engine, a data source, and a concurrent connector, wherein the concurrent connector includes a data processing device of the concurrent connector according to claim 8, wherein: The engine master node of the computing engine receives the data query request sent by the terminal, parses the data query request, generates the data read request and sends it to the concurrent connector, wherein the data read request includes the data to be read, the data source identifier of the data source and the engine identifier; In response to the data read request, the concurrent connector searches a local concurrent configuration table for a data source address corresponding to the data source identifier and an engine address corresponding to the engine identifier, and accesses a data source master node corresponding to the data source address; The data source master node of the data source allocates first node information to the data to be read; The concurrent connector receives the first node information, accesses the engine master node corresponding to the engine address, obtains the second node information of the computing engine, and connects the data source and the computing engine using the first node information and the second node information, so that the computing engine concurrently collects the data to be read from the data source; The data processing system further includes: The engine master node of the computing engine allocates a second target working node for the data to be written, obtains third node information, generates a data write request using the data to be written, the data source identifier of the data source, and the third node information, and sends the data write request to the concurrent connector; In response to the data write request, the concurrent connector searches for a data source address corresponding to the data source identifier from a local concurrent configuration table, accesses a data source master node corresponding to the data source address, and obtains fourth node information from the data source configuration table of the data source master node; The concurrent connector connects the data source and the computing engine according to the third node information and the fourth node information, so that the computing engine concurrently writes the to-be-written data into the data source.

10. The data processing system according to claim 9, wherein: The data source master node of the data source allocates first node information to the data to be read, including: The data source master node receives the data to be read, and locates a plurality of first target segment nodes to which the data to be read belongs in the data source according to the file identifier of the data to be read; The data source master node generates the first node information using the segment node identifier and segment node address of the first target segment node.

11. The data processing system according to claim 9, wherein: The concurrent connector connects the data source and the computing engine according to the third node information and the fourth node information, so that the computing engine concurrently writes the to-be-written data into the data source, including: The concurrent connector determines the working node identifier, working node address and third quantity of each second target working node according to the third node information; The concurrent connector determines the segment node identifiers, segment node addresses, and fourth quantities of all segment nodes according to the fourth node information; The concurrent connector selects second target segment nodes equal to the third number from the fourth number of all segment nodes; The concurrent connector assigns a second target segment node to each of the second target working nodes to obtain a second corresponding relationship; wherein the second corresponding relationship is a corresponding relationship between the working node identifier and working node address of the second target working node and the segment node identifier and segment node address of the second target segment node; According to the second corresponding relationship, the concurrent connector connects each of the second target working nodes and the second target segment nodes through a concurrent channel, so that each of the target working nodes concurrently writes the data to be written into the data source through the corresponding concurrent channel.

12. The data processing system according to claim 11, wherein: Each of the target working nodes concurrently writes the data to be written into the data source through a corresponding concurrent channel, including: The data source master node creates a temporary table and a target write table, and synchronizes the temporary table to each of the second target segment nodes; wherein the primary key of the temporary table is a random distribution key; Each of the second target working nodes concurrently writes the data to be written into a temporary table of the corresponding second target segment node through each of the concurrent channels; The data source master node aggregates the temporary tables of each second target segment node into the target write table to obtain second target data.

13. An electronic device comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, enable the processor to execute the data processing method of the concurrent connector according to any one of claims 1 to 7.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the data processing method of the concurrent connector according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Data lake-based data management system and method

    CN112307122A

  • Request response method, device and system and readable storage medium

    CN114116223A