Data processing method, device, equipment, medium and program product

Abstract

The present disclosure provides a data processing method, apparatus, device, medium, and program product for a data processing system. The method comprises: receiving a data processing request, wherein the data processing request includes a read instruction or a write instruction for data to be processed; sending a processing policy request to a scheduling center based on the data processing request; receiving the data processing policy sent by the scheduling center, wherein the data processing policy includes resource scheduling information for executing the data processing request; and based on the resource scheduling information, invoking corresponding resources to execute the read instruction or the write instruction for the data to be processed. The present disclosure also provides a data processing method, apparatus, device, medium, and program product for a scheduling center.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and more particularly to a data processing method, apparatus, device, medium, and program product. Background Art

[0002] With the advancement of technology, the amount of data generated in our daily lives and work is growing significantly. To meet this demand, data computing applications are currently being used to process this data. Different data processing requirements correspond to different data processing volumes. For example, for larger volumes of data to be processed, data computing applications will increase computing power to meet these needs.

[0003] In related technologies, there is an access bottleneck in reading or writing the data to be processed. As the computing power of data computing applications increases to a certain level, the reading or writing speed will not be improved due to the limitation of the access bottleneck, resulting in no improvement in data processing efficiency and failure to meet user needs. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a data processing method, apparatus, device, medium and program product that can dynamically schedule resources based on data processing requests to improve data processing efficiency.

[0005] One aspect of an embodiment of the present disclosure provides a data processing method for a data processing system, comprising: receiving a data processing request, wherein the data processing request includes a read instruction or a write instruction for data to be processed; sending a processing policy request to a scheduling center based on the data processing request, wherein the scheduling center is configured to obtain data processing information of the data processing system in response to the processing policy request, and generate a data processing policy based on the data processing information; receiving the data processing policy sent by the scheduling center, wherein the data processing policy includes resource scheduling information for executing the data processing request; and based on the resource scheduling information, calling corresponding resources to execute the read instruction or the write instruction for the data to be processed.

[0006] According to an embodiment of the present disclosure, when the data to be processed is stored in a cloud storage system, it includes: creating a local storage warehouse of the cloud storage system; wherein, executing a read instruction or a write instruction for the data to be processed includes: writing the data to be processed into the local storage warehouse; or reading the data to be processed from the local storage warehouse, wherein, the data to be processed is obtained based on the cloud storage path of the cloud storage system.

[0007] According to an embodiment of the present disclosure, the data processing system includes a master node and N slave nodes corresponding to the master node. Calling the corresponding resources based on the resource scheduling information includes: calling M slave nodes among the N slave nodes through the master node based on the resource scheduling information, wherein N and M are integers greater than or equal to 1 respectively, and M is less than or equal to N.

[0008] According to an embodiment of the present disclosure, the data processing system also includes a client, and calling M slave nodes among the N slave nodes through the master node includes: the client sends the resource scheduling information to the master node, wherein the client is used to receive the data processing strategy, and the resource scheduling information includes the pointing information of the M slave nodes; the master node sends the available status of the M slave nodes to the client, wherein the master node determines the corresponding available status based on the pointing information of the M slave nodes; and the client calls the M slave nodes based on the available status of the M slave nodes.

[0009] According to an embodiment of the present disclosure, the data processing system also includes a client, and calling M slave nodes among the N slave nodes through the master node includes: the master node sends the pointing information of the M slave nodes to the client, wherein the master node is used to receive the data processing strategy, and the resource scheduling information includes the pointing information of the M slave nodes; the client calls the M slave nodes based on the pointing information of the M slave nodes.

[0010] Another aspect of an embodiment of the present disclosure provides a data processing method for a scheduling center, comprising: receiving a processing policy request sent by a data processing system, wherein the data processing system is configured to send the processing policy request in response to the received data processing request, and the data processing request includes a read instruction or a write instruction for the data to be processed; obtaining data processing information of the data processing system in response to the processing policy request; generating a data processing policy based on the data processing information, wherein the data processing policy includes resource scheduling information for executing the data processing request; and sending the data processing policy to the data processing system so that the data processing system calls corresponding resources to execute the read instruction or the write instruction for the data to be processed based on the resource scheduling information.

[0011] According to an embodiment of the present disclosure, before obtaining the data processing information of the data processing system in response to the processing policy request, it also includes: setting an information monitoring tool in the data processing system, wherein the information monitoring tool is used to monitor resource usage information of the data processing system in processing data; obtaining the data processing information of the data processing system in response to the processing policy request includes: receiving the resource usage information sent by the information monitoring tool.

[0012] According to an embodiment of the present disclosure, the information monitoring tool includes a first monitoring tool, the data processing system includes a master node and N slave nodes corresponding to the master node, and setting the information monitoring tool in the data processing system includes: setting the first monitoring tool in the N slave nodes; wherein, receiving the resource usage information sent by the information monitoring tool includes: receiving the resource usage information of the N slave nodes sent by the first monitoring tool.

[0013] According to an embodiment of the present disclosure, generating a data processing strategy based on the data processing information includes: determining M slave nodes based on resource usage information of at least one slave node among the N slave nodes, wherein N and M are integers greater than or equal to 1, and M is less than or equal to N; and generating the data processing strategy based on the directional information of the M slave nodes.

[0014] According to an embodiment of the present disclosure, the information monitoring tool includes a second monitoring tool, and the method includes: setting the second monitoring tool on the master node; receiving resource usage information of the master node sent by the second monitoring tool; and adjusting the resource configuration of the master node based on the resource usage information of the master node.

[0015] According to an embodiment of the present disclosure, the data processing system also includes a client, the information monitoring tool includes a third monitoring tool, and the method includes: setting the third monitoring tool on the client; receiving resource usage information of the client sent by the third monitoring tool; and adjusting the resource configuration of the client based on the resource usage information of the client.

[0016] Another aspect of an embodiment of the present disclosure provides a data processing device for a data processing system, comprising: a first receiving module for receiving a data processing request, wherein the data processing request includes a read instruction or a write instruction for the data to be processed; a first sending module for sending a processing policy request to a scheduling center based on the data processing request, wherein the scheduling center is configured to obtain data processing information of the data processing system in response to the processing policy request, and generate a data processing policy based on the data processing information; a second receiving module for receiving the data processing policy sent by the scheduling center, wherein the data processing policy includes resource scheduling information for executing the data processing request; and a resource scheduling module for calling corresponding resources to execute the read instruction or the write instruction for the data to be processed based on the resource scheduling information.

[0017] Another aspect of an embodiment of the present disclosure provides a data processing device for a scheduling center, including: a third receiving module, used to receive a processing policy request sent by a data processing system, wherein the data processing system is configured to send the processing policy request in response to the received data processing request, and the data processing request includes a read instruction or a write instruction for the data to be processed; an information acquisition module, used to obtain data processing information of the data processing system in response to the processing policy request; a policy generation module, used to generate a data processing policy based on the data processing information, wherein the data processing policy includes resource scheduling information for executing the data processing request; a second sending module, used to send the data processing policy to the data processing system, so that the data processing system calls corresponding resources to execute the read instruction or write instruction for the data to be processed based on the resource scheduling information.

[0018] Another aspect of an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0019] Another aspect of the embodiments of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method described above.

[0020] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0021] One or more of the above-described embodiments have the following beneficial effects: by introducing a scheduling center, after a received data processing request is sent to the scheduling center, the scheduling center can generate a data processing policy based on the information about the data to be processed and the specific read or write instructions in the data processing request, combined with the acquired data processing information. This enables the data processing system to dynamically schedule resources to adapt to the data processing request based on the resource scheduling information in the data processing policy, thus avoiding the access bottleneck in reading or writing the data to be processed that occurs in related technologies, thus failing to meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 Schematically illustrates an application scenario diagram suitable for implementing the data processing method according to an embodiment of the present disclosure;

[0024] Figure 2 Schematically shows an architecture diagram suitable for implementing a data processing method according to an embodiment of the present disclosure;

[0025] Figure 3 A flowchart schematically illustrates a data processing method for a data processing system according to an embodiment of the present disclosure;

[0026] Figure 4 Schematically shows a flow chart of calling M slave nodes according to an embodiment of the present disclosure;

[0027] Figure 5 Schematically shows a flowchart of calling M slave nodes according to another embodiment of the present disclosure;

[0028] Figure 6 A flowchart of a data processing method for a dispatching center according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 7 Schematically shows a flow chart of generating a data processing strategy according to an embodiment of the present disclosure;

[0030] Figure 8 The flowchart of adjusting the resource configuration of the master node according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 9 A flowchart of adjusting resource configuration of a client according to an embodiment of the present disclosure is schematically shown;

[0032] Figure 10 The following schematically shows a structural block diagram of a data processing device for a data processing system according to an embodiment of the present disclosure;

[0033] Figure 11 The following schematically shows a structural block diagram of a data processing device for a dispatching center according to an embodiment of the present disclosure;

[0034] Figure 12 The block diagram schematically shows an electronic device suitable for implementing the data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0036] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0037] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0038] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0039] In the technical solution disclosed herein, the acquisition, collection, storage, use, processing, transmission, provision, disclosure and application of data comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0040] Figure 1 An application scenario diagram suitable for implementing the data processing method according to an embodiment of the present disclosure is schematically shown. Figure 2 The following schematically shows an architecture diagram suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0041] like Figure 1 As shown, the application scenario 100 according to this embodiment may include an application layer, a computing layer, an access layer, and a storage layer. The application layer is used to provide functional usage for users and may include terminal devices 111, 112, 113, and a network 114. The computing layer is used to provide operational support for data computing applications and may include a first server 120. The access layer is used to provide access acceleration services between the computing layer and the storage layer and may include a second server 131, a third server 132, N fourth servers 1331-133N, and a fifth server 134. The storage layer may include a local database 141 and a cloud storage system 151.

[0042] The network 114 is a medium for providing a communication link between the terminal devices 111, 112, and 113 and the first server 120. The network 114 can include various connection types, such as wired or wireless communication links or fiber optic cables. It should be understood that the network 114 can also be provided between the application layer, computing layer, access layer, and storage layer, or between devices or servers at each layer.

[0043] Users can use terminal devices 111, 112, and 113 to interact with server 105 via network 114 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 111, 112, and 113, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0044] The terminal devices 111 , 112 , and 113 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0045] The first server 120, the second server 131, the third server 132, the N fourth servers 1331-133N, and the fifth server 134 can be servers or server clusters that provide various services. For example, the first server 120 can be a backend management server that supports websites browsed by users using terminal devices 111, 112, and 113 (for example only). The backend management server can analyze and process received data such as user requests, and provide feedback (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal device.

[0046] Reference Figure 1 and Figure 2First server 120 can be deployed with data computing applications, such as Spark distributed applications, Hive distributed applications, or Flink distributed applications. Second server 131, third server 132, and N fourth servers 1331-133N can be used to implement a data processing system. Second server 131 can be used to deploy clients, third server 132 can serve as a master node, and N fourth servers 1331-133N can serve as N slave nodes. Fifth server 134 can be deployed with a scheduling center.

[0047] N slave nodes can each create local storage repositories 1 to N of the cloud storage system locally on the server, where the underlying data of the local storage repositories and local database 141 can be stored in the local memory or disk managed by the slave nodes. The cloud storage system 151 can be implemented through a cloud server, or across cloud servers. It can also be a cloud distributed storage system based on a distributed storage network, such as the InterPlanetary File System (IPFS). IPFS can use a computer device as a storage node and replace domain-based addresses with content-based addresses. In other words, users are not looking for a specific address but for content stored somewhere. There is no need to verify the identity of the sender, but only the hash of the content. The cloud distributed storage system can also be a BitTorrent system, an Arweave system, or a Stoyj system.

[0048] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0049] The following will be based on Figure 1 The scene described by Figures 2 to 9 The data processing method of the embodiment of the present disclosure is described in detail.

[0050] Figure 3 The flowchart schematically shows a data processing method for a data processing system according to an embodiment of the present disclosure.

[0051] like Figure 3 As shown, the data processing method of this embodiment includes operations S310 to S340.

[0052] In operation S310 , a data processing request is received, where the data processing request includes a read instruction or a write instruction for data to be processed.

[0053] Reference Figure 1 and Figure 2When a user has a data processing requirement, he or she can create a data processing task by operating the front-end interface of a data computing application (such as Spark) through terminal devices 111, 112, and 113. Then, based on the created data processing task, the data computing application in the first server 120 sends a data processing request to the data processing system of the access layer.

[0054] For example, if a bank user needs to collect statistics on a day's loan transactions, the data processing request includes instructions to read the data. The data computing application then performs statistical analysis on the data. After obtaining the statistical results, the data computing application sends another data processing request, which may include instructions to write the statistical analysis data.

[0055] In operation S320 , a processing policy request is sent to the scheduling center based on the data processing request, wherein the scheduling center is configured to obtain data processing information of the data processing system in response to the processing policy request and generate a data processing policy based on the data processing information.

[0056] The processing strategy request allows the data processing system to request a specific resource scheduling strategy from the scheduling center to execute the data processing request. The processing strategy request may include the data size, data type, and read or write instructions for the data to be processed.

[0057] In operation S330 , a data processing policy sent by the scheduling center is received, where the data processing policy includes resource scheduling information for executing the data processing request.

[0058] The data processing strategy may include the above resource scheduling information, and may also include information such as data storage distribution, network routing, and execution tools. Resource scheduling information may include the resources, resource attributes, resource location, and resource status required to execute this data processing request. Figure 2 ,The above resources may refer to the number and performance of the ,running servers of the client, master node, or slave node.

[0059] In operation S340 , based on the resource scheduling information, corresponding resources are called to execute a read instruction or a write instruction for the data to be processed.

[0060] Without the aid of a scheduling center, a data processing system can schedule resources based on its own algorithms. However, the resource scheduling results of these algorithms are fixed and subject to access bottlenecks. By introducing a scheduling center, after receiving a data processing request and sending it to the scheduling center, the scheduling center can generate a data processing strategy based on the information about the data to be processed in the data processing request, as well as the specific read or write instructions, combined with the acquired data processing information. This allows the data processing system to dynamically schedule resources based on the resource scheduling information in the data processing strategy, avoiding the access bottlenecks in reading or writing data to be processed that often occur in related technologies, resulting in an inability to meet user needs.

[0061] According to an embodiment of the present disclosure, when the data to be processed is stored in a cloud storage system, a local storage repository of the cloud storage system is created. In operation S340, executing a read instruction or a write instruction for the data to be processed includes writing the data to be processed to the local storage repository or reading the data to be processed from the local storage repository, wherein the data to be processed is obtained based on the cloud storage path of the cloud storage system.

[0062] To maximize the value of data, computations such as Spark, Hive, and TensorFlow rely on underlying storage support. Accessing different storage systems requires application-specific configuration. By introducing a data processing system, data computing applications and storage systems can be connected, eliminating the need to manually switch between local and cloud storage systems and reducing the configuration process.

[0063] Reference Figure 1 The application layer, computing layer, and access layer can be set up in a unified data center to perform data processing tasks. The cloud storage system 151 in the storage layer can be a distributed storage network. For example, after data is generated on the edge side, it is stored on the edge side and uploaded to the cloud storage system. The edge side refers to the side that generates data, and the data center refers to the side that processes data.

[0064] The edge side uploads data to the cloud storage system by creating a local storage warehouse in the cloud storage system as a node in the cloud storage system. The edge side can directly access the data locally, and the data processing system can also remotely access the data of the node.

[0065] Currently, cloud storage systems only support cold data storage and lack support for real-time network transmission of data (especially hot data). In related technologies, with the explosive growth of data at the edge and the concentration of computing power in data centers, long-distance data transmission is necessary. If the data to be processed is stored in the cloud at the edge and users need to frequently process cloud data, the computing process must frequently access data from the cloud. However, long-distance data transmission slows down access speeds.

[0066] In scenarios such as 5G and the Internet of Things, a significant amount of computing power is concentrated in data centers, while a large amount of fragmented data is stored at the edge. This creates challenges in synergizing computing power and data. The purpose of a data processing system creating a local storage warehouse within a cloud storage system is to cache frequently accessed data from the cloud locally, essentially storing it in the local storage warehouse first. This improves access speed and reduces bandwidth costs. The local storage warehouse can also serve as a node in the cloud storage system, receiving both local and remote access from the data processing system.

[0067] Taking IPFS as an example, refer to Figure 2 , slave nodes 1~N can all create local storage warehouses as N nodes of IPFS. For example, when using slave node 1 to execute a write instruction, the data to be processed is written to local storage warehouse 1. The data will be added to local storage warehouse 1 in the form of a file. IPFS will generate a hash value based on the content of the file, and slave nodes 1~N can all use the hash value to access the file in local storage warehouse 1. In addition, when writing the data to be processed to the local storage warehouse 1, if no storage space is found, you can continue to request a storage strategy from the scheduling center, and the scheduling center will determine the data with lower access frequency, release this data, and write the data to be processed.

[0068] For example, when executing a read command from slave node 1, it can first query whether there is any data to be processed locally on slave node 1. If not, slave node 1 obtains the data to be processed based on the hash value of the data in IPFS (i.e., the cloud storage path) and caches it in the local storage warehouse. Finally, the data to be processed is read from the local storage warehouse.

[0069] According to the embodiments of the present disclosure, based on the characteristics of cloud storage systems, the local storage warehouse of the cloud storage system is encapsulated, and read and write operations are accelerated according to the storage and computing separation architecture. This converts the cold data of the cloud storage system into hot data, improves data access efficiency, and realizes accelerated computing of central computing power and edge data. Cold data refers to data with low read or write frequency, while hot data refers to data with high read or write frequency.

[0070] The relationship between cloud storage systems and data processing systems is similar to that between logistics and warehousing and express delivery companies: the former is responsible for logistics and warehousing, while the latter can correspond to different types of express delivery companies. For upper-level computing scenarios, "data acceleration services" can be built for different scenarios (i.e., different data computing applications).

[0071] According to an embodiment of the present disclosure, the data processing system includes a master node and N slave nodes corresponding to the master node. In operation S340, calling the corresponding resources based on resource scheduling information includes: based on the resource scheduling information, calling M slave nodes among the N slave nodes through the master node, where N and M are integers greater than or equal to 1, respectively, and M is less than or equal to N.

[0072] The master node is used to manage global metadata, which may include metadata of the storage system that manages the database (such as the file system node tree), data block metadata (such as data block location), and capacity data of the slave node (such as free or used space).

[0073] Slave nodes manage local storage resources, such as memory, SSDs, and HDDs. They employ a block storage structure to manage data in memory. All slave nodes synchronize their heartbeats with the master node at intervals specified in the configuration file. The master node manages events related to data processing requests, improving data access stability. Therefore, after receiving resource scheduling information, it can invoke M slave nodes based on the master node's management information.

[0074] Reference Figure 1 and Figure 2 When the computing power of the computing layer increases, it can provide better access to I / O for the data processing request, more slave nodes, and solve the data access bottleneck problem. When the computing power of the computing layer decreases, the number of cache nodes is reduced to reduce IO overhead. Figure 4 and Figure 5 Two optional implementations of calling M slave nodes through a master node are described.

[0075] Figure 4 The flowchart of calling M slave nodes according to an embodiment of the present disclosure is schematically shown.

[0076] like Figure 4 As shown, calling M slave nodes among N slave nodes through the master node may include operations S410 to S430.

[0077] In operation S410 , the client sends resource scheduling information to the master node, wherein the client is used to receive a data processing policy, and the resource scheduling information includes directional information of M slave nodes.

[0078] The client is used to communicate with the master node to execute read or write instructions for the data to be processed.

[0079] Reference Figure 2 Before executing operation S410, an optional implementation method is that the client can receive the data processing request, send a processing policy request to the scheduling center, and then receive the data processing policy. Another optional method is that the client receives the data processing request and then sends it to the master node, the master node sends the processing policy request to the scheduling center, and finally, the client receives the data processing policy.

[0080] In operation S420 , the master node sends the availability status of the M slave nodes to the client, wherein the master node determines the corresponding availability status based on the pointing information of the M slave nodes.

[0081] Available states can include free and unavailable, respectively indicating whether the data processing request can be executed. The master node can determine the available state by obtaining information about the storage data type, application scenario, free space, and execution task information of the M slave nodes based on local association information. Alternatively, the master node can send a verification request to the slave node and parse the slave node's response to confirm the state. For example, the pointing information includes information that can identify the M slave nodes, such as their identifiers, names, and IP addresses.

[0082] In operation S430, the client calls the M slave nodes based on the availability status of the M slave nodes. If some of the M slave nodes are available and some are unavailable, the client can call the available nodes.

[0083] After the client receives the data processing strategy, it can obtain the resource retrieval information therein, thereby determining the directional information of the M slave nodes. In some embodiments, the client can directly make calls based on the directional information of the M slave nodes. The purpose of performing operations S410 to S430 is to improve the success rate of the call by determining the available status in advance. For example, if a direct call is made, the call will fail if a slave node is unavailable. In order to complete the data processing request, the call instruction may be repeatedly issued, resulting in a waste of time and resources.

[0084] Figure 5 The flowchart of calling M slave nodes according to another embodiment of the present disclosure is schematically shown.

[0085] like Figure 5 As shown, calling M slave nodes among N slave nodes through the master node may include operations S510 to S520.

[0086] In operation S510 , the master node sends the pointing information of the M slave nodes to the client, wherein the master node is used to receive the data processing strategy, and the resource scheduling information includes the pointing information of the M slave nodes.

[0087] Before executing operation S510, after receiving the data processing strategy, the master node may determine the availability status of each node based on the pointing information of the M slave nodes, and return the pointing information to the client if each node is available.

[0088] In operation S520 , the client calls the M slave nodes based on the pointing information of the M slave nodes.

[0089] For example, calling M slave nodes means utilizing them to execute a read or write instruction. Specifically, when reading data to be processed, the M slave nodes can obtain the data from a cloud storage system or a local database. The client then reads the data from the M slave nodes and passes it to the data computing application.

[0090] Compared to the client receiving the data processing strategy in operations S410 through S430, the master node receiving the data processing strategy in operations S510 through S520 reduces the client's interaction with the dispatch center by one step. The master node handles the data interaction process for receiving the strategy, which reduces the client's operational burden. It should be noted that the choice of master node or client data processing strategy can be determined based on practical circumstances and is not specifically limited in this disclosure.

[0091] Figure 6 The flowchart of the data processing method for the dispatching center according to the embodiment of the present disclosure is schematically shown.

[0092] like Figure 6 As shown, the data processing method of this embodiment includes operations S610 to S640.

[0093] In operation S610 , a processing policy request sent by a data processing system is received, wherein the data processing system is configured to send the processing policy request in response to the received data processing request, and the data processing request includes a read instruction or a write instruction for data to be processed.

[0094] In operation S620 , data processing information of the data processing system is obtained in response to the processing policy request.

[0095] In operation S630 , a data processing policy is generated based on the data processing information, wherein the data processing policy includes resource scheduling information for executing the data processing request.

[0096] In operation S640 , the data processing policy is sent to the data processing system, so that the data processing system calls corresponding resources to execute a read instruction or a write instruction for the data to be processed based on the resource scheduling information.

[0097] Reference Figure 2,The scheduling center can obtain data such as tasks of the client, the master node, and N slave nodes, IO load, data distribution, network routing, and other data, and return the ,optimal scheduling strategy instructions (i.e., data processing ,strategy) to the data processing system through operations such as data ,analysis, baseline model, and intelligent strategies.

[0098] For example, a Spark application sends a data processing request to a client, which includes multiple data processing tasks, such as 5 SQL statements. The processing strategy request may include relevant information about the 5 SQL statements, such as the number of tasks being 5, the type being read, and the amount of data read by each SQL statement. The scheduling center can analyze and process the number of slave nodes called by the 5 SQL statements in combination with a preset model, and can determine which slave nodes will execute the corresponding data processing tasks in combination with the data processing information. Among them, the increase in the computing power of the Spark application means that more data processing tasks can be executed simultaneously, such as executing more SQL statements, or a larger amount of data to be processed. Dynamic scheduling by the scheduling center to adapt to data processing requests avoids the situation in related technologies where there is an access bottleneck in reading or writing data to be processed, which fails to meet user needs.

[0099] According to an embodiment of the present disclosure, before performing operation S620 to obtain data processing information of the data processing system in response to the processing policy request, the method further includes: providing an information monitoring tool in the data processing system, wherein the information monitoring tool is used to monitor resource usage information of data processed by the data processing system. Obtaining the data processing information of the data processing system in response to the processing policy request includes receiving the resource usage information sent by the information monitoring tool.

[0100] Information monitoring tools can be implemented using technologies such as code intrusion tracking, bytecode enhancement tracking, aspect technology, or interconnected data. Figure 2 The information monitoring tool in the client, master node and N slave nodes is the Agent probe embedding tool, which can be installed on each server in the form of a software development kit SDK to realize information monitoring.

[0101] According to an embodiment of the present disclosure, the information monitoring tool includes a first monitoring tool, and setting the information monitoring tool in the data processing system includes setting the first monitoring tool on N slave nodes. Receiving resource usage information sent by the information monitoring tool includes receiving resource usage information of the N slave nodes sent by the first monitoring tool.

[0102] The first monitoring tool includes an SDK adapted for slave nodes, which uses agent probes to monitor slave node information. Resource usage information for slave nodes may include information such as currently executed tasks, current availability, I / O load, local database idleness, local storage warehouse status, hardware CPU performance, space utilization, and data access counts. Based on each slave node's resource usage information and combined with data processing requests, the dispatch center can determine the M slave nodes to call.

[0103] Figure 7 The flowchart of generating a data processing strategy in operation S630 according to an embodiment of the present disclosure is schematically shown.

[0104] like Figure 7 As shown, generating a data processing strategy based on data processing information in operation S630 includes operations S710 to S720.

[0105] In operation S710 , M slave nodes are determined based on resource usage information of at least one slave node among N slave nodes, where N and M are integers greater than or equal to 1, and M is less than or equal to N.

[0106] In operation S720, a data processing strategy is generated based on the directional information of the M slave nodes.

[0107] Reference Figure 2 For example, the data processing request is used to process the loan data of the current day. The type of data to be processed obtained by the dispatch center is loan data. First, the management information is obtained from the master node to determine the slave nodes used to access the loan data, such as slave nodes 1 to 5. Then, the resource usage information of slave nodes 1 to 5 is obtained. Then, based on the current availability status of slave nodes 1 to 5, it can be determined that slave nodes 1 to 3 are available. Then, it is determined that two slave nodes can be assigned to process the loan data of the current day. Then, the space utilization rate of slave nodes 1 to 3 is continued to be sorted to determine slave nodes 1 and 2 with lower space utilization rate. Finally, a data processing strategy is generated based on the directional information of slave nodes 1 and 2, for example, the directional information is encapsulated as resource scheduling information. Among them, the resource scheduling information can also include the calling interface, calling instructions, server model and other information of slave nodes 1 and 2.

[0108] It should be noted that the above processing process and order of the dispatch center generating the data processing strategy are only examples, and the present disclosure is not limited thereto.

[0109] Figure 8 A flowchart for adjusting resource configuration of a master node according to an embodiment of the present disclosure is schematically shown.

[0110] like Figure 8As shown, the process of adjusting the resource configuration of the master node in this embodiment may include operations S810 to S830. Among them, the above-mentioned setting of the information monitoring tool in the data processing system includes operation S810. The above-mentioned receiving the resource usage information sent by the information monitoring tool includes operation S820.

[0111] In operation S810 , a second monitoring tool is set at a master node.

[0112] The second monitoring tool includes an SDK adapted to the master node, which uses an Agent probe to implement information monitoring of the master node.

[0113] In operation S820 , resource usage information of the master node sent by the second monitoring tool is received.

[0114] The resource usage information of the master node may include management information and the current performance status of the master node. The current performance status can be reflected by the CPU usage of the server. If the usage is too high, it indicates a performance shortage.

[0115] In operation S830 , resource configuration of the master node is adjusted based on the resource usage information of the master node.

[0116] Resource configuration refers to the server configuration supporting the master node. For example, during normal data processing, the master node can be supported by six servers in a server cluster. However, as the computing power of the data computing application increases, if the current performance of the single server cluster is limited, the resource configuration of the master node can be adjusted by increasing the number of server clusters to three (for example only). By dynamically adjusting the resource configuration of the master node, horizontal resource expansion of the master node is achieved, providing more flexible operational performance.

[0117] Figure 9 A flowchart for adjusting resource configuration of a client according to an embodiment of the present disclosure is schematically shown.

[0118] like Figure 9 As shown, the process of adjusting the resource configuration of the client in this embodiment may include operations S910 to S930. Among them, the above-mentioned setting of the information monitoring tool in the data processing system includes operation S910. The above-mentioned receiving the resource usage information sent by the information monitoring tool includes operation S920.

[0119] In operation S910 , a third monitoring tool is set at a client.

[0120] The third monitoring tool includes an SDK adapted to the client, which uses Agent probes to implement client information monitoring.

[0121] In operation S920 , resource usage information of the client sent by the third monitoring tool is received.

[0122] In operation S930 , resource configuration of the client is adjusted based on the resource usage information of the client.

[0123] Client resource usage information can include the client's current operational performance. For example, if a client receives a high number of data processing requests simultaneously, exceeding normal thresholds, the normal operation of a single server cluster may strain current operational performance. Therefore, the master node's resource allocation can be adjusted by increasing the number of server clusters to three (for example only). By dynamically adjusting the client's resource allocation and achieving horizontal resource expansion, the master node can achieve more flexible operational performance.

[0124] Based on the above data processing method, the present disclosure provides a data processing device for a data processing system and a data processing device for a dispatching center. Figure 10 and Figure 11 The device is described in detail.

[0125] Figure 10 The structure block diagram of a data processing device used in a data processing system according to an embodiment of the present disclosure is schematically shown.

[0126] like Figure 10 As shown, the data processing device 1000 of this embodiment includes a first receiving module 1010 , a first sending module 1020 , a second receiving module 1030 and a resource scheduling module 1040 .

[0127] The first receiving module 1010 may perform operation S310 to receive a data processing request, where the data processing request includes a read instruction or a write instruction for data to be processed.

[0128] The first sending module 1020 may perform operation S320 for sending a processing strategy request to the scheduling center based on the data processing request, wherein the scheduling center is configured to obtain data processing information of the data processing system in response to the processing strategy request and generate a data processing strategy based on the data processing information.

[0129] The second receiving module 1030 may perform operation S330 to receive a data processing policy sent by the scheduling center, where the data processing policy includes resource scheduling information for executing the data processing request.

[0130] The resource scheduling module 1040 may perform operation S340 to call corresponding resources to execute a read instruction or a write instruction for the data to be processed based on the resource scheduling information.

[0131] Figure 11 The structural block diagram of the data processing device for the dispatching center according to an embodiment of the present disclosure is schematically shown.

[0132] like Figure 11 As shown, the data processing device 1100 of this embodiment includes a third receiving module 1110 , an information obtaining module 1120 , a policy generating module 1130 and a second sending module 1140 .

[0133] The third receiving module 1110 can execute operation S610 to receive a processing policy request sent by the data processing system, wherein the data processing system is configured to send a processing policy request in response to the received data processing request, and the data processing request includes a read instruction or a write instruction for the data to be processed.

[0134] The information obtaining module 1120 may perform operation S620 for obtaining data processing information of the data processing system in response to the processing policy request.

[0135] The policy generating module 1130 may perform operation S630 to generate a data processing policy based on the data processing information, wherein the data processing policy includes resource scheduling information for executing the data processing request.

[0136] The second sending module 1140 may perform operation S640 to send the data processing policy to the data processing system, so that the data processing system calls corresponding resources to execute a read instruction or a write instruction for the data to be processed based on the resource scheduling information.

[0137] According to an embodiment of the present disclosure, any multiple modules in the data processing device 1000 or the data processing device 1100 may be combined into a single module for implementation, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module.

[0138] According to an embodiment of the present disclosure, at least one module in the data processing device 1000 or the data processing device 1100 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable means of integrating or packaging circuits, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one module in the data processing device 1000 or the data processing device 1100 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0139] Figure 12 The block diagram schematically shows an electronic device suitable for implementing the data processing method according to an embodiment of the present disclosure.

[0140] like Figure 12 As shown, the electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage portion 1208 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0141] Various programs and data required for the operation of the electronic device 1200 are stored in the RAM 1203. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. The processor 1201 performs various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1202 and / or the RAM 1203. It should be noted that the programs may also be stored in one or more memories other than the ROM 1202 and the RAM 1203. The processor 1201 may also perform various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0142] According to an embodiment of the present disclosure, electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to bus 1204. Electronic device 1200 may also include one or more of the following components connected to I / O interface 1205: an input unit 1206 including a keyboard, mouse, etc.; an output unit 1207 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage unit 1208 including a hard disk; and a communication unit 1209 including a network interface card such as a LAN card or modem. Communication unit 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1210 as needed, so that computer programs read from the removable media can be installed into storage unit 1208 as needed.

[0143] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments. Alternatively, the computer-readable storage medium may exist independently, without being incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the methods according to the embodiments of the present disclosure.

[0144] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above, and / or one or more memories other than ROM 1202 and RAM 1203.

[0145] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the item recommendation method provided by the embodiments of the present disclosure.

[0146] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 1201 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0147] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1209, and / or installed from the removable medium 1211. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0148] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209 and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0149] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0152] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, used in a data processing system, comprising: receiving a data processing request from at least one of a plurality of data computing applications, wherein the data processing request includes a read instruction or a write instruction for data to be processed, each data computing application being a distributed application; sending a processing policy request to a scheduling center based on the data processing request, wherein the scheduling center is configured to obtain data processing information of the data processing system in response to the processing policy request and generate a data processing policy based on the data processing information; receiving the data processing policy sent by the scheduling center, wherein the data processing policy includes resource scheduling information for executing the data processing request; Based on the resource scheduling information, calling corresponding resources to execute a read instruction or a write instruction for the data to be processed; The data processing system includes a client, a master node, and N slave nodes corresponding to the master node, and the scheduling center is configured as follows: Obtain resource usage information of the client, the master node, and the N slave nodes through a monitoring tool; generating the data processing strategy based on the resource usage information of the client, the master node, and the N slave nodes and the data processing information; The calling of corresponding resources based on the resource scheduling information includes: Based on the resource scheduling information, M slave nodes among the N slave nodes are called through the master node, specifically including: the master node sends the pointing information of the M slave nodes to the client, wherein the master node is used to receive the data processing strategy, and the resource scheduling information includes the pointing information of the M slave nodes; the client calls the M slave nodes based on the pointing information of the M slave nodes; wherein N and M are integers greater than or equal to 1 respectively, and M is less than or equal to N.

2. The method according to claim 1, wherein In the case where the data to be processed is stored in a cloud storage system, it includes: Creating a local storage repository for the cloud storage system; The executing of a read instruction or a write instruction for the data to be processed includes: Writing the data to be processed into the local storage warehouse; or The data to be processed is read from the local storage warehouse, wherein the data to be processed is obtained based on a cloud storage path of the cloud storage system.

3. The method according to claim 1, wherein The data processing system further includes a client, and calling M slave nodes among the N slave nodes through the master node includes: The client sends the resource scheduling information to the master node, wherein the client is used to receive the data processing policy, and the resource scheduling information includes directional information of the M slave nodes; The master node sends the availability status of the M slave nodes to the client, wherein the master node determines the corresponding availability status based on the pointing information of the M slave nodes; The client calls the M slave nodes based on the available status of the M slave nodes.

4. A data processing method for a dispatching center, comprising: receiving a processing policy request sent by a data processing system, wherein the data processing system is configured to send the processing policy request in response to a data processing request received from at least one of a plurality of data computing applications, the data processing request including a read instruction or a write instruction for data to be processed, each data computing application being a distributed application; obtaining data processing information of the data processing system in response to the processing policy request; generating a data processing policy based on the data processing information, wherein the data processing policy includes resource scheduling information for executing the data processing request; Sending the data processing strategy to the data processing system so that the data processing system calls corresponding resources to execute a read instruction or a write instruction for the data to be processed based on the resource scheduling information; The data processing system includes a client, a master node, and N slave nodes corresponding to the master node, and the scheduling center is configured as follows: Obtain resource usage information of the client, the master node, and the N slave nodes through a monitoring tool; generating the data processing strategy based on the resource usage information of the client, the master node, and the N slave nodes and the data processing information; The data processing system calling corresponding resources based on the resource scheduling information includes: Based on the resource scheduling information, M slave nodes among the N slave nodes are called through the master node, specifically including: the master node sends the pointing information of the M slave nodes to the client, wherein the master node is used to receive the data processing strategy, and the resource scheduling information includes the pointing information of the M slave nodes; the client calls the M slave nodes based on the pointing information of the M slave nodes; wherein N and M are integers greater than or equal to 1 respectively, and M is less than or equal to N.

5. The method according to claim 4, wherein Before obtaining the data processing information of the data processing system in response to the processing policy request, the method further includes: Setting an information monitoring tool in the data processing system, wherein the information monitoring tool is used to monitor resource usage information of data processed by the data processing system; The obtaining, in response to the processing policy request, data processing information of the data processing system includes: Receive the resource usage information sent by the information monitoring tool.

6. The method according to claim 5, wherein: The information monitoring tool includes a first monitoring tool, the data processing system includes a master node and N slave nodes corresponding to the master node, and setting the information monitoring tool in the data processing system includes: Setting the first monitoring tool on the N slave nodes; The receiving of the resource usage information sent by the information monitoring tool includes: Receive resource usage information of N slave nodes sent by the first monitoring tool.

7. The method according to claim 6, wherein: Generating a data processing strategy based on the data processing information includes: Determining M slave nodes based on resource usage information of at least one slave node among the N slave nodes, where N and M are integers greater than or equal to 1, and M is less than or equal to N; The data processing strategy is generated based on the directional information of the M slave nodes.

8. The method according to claim 6, wherein: The information monitoring tool includes a second monitoring tool, and the method includes: Setting the second monitoring tool on the master node; receiving resource usage information of the master node sent by the second monitoring tool; Adjust resource configuration of the master node based on the resource usage information of the master node.

9. The method according to claim 6, wherein: The data processing system further includes a client, the information monitoring tool includes a third monitoring tool, and the method includes: Setting the third monitoring tool on the client; receiving resource usage information of the client sent by the third monitoring tool; Adjust resource configuration of the client based on the resource usage information of the client.

10. A data processing device, used in a data processing system, comprising: a first receiving module, configured to receive a data processing request from at least one of a plurality of data computing applications, wherein the data processing request includes a read instruction or a write instruction for data to be processed, and each data computing application is a distributed application; a first sending module, configured to send a processing strategy request to a scheduling center based on the data processing request, wherein the scheduling center is configured to obtain data processing information of the data processing system in response to the processing strategy request, and generate a data processing strategy based on the data processing information; a second receiving module, configured to receive the data processing policy sent by the scheduling center, wherein the data processing policy includes resource scheduling information for executing the data processing request; A resource scheduling module, configured to call corresponding resources to execute a read instruction or a write instruction for the data to be processed based on the resource scheduling information; The data processing system includes a client, a master node, and N slave nodes corresponding to the master node, and the scheduling center is configured as follows: Obtain resource usage information of the client, the master node, and the N slave nodes through a monitoring tool; generating the data processing strategy based on the resource usage information of the client, the master node, and the N slave nodes and the data processing information; The calling of corresponding resources based on the resource scheduling information includes: Based on the resource scheduling information, M slave nodes among the N slave nodes are called through the master node, specifically including: the master node sends the pointing information of the M slave nodes to the client, wherein the master node is used to receive the data processing strategy, and the resource scheduling information includes the pointing information of the M slave nodes; the client calls the M slave nodes based on the pointing information of the M slave nodes; wherein N and M are integers greater than or equal to 1 respectively, and M is less than or equal to N.

11. A data processing device for a dispatching center, comprising: a third receiving module, configured to receive a processing strategy request sent by a data processing system, wherein the data processing system is configured to send the processing strategy request in response to the received data processing request, wherein the data processing request includes a read instruction or a write instruction for data to be processed; an information obtaining module, configured to obtain data processing information of the data processing system in response to the processing strategy request; a policy generating module, configured to generate a data processing policy based on the data processing information, wherein the data processing policy includes resource scheduling information for executing the data processing request; A second sending module is configured to send the data processing strategy to the data processing system, so that the data processing system calls corresponding resources to execute a read instruction or a write instruction for the data to be processed based on the resource scheduling information; The data processing system includes a client, a master node, and N slave nodes corresponding to the master node, and the scheduling center is configured as follows: Obtain resource usage information of the client, the master node, and the N slave nodes through a monitoring tool; generating the data processing strategy based on the resource usage information of the client, the master node, and the N slave nodes and the data processing information; The data processing system calling corresponding resources based on the resource scheduling information includes: Based on the resource scheduling information, M slave nodes among the N slave nodes are called through the master node, specifically including: the master node sends the pointing information of the M slave nodes to the client, wherein the master node is used to receive the data processing strategy, and the resource scheduling information includes the pointing information of the M slave nodes; the client calls the M slave nodes based on the pointing information of the M slave nodes; wherein N and M are integers greater than or equal to 1 respectively, and M is less than or equal to N.

12. A data processing device comprising: Multiple data computing applications, each of which is a distributed application; A data processing system for executing the method according to any one of claims 1 to 3, wherein the data computing application is configured to send a data processing request to the data processing system based on a data processing task; A dispatching center, configured to execute the method according to any one of claims 4 to 9; Cloud-based distributed storage system based on distributed storage network; Among them, the N slave nodes of the data processing system are used to locally create N local storage warehouses corresponding to the cloud distributed storage system, and N is an integer greater than or equal to 1.

13. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the method according to any one of claims 1 to 9.

14. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 9.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images