Data processing method, device and system
By introducing a dynamic adjustment mechanism of the first transit node and processing node in the data processing system, the problem of single data processing mode in the prior art is solved, high throughput and flexible performance adjustment is achieved, and it is suitable for massive data processing.
Patent Information
- Application Number
- CN202210719647.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-06-23
AI Technical Summary
The existing data processing methods are relatively single, resulting in small throughput, low efficiency and difficult to improve processing capabilities when processing massive data. The hardware of the equipment is fixed, making it difficult to flexibly adjust performance.
Provide a data processing system, including a first transit node, a request node and a processing node, transmits data processing requests through the first transit node, and dynamically adjusts the number of processing units according to the frequency and number of processing requests, and processes the requests in parallel using multiple threads. , supports the cache and queue management of data processing results, and realizes flexible expansion and performance adjustment of data processing.
It improves the throughput of the data processing system, supports massive data processing, reduces resource consumption and failure risks, and achieves flexible adjustment of processing capabilities and performance improvement.
Smart Images

Figure CN115002209B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device and system. Background Art
[0002] With the development of computer technology, data processing has become an indispensable link in various fields. For example, in the field of Internet of Things, multiple IoT devices usually use data in different formats, so it is necessary to process these data in different formats to unify the format of these data.
[0003] In the related art, a data processing device is used to receive data to be processed and process the data. For example, a data processing device is used to receive data in different formats sent by multiple IoT devices, and the data processing device performs format conversion on each received data to convert the formats of the data into a unified format.
[0004] However, the current data processing method is relatively simple. Summary of the invention
[0005] The present application provides a data processing method, device and system, which can solve the problem that the current data processing method is relatively single. The technical solution is as follows:
[0006] In a first aspect, a data processing method is provided. The method is used for a target processing node in a data processing system. The data processing system includes: a first transit node, at least one request node, and at least one processing node. The target processing node is any processing node in the at least one processing node. The method includes:
[0007] receiving a data processing request sent by the first transfer node, wherein the data processing request is sent by a target requesting node to the first transfer node according to the data to be processed by the data processing system, and the target requesting node is any one of the at least one requesting node;
[0008] Execute the data processing operation requested by the data processing request;
[0009] The processing result of the data processing operation is transmitted to the target requesting node.
[0010] Optionally, the data transmission system further includes a second transfer node, which transmits the processing result of the data processing operation to the target request node, including:
[0011] Sending the processing result to the second transfer node;
[0012] A data processing response of the data processing request is sent to the first transfer node, so that the first transfer node sends the data processing response to the target request node, and the target request node obtains the processing result from the second transfer node according to the data processing response, and the data processing response is used to indicate that the processing result has been obtained.
[0013] Optionally, the data processing request carries: an identifier of a script corresponding to the data processing request; the script corresponding to the data processing request is: a script required to be run to execute the data processing operation requested by the data processing request;
[0014] Executing the data processing operation requested by the data processing request includes:
[0015] The data processing operation is performed using a processing unit created in the processing node and used to run a script corresponding to the data processing request.
[0016] Optionally, the data processing request sent by the first transfer node includes: different data processing requests corresponding to the same script; the data processing operations requested by the different data processing requests are executed by the same processing unit.
[0017] Optionally, the method further comprises:
[0018] Determining a first number of the data processing requests received within a most recent first time period multiple times;
[0019] When the growth rate of the first number is greater than a first probability threshold, adding the processing unit in the target processing node, the first probability threshold being greater than zero;
[0020] When the growth rate of the first number is less than a second probability threshold, reducing the processing unit in the target processing node, the second probability threshold being less than zero.
[0021] Optionally, the method further comprises:
[0022] Determining multiple times a second number of the data processing requests corresponding to the same script received within a recent second time period;
[0023] When the growth rate of the second number is greater than a third probability threshold, adding a processing unit in the target processing node for running the same script, and the third probability threshold is greater than zero;
[0024] When the growth rate of the second number is less than a fourth probability threshold, reducing the processing unit in the target processing node for running the same script, and the fourth probability threshold is less than zero.
[0025] Optionally, the data processing system further includes: a second transfer node, and the target processing node is used to:
[0026] When m processing units are added to the processing node, x script acquisition operations are performed to obtain n scripts for the m processing units to run; wherein m≥x≥n≥1; one script acquisition operation corresponds to one script among the n scripts, and the script acquisition operation is used to: when a corresponding script is stored in the memory of the target processing unit, acquire the corresponding script from the memory; when the corresponding script is not stored in the memory, send an acquisition request for the corresponding script to the second transfer node, and receive the corresponding script sent by the second transfer node according to the acquisition request;
[0027] The m processing units are created according to the n scripts.
[0028] Optionally, m>x=n.
[0029] Optionally, the method further comprises:
[0030] Counting the frequency of execution of each script stored in the memory and the second transfer node;
[0031] When the frequency at which the first script stored in the second transfer node is executed is greater than a first frequency threshold, moving the first script from the second transfer node to the memory;
[0032] When the frequency at which the second script stored in the memory is executed is greater than a second frequency threshold and less than or equal to the first frequency threshold, the second script is moved from the memory to the second transit node.
[0033] Optionally, before executing the data processing operation requested by the data processing request, the method further includes:
[0034] When multiple data processing requests sent by the first transit node are received, multiple threads are used in parallel to determine the processing units in the processing node for respectively running the scripts corresponding to the multiple data processing requests according to the multiple data processing requests.
[0035] In a second aspect, a data processing method is provided, the method being used for a first transit node in a data processing system, the data processing system further comprising: at least one requesting node and at least one processing node, the method comprising:
[0036] Receiving a data processing request sent by a target requesting node, wherein the data processing request is sent by the target requesting node according to the data to be processed by the data processing system, and the target requesting node is any one of the at least one requesting node;
[0037] The data processing request is sent to a target processing node so that the target processing node performs the data processing operation requested by the data processing request and transmits the processing result of the data processing operation to the target request node, and the target processing node is any processing node among the at least one processing node.
[0038] Optionally, the data transmission system further includes a second transfer node, and the target processing node is used to send the processing result to the second transfer node. The method further includes:
[0039] receiving a data processing response to the data processing request sent by the target processing node, wherein the data processing response is used to indicate that the processing result has been obtained;
[0040] The data processing response is sent to the target requesting node, so that the target requesting node obtains the processing result from the second transit node according to the data processing response.
[0041] Optionally, sending the data processing request to the target processing node includes:
[0042] Upon receiving multiple data processing requests sent by the target requesting node, sending a first data processing request among the multiple data processing requests to the target processing node;
[0043] After receiving the data processing response to the first data processing request, a second data processing request among the multiple data processing requests is sent to the target processing node.
[0044] Optionally, after receiving the data processing request sent by the target requesting node, the method further includes:
[0045] Adding the data processing request to a first queue;
[0046] Sending the data processing request to the target processing node includes:
[0047] Sending the data processing request in the first queue to the target processing node;
[0048] After receiving a data processing response to the data processing request sent by the target processing node, the method further includes:
[0049] adding the data processing response to a second queue;
[0050] Sending the data processing response to the target requesting node includes:
[0051] The data processing response in the second queue is sent to the target requesting node.
[0052] In a third aspect, a data processing method is provided, the method being used for a target request node in a data processing system, the data processing system comprising: a first transit node, at least one request node, and at least one processing node, the target request node being any request node in the at least one request node, the method comprising:
[0053] Acquiring data to be processed by the data processing system;
[0054] Sending a data processing request to the first transit node according to the data to be processed, so that the first transit node sends the data processing request to a target processing node, and the target processing node performs the data processing operation requested by the data processing request, the target processing node being any processing node among the at least one processing node;
[0055] A processing result of the data processing operation is obtained from the target processing node.
[0056] Optionally, the data transmission system further includes a second transfer node, the target processing node is used to send the processing result to the second transfer node, and send a data processing response of the data processing request to the first transfer node, the data processing response is used to indicate that the processing result has been obtained;
[0057] The obtaining of the processing result of the data processing operation from the target processing node includes:
[0058] Receiving the data processing response sent by the first transfer node;
[0059] According to the data processing response, the processing result is obtained from the second transfer node.
[0060] In a fourth aspect, a data processing device is provided, comprising: a processor and a memory, wherein the memory stores a program, and the processor is used to execute the program stored in the memory to implement the data processing method described in any one of the first aspect, the second aspect or the third aspect.
[0061] In a fifth aspect, a data processing system is provided, the data processing system comprising: a first transit node, at least one request node and at least one processing node,
[0062] The processing node is used to execute the data processing method described in any one of the designs of the first aspect;
[0063] The first transfer node is used to execute the data processing method described in any design of the second aspect;
[0064] The requesting node is used to execute the data processing method described in any design of the third aspect.
[0065] In a sixth aspect, a non-volatile computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the data processing method as described in any one of the designs of the first aspect, the second aspect or the third aspect.
[0066] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute a data processing method as described in any one of the first aspect, the second aspect or the third aspect.
[0067] In summary, the data processing system provided by the present application includes: a first transit node, at least one request node and at least one processing node. Each request node can send a data processing request to any processing node through the first transit node, so that the processing node performs the data processing operation requested by the data processing request, and transmits the result of the data processing operation to the request node. It can be seen that on the basis of the data processing system provided by the present application, the present application provides a new data processing method, which enriches the data processing method.
[0068] In addition, the data processing system and data processing method provided by the present application can decouple the request node from the processing node, so that the number of both the request node and the processing node can be expanded. Moreover, when the number of at least one of the request node and the processing node is expanded, the performance of the data processing system will change. Therefore, the present application can support flexible adjustment of the performance of the data processing system. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] Figure 1 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;
[0070] Figure 2 A schematic diagram of the structure of another data processing system provided in an embodiment of the present application;
[0071] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;
[0072] Figure 4 A schematic diagram of the relationship between a time unit and a time period provided in an embodiment of the present application;
[0073] Figure 5 A schematic diagram of the relationship between another time unit and time period provided in an embodiment of the present application;
[0074] Figure 6 A schematic diagram of unit division of a target processing node provided in an embodiment of the present application;
[0075] Figure 7 A flowchart of another data processing method provided in an embodiment of the present application;
[0076] Figure 8 A flowchart of another data processing method provided in an embodiment of the present application;
[0077] Fig. 9 A flowchart of another data processing method provided in an embodiment of the present application;
[0078] Fig.10 A schematic diagram of the relationship between multiple threads and multiple data processing requests provided in an embodiment of the present application;
[0079] Fig.11 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0080] Fig.12 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;
[0081] Fig.13 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;
[0082] Fig.14 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;
[0083] Fig.15 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;
[0084] Fig.16 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0085] In order to make the principles, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0086] Data processing is an indispensable link in various fields.
[0087] For example, in the field of the Internet of Things, multiple IoT devices use data in different formats, and it is necessary to unify the formats of these data in different formats. For example, the above different formats may include: JS object notation (JSON) format (a lightweight data exchange format), test expression (test) format, binary format, etc. The data in different formats may be different or the same communication protocols, such as the transmission control protocol (TCP), message queuing telemetry transport (MQTT) protocol, user datagram protocol (UDP), constrained application protocol (CoAP), etc.
[0088] For another example, in the field of communications, a communications device needs to process request data received from another communications device, obtain response data for the request data, and feed the response data back to the other communications device. For example, when the request data is authentication data, the response data for the request data may be an authentication result of the authentication data.
[0089] In the related art, a data processing device is used to receive the data to be processed and process the data. However, the current data processing method is relatively simple. In addition, when a large amount of data needs to be processed, the throughput of a data processing device is small. The efficiency of using a data processing device to process a large amount of data is low, and it is easy to cause the data processing device to malfunction. In addition, after the data processing device leaves the factory, the hardware of the data processing device is difficult to change. Therefore, the processing capacity of the data processing device is fixed, and it is difficult to improve the performance of the data processing device.
[0090] Based on this, an embodiment of the present application provides a data processing system, which can process data, and the way in which the data processing system processes data is different from the way in which the above-mentioned data processing device processes data, enriching the way of data processing. In addition, the data processing system has a large throughput, can support the processing of massive data, and the data processing system is not prone to failure. Furthermore, during the use of the data processing system, the processing capacity of the data processing system can be flexibly adjusted as needed.
[0091] For example, Figure 1A structural diagram of a data processing system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the data processing system includes: a first transit node 01, at least one request node 02, and at least one processing node 03. Both the request node 02 and the processing node 03 are in communication connection with the first transit node 01.
[0092] It should be noted that the number of request nodes 02 in the data processing system may be greater than or equal to 1, and the number of processing nodes 03 may also be greater than or equal to 1. Figure 1 In the example, the number of request nodes 02 and processing nodes 03 is 1. The nodes in the data processing system (such as the first transit node, the request node, and the processing node, etc.) can be a device, such as a server, a server cluster, etc., or a part of a device, such as an interface board in a device, etc., which is not limited in the embodiments of the present application.
[0093] Optionally, please continue to refer to Figure 1 The data processing system may further include a second transfer node 04 , and both the request node 02 and the processing node 03 are communicatively connected to the second transfer node 04 .
[0094] Optionally, the data processing system may further include a storage node 05, and the second transit node 04 may also be connected to the storage node 05 for communication. Figure 2 As shown, the storage node 05 may also be not in communication connection with the second transit node 04 , but in communication connection with the processing node 03 .
[0095] The following will take the data processing method provided in the embodiment of the present application as an example to introduce the functions of the data processing system.
[0096] For example, Figure 3 A flowchart of a data processing method provided in an embodiment of the present application. The data processing method can be used for a target processing node in a data processing system provided in an embodiment of the present application, and the target processing node is any processing node of at least one processing node in the data processing system. The data processing method used to execute by other processing nodes other than the target processing node in the at least one processing node can refer to the data processing method used to execute by the target processing node.
[0097] like Figure 3 As shown, the data processing method may include:
[0098] Step 201: Receive a data processing request sent by a first transit node. The data processing request is sent by a target request node to the first transit node according to the data to be processed by a data processing system. The target request node is any request node among at least one request node.
[0099] The target request node can obtain the data to be processed of the data processing system, and generate a data processing request based on the data to be processed, and the data processing request is used to request the execution of a processing operation of the data to be processed. The processing operation can be any operation of processing data, such as converting the format of data, or generating response data of data, etc.
[0100] After receiving the data processing request, the target requesting node may also send the data processing request to the first transit node, and the first transit node may send the data processing request to the target processing node.
[0101] Step 202: Execute the data processing operation requested by the data processing request.
[0102] Step 203: Transmit the processing result of the data processing operation to the target requesting node.
[0103] After executing the data processing operation requested by the data processing request, the target processing node may transmit the processing result of the obtained data processing operation to the target requesting node, so that the target requesting node obtains the processing result.
[0104] In summary, the data processing system provided by the present application includes: a first transit node, at least one request node and at least one processing node. Each request node can send a data processing request to any processing node through the first transit node, so that the processing node performs the data processing operation requested by the data processing request, and transmits the result of the data processing operation to the request node. It can be seen that on the basis of the data processing system provided by the present application, the present application provides a new data processing method, which enriches the data processing method.
[0105] In addition, the data processing system and data processing method provided by the present application can decouple the request node from the processing node, so that the number of both the request node and the processing node can be expanded. Moreover, when the number of at least one of the request node and the processing node is expanded, the performance of the data processing system will change. Therefore, the present application can support flexible adjustment of the performance of the data processing system.
[0106] Furthermore, in the above step 203, the target processing node may transmit the processing result to the target requesting node in any of a variety of ways.
[0107] For example, the target processing node may be in communication connection with the target requesting node, and the target processing node may directly send the processing node to the target requesting node.
[0108] For another example, when the data transmission system also includes a second transfer node, the target processing node can send the processing result to the second transfer node, and send a data processing response of the data processing request to the first transfer node, and the data processing response is used to indicate that the processing result has been obtained. After receiving the data processing response, the first transfer node can send the data processing response to the target request node. After that, the target request node can obtain the processing result from the second transfer node based on the data processing response. In this case, the processing result does not need to be transmitted on the link between the target request node, the first transfer node and the target processing node, thereby reducing the transmission load of the link, reducing the bandwidth of the link, reducing the resource consumption of the data processing system, and improving the throughput of the data processing system.
[0109] Optionally, the data processing request carries: an identifier of a script corresponding to the data processing request; the script corresponding to the data processing request is: a script required to execute the data processing operation requested by the data processing request. When the target processing node executes step 202, it can use the processing unit created in the processing node for running the script corresponding to the data processing request to execute the data processing operation requested by the data processing request. It can be seen that the data processing request in the present application carries a script identifier, so there is no need to carry a script in the data processing request, so that the data volume of the data processing request is low, the transmission load of the link between the target request node, the first transit node and the target processing node is low, the bandwidth of the link is reduced, the resource consumption of the data processing system is reduced, and the throughput of the data processing system is improved.
[0110] Optionally, the data processing request may also carry metadata required to run the script corresponding to the data processing request, and the processing unit may run the script corresponding to the data processing request according to the metadata. Of course, the data processing request may not carry the metadata, and this embodiment of the application does not limit this.
[0111] Optionally, the data processing request may also carry the data to be processed. It is understandable that the data processing request may not carry the data to be processed, but may carry the identification of the data to be processed. In this case, the target requesting node may also transmit the data to be processed to the second transit node, and after receiving the data processing request, the target processing node may obtain the data to be processed from the second transit node according to the identification of the data to be processed carried by the data processing request, and then process the data to be processed.
[0112] Furthermore, the data processing request sent by the first transit node includes: when the data processing requests for the same script are different, the data processing operations requested by the different data processing requests can be executed by the same processing unit in the target processing node. In this way, the requests for the same script use the same processing unit to execute the data processing operations, thereby avoiding the creation of multiple identical processing units, reducing the load of the processing node in creating the processing unit, reducing the resource consumption of the data processing system, and improving the throughput of the data processing system.
[0113] It is understandable that the data processing operations requested by the requests corresponding to the same script may also be executed by different processing units. Alternatively, a part of the data processing operations requested by different data processing requests corresponding to the same script may be executed by the same processing unit, and another part of the data processing operations requested by different data processing requests corresponding to the same script may be executed by different processing units.
[0114] Furthermore, the target processing node may manage the number of processing units therein.
[0115] By way of example, the target processing node may also determine multiple times the first number of data processing requests received within a most recent first time period; when the growth rate of the first number is greater than a first probability threshold, the target processing node may increase the processing units in the target processing node, and the first probability threshold is greater than zero; when the growth rate of the first number is less than a second probability threshold (the decrease rate of the first number is higher), the target processing node may reduce the processing units in the target processing node, and the second probability threshold is less than zero.
[0116] It can be seen that the target processing node can appropriately increase or decrease the number of processing units in the target processing node according to the number of data processing requests received, so as to minimize power consumption and achieve reasonable utilization of the resources of the target processing node while ensuring the processing speed.
[0117] Optionally, before increasing (or decreasing) the processing units in the target processing node, the target processing node may determine the number of processing units to be increased (or decreased) according to the growth rate of the first number. Alternatively, the number of processing units increased (or decreased) by the target processing node each time is a fixed value.
[0118] Optionally, each time the target processing node determines the first number of data processing requests received in the most recent first time period, it can also determine the auxiliary number of data processing requests corresponding to the same script received in the first time period. Before increasing (or decreasing) the processing units in the target processing node, the target processing node can determine whether it is necessary to increase (or decrease) the processing units used to run the same script based on the growth rate of the auxiliary number. For example, when the growth rate of the auxiliary number is greater than the third probability threshold, the processing units used to run the same script in the target processing node are increased, and the third probability threshold is greater than zero; when the growth rate of the auxiliary number is less than the fourth probability threshold (the reduction rate of the auxiliary number is high), the processing units used to run the same script in the target processing node are reduced, and the fourth probability threshold is less than zero.
[0119] As another example, the target processing node may also multiple times determine the second number of data processing requests corresponding to the same script received within a most recent second time period; when the growth rate of the second number is greater than a third probability threshold, increase the number of processing units in the target processing node for running the same script; when the growth rate of the second number is less than a fourth probability threshold (the decrease rate of the second number is higher), reduce the number of processing units in the target processing node for running the same script.
[0120] It can be seen that in the embodiment of the present application, the target processing node can appropriately increase or decrease the number of processing units used to run the script according to the number of requests received corresponding to the same script, so as to minimize the power consumption of the target processing node while ensuring data processing efficiency.
[0121] Optionally, the target processing node may use a sliding time window algorithm to determine the first number, the second number, and the auxiliary number.
[0122] For example, taking the first number as an example, the target processing node can perform a first statistic on the number of data processing requests received in each first time unit, and the first time period includes a plurality of consecutive first time units. When the target processing node determines the first number using a sliding time window algorithm, the target processing node can determine the first number based on the first statistic.
[0123] like Figure 4As shown, assuming that the first time unit is 10 seconds and the first time period is 1 minute (including 6 first time units), the target processing node can count the number of data processing requests received every 10 seconds. Each time the first number is determined, the target processing node can obtain the numbers counted in the most recent 6 first time units and determine the sum of these numbers as the first number. The most recent 6 first time units determined by the target processing node when determining the first number multiple times will change over time. For example, when the target processing node determines the first number for the first time, the most recent 6 first time units determined by the target processing node are as follows: Figure 4 As shown, the target processing node determines the most recent 6 first time units when determining the first number for the second time as follows Figure 5 As shown, the time for determining the first number for the second time differs from the time for determining the first number for the first time by a first time unit (10 seconds).
[0124] Similarly, for the auxiliary number, the data processing method further includes: the target processing node performs a second statistic on the number of data processing requests corresponding to the same script received in each first time unit, and the first time period includes multiple consecutive first time units; when the target processing node uses the sliding time window algorithm to determine the auxiliary number, the target processing node can determine the auxiliary number based on the second statistic. For the second number, the data processing method further includes: the target processing node performs a third statistic on the number of data processing requests corresponding to the same script received in each second time unit, and the second time period includes multiple consecutive second time units; when the target processing node uses the sliding time window algorithm to determine the second number, the target processing node can determine the second number based on the third statistic.
[0125] Further, according to the foregoing content, the target processing node uses the created processing unit to perform data processing operations. The process of the target processing node creating a processing unit can be called the process of the target processing node adding a processing unit. When the target processing node adds m processing units, it needs to perform x script acquisition operations to obtain n scripts for the m processing units to run (where each processing unit is used to run one script), and then create the m processing units according to the n scripts, m≥x≥n≥1.
[0126] For example, a script acquisition operation corresponds to one script among n scripts, and the script acquisition operation is used to: when the corresponding script is stored in the memory of the target processing unit, acquire the corresponding script from the memory; when the corresponding script is not stored in the memory, send an acquisition request for the corresponding script to the second transfer node, and receive the corresponding script sent by the second transfer node according to the acquisition request.
[0127] On the one hand, if the second transit node is connected to the storage node in communication, then after receiving the acquisition request, the second transit node may first check whether the script for which the acquisition request is to be acquired is stored locally. If the script is not stored locally, the second transit node may acquire the script from the connected storage node, store the script locally in the second transit node, and send the script to the target processing node.
[0128] On the other hand, if the second transit node is not in communication connection with the storage node, and the storage node is in communication connection with the target processing node, then when the second transit node does not store a script requested by the acquisition request locally, the second transit node may also send an acquisition response to the target processing node indicating that the script is not stored. At this time, the target processing node may send the acquisition request to the storage node, and receive the script sent by the storage node. Afterwards, the target processing node may send the script to the second transit node again, so that the second transit node stores the script.
[0129] Optionally, the memory of the target processing node and the second transfer node can store information (such as scripts) in a cached manner. Therefore, the scripts stored in the memory or the second transfer node have an effective period, and the effective period will decrease over time. When the effective period is reduced to zero, the script will be released, thereby avoiding the occupation of the storage location by the script and reducing the load of the target processing node or the second transfer node. In addition, after the script is read, the effective period of the script will be reset to the initial value (greater than zero).
[0130] The storage node may store information (such as scripts) in a non-cached manner (such as a persistent storage manner), which is not limited in the present embodiment of the application. The storage node may be a relational database management system (MySQL) node.
[0131] Optionally, m>x=n. When m>n, there are multiple processing units for running the same script among the m processing units. Since x=n, the number of times the target processing node performs the script acquisition operation is the same as the number n of scripts that the m processing units are used to run. In addition, for multiple processing units for running the same script, the target processing node only needs to perform a script acquisition operation for the same script once, without having to perform multiple script acquisition operations, thereby avoiding the problem of repeatedly acquiring the same script on the memory, the second transit node or the storage node affecting the performance of the memory and the second transit node.
[0132] When the second transfer node stores information in a cached manner, if the same script is repeatedly obtained on the second transfer node, and the script is not cached on the second transfer node, the phenomenon of repeatedly obtaining the script on the storage node is called cache breakdown. Cache breakdown can cause a thundering herd effect and affect the performance of the entire data processing system. The solution provided by the embodiment of the present application can avoid cache breakdown, avoid the thundering herd effect, and ensure the performance of the entire data processing system. In addition, it can also avoid the second transfer node or the target processing node from repeatedly caching the same script on the second transfer node, thereby reducing the load on the second transfer node or the target processing node.
[0133] Furthermore, the target processing node may also count the frequency at which each script stored in the memory and the second transfer node is executed (the frequency at which the script is executed by the processing unit); when the frequency at which the first script stored in the second transfer node is executed is greater than the first frequency threshold, the target processing node may move the first script from the second transfer node to the memory; when the frequency at which the second script stored in the memory is executed is greater than the second frequency threshold and less than or equal to the first frequency threshold (the first frequency threshold may be greater than or equal to zero), the target processing node may move the second script from the memory to the second transfer node.
[0134] It can be seen that for the scripts stored in the memory and the second transfer node, the target processing node can adjust the storage location of the script according to the frequency at which the script is run, so that the first script with a higher running frequency is stored in the memory, thereby improving the efficiency of the target processing node in reading the first script. In addition, the second script with a lower running frequency can also be stored in the second transfer node to avoid the second script occupying the memory of the target processing node.
[0135] It is understandable that when the script is stored in the memory and the second transfer node in a cache manner, the script has a validity period. When the target processing node transfers the script between the memory and the second transfer node, the validity period of the script is reset.
[0136] Optionally, before executing the data processing operation requested by the data processing request, the target processing node also needs to determine (such as creating or searching) a processing unit for executing the data processing operation requested by the data processing request. For example, when the target processing node receives multiple data processing requests sent by the first transit node, it can use multiple threads in parallel to determine the processing units in the processing node that are used to run the scripts corresponding to the multiple data processing requests. In this way, by determining the processing unit in parallel through multiple threads, efficiency can be improved, and system throughput can be further improved, so that the data processing system can support the processing of massive data.
[0137] For example, according to the functions of the target processing node provided in the embodiments of the present application, the target processing node can be divided into multiple units. Figure 6 As shown, the target processing node may include: a main control unit, a statistical unit, a processing unit and a cache unit.
[0138] The statistical unit is used to count the first number, the second number, the auxiliary number, the frequency of script execution, etc.
[0139] The processing unit is used to run the script to execute the data processing operation corresponding to the data processing request and obtain the processing result.
[0140] The cache unit is used to adjust the storage location of the script in the memory and on the second transfer node according to the frequency at which the script is executed. The cache unit can also be used to perform the above script acquisition operation.
[0141] In the method for the target processing unit, all operations except the operations performed by the statistical unit, the processing unit and the cache unit can be performed by the main control unit. For example, the main control unit is used to receive a data processing request sent by a first transfer node; determine a processing unit for performing the data processing operation requested by the data processing request according to the data processing request; send a processing result to the second transfer node; send a data processing response to the first transfer node; and adjust the number of processing units according to the statistical result of the statistical unit.
[0142] Figure 7 This is a flowchart of another data processing method provided in an embodiment of the present application. The data processing method can be used for the first transit node in the data processing system provided in an embodiment of the present application. Figure 7 As shown, the data processing method may include:
[0143] Step 301: Receive a data processing request sent by a target request node. The data processing request is sent by the target request node according to the data to be processed in the data processing system. The target request node is any request node among at least one request node in the data processing system.
[0144] Step 302: Send a data processing request to a target processing node so that the target processing node performs the data processing operation requested by the data processing request and transmits the processing result of the data processing operation to the target requesting node. The target processing node is any processing node in at least one processing node in the data processing system.
[0145] In summary, the data processing system provided by the present application includes: a first transit node, at least one request node and at least one processing node. Each request node can send a data processing request to any processing node through the first transit node, so that the processing node performs the data processing operation requested by the data processing request, and transmits the result of the data processing operation to the request node. It can be seen that on the basis of the data processing system provided by the present application, the present application provides a new data processing method, which enriches the data processing method.
[0146] In addition, the data processing system and data processing method provided by the present application can decouple the request node from the processing node, so that the number of both the request node and the processing node can be expanded. Moreover, when the number of at least one of the request node and the processing node is expanded, the performance of the data processing system will change. Therefore, the present application can support flexible adjustment of the performance of the data processing system.
[0147] Optionally, the data transmission system also includes a second transfer node, and the target processing node is used to send the processing result to the second transfer node. At this time, the data processing method also includes: the first transfer node receives a data processing response to the data processing request sent by the target processing node, and the data processing response is used to indicate that the processing result has been obtained; thereafter, the first transfer node sends a data processing response to the target request node, so that the target request node obtains the processing result from the second transfer node according to the data processing response. In this case, the processing result does not need to be transmitted on the link between the target request node, the first transfer node and the target processing node, thereby reducing the transmission load of the link, reducing the bandwidth of the link, reducing the resource consumption of the data processing system, and improving the throughput of the data processing system.
[0148] In the embodiment of the present application, the target processing node transmits the processing result to the target request node through the second transfer node as an example. Optionally, the target processing node can also directly transmit the processing result to the target request node. In this case, the first transfer node does not need to receive the data processing response, nor does it need to send the data processing response to the target request node. The second transfer node also does not need to store the above processing result.
[0149] Optionally, after receiving the data processing request, the first transit node may also parse the data processing request to obtain information carried by the data processing request, such as the identifier of the script corresponding to the data processing request. The first transit node may also package this information into a message structure object (also called request msg) in a certain format to update the data processing request. Subsequently, the first transit node may send the updated data processing request to the target processing node. In this way, all data processing requests received by the target processing node adopt the structure of the message structure object, which facilitates the target processing node to parse and process the data processing request.
[0150] When the data processing request carries the identifier of the script corresponding to the data processing request, there is no need to carry the script in the data processing request, so that the data volume of the data processing request is relatively low. The transmission load of the link between the target request node, the first transit node and the target processing node is relatively low, which reduces the bandwidth of the link, reduces the resource consumption of the data processing system, and improves the throughput of the data processing system.
[0151] Optionally, the data processing request may also carry metadata required for running the script corresponding to the data processing request. Of course, the data processing request may not carry the metadata, and this embodiment of the present application does not limit this.
[0152] Optionally, the data processing request may also carry the data to be processed. It is understandable that the data processing request may not carry the data to be processed, but may carry the identification of the data to be processed. In this case, the target requesting node may also transmit the data to be processed to the second transit node, and after receiving the data processing request, the target processing node may obtain the data to be processed from the second transit node according to the identification of the data to be processed carried by the data processing request, and then process the data to be processed.
[0153] Optionally, in step 302, when the first transit node receives multiple data processing requests sent by the target request node, it may send (one or more) first data processing requests among the multiple data processing requests to the target processing node; after receiving the data processing response to the first data processing request, it may send (one or more) second data processing requests among the multiple data processing requests to the target processing node. It can be seen that the first transit node may send a group of data processing requests to the target processing node each time, and after receiving the data processing response to the group of data processing requests sent by the target processing node, it may send the next group of data processing requests to the target processing node.
[0154] In this way, a large number of data processing requests will not flow into the processing node, causing the processing node to fail, which has the effect of reducing the peak of data processing requests. In addition, multiple data processing requests to be sent to the target processing node can be buffered on the first transfer node, allowing the data processing system to process more data, thereby improving the throughput of the entire data processing system.
[0155] Optionally, multiple queues may be maintained in the first transfer node. After receiving a data processing request sent by the target request node, the first transfer node may add the data processing request to the first queue, and at this time, when the first transfer node sends the data processing request to the target processing node, it may send the data processing request in the first queue to the target processing node. After receiving a data processing response to the data processing request sent by the target processing node, the first transfer node may add the data processing response to the second queue among the multiple queues, and at this time, when the first transfer node sends the data processing response to the target request node, it may send the data processing response in the second queue to the target request node.
[0156] The first transit node caches the received data processing requests and data processing responses through a queue, which can maintain the order of these data processing requests and data processing responses. In addition, the data processing requests and data processing responses added to the queue are generally not deleted before the queue is full, so the persistence of the data processing requests and data processing responses on the first transit node can be achieved.
[0157] Optionally, both the first queue and the second queue can be determined by subscription. For example, the target processing node can send the identifier of the first queue to the first transfer node to subscribe the first queue to the first transfer node. Afterwards, if the first transfer node receives a data processing request that needs to be sent to the target processing node, the data processing request can be added to the first queue, and then the data processing request in the first queue is sent to the target processing node. Similarly, the target request node can send the identifier of the second queue to the first transfer node to subscribe the second queue to the first transfer node. Afterwards, if the first transfer node receives a data processing response that needs to be sent to the target request node, the data processing response can be added to the second queue, and then the data processing response in the second queue is sent to the target request node.
[0158] The number of the first queue and the second queue can be one or more, and the embodiments of the present application do not limit this. For example, multiple groups of queues can be maintained in the first transfer node, including a group of first queues and a group of second queues, and each group of queues includes at least one queue. When the target processing node subscribes to the first queue, it can send the identifiers of each queue in the group of first queues to the first transfer node, or send the identifier of the group of first queues. When the target request node subscribes to the second queue, it can send the identifiers of each queue in the group of second queues to the first transfer node, or send the identifier of the group of second queues. Among them, the identifier of a group of queues can be called a topic.
[0159] Furthermore, when there are multiple first queues, the target request node may also specify which first queue the data processing request enters (for example, the data processing request carries the identifier of the first queue it needs to enter). Alternatively, the target request node may not specify which first queue the data processing request enters, but the first transit node may determine (such as by random means, or according to the lengths of each first queue, etc.) the first queue the data processing request enters. Similarly, when there are multiple second queues, the target processing node may also specify which second queue the data processing response enters (for example, the data processing response carries the identifier of the second queue it needs to enter). Alternatively, the target processing node may not specify which second queue the data processing response enters, but the first transit node may determine (such as by random means, or according to the lengths of each second queue, etc.) the second queue the target processing response enters.
[0160] When the first transfer node maintains the above-mentioned multiple queues, the first transfer node can be a Kafka (an open source stream processing platform) node or a NATS (a fire-and-forget messaging system) node. When the second transfer node stores information in a cache manner, the second transfer node can be a Remote Dictionary Server (Redis) node.
[0161] Figure 8 A flowchart of another data processing method provided in an embodiment of the present application, the data processing method can be used for a target request node in a data processing system provided in an embodiment of the present application. The target request node can be any request node in at least one request node in the data processing system.
[0162] Step 401: Obtain data to be processed from a data processing system.
[0163] The requesting node may be communicatively connected to a device outside the data processing system, and the requesting node may receive data to be processed sent by the device outside the data processing system.
[0164] For example, the devices outside the data processing system may include multiple IoT devices, and at least some of the multiple IoT devices use different data formats. Therefore, the IoT devices need to send the data they use to the data processing system for data format conversion. At this time, the IoT devices can send the data they use to the target request node so that the target request node can obtain the data to be processed, and then process the data to be processed through subsequent steps.
[0165] It is understandable that the data to be processed may not be data sent by a device outside the data processing system. For example, the data to be processed is data input by a user into a target request node, which is not limited in this embodiment of the present application.
[0166] Step 402: Send a data processing request to the first transit node according to the data to be processed, so that the first transit node sends the data processing request to the target processing node, and the target processing node performs the data processing operation requested by the data processing request, and the target processing node is any processing node among the at least one processing node.
[0167] The target processing node may encapsulate a data processing request according to the data to be processed, and then send the data processing request to the first transfer node.
[0168] Step 403: Obtain the processing result of the data processing operation from the target processing node.
[0169] After obtaining the processing result, the data requesting node may transmit the processing result to other devices, or further process the processing result, etc. The embodiment of the present application does not limit the operation of the data requesting node after obtaining the processing result.
[0170] In summary, the data processing system provided by the present application includes: a first transit node, at least one request node and at least one processing node. Each request node can send a data processing request to any processing node through the first transit node, so that the processing node performs the data processing operation requested by the data processing request, and transmits the result of the data processing operation to the request node. It can be seen that on the basis of the data processing system provided by the present application, the present application provides a new data processing method, which enriches the data processing method.
[0171] In addition, the data processing system and data processing method provided by the present application can decouple the request node from the processing node, so that the number of both the request node and the processing node can be expanded. Moreover, when the number of at least one of the request node and the processing node is expanded, the performance of the data processing system will change. Therefore, the present application can support flexible adjustment of the performance of the data processing system.
[0172] Optionally, the data transmission system also includes a second transfer node, and the target processing node is used to send the processing result to the second transfer node, and send a data processing response to the data processing request to the first transfer node, and the data processing response is used to indicate that the processing result has been obtained; in step 403, the target request node can obtain the processing result from the second transfer node according to the data processing response after receiving the data processing response sent by the first transfer node. In this case, the processing result does not need to be transmitted on the link between the target request node, the first transfer node and the target processing node, thereby reducing the transmission load of the link, reducing the bandwidth of the link, reducing the resource consumption of the data processing system, and improving the throughput of the data processing system.
[0173] In the embodiment of the present application, the target processing node transmits the processing result to the target request node through the second transfer node as an example. Optionally, the target processing node can also be connected to the target request node for communication, and the target request node can directly receive the processing result sent by the target processing node, which is not limited in the embodiment of the present application.
[0174] Optionally, the above data processing request carries: the identifier of the script corresponding to the data processing request; the script corresponding to the data processing request is: the script required to run to execute the data processing operation requested by the data processing request; the target processing node can use the processing unit created in the processing node for running the script corresponding to the data processing request to execute the data processing operation requested by the data processing request. It can be seen that the data processing request in the present application carries the script identifier, so there is no need to carry the script in the data processing request, so that the data volume of the data processing request is low, the transmission load of the link between the target request node, the first transit node and the target processing node is low, the bandwidth of the link is reduced, the resource consumption of the data processing system is reduced, and the throughput of the data processing system is improved.
[0175] Optionally, the data processing request may also carry metadata required to run the script corresponding to the data processing request, and the processing unit may run the script corresponding to the data processing request according to the metadata. Of course, the data processing request may not carry the metadata, and this embodiment of the application does not limit this.
[0176] Optionally, the data processing request may also carry the data to be processed. It is understandable that the data processing request may not carry the data to be processed, but may carry the identification of the data to be processed. In this case, the target requesting node may also transmit the data to be processed to the second transit node, and after receiving the data processing request, the target processing node may obtain the data to be processed from the second transit node according to the identification of the data to be processed carried by the data processing request, and then process the data to be processed.
[0177] Fig. 9 Another data processing method provided in the embodiment of the present application can be used in the data processing system provided in the embodiment of the present application. Fig. 9 As shown, the data processing method may include:
[0178] Step 501: The target requesting node obtains the data to be processed from the data processing system.
[0179] Step 501 can refer to Figure 8 The relevant explanations in the illustrated embodiment (such as step 401, etc.) will not be repeated in detail in the embodiment of the present application.
[0180] The data to be processed obtained by the target request node may be data to be converted into a format, or the data to be processed may be request data for which response data is to be obtained.
[0181] For example, the data to be processed may be uplink data sent by an IoT device outside the data processing system to the IoT platform. In this case, the data processing system needs to convert the format of the data to be processed into the format used by the IoT platform.
[0182] As another example, the data to be processed may be downlink data used by the IoT platform to be sent to the IoT device. In this case, the data processing system needs to convert the format of the data to be processed into the format used by the IoT device.
[0183] Optionally, when the data to be processed is related to the Internet of Things platform, at least one node in the data processing system provided in the embodiment of the present application can be integrated with the Internet of Things platform, or the entire data processing system can be independent of the Internet of Things platform. Among them, the Internet of Things platform is a platform based on cloud computing technology, which can provide multi-protocol / multi-type device access, device management, device monitoring, unified device model (meaning that the data of devices of different models are formatted in a unified manner), device shadow, and data processing and analysis capabilities.
[0184] As another example, the data to be processed is authentication data, and the data processing system needs to process the authentication data to obtain an authentication result of the authentication data.
[0185] Step 502: The target requesting node sends a data processing request to the first transit node according to the acquired data to be processed.
[0186] Step 502 can refer to Figure 8 The relevant explanations in the illustrated embodiment (such as step 402, etc.) will not be repeated in detail in the embodiment of the present application.
[0187] The target processing node may encapsulate a data processing request according to the data to be processed, and then send the data processing request to the first transfer node. For example, the data processing request (such as the payload of the data processing request) may carry: the data to be processed (msg), the identifier of the data processing request (request ID), the identifier of the script corresponding to the data processing request (ID), and the metadata required to run the script (metadata).
[0188] Step 503: The first transit node sends a data processing request to the target processing node.
[0189] Step 503 can refer to Figure 7 The relevant explanations in the illustrated embodiment (such as step 302, etc.) will not be repeated in detail in the embodiment of the present application.
[0190] For example, multiple queues are maintained in the first transfer node. Before step 503, the target processing node may send a subscription request to the first transfer node, and the subscription request carries the identifier of a queue group (including at least one first queue) in the multiple queues. After the first transfer node receives the data processing request sent by the target request node in step 502, it can add the data processing request to a first queue in the queue group according to the identifier of the queue group. In addition, the first transfer node can send the data processing request in the queue group to the target processing node in step 503.
[0191] When the first transit node sends a data processing request in a first queue to the target processing node, it may send a group of data processing requests in the first queue to the target processing node each time, and after receiving a data processing response to the group of data processing requests sent by the target processing node, it may continue to send the next group of data processing requests in the first queue to the target processing node. Each group of data processing requests includes at least one continuous data processing request.
[0192] Step 504: The target processing node executes the data processing operation requested by the data processing request.
[0193] Step 504 can refer to Figure 3 The relevant explanations in the illustrated embodiment (such as step 202, etc.) will not be repeated in detail in the embodiment of the present application.
[0194] The target processing node will receive one or more data processing requests at the same time. For a data processing request received by the target processing node, the target processing node will first parse the data processing request to determine the data to be processed carried by the data processing request, the identifier of the data processing request, the identifier of the script corresponding to the data processing request, and the metadata required to run the script. After that, the target processing node needs to determine the processing unit in the target processing node for running the script according to the identifier of the script corresponding to the data processing request. Finally, the target processing node can use the processing unit to perform the data processing operation requested by the data processing request according to the metadata.
[0195] It should be noted that a processing unit is a unit with processing functions in a target processing node, and a processing unit can be created and destroyed by the target processing node. When a processing unit executes a data processing operation requested by a data processing request, it will run the script corresponding to the data processing request, build an environment for executing the data processing operation, and then execute the data processing operation to process the data to be processed in the data processing request.
[0196] The target processing node may or may not have the processing unit created. When the processing unit is not created in the target processing node, the target processing node may create the processing unit.
[0197] The target processing node can immediately create the processing unit when it obtains the identifier of the script through parsing and determines that the processing unit has not been created in the target processing node. Figure 3 In the scheme of adding a processing unit in the embodiment shown, when adding a processing unit, a processing unit for running the script is created, which is not limited in the embodiment of the present application.
[0198] Optionally, when the target processing node receives multiple data processing requests simultaneously, the target processing node may parse the data processing requests in parallel using multiple threads, and determine a processing unit for running the script corresponding to the data processing request. Fig.10 As shown, it is assumed that the first transit node sends the data processing requests 1, 2, 3, and 4 in the first queue to the target processing node at the same time. The target processing node can use threads 1, 2, 3, and 4 to parse the data processing requests 1, 2, 3, and 4 one by one, and determine the processing units for running the scripts corresponding to the data processing requests 1, 2, 3, and 4, wherein threads 1, 2, 3, and 4 work in parallel.
[0199] Step 505: The target processing node sends the processing result to the second transfer node.
[0200] Step 505 can refer to Figure 3The relevant explanations in the illustrated embodiments are not repeated here in the embodiments of the present application.
[0201] After obtaining the processing result of the data processing operation requested by a data processing request, the target processing node can send the processing result to the second transfer node. For example, the processing result can carry the processed data to be processed and the identifier of the data processing request, and the identifier of the data processing request is used to indicate that the processing result is the processing result of the data processing request.
[0202] After receiving the processing result, the second transit node may store the processing result so that a subsequent target requesting node can read the processing result.
[0203] Step 506: The target processing node sends a data processing response to the data processing request to the first transit node, where the data processing response is used to indicate that the processing result has been obtained.
[0204] Step 506 can refer to Figure 3 The relevant explanations in the illustrated embodiments are not repeated here in the embodiments of the present application.
[0205] After obtaining the processing result, the target processing node can also send a data processing response to the first transit node. The data processing response does not carry the processing result (such as the processed data to be processed), and the data processing response can carry the identifier of the data processing request and an indication that the processing result has been obtained.
[0206] Step 507: The first transit node sends a data processing response to the target requesting node.
[0207] For example, multiple queues are maintained in the first transfer node. Before step 507, the target request node may send a subscription request to the first transfer node, and the subscription request carries the identifier of a queue group (including at least one second queue) in the multiple queues. After the first transfer node receives the data processing response sent by the target processing node in step 506, it can add the data processing response to a second queue in the queue group according to the identifier of the queue group. In addition, the first transfer node can send the data processing response in the queue group to the target request node in step 507.
[0208] Step 508: The target requesting node obtains the processing result from the second transit node according to the data processing response.
[0209] After receiving the data processing response, the target requesting node can obtain the processing result carrying the identifier of the data processing request from the second transit node according to the identifier of the data processing request in the data processing response.
[0210] For example, the target requesting node may send a read request to the second transit node according to the data processing response, and the read request carries the identifier of the data processing request. After receiving the read request, the second transit node may search for a processing result carrying the identifier of the data processing request, and send the processing result to the target requesting node.
[0211] In summary, the data processing system provided by the present application includes: a first transit node, at least one request node and at least one processing node. Each request node can send a data processing request to any processing node through the first transit node, so that the processing node performs the data processing operation requested by the data processing request, and transmits the result of the data processing operation to the request node. It can be seen that on the basis of the data processing system provided by the present application, the present application provides a new data processing method, which enriches the data processing method.
[0212] In addition, the data processing system and data processing method provided by the present application can decouple the request node from the processing node, so that the number of both the request node and the processing node can be expanded. Moreover, when the number of at least one of the request node and the processing node is expanded, the performance of the data processing system will change. Therefore, the present application can support flexible adjustment of the performance of the data processing system.
[0213] exist Fig. 9 In the embodiment shown, the target processing node can also manage the number of processing units in the target processing node, for example, Figure 3 In the embodiment shown, the number of processing units is increased or decreased according to the first number, the second number and the auxiliary number counted. Figure 3 The illustrated embodiment moves the script between memory and a second transit node.
[0214] It should be noted that Figure 3 , Figure 7 , Figure 8 and Fig. 9 The solutions in the four embodiments shown can be combined with each other, and the explanations and effects of similar solutions can be referenced to each other, and the embodiments of the present application will not be described in detail here.
[0215] According to the above embodiment of the data transmission method, the target request node is used to obtain the data to be processed of the data processing system, and send a data processing request to the first transit node according to the data to be processed.
[0216] The first transfer node is used to send a data processing request to the target processing node.
[0217] The target processing node is used to execute the data processing operation requested by the data processing request and transmit the processing result of the data processing operation to the target requesting node;
[0218] The target requesting node is used to obtain the processing result of the data processing operation from the target processing node.
[0219] Optionally, the data transmission system further includes a second transfer node, and the target processing node is used to send the processing result to the second transfer node, and send a data processing response of the data processing request to the first transfer node.
[0220] The first transfer node is further used to send the data processing response to the target requesting node.
[0221] The target requesting node is used to obtain the processing result from the second transfer node according to the data processing response, and the data processing response is used to indicate that the processing result has been obtained.
[0222] Optionally, the data processing request carries: an identifier of a script corresponding to the data processing request; the script corresponding to the data processing request is: a script required to be run to execute the data processing operation requested by the data processing request;
[0223] The target processing nodes are used to:
[0224] The data processing operation is performed using a processing unit created in the processing node and used to run a script corresponding to the data processing request.
[0225] Optionally, the data processing request sent by the first transfer node includes: different data processing requests corresponding to the same script; the data processing operations requested by the different data processing requests are executed by the same processing unit.
[0226] Optionally, the target processing node is further configured to:
[0227] Determining a first number of the data processing requests received within a most recent first time period multiple times;
[0228] When the growth rate of the first number is greater than a first probability threshold, adding the processing unit in the target processing node, the first probability threshold being greater than zero;
[0229] When the growth rate of the first number is less than a second probability threshold, reducing the processing unit in the target processing node, the second probability threshold being less than zero.
[0230] Optionally, the target processing node is further configured to:
[0231] Determining multiple times a second number of the data processing requests corresponding to the same script received within a recent second time period;
[0232] When the growth rate of the second number is greater than a third probability threshold, adding a processing unit in the target processing node for running the same script, and the third probability threshold is greater than zero;
[0233] When the growth rate of the second number is less than a fourth probability threshold, reducing the processing unit in the target processing node for running the same script, and the fourth probability threshold is less than zero.
[0234] Optionally, the data processing system further includes: a second transfer node, and the target processing node is used to:
[0235] When m processing units are added to the processing node, x script acquisition operations are performed to obtain n scripts for the m processing units to run; wherein m≥x≥n≥1; one script acquisition operation corresponds to one script among the n scripts, and the script acquisition operation is used to: when a corresponding script is stored in the memory of the target processing unit, acquire the corresponding script from the memory; when the corresponding script is not stored in the memory, send an acquisition request for the corresponding script to the second transfer node, and receive the corresponding script sent by the second transfer node according to the acquisition request;
[0236] The m processing units are created according to the n scripts.
[0237] Optionally, m>x=n.
[0238] Optionally, the target processing node is further configured to:
[0239] Counting the frequency of execution of each script stored in the memory and the second transfer node;
[0240] When the frequency at which the first script stored in the second transfer node is executed is greater than a first frequency threshold, moving the first script from the second transfer node to the memory;
[0241] When the frequency at which the second script stored in the memory is executed is greater than a second frequency threshold and less than or equal to the first frequency threshold, the second script is moved from the memory to the second transit node.
[0242] Optionally, the target processing node is also used to: when receiving multiple data processing requests sent by the first transit node, use multiple threads in parallel to determine the processing units in the processing node for running the scripts corresponding to the multiple data processing requests respectively.
[0243] Optionally, the first transit node is used for:
[0244] Upon receiving multiple data processing requests sent by the target requesting node, sending a first data processing request among the multiple data processing requests to the target processing node;
[0245] After receiving the data processing response to the first data processing request, a second data processing request among the multiple data processing requests is sent to the target processing node.
[0246] Optionally, the first transit node is used to: after receiving a data processing request sent by a target request node, add the data processing request to a first queue, and send the data processing request in the first queue to the target processing node; after receiving a data processing response to the data processing request sent by the target processing node, add the data processing response to a second queue, and send the data processing response in the second queue to the target request node.
[0247] Fig.11 The present invention provides a structural diagram of a data processing device provided in an embodiment of the present invention, wherein the data processing device is used for a target processing node in a data processing system, wherein the data processing system comprises: a first transit node, at least one request node, and at least one processing node, wherein the target processing node is any processing node in the at least one processing node. Fig.11 As shown, the data processing device includes:
[0248] The receiving module 1001 is used to receive a data processing request sent by the first transfer node, wherein the data processing request is sent by a target requesting node to the first transfer node according to the data to be processed by the data processing system, and the target requesting node is any requesting node among the at least one requesting node; the function of the receiving module 1001 can refer to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to the target processing node receiving the data processing request are not described in detail in this embodiment of the present application.
[0249] The execution module 1002 is used to execute the data processing operation requested by the data processing request; the function of the execution module 1002 can refer to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to the target processing node executing the data processing request are not described in detail in this embodiment of the present application.
[0250] The transmission module 1003 is used to transmit the processing result of the data processing operation to the target request node. The function of the transmission module 1003 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to the target processing node transmitting the data processing request to the target request node are not elaborated in the embodiments of the present application.
[0251] Optionally, the data transmission system further includes a second transfer node, and the transmission module 1003 is used to:
[0252] Sending the processing result to the second transfer node;
[0253] A data processing response of the data processing request is sent to the first transfer node, so that the first transfer node sends the data processing response to the target request node, and the target request node obtains the processing result from the second transfer node according to the data processing response, and the data processing response is used to indicate that the processing result has been obtained.
[0254] Optionally, the data processing request carries: data to be processed, and an identifier of a script corresponding to the data processing request; the script corresponding to the data processing request is: a script required to execute the data processing operation requested by the data processing request; the execution module 1002 is used to: execute the data processing operation using a processing unit created in the processing node for running the script corresponding to the data processing request.
[0255] Optionally, the data processing request sent by the first transfer node includes: different data processing requests corresponding to the same script; the data processing operations requested by the different data processing requests are executed by the same processing unit.
[0256] Alternatively, if Fig.12 As shown, in Fig.11 On the basis of, the data processing device further includes:
[0257] The first determination module 1004 is used to repeatedly determine the first number of the data processing requests received within the most recent first time period; the function of the first determination module 1004 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to determining the first number by the target processing node are not described in detail in this embodiment of the present application.
[0258] The first adding module 1005 is used to increase the processing unit in the target processing node when the growth rate of the first number is greater than a first probability threshold, and the first probability threshold is greater than zero; the function of the first adding module 1005 can refer to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to adding a processing unit to the target processing node when the growth rate of the first number is greater than the first probability threshold are not described in detail in the embodiments of the present application.
[0259] The first reduction module 1006 is used to reduce the processing unit in the target processing node when the growth rate of the first number is less than a second probability threshold, and the second probability threshold is less than zero. The function of the first reduction module 1006 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to adding a processing unit to the target processing node when the growth rate of the first number is less than the second probability threshold are not elaborated in the embodiments of the present application.
[0260] Alternatively, if Fig.12 As shown, the data processing device also includes:
[0261] The second determination module 1007 is used to determine multiple times the second number of the data processing requests corresponding to the same script received in the most recent second time period; the function of the second determination module 1007 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to determining the second number by the target processing node are not described in detail in the embodiments of the present application.
[0262] The second adding module 1008 is used to add a processing unit for running the same script in the target processing node when the growth rate of the second number is greater than a third probability threshold, and the third probability threshold is greater than zero; the function of the second adding module 1008 can refer to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to adding a processing unit to the target processing node when the growth rate of the second number is greater than the third probability threshold are not described in detail in the embodiments of the present application.
[0263] The second reduction module 1009 is used to reduce the processing units used to run the same script in the target processing node when the growth rate of the second number is less than a fourth probability threshold, and the fourth probability threshold is less than zero. The function of the second reduction module 1009 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to adding a processing unit to the target processing node when the growth rate of the second number is less than the fourth probability threshold are not elaborated in the embodiments of the present application.
[0264] Optionally, the data processing system also includes: a second transfer node, the target processing node is used for (such as the above-mentioned first adding module or the second adding module is used for): when m processing units are added to the processing node, performing x script acquisition operations to obtain n scripts used to run by the m processing units; wherein, m≥x≥n≥1; one script acquisition operation corresponds to one script among the n scripts, and the script acquisition operation is used for: when a corresponding script is stored in the memory of the target processing unit, acquiring the corresponding script from the memory; when the corresponding script is not stored in the memory, sending a request for acquiring the corresponding script to the second transfer node, and receiving the corresponding script sent by the second transfer node according to the acquisition request; and creating the m processing units according to the n scripts.
[0265] Optionally, m>x=n.
[0266] Alternatively, if Fig.12 As shown, the data processing device also includes:
[0267] Statistics module 1010, used to count the frequency of execution of each script stored in the memory and the second transfer node; the function of the statistics module can refer to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to the frequency of the target processing node statistics script being executed are not repeated in this embodiment of the present application.
[0268] A first moving module 1011 is configured to move the first script from the second transfer node to the memory when the frequency at which the first script stored in the second transfer node is executed is greater than a first frequency threshold;
[0269] The second moving module 1012 is configured to move the second script stored in the memory from the memory to the second transfer node when the frequency at which the second script stored in the memory is executed is greater than a second frequency threshold and less than or equal to the first frequency threshold.
[0270] The functions of the first mobile module and the second mobile module can refer to Figure 3 , Figure 7 , Figure 8 , Fig. 9The contents related to moving the script between the memory of the target processing node and the second transfer node are not described in detail in this embodiment of the present application.
[0271] Optionally, the data processing device further includes:
[0272] The third determination module 1013 is used to determine the processing units in the processing node for respectively running the scripts corresponding to the multiple data processing requests based on the multiple data processing requests using multiple threads in parallel when receiving the multiple data processing requests sent by the first transfer node. The function of the third determination module can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to the target processing node using multiple threads to determine the processing unit in parallel are not described in detail in this embodiment of the present application.
[0273] In summary, the data processing system provided by the present application includes: a first transit node, at least one request node and at least one processing node. According to the function of the data processing device provided by the embodiment of the present application, each request node can send a data processing request to any processing node through the first transit node, so that the processing node performs the data processing operation requested by the data processing request, and transmits the result of the data processing operation to the request node. It can be seen that on the basis of the data processing system provided by the present application, the present application provides a new data processing method, which enriches the data processing method.
[0274] In addition, the data processing system and data processing method provided by the present application can decouple the request node from the processing node, so that the number of both the request node and the processing node can be expanded. Moreover, when the number of at least one of the request node and the processing node is expanded, the performance of the data processing system will change. Therefore, the present application can support flexible adjustment of the performance of the data processing system.
[0275] Fig.13 This is a schematic diagram of the structure of another data processing device provided in an embodiment of the present application, wherein the data processing device is used as a first transit node in a data processing system, and the data processing system further includes: at least one request node and at least one processing node. Fig.13 As shown, the data processing device includes:
[0276] The first receiving module 1101 is used to receive a data processing request sent by a target requesting node, wherein the data processing request is sent by the target requesting node according to the data to be processed by the data processing system, and the target requesting node is any requesting node among the at least one requesting node; the function of the first receiving module 1101 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The content related to the first transit node receiving the data processing request, such as step 301, will not be repeated here in this embodiment of the present application.
[0277] The first sending module 1102 is used to send the data processing request to the target processing node, so that the target processing node performs the data processing operation requested by the data processing request and transmits the processing result of the data processing operation to the target request node. The target processing node is any processing node in the at least one processing node. The function of the first sending module 1102 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The content related to the first transit node sending a data processing request, such as step 302, will not be repeated here in this embodiment of the present application.
[0278] Optionally, the data transmission system further includes a second transfer node, and the target processing node is used to send the processing result to the second transfer node. Fig.14 As shown, in Fig.13 On the basis of, the data processing device further includes:
[0279] The second receiving module 1103 is used to receive the data processing response of the data processing request sent by the target processing node, and the data processing response is used to indicate that the processing result has been obtained; the function of the second receiving module can refer to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to the first transfer node receiving the data processing response are not described in detail in this embodiment of the present application.
[0280] The second sending module 1104 is used to send the data processing response to the target request node, so that the target request node can obtain the processing result from the second transfer node according to the data processing response. Figure 3 , Figure 7 , Figure 8 , Fig. 9 The contents related to the first transfer node sending a data processing response are not described in detail in this embodiment of the present application.
[0281] Optionally, the first sending module 1101 is used to:
[0282] Upon receiving multiple data processing requests sent by the target requesting node, sending a first data processing request among the multiple data processing requests to the target processing node;
[0283] After receiving the data processing response to the first data processing request, a second data processing request among the multiple data processing requests is sent to the target processing node.
[0284] Alternatively, if Fig.14 As shown, the data processing device also includes:
[0285] The first adding module 1105 is used to add the data processing request to the first queue; the function of the first adding module can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The content related to the first transit node adding the data processing request to the first queue is not repeated here in this embodiment of the present application.
[0286] The second adding module 1106 is used to add the data processing response to the second queue; the function of the second adding module can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The content related to the first transfer node adding the data processing response to the second queue is not repeated in this embodiment of the present application.
[0287] The first sending module 1101 is used for: sending the data processing request in the first queue to the target processing node;
[0288] The second sending module 1104 is used to send the data processing response in the second queue to the target requesting node.
[0289] In summary, the data processing system provided by the present application includes: a first transit node, at least one request node and at least one processing node. According to the function of the data processing device provided by the embodiment of the present application, each request node can send a data processing request to any processing node through the first transit node, so that the processing node performs the data processing operation requested by the data processing request, and transmits the result of the data processing operation to the request node. It can be seen that on the basis of the data processing system provided by the present application, the present application provides a new data processing method, which enriches the data processing method.
[0290] In addition, the data processing system and data processing method provided by the present application can decouple the request node from the processing node, so that the number of both the request node and the processing node can be expanded. Moreover, when the number of at least one of the request node and the processing node is expanded, the performance of the data processing system will change. Therefore, the present application can support flexible adjustment of the performance of the data processing system.
[0291] Fig.15A schematic diagram of the structure of another data processing device provided in an embodiment of the present application, wherein the data processing device is used for a target request node in a data processing system, wherein the data processing system comprises: a first transit node, at least one request node and at least one processing node, wherein the target request node is any request node in the at least one request node. Fig.15 As shown, the data processing device includes:
[0292] The first acquisition module 1201 is used to acquire the data to be processed by the data processing system; the function of the first acquisition module 1201 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The content related to the target request node obtaining the data to be processed, such as step 401, will not be described in detail in this embodiment of the present application.
[0293] The sending module 1202 is used to send a data processing request to the first transfer node according to the data to be processed, so that the first transfer node sends the data processing request to the target processing node, and the target processing node performs the data processing operation requested by the data processing request, and the target processing node is any processing node in the at least one processing node; the function of the sending module 1202 can refer to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The content related to the target request node sending a data processing request to the first transit node, such as step 402, will not be repeated here in this embodiment of the present application.
[0294] The second acquisition module 1203 is used to obtain the processing result of the data processing operation from the target processing node. The function of the second acquisition module 1203 can be referred to Figure 3 , Figure 7 , Figure 8 , Fig. 9 The content related to the target requesting node obtaining the processing result from the first transfer node, such as step 403, will not be repeated in this embodiment of the present application.
[0295] Optionally, the data transmission system further includes a second transfer node, the target processing node is used to send the processing result to the second transfer node, and send a data processing response of the data processing request to the first transfer node, the data processing response is used to indicate that the processing result has been obtained;
[0296] The second acquisition module 1203 is used to: receive the data processing response sent by the first transfer node; and acquire the processing result from the second transfer node according to the data processing response.
[0297] In summary, the data processing system provided by the present application includes: a first transit node, at least one request node and at least one processing node. According to the function of the data processing device provided by the embodiment of the present application, each request node can send a data processing request to any processing node through the first transit node, so that the processing node performs the data processing operation requested by the data processing request, and transmits the result of the data processing operation to the request node. It can be seen that on the basis of the data processing system provided by the present application, the present application provides a new data processing method, which enriches the data processing method.
[0298] In addition, the data processing system and data processing method provided by the present application can decouple the request node from the processing node, so that the number of both the request node and the processing node can be expanded. Moreover, when the number of at least one of the request node and the processing node is expanded, the performance of the data processing system will change. Therefore, the present application can support flexible adjustment of the performance of the data processing system.
[0299] Fig.16 Another data processing device provided in an embodiment of the present application is as follows: Fig.16 As shown, the data processing device includes: a processor 1301 and a memory 1302, the memory 1302 stores a program, and the processor 1301 is used to execute the program stored in the memory 1302 to implement any data processing method provided in the embodiment of the present application for any node (such as a target request node, a first transfer node, a target processing node, etc.) in the data processing system, such as Figure 3 , Figure 7 , Figure 8 or Fig. 9 The data processing method shown is used for any node.
[0300] The present application also provides a non-transitory computer-readable storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes any data processing method provided in the present application for any node (such as a target request node, a first transfer node, a target processing node, etc.) in a data processing system, such as Figure 3 , Figure 7 , Figure 8 or Fig. 9 The data processing method shown is used for any node.
[0301] The present application also provides a computer program product including instructions. When the computer program product is executed on a computer, the computer executes any data processing method provided in the present application for any node (such as a target request node, a first transfer node, a target processing node, etc.) in a data processing system. Figure 3 , Figure 7 , Figure 8 or Fig. 9 The data processing method shown is used for any node.
[0302] It should be noted that the data involved in this application (including but not limited to data for processing, data for storage, etc.) are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the data to be processed involved in this application are all obtained with full authorization.
[0303] In the present disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.
[0304] It should be noted that the method embodiments provided in the embodiments of the present application can be referenced with the corresponding device embodiments, and the embodiments of the present application are not limited thereto. The sequence of the steps of the method embodiments provided in the embodiments of the present application can be appropriately adjusted, and the steps can also be increased or decreased accordingly according to the circumstances. Any technician familiar with the technical field can easily think of the changed methods within the technical scope disclosed in the present application, and they should be included in the protection scope of the present application, so they will not be repeated.
Claims
1. A data processing method, characterized in that: The method is used for a target processing node in a data processing system, the data processing system comprising: a first transit node, at least one request node and at least one processing node, the target processing node being any processing node in the at least one processing node, the method comprising: receiving a data processing request sent by the first transfer node, wherein the data processing request is sent by a target request node to the first transfer node according to the data to be processed by the data processing system, and the target request node is any one of the at least one request node; the data processing request carries: an identifier of a script corresponding to the data processing request; Execute the data processing operation requested by the data processing request; transmitting the processing result of the data processing operation to the target requesting node; Determining a first number of the data processing requests received within a most recent first time period multiple times; When the growth rate of the first number is greater than a first probability threshold, adding a processing unit in the target processing node, the first probability threshold being greater than zero; The target processing node is used for: When m processing units are added to the processing node, script acquisition operations are performed x times to obtain n scripts for the m processing units to run; wherein m>x=n≥1; one script acquisition operation corresponds to one script among the n scripts; there are multiple processing units for running the same script among the m processing units, and for the multiple processing units for running the same script, the target processing node performs one script acquisition operation for the same script; The m processing units are created according to the n scripts.
2. The method according to claim 1, characterized in that The data transmission system further includes a second transfer node, which transmits the processing result of the data processing operation to the target request node, including: Sending the processing result to the second transfer node; A data processing response of the data processing request is sent to the first transfer node, so that the first transfer node sends the data processing response to the target request node, and the target request node obtains the processing result from the second transfer node according to the data processing response, and the data processing response is used to indicate that the processing result has been obtained.
3. The method according to claim 1 or 2, characterized in that: The script corresponding to the data processing request is: a script required to be run to execute the data processing operation requested by the data processing request; Executing the data processing operation requested by the data processing request includes: The data processing operation is performed using a processing unit created in the processing node and used to run a script corresponding to the data processing request.
4. The method according to claim 3, characterized in that The data processing requests sent by the first transfer node include: different data processing requests corresponding to the same script; the data processing operations requested by the different data processing requests are executed by the same processing unit.
5. The method according to claim 3 or 4, characterized in that: The method further comprises: When the growth rate of the first number is less than a second probability threshold, reducing the processing unit in the target processing node, the second probability threshold being less than zero.
6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: Determining multiple times a second number of the data processing requests corresponding to the same script received within a recent second time period; When the growth rate of the second number is greater than a third probability threshold, adding a processing unit in the target processing node for running the same script, and the third probability threshold is greater than zero; When the growth rate of the second number is less than a fourth probability threshold, reducing the processing unit in the target processing node for running the same script, and the fourth probability threshold is less than zero.
7. The method according to claim 5 or 6, characterized in that: The data processing system further includes: a second transfer node, The script acquisition operation is used to: when the corresponding script is stored in the memory of the target processing unit, acquire the corresponding script from the memory; when the corresponding script is not stored in the memory, send an acquisition request for the corresponding script to the second transfer node, and receive the corresponding script sent by the second transfer node according to the acquisition request.
8. The method according to claim 7, characterized in that The method further comprises: Counting the frequency of execution of each script stored in the memory and the second transfer node; When the frequency at which the first script stored in the second transfer node is executed is greater than a first frequency threshold, moving the first script from the second transfer node to the memory; When the frequency at which the second script stored in the memory is executed is greater than a second frequency threshold and less than or equal to the first frequency threshold, the second script is moved from the memory to the second transit node.
9. The method according to any one of claims 3 to 8, characterized in that: Before executing the data processing operation requested by the data processing request, the method further includes: When multiple data processing requests sent by the first transfer node are received, multiple threads are used in parallel to determine the processing units in the processing node for respectively running the scripts corresponding to the multiple data processing requests according to the multiple data processing requests.
10. A data processing method, characterized in that: The method is used for a first transit node in a data processing system, the data processing system further comprising: at least one request node and at least one processing node, the method comprising: receiving a data processing request sent by a target requesting node, wherein the data processing request is sent by the target requesting node according to the data to be processed by the data processing system, and the target requesting node is any one of the at least one requesting node; the data processing request carries: an identifier of a script corresponding to the data processing request; Sending the data processing request to a target processing node, so that the target processing node performs the data processing operation requested by the data processing request and transmits the processing result of the data processing operation to the target requesting node, the target processing node being any processing node among the at least one processing node; The target processing node is used for: Determining a first number of the data processing requests received within a most recent first time period multiple times; When the growth rate of the first number is greater than a first probability threshold, adding a processing unit in the target processing node, the first probability threshold being greater than zero; When m processing units are added to the processing node, script acquisition operations are performed x times to obtain n scripts for the m processing units to run; wherein m>x=n≥1; one script acquisition operation corresponds to one script among the n scripts; there are multiple processing units for running the same script among the m processing units, and for the multiple processing units for running the same script, the target processing node performs one script acquisition operation for the same script; The m processing units are created according to the n scripts.
11. The method according to claim 10, characterized in that The data transmission system further includes a second transfer node, and the target processing node is used to send the processing result to the second transfer node. The method further includes: receiving a data processing response to the data processing request sent by the target processing node, wherein the data processing response is used to indicate that the processing result has been obtained; The data processing response is sent to the target requesting node, so that the target requesting node obtains the processing result from the second transit node according to the data processing response.
12. The method according to claim 11, characterized in that The sending the data processing request to the target processing node includes: Upon receiving multiple data processing requests sent by the target requesting node, sending a first data processing request among the multiple data processing requests to the target processing node; After receiving the data processing response to the first data processing request, a second data processing request among the multiple data processing requests is sent to the target processing node.
13. The method according to claim 11 or 12, characterized in that: After receiving the data processing request sent by the target requesting node, the method further includes: Adding the data processing request to a first queue; Sending the data processing request to the target processing node includes: Sending the data processing request in the first queue to the target processing node; After receiving a data processing response to the data processing request sent by the target processing node, the method further includes: adding the data processing response to a second queue; Sending the data processing response to the target requesting node includes: The data processing response in the second queue is sent to the target requesting node.
14. A data processing method, characterized in that: The method is used for a target request node in a data processing system, the data processing system comprising: a first transit node, at least one request node and at least one processing node, the target request node being any request node in the at least one request node, the method comprising: Acquiring data to be processed by the data processing system; Sending a data processing request to the first transfer node according to the data to be processed, so that the first transfer node sends the data processing request to a target processing node, and the target processing node performs the data processing operation requested by the data processing request, the target processing node being any processing node among the at least one processing node; the data processing request carries: an identifier of a script corresponding to the data processing request; Obtaining a processing result of the data processing operation from the target processing node; The target processing node is used for: Determining a first number of the data processing requests received within a most recent first time period multiple times; When the growth rate of the first number is greater than a first probability threshold, adding a processing unit in the target processing node, the first probability threshold being greater than zero; When m processing units are added to the processing node, script acquisition operations are performed x times to obtain n scripts for the m processing units to run; wherein m>x=n≥1; one script acquisition operation corresponds to one script among the n scripts; there are multiple processing units for running the same script among the m processing units, and for the multiple processing units for running the same script, the target processing node performs one script acquisition operation for the same script; The m processing units are created according to the n scripts.
15. The method according to claim 14, characterized in that The data transmission system further includes a second transfer node, the target processing node is used to send the processing result to the second transfer node, and send a data processing response of the data processing request to the first transfer node, the data processing response is used to indicate that the processing result has been obtained; The obtaining of the processing result of the data processing operation from the target processing node includes: Receiving the data processing response sent by the first transfer node; According to the data processing response, the processing result is obtained from the second transfer node.
16. A data processing device, characterized in that: The data processing device includes: a processor and a memory, wherein the memory stores a program, and the processor is used to execute the program stored in the memory to implement the data processing method described in any one of claims 1 to 15.
17. A data processing system, characterized in that: The data processing system comprises: a first transfer node, at least one request node and at least one processing node. The processing node is used to execute the data processing method according to any one of claims 1 to 9; The first transfer node is used to execute the data processing method according to any one of claims 10 to 13; The requesting node is used to execute the data processing method described in claim 14 or 15.
18. A non-transitory computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the data processing method according to any one of claims 1 to 15.
Citation Information
Patent Citations
Method and equipment for processing data
CN102880457A
A method and apparatus for processing data
CN109144735A
Data processing method and device
CN109302448A
Data processing method, device, storage medium and system
CN114398397A
Distributed processing system and distributed processing method
JP2014219859A
