A data processing method and device

Abstract

The present invention discloses a data processing method and device, and relates to the field of computer technology. A specific implementation of the method includes: accessing a real-time data stream, detecting the resource usage rate of each machine on which a logic processing operator depends, and obtaining a total resource usage rate; comparing the total resource usage rate with a preset usage rate range, and using a processing strategy corresponding to the comparison result to adjust the parallelism of each machine on which the logic processing operator depends; using the logic processing operator after adjusting the parallelism to process the real-time data stream, transmitting the processing result to the next logic processing operator for reprocessing, repeating the above parallelism adjustment and data processing process until there is no logic processing operator, and obtaining a total processing result. This implementation automatically adjusts the parallelism of the operator's use of resources according to the size of the data magnitude and the processing speed, without the need for data developers to adjust.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] In the field of big data development technology, real-time computing is increasingly used in a wide range of scenarios, such as real-time product recommendations, real-time decision-making, and real-time matching. Real-time computing engines such as Strom, Spark Streaming, and Flink have emerged one after another, among which Flink has a more prominent advantage. Flink is distributed, and the amount of computing resources used by a task is determined by each operator and the shared chain strategy.

[0003] How to use the least resources to complete real-time computing tasks is a question that needs to be explored. Currently, real-time computing developers usually estimate the amount of data in advance, and then combine the code logic and previously accumulated project experience to formulate the parallelism and shared chain strategies of each operator. The strategy takes effect after the task is started and does not change during the task operation.

[0004] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems: it is difficult to formulate reasonable operator parallelism and shared chain strategies, which requires developers to make multiple adjustments; the effect of coping with data peak and trough periods needs to be improved; resources are continuously occupied and excess resources will not be automatically released. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data processing method and device, which can at least solve the problems in the prior art that it is difficult to formulate reasonable operator parallelism and sharing chain strategies, the effect of coping with data peak and trough periods needs to be improved, and resources are not used reasonably.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0007] Access the real-time data stream, detect the resource usage of each machine that the logical processing operator depends on, and obtain the total resource usage;

[0008] Compare the total resource usage rate with the preset usage rate range, use the processing strategy corresponding to the comparison result, and adjust the parallelism of each machine on which the logical processing operator depends;

[0009] The real-time data stream is processed by the logic processing operator after adjusting the parallelism, and the processing result is transmitted to the next logic processing operator for reprocessing. The above parallelism adjustment and data processing process are repeated until there are no logic processing operators, and the total processing result is obtained.

[0010] Optionally, obtaining the total resource usage rate includes:

[0011] A first preset number of machines with the highest resource utilization rates are determined and averaged to obtain a total resource utilization rate; wherein the processing logic of each machine is the same.

[0012] Optionally, the resources include CPU and memory, and the preset usage range is greater than or equal to a first threshold and less than a second threshold;

[0013] The comparing the total resource usage rate and the preset usage rate range, and using a processing strategy corresponding to the comparison result to adjust the parallelism of each machine on which the logic processing operator depends, includes:

[0014] When the CPU usage is less than the first threshold, if the memory usage is less than the second threshold, the parallelism of the operator is reduced by a second preset amount, otherwise the slot configuration ratio of the operator is reduced; wherein the configuration ratio is CPU configuration amount / memory configuration amount;

[0015] When the CPU usage is within the preset usage range, if the memory usage is less than the first threshold, the operator's parallelism is reduced by a second preset number; if it is greater than or equal to the second threshold, the operator's parallelism is increased by a third preset number; otherwise, no processing is performed;

[0016] When the CPU usage is greater than or equal to the second threshold, if the memory usage is greater than or equal to the first threshold, a third preset number of parallelism is added to the operator, otherwise the slot configuration ratio of the operator is increased.

[0017] Optionally, using a processing strategy corresponding to the comparison result to adjust the parallelism of each machine on which the logic processing operator depends includes:

[0018] If the processing strategy is to reduce the slot configuration ratio of the operator, then the slot configuration ratio of all machines under the operator is reduced; or

[0019] If the processing strategy is to increase the slot configuration ratio of the operator, then the slot configuration ratio of all machines under the operator is increased; or

[0020] If the processing strategy is to reduce the second preset number of parallelism or increase the third preset number of parallelism, the processing strategy, the current number of machines and the number of parallelism under the operator are fed back to the task resource manager, so that the task resource manager can readjust the number of machines under the operator and the number of parallelism under each machine.

[0021] Optionally, also include:

[0022] If the processing strategy is to reduce the second preset number of parallelism for the operator or increase the slot configuration ratio of the operator, the freed resources are released to the shared cluster; or

[0023] If the processing strategy is to increase the third preset number of parallelism for the operator or reduce the slot configuration ratio of the operator, resources are pulled from the shared cluster.

[0024] Optionally, it also includes: determining whether the current resource occupancy has reached a preset resource occupancy upper limit, and if so, stopping pulling resources from the shared cluster.

[0025] To achieve the above object, according to another aspect of an embodiment of the present invention, a data processing device is provided, comprising:

[0026] The detection module is used to access the real-time data stream, detect the resource usage of each machine that the logic processing operator depends on, and obtain the total resource usage;

[0027] An adjustment module is used to compare the total resource usage rate with a preset usage rate range, and use a processing strategy corresponding to the comparison result to adjust the parallelism of each machine on which the logic processing operator depends;

[0028] The processing module is used to process the real-time data stream using the logic processing operator after adjusting the parallelism, transmit the processing result to the next logic processing operator for reprocessing, repeat the above parallelism adjustment and data processing process until there are no logic processing operators, and obtain the total processing result.

[0029] Optionally, the detection module is used to: determine a first preset number of machines with the highest resource utilization and take an average to obtain a total resource utilization; wherein the processing logic of each machine is the same.

[0030] Optionally, the resources include CPU and memory, and the preset usage range is greater than or equal to a first threshold and less than a second threshold;

[0031] The adjustment module is used for:

[0032] When the CPU usage is less than the first threshold, if the memory usage is less than the second threshold, the parallelism of the operator is reduced by a second preset amount, otherwise the slot configuration ratio of the operator is reduced; wherein the configuration ratio is CPU configuration amount / memory configuration amount;

[0033] When the CPU usage is within the preset usage range, if the memory usage is less than the first threshold, the operator's parallelism is reduced by a second preset number; if it is greater than or equal to the second threshold, the operator's parallelism is increased by a third preset number; otherwise, no processing is performed;

[0034] When the CPU usage is greater than or equal to the second threshold, if the memory usage is greater than or equal to the first threshold, a third preset number of parallelism is added to the operator, otherwise the slot configuration ratio of the operator is increased.

[0035] Optionally, the adjustment module is used to:

[0036] If the processing strategy is to reduce the slot configuration ratio of the operator, then the slot configuration ratio of all machines under the operator is reduced; or

[0037] If the processing strategy is to increase the slot configuration ratio of the operator, then the slot configuration ratio of all machines under the operator is increased; or

[0038] If the processing strategy is to reduce the second preset number of parallelism or increase the third preset number of parallelism, the processing strategy, the current number of machines and the number of parallelism under the operator are fed back to the task resource manager, so that the task resource manager can readjust the number of machines under the operator and the number of parallelism under each machine.

[0039] Optionally, it also includes a resource release / pull module for:

[0040] If the processing strategy is to reduce the second preset number of parallelism for the operator or increase the slot configuration ratio of the operator, the freed resources are released to the shared cluster; or

[0041] If the processing strategy is to increase the third preset number of parallelism for the operator or reduce the slot configuration ratio of the operator, resources are pulled from the shared cluster.

[0042] Optionally, the resource releasing / pulling module is further used to:

[0043] Determine whether the current resource usage has reached the preset resource usage upper limit. If so, stop pulling resources from the shared cluster.

[0044] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing electronic device is provided.

[0045] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned data processing methods.

[0046] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above data processing methods is implemented.

[0047] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: Based on Flink real-time computing tasks, the parallelism of the operator's use of resources is automatically adjusted according to the size of the data level and the processing speed, without the need for data developers to adjust. During data peak periods, operators that occupy large resources can be flexibly expanded to avoid full CPU or memory overflow; during data trough periods, resource usage is reduced.

[0048] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0050] Figure 1 is a main flow chart of a data processing method according to an embodiment of the present invention;

[0051] Figure 2(a) is a schematic diagram of a data flow diagram;

[0052] Figure 2(b) is a simplified diagram of the internal structure of the Flink real-time computing project;

[0053] Figure 3 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0054] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0055] Figure 5 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0057] The problems existing in the prior art are described in detail here:

[0058] 1) It is difficult to formulate reasonable operator parallelism and shared chain strategies

[0059] Real-time computing projects usually include multiple operators, each of which plays a different role and processes different amounts of data and logic. Project developers formulate operator parallelism and shared chain strategies based on the logic and magnitude of data processing by the operator. It is often difficult to achieve the best effect with just one setting, and developers need to make multiple adjustments to ensure that the overall resource (CPU and memory) usage of the project is within a reasonable range. For inexperienced developers, it will take longer to adjust the project.

[0060] 2) The effectiveness of coping with data peaks and troughs needs to be improved

[0061] A very common scenario for applying real-time computing is processing real-time user behavior data in e-commerce apps. The amount of user behavior data changes with time periods, with peaks and valleys. The amount of data is large during the day and small at night. According to the existing method, the project uses a constant operator parallelism, that is, the CPU and memory resources occupied by the system are constant throughout the entire period. The CPU and memory usage rates are positively correlated with the amount of data. During the day, the amount of data is large, and the CPU and memory usage rates are relatively reasonable, but it is easy to have a short-term peak data volume that is too large, resulting in memory overflow and overall program downtime. At night, the amount of data is small, and the resource utilization rate is usually very low, which is a waste.

[0062] 3) Resources are continuously occupied and excess resources will not be automatically released

[0063] When the Flink real-time computing project starts running, it will allocate computing resources to each operator according to the user-defined operator parallelism and operator sharing chain strategy; after the program starts, the computing resources occupied by each operator are constant and will not change with the change of the processing data level. Nowadays, Flink adopts a shared cluster resource mode, that is, multiple Flink computing clusters share a cluster pool. If the resource utilization rate of a project is low, it means that the project could have used fewer resources to complete the computing task, but it occupied too many computing resources. Therefore, if the occupied resources can be reduced, these resources can be used by other computing resources.

[0064] See also Figure 1 , which shows a main flow chart of a data processing method provided by an embodiment of the present invention, comprising the following steps:

[0065] S101: access the real-time data stream, detect the resource usage rate of each machine on which the logic processing operator depends, and obtain the total resource usage rate;

[0066] S102: comparing the total resource usage rate with a preset usage rate range, and using a processing strategy corresponding to the comparison result to adjust the parallelism of each machine on which the logic processing operator depends;

[0067] S103: Process the real-time data stream using the logic processing operator after adjusting the parallelism, transmit the processing result to the next logic processing operator for reprocessing, repeat the above parallelism adjustment and data processing process until there is no logic processing operator, and obtain the total processing result.

[0068] In the above implementation, for step S101, Flink, a distributed open source stream data processing engine, is now widely used in big data real-time computing scenarios. Flink real-time computing engineering has many steps to process data, such as the step of receiving data, the step of processing data, and the step of storing data. Each step requires a special component to process, which is called an operator. Each component requires a certain computer machine to process data, and different components have different pressures when processing data. Therefore, when working normally, the required machine resources are also different.

[0069] The Flink real-time computing project is connected to the real-time data stream. Each operator in the project is equipped with a plug-in for dynamically adjusting the operator parallelism. The operator parallelism plug-in is called to first detect the CPU and memory usage of each machine under the operator. This step can be done by directly calling system commands, such as the top command. Among them, this solution does not consider the GPU situation.

[0070] The processing logic of each machine in the same operator is the same, so there will not be a situation where one machine has high CPU usage and low memory usage, while another machine has low CPU usage and high memory usage. The overall situation is basically similar. After collecting the resource usage of each machine, select the 10 machines with the highest resource usage (i.e., the first preset number, which is only an example and can be adjusted in practice) and take the average to obtain the CPU utilization and memory utilization of the entire operator.

[0071] For step S102, in a normal cluster, the CPU and memory ratio of a single computing slot is "1 core with 2G memory", and the CPU and memory utilization should be between 30% and 80%. The query results and processing methods are shown in the following table:

[0072] Table 1

[0073]

[0074] The above lists all the situations. Scenario 1 to Scenario 3 correspond to the situation where the CPU usage is <30%, Scenario 4 to Scenario 6 correspond to the situation where 30%≤CPU usage is <80%, and Scenario 7 to Scenario 9 correspond to the situation where 80%≤CPU usage. Here, 30% and 80% correspond to the first threshold and the second threshold in the preset usage range, respectively. Reducing the parallelism by 5 and increasing the parallelism by 3 correspond to the second preset number and the third preset number, respectively. These values ​​are only examples and can be set according to the staff.

[0075] The operator parallelism is the number of slots. For example, reducing the parallelism by 5 means reducing 5 slots, and increasing the parallelism by 3 means increasing 3 slots. When Fink creates a cluster, it sets the CPU and memory specifications of a single machine, such as 4 cores and 8G. Slots are a further subdivision of a single machine, such as dividing a single machine into 4 slots, each with 1 core and 2G resources, and the slots are isolated from each other.

[0076] The slot configuration ratio is the CPU / memory of the slot. For example, if CPU / memory = 1 / 2, it can be changed to 1 / 1 if it is increased, and to 1 / 4 if it is decreased. Usually, if the ratio is increased or decreased by a certain ratio (the two ratios can be different), the resource ratio will become the other scenarios in the table, and it will not increase or decrease indefinitely.

[0077] For step S103, a Flink computing program includes multiple computing components processing data flow, the simplest flow is receiving data - processing data - storing data, corresponding to the Source operator ---- {a series of logical processing operators} ---- Sink operator, where the Source operator is the operator that receives data, and the Sink operator is the operator that stores data. Usually, the parallelism of the Source operator is consistent with the number of partitions of the data source (data received by Flink), that is, the data in one partition is processed by one slot.

[0078] This solution mainly adjusts the parallelism of the logic processing operator. Flink has a back pressure mechanism in the calculation. The consumption speed of the operator is affected by the previous and next operators. The subsequent logic processing operators are in the order of the calculation process. The logic processing operators close to the source operator will first call the operator parallelism plug-in. After obtaining the processing strategy, it will be fed back to the Flink task resource manager, which will readjust the number of machines under the operator and the number of parallelism under each machine. After the real-time data stream is processed by the logic processing operator with adjusted parallelism, the processing result will be passed to the next logic processing operator for further processing, as shown in Figure 2(a) and Figure 2(b).

[0079] In Flink, there are JM (job Manager) and TM (task Manager). JM is equivalent to the program brain and is responsible for the overall allocation of resources. TM is equivalent to the actual work executor. What to do and how to do it require unified coordination by JM.

[0080] Each operator in the entire Flink computing task is relatively independent but connected to each other. The independence lies in the independence of resources and work content. The connection lies in that the amount of data processed by the operator will be affected by the speed of data processing by the operators before and after. For example, when the Source operator processes data slowly, it will adaptively reduce the rate of receiving data, which is also coordinated by JM. Therefore, when the speed of data processing by a certain operator changes, the amount of data received by the downstream operator may change accordingly. The operators behind should be processed one by one according to this logic in the order of process processing. In some scenarios, the amount of data in the data stream will change over time. To solve this problem, the Flink real-time computing project will periodically adjust the operator parallelism. The cycle size is recommended to be determined according to the actual scenario, such as uniform adjustment at 12 o'clock every night when Flink is idle.

[0081] There are multiple Flink real-time computing projects running simultaneously in the shared cluster. After adjusting the parallelism of an operator in a Flink real-time computing project, the excess resources will be released to the shared cluster, and the insufficient part will be requested from the shared cluster.

[0082] For example, if the parallelism of the Source operator is 10, the parallelism of the logic processing operator is 20, and the parallelism of the Sink operator is 10, and the strategy between operators is non-sharing, one parallelism (i.e. one slot) is allocated 1 core and 2G, and the resources required for the Flink real-time computing project are 40 cores and 80G. According to the settings of this solution, resources can be changed flexibly, but a resource occupancy upper limit value must be set for the Flink real-time computing project. When the overall resource occupancy of the project reaches this value, resources can no longer be requested from the shared cluster to avoid the situation where a Flink real-time computing project occupies all computing resources.

[0083] In addition, through the above method, during data peak periods, operators that occupy large resources can be flexibly expanded to avoid full CPU or memory overflow; during data trough periods, resource usage can be reduced.

[0084] The method provided in the above embodiment sets a plug-in for dynamically adjusting the parallelism of operators. This plug-in can dynamically adjust the parallelism of operators and the slot resource allocation ratio according to the actual resource utilization rate of each operator when Flink is actually working, so as to achieve the purpose of adjusting the parallelism of operators without the need for developers.

[0085] Compared with the prior art, the method provided by the embodiment of the present invention has at least the following beneficial effects:

[0086] 1. The parallelism of resources used by operators is automatically adjusted according to the size and processing speed of data, without the need for data developers to make adjustments themselves.

[0087] 2. During data peak periods, operators that occupy large resources can be flexibly expanded to avoid full CPU or memory overflow; during data trough periods, resource usage can be reduced.

[0088] 3. In the shared cluster resource mode, excess resources are allocated to the shared cluster, and insufficient resources are pulled from the shared cluster, which can greatly improve machine resource utilization.

[0089] See also Figure 3 , which shows a schematic diagram of main modules of a data processing device 300 provided in an embodiment of the present invention, including:

[0090] The detection module 301 is used to access the real-time data stream, detect the resource usage rate of each machine on which the logic processing operator depends, and obtain the total resource usage rate;

[0091] An adjustment module 302 is used to compare the total resource usage rate with a preset usage rate range, and use a processing strategy corresponding to the comparison result to adjust the parallelism of each machine on which the logic processing operator depends;

[0092] The processing module 303 is used to process the real-time data stream using the logic processing operator after adjusting the parallelism, transmit the processing result to the next logic processing operator for reprocessing, repeat the above parallelism adjustment and data processing process until there are no logic processing operators, and obtain the total processing result.

[0093] In the implementation device of the present invention, the detection module 301 is used to:

[0094] A first preset number of machines with the highest resource utilization rates are determined and averaged to obtain a total resource utilization rate; wherein the processing logic of each machine is the same.

[0095] In the implementation device of the present invention, the resources include CPU and memory, and the preset usage rate range is greater than or equal to the first threshold and less than the second threshold;

[0096] The adjustment module 302 is used to:

[0097] When the CPU usage is less than the first threshold, if the memory usage is less than the second threshold, the parallelism of the operator is reduced by a second preset amount, otherwise the slot configuration ratio of the operator is reduced; wherein the configuration ratio is CPU configuration amount / memory configuration amount;

[0098] When the CPU usage is within the preset usage range, if the memory usage is less than the first threshold, the operator's parallelism is reduced by a second preset number; if it is greater than or equal to the second threshold, the operator's parallelism is increased by a third preset number; otherwise, no processing is performed;

[0099] When the CPU usage is greater than or equal to the second threshold, if the memory usage is greater than or equal to the first threshold, a third preset number of parallelism is added to the operator, otherwise the slot configuration ratio of the operator is increased.

[0100] In the implementation device of the present invention, the adjustment module 302 is used to:

[0101] If the processing strategy is to reduce the slot configuration ratio of the operator, then the slot configuration ratio of all machines under the operator is reduced; or

[0102] If the processing strategy is to increase the slot configuration ratio of the operator, then the slot configuration ratio of all machines under the operator is increased; or

[0103] If the processing strategy is to reduce the second preset number of parallelism or increase the third preset number of parallelism, the processing strategy, the current number of machines and the number of parallelism under the operator are fed back to the task resource manager, so that the task resource manager can readjust the number of machines under the operator and the number of parallelism under each machine.

[0104] The implementation device of the present invention also includes a resource release / pull module, which is used to:

[0105] If the processing strategy is to reduce the second preset number of parallelism for the operator or increase the slot configuration ratio of the operator, the freed resources are released to the shared cluster; or

[0106] If the processing strategy is to increase the third preset number of parallelism for the operator or reduce the slot configuration ratio of the operator, resources are pulled from the shared cluster.

[0107] In the implementation device of the present invention, the resource release / pull module is also used to:

[0108] Determine whether the current resource usage has reached the preset resource usage upper limit. If so, stop pulling resources from the shared cluster.

[0109] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0110] Figure 4 An exemplary system architecture 400 to which embodiments of the present invention may be applied is shown, including terminal devices 401 , 402 , 403 , a network 404 and a server 405 (only an example).

[0111] Terminal devices 401, 402, 403 may be various electronic devices with display screens and supporting web browsing, and may be installed with various communication client applications. Users may use terminal devices 401, 402, 403 to interact with server 405 via network 404 to receive or send messages, etc.

[0112] The network 404 is used to provide a medium for communication links between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0113] Server 405 may be a server that provides various services, and is used to perform operations such as calculating operator resource utilization, adjusting operator parallelism, processing data, and transmitting processing results.

[0114] It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the device is generally set in the server 405 .

[0115] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0116] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device suitable for implementing an embodiment of the present invention. Figure 5 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0118] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0119] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0120] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a detection module, an adjustment module, and a processing module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the adjustment module may also be described as an "operator parallelism adjustment module".

[0123] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes the data processing method provided in the embodiment of the present invention.

[0124] Compared with the prior art, the technical solution of the embodiment of the present invention has at least the following beneficial effects:

[0125] 1. The parallelism of resources used by operators is automatically adjusted according to the size and processing speed of data, without the need for data developers to make adjustments themselves.

[0126] 2. During data peak periods, operators that occupy large resources can be flexibly expanded to avoid full CPU or memory overflow; during data trough periods, resource usage can be reduced.

[0127] 3. In the shared cluster resource mode, excess resources are allocated to the shared cluster, and insufficient resources are pulled from the shared cluster, which can greatly improve machine resource utilization.

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Access the real-time data stream, detect the resource usage of each machine that the logical processing operator depends on, and obtain the total resource usage; Compare the total resource usage rate with the preset usage rate range, use the processing strategy corresponding to the comparison result, and adjust the parallelism of each machine on which the logic processing operator depends; the resources include CPU and memory, and the preset usage rate range is greater than or equal to the first threshold and less than the second threshold; the process includes: when the CPU usage rate is less than the first threshold, if the memory usage rate is less than the second threshold, then reduce the parallelism of the operator by a second preset number, otherwise reduce the slot configuration ratio of the operator; wherein the configuration ratio is CPU configuration amount / memory configuration amount; When the CPU usage is within the preset usage range, if the memory usage is less than the first threshold, the operator's parallelism is reduced by a second preset number; if it is greater than or equal to the second threshold, the operator's parallelism is increased by a third preset number; otherwise, no processing is performed; When the CPU usage is greater than or equal to the second threshold, if the memory usage is greater than or equal to the first threshold, increase the third preset number of parallelism for the operator, otherwise increase the slot configuration ratio of the operator; The real-time data stream is processed by the logic processing operator after adjusting the parallelism, and the processing result is transmitted to the next logic processing operator for reprocessing. The above parallelism adjustment and data processing process are repeated until there are no logic processing operators, and the total processing result is obtained.

2. The method according to claim 1, characterized in that The total resource utilization rate is obtained, including: A first preset number of machines with the highest resource utilization rates are determined and averaged to obtain a total resource utilization rate; wherein the processing logic of each machine is the same.

3. The method according to claim 1, characterized in that The using of a processing strategy corresponding to the comparison result to adjust the parallelism of each machine on which the logic processing operator depends includes: If the processing strategy is to reduce the slot configuration ratio of the operator, then the slot configuration ratio of all machines under the operator is reduced; or If the processing strategy is to increase the slot configuration ratio of the operator, then the slot configuration ratio of all machines under the operator is increased; or If the processing strategy is to reduce the second preset number of parallelism or increase the third preset number of parallelism, the processing strategy, the current number of machines and the number of parallelism under the operator are fed back to the task resource manager, so that the task resource manager can readjust the number of machines under the operator and the number of parallelism under each machine.

4. The method according to claim 1, characterized in that: Also includes: If the processing strategy is to reduce the second preset number of parallelism for the operator or increase the slot configuration ratio of the operator, the freed resources are released to the shared cluster; or If the processing strategy is to increase the third preset number of parallelism for the operator or reduce the slot configuration ratio of the operator, resources are pulled from the shared cluster.

5. The method according to claim 4, characterized in that Also includes: Determine whether the current resource usage has reached the preset resource usage upper limit. If so, stop pulling resources from the shared cluster.

6. A data processing device, characterized in that: include: The detection module is used to access the real-time data stream, detect the resource usage of each machine that the logic processing operator depends on, and obtain the total resource usage; An adjustment module is used to compare the total resource usage rate with a preset usage rate range, and use a processing strategy corresponding to the comparison result to adjust the parallelism of each machine on which the logic processing operator depends; The resources include CPU and memory, and the preset usage range is greater than or equal to a first threshold and less than a second threshold; the process includes: when the CPU usage is less than the first threshold, if the memory usage is less than the second threshold, then reducing the parallelism of the operator by a second preset number, otherwise reducing the slot configuration ratio of the operator; wherein the configuration ratio is CPU configuration amount / memory configuration amount; When the CPU usage is within the preset usage range, if the memory usage is less than the first threshold, the operator's parallelism is reduced by a second preset number; if it is greater than or equal to the second threshold, the operator's parallelism is increased by a third preset number; otherwise, no processing is performed; When the CPU usage is greater than or equal to the second threshold, if the memory usage is greater than or equal to the first threshold, increase the third preset number of parallelism for the operator, otherwise increase the slot configuration ratio of the operator; The processing module is used to process the real-time data stream using the logic processing operator after adjusting the parallelism, transmit the processing result to the next logic processing operator for reprocessing, repeat the above parallelism adjustment and data processing process until there are no logic processing operators, and obtain the total processing result.

7. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images