Distributed system data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a distributed system data processing method and device, computer equipment, a computer readable storage medium and a computer program product, and is applied to the technical field of big data, the method is applied to a distributed system, and the distributed system comprises a dispatching center and an actuator. Comprising the following steps: generating a processing task according to the data content of to-be-processed data through the dispatching center; an execution strategy of the processing task is constructed through the dispatching center, and the execution strategy comprises execution time and an execution object; sending the to-be-processed data and the execution time to an executor corresponding to the execution object through the dispatching center; and processing the received data to be processed through the actuator according to the received execution time. By adopting the method, the data processing efficiency of the distributed system can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a distributed system data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] As the degree of business informatization construction continues to develop, the demand for data processing is also increasing, and the corresponding processing tasks are becoming more and more complicated. At present, staff usually write execution codes for the processing tasks of heavy data to be processed in distributed systems, and run the execution codes in distributed systems. However, when this method is used, when the processing tasks are more complex and involve multi-threaded and multi-time point processing, the efficiency of writing execution codes depends on the professional level of the staff, which can easily cause the code writing to take a long time, resulting in low data processing efficiency in distributed systems. Summary of the invention

[0003] Based on this, it is necessary to provide a distributed system data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the data processing efficiency of the distributed system in response to the above technical problems.

[0004] In a first aspect, the present application provides a distributed system data processing method, which is applied to a distributed system, wherein the distributed system includes a scheduling center and an executor, including:

[0005] Generate a processing task according to the data content of the data to be processed by the dispatch center;

[0006] Constructing an execution strategy for the processing task through the scheduling center, wherein the execution strategy includes an execution time and an execution object;

[0007] Sending the data to be processed and the execution time to the executor corresponding to the execution object through the scheduling center;

[0008] The executor processes the received data to be processed according to the received execution time.

[0009] In one embodiment, constructing the execution strategy of the processing task includes: allocating a processing thread pool to the processing task according to the task processing attributes of the processing task; generating the execution time of the processing task according to the processing thread pool corresponding to the processing task and the task processing attributes of the processing task; and selecting an execution object matching the processing task from the multiple preset execution objects according to the task processing attributes of the processing task and execution information of multiple preset execution objects.

[0010] In one embodiment, the processing thread pool includes a fast processing thread pool or a slow processing thread pool; the processing thread pool is allocated to the processing task according to the task processing attribute of the processing task, including: if the task processing attribute of the processing task indicates that the processing task meets the preset execution processing condition, then the fast processing thread pool is allocated as the processing thread pool of the processing task; if the task processing attribute of the processing task indicates that the processing task does not meet the preset execution processing condition, then the slow processing thread pool is allocated as the processing thread pool of the processing task.

[0011] In one of the embodiments, the execution time of the processing task is generated according to the processing thread pool corresponding to the processing task and the task processing attributes of the processing task, including: assigning a priority to the processing task according to the task processing attributes of the processing task, and, according to the task processing attributes of the processing task, establishing an association relationship between the processing task and other processing tasks; generating the execution time of the processing task according to the priority of the processing task and the association relationship between the processing task and other processing tasks.

[0012] In one embodiment, the execution information includes an execution task type and a current execution status; the selecting an execution object matching the processing task from the multiple preset execution objects based on the task processing attributes of the processing task and the execution information of multiple preset execution objects includes: selecting multiple matching objects matching the processing task attributes of the processing task from the multiple preset execution objects based on the execution task types of the multiple preset execution objects; and selecting an execution object whose corresponding current execution status satisfies a preset status condition from the multiple matching objects.

[0013] In one embodiment, the method further includes at least one of the following items: if the current execution state represents that the execution state of the first execution object is an idle state, then determining that the current execution state corresponding to the first execution object satisfies a preset state condition; if the current execution state represents that the execution state of the first execution object is a normal state of the object, then determining that the current execution state corresponding to the first execution object satisfies the preset state condition.

[0014] In a second aspect, the present application further provides a distributed system data processing device, which is applied to a distributed system, wherein the distributed system includes a scheduling center and an executor, including:

[0015] An acquisition module, used to acquire charging and discharging data and grid operation data of the grid to be optimized;

[0016] An evaluation module, configured to evaluate the carbon emission of the power grid to be optimized according to the charging and discharging data and the power grid operation data, and obtain carbon emission information, wherein the carbon emission information includes carbon emission amount and carbon emission object;

[0017] The optimization module is used to optimize the grid charging and discharging strategy of the grid to be optimized according to the carbon emissions, and to optimize the grid power control of the grid to be optimized according to the carbon emissions and the carbon emission objects.

[0018] In a third aspect, the present application further provides a computer device, which is applied to a distributed system, wherein the distributed system includes a scheduling center and an executor, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0019] Generate a processing task according to the data content of the data to be processed by the dispatch center;

[0020] Constructing an execution strategy for the processing task through the scheduling center, wherein the execution strategy includes an execution time and an execution object;

[0021] Sending the data to be processed and the execution time to the executor corresponding to the execution object through the scheduling center;

[0022] The executor processes the received data to be processed according to the received execution time.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium, which is applied to a distributed system, wherein the distributed system includes a scheduling center and an executor, and a computer program is stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0024] Generate a processing task according to the data content of the data to be processed by the dispatch center;

[0025] Constructing an execution strategy for the processing task through the scheduling center, wherein the execution strategy includes an execution time and an execution object;

[0026] Sending the data to be processed and the execution time to the executor corresponding to the execution object through the scheduling center;

[0027] The executor processes the received data to be processed according to the received execution time.

[0028] In a fifth aspect, the present application further provides a computer program product, which is applied to a distributed system, wherein the distributed system includes a scheduling center and an executor, and includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0029] Generate a processing task according to the data content of the data to be processed by the dispatch center;

[0030] Constructing an execution strategy for the processing task through the scheduling center, wherein the execution strategy includes an execution time and an execution object;

[0031] Sending the data to be processed and the execution time to the executor corresponding to the execution object through the scheduling center;

[0032] The executor processes the received data to be processed according to the received execution time.

[0033] The above-mentioned distributed system data processing method, device, computer equipment, computer-readable storage medium and computer program product are applied to a distributed system, wherein the distributed system includes a scheduling center and an executor, and the scheduling center generates a processing task according to the data content of the data to be processed; the scheduling center constructs an execution strategy for the processing task, wherein the execution strategy includes an execution time and an execution object; the scheduling center sends the data to be processed and the execution time to the executor corresponding to the execution object; the executor processes the received data to be processed according to the received execution time, and the scheduling center constructs an execution strategy according to the data content of the data to be processed, and then interacts with the executor to realize the automatic processing of the data to be processed in the distributed system. The execution strategy includes the execution time, so even if the processing task is more complex and involves multi-threaded and multi-time point processing, the execution time pre-determined by the scheduling center can be used to realize the on-time processing of the data to be processed, thereby improving the data processing efficiency of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 An application environment diagram of a distributed system data processing method in one embodiment;

[0036] Figure 2A schematic diagram of a flow chart of a distributed system data processing method in one embodiment;

[0037] Figure 3 A flowchart of steps for constructing an execution strategy for processing a task in one embodiment;

[0038] Figure 4 is a structural block diagram of a distributed system data processing device in one embodiment;

[0039] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] It should be noted that the information (including but not limited to data content, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use and processing of relevant data are in compliance with the relevant provisions of national laws and regulations. For content pushed to users (for example, power risk assessment results, etc.), users can refuse or conveniently refuse content push, etc. In the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0042] The distributed system data processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 controls the dispatch center to generate processing tasks according to the data content of the data to be processed; the control dispatch center builds the execution strategy of the processing task, wherein the execution strategy includes the execution time and the execution object; the control dispatch center sends the data to be processed and the execution time to the executor corresponding to the execution object; the server 104 controls the executor to process the received data to be processed according to the received execution time. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0043] In an exemplary embodiment, Figure 2 As shown, a distributed system data processing method is provided, and the method is applied to Figure 1 The server 104 in the example is used to describe the distributed system, the distributed system includes a dispatch center and an executor, and includes the following steps 202 to 208. Among them:

[0044] Step 202: Generate a processing task according to the data content of the data to be processed by the scheduling center.

[0045] The distributed system may be a xx-job system or a xxl-job system, and there is no limitation here.

[0046] The data to be processed in step 202 is data waiting for data processing. Specifically, the data processing type of the data processing includes but is not limited to data analysis type, data cleaning type, data verification type, data conversion type, data verification type and data model building type. The number of data to be processed can be single or multiple, and is not limited here.

[0047] Exemplarily, step 202 includes: allocating corresponding data processing types to the data to be processed according to the data content of the data to be processed by the scheduling center, and generating processing tasks according to the data processing type by the scheduling center.

[0048] Furthermore, the dispatch center generates processing tasks according to the data processing type, including: the dispatch center generates processing tasks according to the data processing method corresponding to the data to be processed and the data to be processed, wherein the data processing method includes but is not limited to data parsing method, data cleaning method, data verification method, data conversion method, data verification method and data model building method.

[0049] Step 204: construct an execution strategy for the processing task through the scheduling center, wherein the execution strategy includes an execution time and an execution object.

[0050] The processing task in step 204 may be single or multiple, and there is no limitation here.

[0051] Exemplarily, step 204 includes: identifying, by the scheduling center, a task processing attribute of the processing task according to a processing type corresponding to the processing task, wherein the task processing attribute represents at least one of a processing duration and processing resource consumption information of the processing task; generating, by the scheduling center, an execution time of the processing task according to the task processing attribute of the processing task, and allocating, by the scheduling center, an execution object to the processing task according to the task processing attribute of the processing task.

[0052] Step 206: Send the data to be processed and the execution time to the executor corresponding to the execution object through the scheduling center.

[0053] The plurality of executors communicate with the dispatch center. Specifically, the plurality of executors are registered with the dispatch center.

[0054] Exemplarily, step 206 includes: matching multiple registered executors with the execution object through the scheduling center to obtain the executor corresponding to the execution object; and sending the data to be processed and the execution time to the executor corresponding to the execution object through the scheduling center.

[0055] Furthermore, the multiple registered executors are matched with the execution object through the dispatch center to obtain the executor corresponding to the execution object, including: matching the executor names of the multiple registered executors with the object name of the execution object to obtain the name matching degree, and determining the executor among the multiple registered executors whose corresponding name matching degree is greater than a preset name matching degree threshold as the executor corresponding to the execution object.

[0056] It is understandable that there may be situations where the dispatch center and the executor do not have a timing function. Therefore, it is easy for the executor to fail to process data on time according to the execution time, resulting in low data processing accuracy.

[0057] As an embodiment, step 206 includes: the distributed system also includes a queue processing server, which sends the data to be processed, the execution time corresponding to the data to be processed, and the executor to the queue processing server through the scheduling center, and the queue processing server sends the data to be processed to the executor corresponding to the data to be processed according to the execution time corresponding to the data to be processed.

[0058] Furthermore, the queue processing server may be a rabbitMQ (a popular open source message broker and queue server) server.

[0059] In this way, even if the dispatch center and the executor do not have a timing function, the queue processing server with a timing function can interact with the dispatch center and the executor to achieve the processing of the data to be processed at an accurate time.

[0060] Optionally, the method also includes: the scheduling center visually displays the processing task on the interface, and when receiving the user's editing operation on the processing task displayed on the interface, modifies the processing task according to the editing operation, wherein the editing operation includes but is not limited to adding task operations, deleting task operations and modifying task operations.

[0061] Step 208: Process the received data to be processed according to the received execution time by the executor.

[0062] Exemplarily, step 208 includes: processing the received data to be processed according to the data processing type of the received data to be processed by the executor at the received execution time.

[0063] Furthermore, the received data to be processed is processed according to the data processing type of the received data to be processed, including: matching the data processing method for the data to be processed according to the data processing type of the received data to be processed; and processing the data to be processed according to the data processing method corresponding to the data to be processed.

[0064] Optionally, after step 208, the method further includes: the executor returns the execution result to the dispatch center.

[0065] Furthermore, the method also includes: the dispatch center updates the execution status of the executor according to the execution result.

[0066] Furthermore, the method also includes: the dispatch center monitors the execution process of the executor according to the execution result to obtain the monitoring result, and sends the executor execution abnormality alarm when the monitoring result indicates that the execution is abnormal.

[0067] The above-mentioned distributed system data processing method is applied to a distributed system, and the distributed system includes a scheduling center and an executor. The scheduling center generates a processing task according to the data content of the data to be processed; the scheduling center constructs an execution strategy for the processing task, wherein the execution strategy includes an execution time and an execution object; the scheduling center sends the data to be processed and the execution time to the executor corresponding to the execution object; the executor processes the received data to be processed according to the received execution time, and the scheduling center constructs an execution strategy according to the data content of the data to be processed, and then interacts with the executor to realize the automatic processing of the data to be processed in the distributed system. The execution strategy includes the execution time, so even if the processing task is more complex and involves multi-threaded and multi-time point processing, the execution time pre-determined by the scheduling center can be used to realize the on-time processing of the data to be processed, thereby improving the data processing efficiency of the distributed system.

[0068] In an exemplary embodiment, Figure 3 As shown, constructing the execution strategy of the processing task includes steps 302 to 306. Among them:

[0069] Step 302: Allocate a processing thread pool for the processing task according to the task processing attribute of the processing task.

[0070] The task processing attribute in step 302 includes at least one of processing time and processing consumption resources.

[0071] As one embodiment, step 302 includes: the processing thread pool includes a fast processing thread pool or a slow processing thread pool; if the task processing attribute of the processing task represents that the processing task meets the preset execution processing conditions, the fast processing thread pool is allocated as the processing thread pool for the processing task; if the task processing attribute of the processing task represents that the processing task does not meet the preset execution processing conditions, the slow processing thread pool is allocated as the processing thread pool for the processing task.

[0072] Furthermore, the method further includes: if the task processing attribute includes processing duration, then when the processing duration is less than a preset duration threshold, determining that the task processing attribute indicates that the processing task meets a preset execution processing condition.

[0073] Furthermore, the method further includes: if the task processing attribute includes processing consumption resources, then when the processing consumption resources are less than a preset resource threshold, determining that the task processing attribute represents that the processing task meets a preset execution processing condition.

[0074] Step 304: Generate the execution time of the processing task according to the processing thread pool corresponding to the processing task and the task processing attribute of the processing task.

[0075] Exemplarily, step 304 includes: assigning a priority to the processing task according to the task processing attributes of the processing task, and building an association relationship between the processing task and other processing tasks according to the task processing attributes of the processing task; generating an execution time of the processing task according to the priority of the processing task and the association relationship between the processing task and other processing tasks.

[0076] Furthermore, priorities are assigned to processing tasks based on task processing attributes of the processing tasks, including: generating the importance of the processing tasks based on the task processing attributes of the processing tasks and the data processing type of the processing tasks, and assigning priorities to the processing tasks based on the importance of the processing tasks, wherein the higher the importance, the higher the assigned priority.

[0077] As one embodiment, the execution time of the processing task is generated according to the priority of the processing task and the association relationship between the processing task and other processing tasks, including: reselecting the associated processing task whose association represented by the corresponding association relationship is greater than a preset association threshold from other processing tasks, and generating the execution time of the processing task according to the data processing type corresponding to the associated processing task, the execution time and the priority of the processing task.

[0078] Furthermore, the execution time of the processing task is generated according to the data processing type corresponding to the associated processing task, the execution time and the priority of the processing task, including: determining the task execution order between the processing task and the associated processing task according to the data processing type corresponding to the associated processing task and the data processing type of the processing task; generating the execution time of the processing task according to the task execution order between the processing task and the associated processing task, the execution time of the associated processing task and the priority of the processing task, wherein the higher the priority of the processing task, the earlier the execution time of the processing task.

[0079] The data processing type of the processing task matches the data processing type of the processing data corresponding to the processing task.

[0080] Step 306 , selecting an execution object that matches the processing task from the multiple preset execution objects according to the task processing attribute of the processing task and the execution information of the multiple preset execution objects.

[0081] Exemplarily, step 306 includes: the execution information includes the execution task type and the current execution status; according to the execution task type of multiple preset execution objects, multiple matching objects that match the processing task attributes of the processing task are selected from the multiple preset execution objects; and the corresponding execution object whose current execution status meets the preset status condition is selected from the multiple matching objects.

[0082] Among them, the execution task types of multiple preset execution objects are matched with the data processing types corresponding to the processing tasks to obtain the type matching degree. When the type matching degree corresponding to the preset execution object is greater than the preset type matching degree threshold, it is determined that the processing task attributes of the preset execution object and the processing task match.

[0083] Furthermore, the method also includes at least one of the following items: if the current execution state represents that the execution state of the first execution object is an idle state, then it is determined that the current execution state corresponding to the first execution object meets the preset state condition; if the current execution state represents that the execution state of the first execution object is a normal state of the object, then it is determined that the current execution state corresponding to the first execution object meets the preset state condition.

[0084] In this way, it is ensured that the executor that issues the execution task is idle and normal, thereby improving the data processing efficiency of the data to be processed.

[0085] In this embodiment, a processing thread pool is allocated to the processing task according to the task processing attributes of the processing task; the execution time of the processing task is generated according to the processing thread pool corresponding to the processing task and the task processing attributes of the processing task; an execution object matching the processing task is selected from multiple preset execution objects according to the task processing attributes of the processing task and the execution information of multiple preset execution objects, and the granularity is refined to first allocate a processing thread pool based on the task processing attributes of the processing task, and then generate the execution time based on the task processing attributes of the processing thread pool processing task, and then select the execution object based on the task processing attributes of the processing task and the execution information of multiple preset execution objects, thereby ensuring the accuracy of the execution time and execution object corresponding to the processing task, that is, improving the accuracy of constructing the execution strategy of the processing task, and thereby improving the processing accuracy of the data to be processed.

[0086] As a detailed embodiment, a processing task is generated according to the data content of the data to be processed by the scheduling center; if the task processing attribute of the processing task indicates that the processing task meets the preset execution processing conditions, the fast processing thread pool is allocated as the processing thread pool of the processing task; if the task processing attribute of the processing task indicates that the processing task does not meet the preset execution processing conditions, the slow processing thread pool is allocated as the processing thread pool of the processing task; according to the task processing attribute of the processing task, a priority is assigned to the processing task, and, according to the task processing attribute of the processing task, an association relationship between the processing task and other processing tasks is established; according to the priority of the processing task and the association relationship between the processing task and other processing tasks, the execution time of the processing task is generated.

[0087] Furthermore, according to the execution task types of multiple preset execution objects, multiple matching objects that match the processing task attributes of the processing task are selected from the multiple preset execution objects; the corresponding execution object whose current execution status meets the preset status condition is selected from the multiple matching objects; the data to be processed and the execution time are sent to the executor corresponding to the execution object through the scheduling center; and the received data to be processed is processed by the executor according to the received execution time.

[0088] In this way, the scheduling center generates a processing task according to the data content of the data to be processed; the scheduling center constructs an execution strategy for the processing task, wherein the execution strategy includes the execution time and the execution object; the scheduling center sends the data to be processed and the execution time to the executor corresponding to the execution object; the executor processes the received data to be processed according to the received execution time. The scheduling center constructs an execution strategy according to the data content of the data to be processed, and then interacts with the executor to realize the automatic processing of the data to be processed in the distributed system. The execution strategy includes the execution time, so even if the processing task is more complex and involves multi-threaded and multi-time point processing, the execution time pre-determined by the scheduling center can be used to realize the on-time processing of the data to be processed, thereby improving the data processing efficiency of the distributed system.

[0089] Furthermore, a processing thread pool is allocated to the processing task according to the task processing attributes of the processing task; the execution time of the processing task is generated according to the processing thread pool corresponding to the processing task and the task processing attributes of the processing task; an execution object matching the processing task is selected from multiple preset execution objects according to the task processing attributes of the processing task and the execution information of multiple preset execution objects, and the granularity is refined to first allocate a processing thread pool based on the task processing attributes of the processing task, and then generate the execution time based on the task processing attributes of the processing thread pool processing task, and then select the execution object based on the task processing attributes of the processing task and the execution information of multiple preset execution objects, thereby ensuring the accuracy of the execution time and execution object corresponding to the processing task, that is, improving the accuracy of constructing the execution strategy of the processing task, and thereby improving the processing accuracy of the data to be processed.

[0090] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0091] Based on the same inventive concept, the embodiment of the present application also provides a distributed system data processing device for implementing the above-mentioned distributed system data processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the one or more distributed system data processing device embodiments provided below can refer to the above-mentioned limitations on the distributed system data processing method, and will not be repeated here.

[0092] In an exemplary embodiment, Figure 5 As shown, a distributed system data processing device 400 is provided, including: a generating module 402, a constructing module 404, a sending module 406 and a processing module 408, wherein:

[0093] A generation module 402, used to generate a processing task according to the data content of the data to be processed through the scheduling center;

[0094] A construction module 404 is used to construct an execution strategy for processing tasks through a scheduling center, wherein the execution strategy includes an execution time and an execution object;

[0095] The sending module 406 is used to send the to-be-processed data and the execution time to the executor corresponding to the execution object through the scheduling center;

[0096] The processing module 408 is used to process the received data to be processed according to the received execution time through the executor.

[0097] In one of the embodiments, the construction module 404 is also used to allocate a processing thread pool to the processing task according to the task processing attributes of the processing task; generate the execution time of the processing task according to the processing thread pool corresponding to the processing task and the task processing attributes of the processing task; and select an execution object that matches the processing task from multiple preset execution objects according to the task processing attributes of the processing task and the execution information of multiple preset execution objects.

[0098] In one embodiment, the processing thread pool includes a fast processing thread pool or a slow processing thread pool; the construction module 404 is also used to allocate the fast processing thread pool as the processing thread pool for the processing task if the task processing attribute of the processing task indicates that the processing task meets the preset execution processing conditions; if the task processing attribute of the processing task indicates that the processing task does not meet the preset execution processing conditions, then allocate the slow processing thread pool as the processing thread pool for the processing task.

[0099] In one embodiment, the construction module 404 is also used to assign a priority to the processing task according to the task processing attributes of the processing task, and to construct an association relationship between the processing task and other processing tasks according to the task processing attributes of the processing task; and to generate an execution time of the processing task according to the priority of the processing task and the association relationship between the processing task and other processing tasks.

[0100] In one embodiment, the execution information includes an execution task type and a current execution status; the construction module 404 is also used to select multiple matching objects that match the processing task attributes of the processing task from multiple preset execution objects according to the execution task types of multiple preset execution objects; and select an execution object whose corresponding current execution status meets the preset status conditions from the multiple matching objects.

[0101] In one embodiment, the construction module 404 is also used to determine that the current execution state corresponding to the first execution object meets the preset state condition if the current execution state represents that the execution state of the first execution object is an idle state; if the current execution state represents that the execution state of the first execution object is a normal state of the object, then determine that the current execution state corresponding to the first execution object meets the preset state condition.

[0102] Each module in the above-mentioned distributed system data processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0103] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a distributed system data processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0104] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0106] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0107] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0109] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0110] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A distributed system data processing method, characterized in that: Applied to a distributed system, the distributed system includes a dispatching center and an executor, and the method includes: Generate a processing task according to the data content of the data to be processed by the dispatch center; Constructing an execution strategy for the processing task through the scheduling center, wherein the execution strategy includes an execution time and an execution object; Sending the data to be processed and the execution time to the executor corresponding to the execution object through the scheduling center; The executor processes the received data to be processed according to the received execution time.

2. The method according to claim 1, characterized in that The constructing the execution strategy of the processing task includes: Allocating a processing thread pool to the processing task according to the task processing attribute of the processing task; Generate an execution time of the processing task according to a processing thread pool corresponding to the processing task and a task processing attribute of the processing task; According to the task processing attribute of the processing task and the execution information of multiple preset execution objects, an execution object matching the processing task is selected from the multiple preset execution objects.

3. The method according to claim 2, characterized in that The processing thread pool includes a fast processing thread pool or a slow processing thread pool; and the allocating a processing thread pool to the processing task according to the task processing attribute of the processing task includes: If the task processing attribute of the processing task indicates that the processing task meets the preset execution processing condition, the fast processing thread pool is allocated as the processing thread pool of the processing task; If the task processing attribute of the processing task indicates that the processing task does not meet the preset execution processing condition, the slow processing thread pool is allocated as the processing thread pool of the processing task.

4. The method according to claim 2, characterized in that: The generating the execution time of the processing task according to the processing thread pool corresponding to the processing task and the task processing attribute of the processing task comprises: assigning a priority to the processing task according to the task processing attribute of the processing task, and establishing an association relationship between the processing task and other processing tasks according to the task processing attribute of the processing task; The execution time of the processing task is generated according to the priority of the processing task and the association relationship between the processing task and other processing tasks.

5. The method according to claim 2, characterized in that: The execution information includes the execution task type and the current execution state; the selecting an execution object matching the processing task from the multiple preset execution objects according to the task processing attribute of the processing task and the execution information of the multiple preset execution objects includes: According to the execution task types of the multiple preset execution objects, selecting multiple matching objects that match the processing task attributes of the processing task from the multiple preset execution objects; An execution object whose corresponding current execution state meets a preset state condition is selected from the multiple matching objects.

6. The method according to claim 5, characterized in that The method further comprises at least one of the following: If the current execution state indicates that the execution state of the first execution object is an idle state, determining that the current execution state corresponding to the first execution object satisfies a preset state condition; If the current execution state indicates that the execution state of the first execution object is a normal state of the object, it is determined that the current execution state corresponding to the first execution object meets a preset state condition.

7. A distributed system data processing device, characterized in that: Applied to a distributed system, the distributed system includes a dispatching center and an executor, and the device includes: A generation module, used to generate a processing task according to the data content of the data to be processed through the scheduling center; A construction module, used to construct an execution strategy for the processing task through the scheduling center, wherein the execution strategy includes an execution time and an execution object; A sending module, used for sending the to-be-processed data and the execution time to the executor corresponding to the execution object through the scheduling center; A processing module is used to process the received data to be processed according to the received execution time through the executor.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images