Data scheduling control method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of cluster scheduling optimization, is applied to a cluster storage node calling scene, and relates to a data scheduling control method, device and equipment and a storage medium thereof. And the node scheduling state, the scheduling weight and the data balance speed are set in the scheduling control component, so that when the storage node fails, data scheduling control is quickly and flexibly performed in combination with the management node, normal data is prevented from being transferred to the failed node, data scheduling adjustment can be performed in time, and the failed storage node can be deleted or repaired in time. According to the data scheduling control method, a large amount of invalid or wrong calling is avoided as much as possible, especially in a financial service data processing scene with complex service data and large data volume, data scheduling control can be rapidly and flexibly carried out in combination with the management node by adopting the data scheduling control method, normal data is prevented from being transferred to a fault node, and the data scheduling efficiency is improved. And the service life of the database and the processing system is prolonged.

Description

Technical Field

[0001] The present application relates to the technical field of cluster scheduling optimization, and is applied to cluster storage node calling scenarios, and in particular to a data scheduling control method, device, equipment and storage medium thereof. Background Art

[0002] At present, cloud storage is being used more and more widely in data storage scenarios, especially in the field of financial technology with a large amount of business data.

[0003] The underlying layer of the distributed database uses distributed storage components to store data. The storage components replicate multiple copies of data based on the Multi-Raft consistency protocol to ensure data reliability. When deployed in a cloud environment, when a storage node is before or after failure recovery, during the data rebalancing process, it is impossible to effectively control its data balancing process, which may affect the stability and consistency of the data. The data rebalancing refers to the process of data migration between different storage nodes under the same management node in order to balance the amount of data stored in different storage nodes under the same management node after the data of the storage node is updated. Specifically, when a storage node fails, the data rebalancing process may be triggered, but this process cannot be effectively monitored and controlled. Therefore, the uncontrolled data rebalancing process may increase the system load and delay, easily affect the normal operation of the system, and reduce the service life of the system. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data scheduling control method, device, equipment and storage medium thereof to solve the problem in the prior art that the uncontrolled data rebalancing process easily increases the load and delay of the system, affects the normal operation of the system, and reduces the service life of the system.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a data scheduling control method, which adopts the following technical solutions:

[0006] A data scheduling control method comprises the following steps:

[0007] Monitoring the execution status of all storage nodes in the target cluster, wherein the storage nodes are used to store data called by the target service according to the service request;

[0008] If it is monitored that the execution state corresponding to the current storage node is a call failure state, a first scheduling command is sent to the management node of the current storage node to generate a data scheduling restriction;

[0009] Identify the invalid data in the current storage node, and obtain the latest available data from the data source of the invalid data;

[0010] Filtering a backup node of the current storage node according to the storage mapping relationship between all storage nodes and the data source, wherein the backup node contains the latest available data transmitted by the data source;

[0011] When a standby node is selected, jump to the standby node to complete the data called by the target service;

[0012] Repairing the current storage node, and adjusting the execution state corresponding to the current storage node to a fault repair state after the repair;

[0013] If it is monitored that the execution state corresponding to the current storage node is adjusted from the call fault state to the fault repair state, a second scheduling command is sent to the management node of the current storage node to release the data scheduling restriction.

[0014] Furthermore, the step of monitoring the execution status of all storage nodes in the target cluster specifically includes:

[0015] Analyze the current task execution log in real time to obtain the latest execution results;

[0016] Perform a key field screening operation on the latest execution result to obtain a screening result;

[0017] If the key field in the screening result is an indication field when a running error is reported, the storage node corresponding to the current task is in a call failure state;

[0018] If the key field in the screening result is an indication field for successful operation, the storage node corresponding to the current task is in a normal calling state.

[0019] Furthermore, before executing the step of monitoring the execution status of all storage nodes in the target cluster, the method further includes:

[0020] Construct a mapping relationship between management nodes and storage nodes;

[0021] Setting all management nodes in the target cluster node as scheduling control objects;

[0022] According to the mapping relationship between the management node and the storage node, different storage nodes under the same management node are set as scheduling control objects of the same management node;

[0023] Set the scheduling state and scheduling weight of all storage nodes and management nodes, wherein the scheduling state includes a schedulable state and a prohibited scheduling state, and the scheduling weight includes a first weight value and a second weight value, the schedulable state corresponds to the first weight value, and the prohibited scheduling state corresponds to the second weight value;

[0024] Initializing the initial scheduling state of all nodes to the schedulable state, and initializing the initial scheduling weights of all nodes to the first weight value;

[0025] The initial data balancing speed of different storage nodes under the same management node is set, wherein the data balancing speed includes the data balancing speeds corresponding to the schedulable state and the prohibited scheduling state respectively.

[0026] Furthermore, when it is detected that the execution state corresponding to the current storage node is a call failure state, the method further includes:

[0027] Setting the scheduling state of the current storage node to a scheduling disabled state;

[0028] Setting the scheduling weight of the current storage node to a second weight value;

[0029] The data balancing speed of the current storage node is set to the data balancing speed corresponding to the scheduling disabled state.

[0030] Furthermore, the step of sending a first scheduling command to the management node of the current storage node to generate a data scheduling restriction specifically includes:

[0031] According to a preset mapping relationship between management nodes and storage nodes, determine the management node corresponding to the current storage node, and determine other storage nodes managed by the management node;

[0032] Sending the first scheduling command to the management node to instruct the management node to filter out a preset first adjustment strategy according to the first scheduling command, wherein the first adjustment strategy includes a scheduling state, a scheduling weight, and a data balancing speed corresponding to when generating a data scheduling restriction;

[0033] Setting the scheduling state of the management node to a schedulable state according to the first adjustment strategy;

[0034] Setting the scheduling weight of the management node to a first weight value according to the first adjustment strategy;

[0035] According to the first adjustment strategy, the data balancing speed of other storage nodes managed by the management node is set to the data balancing speed corresponding to the scheduling prohibited state.

[0036] Further, when it is monitored that the execution state corresponding to the current storage node is adjusted from the call failure state to the failure repair state, the method further includes:

[0037] Adjusting the scheduling state of the current storage node from a scheduling-disabled state to a scheduling-enabled state;

[0038] Adjusting the scheduling weight of the current storage node from the second weight value to the first weight value;

[0039] The data balancing speed of the current storage node is adjusted from the data balancing speed corresponding to the scheduling disabled state to the data balancing speed corresponding to the scheduling enabled state.

[0040] Furthermore, the step of sending a second scheduling command to the management node of the current storage node to lift the data scheduling restriction specifically includes:

[0041] Determine the management node corresponding to the current storage node and other storage nodes managed by the management node according to the mapping relationship between the management node and the storage node;

[0042] Sending the second scheduling command to the management node to instruct the management node to filter out a preset second adjustment strategy according to the second scheduling command, wherein the second adjustment strategy includes a scheduling state, a scheduling weight, and a data balancing speed corresponding to lifting the data scheduling restriction;

[0043] Setting the scheduling state of the management node to a scheduling prohibited state according to the second adjustment strategy;

[0044] Setting the scheduling weight of the management node to a second weight value according to the second adjustment strategy;

[0045] According to the second adjustment strategy, the data balancing speed of other storage nodes managed by the management node is set to the data balancing speed corresponding to the schedulable state.

[0046] In order to solve the above technical problems, the embodiment of the present application also provides a data scheduling control device, which adopts the following technical solution:

[0047] A data scheduling control device, comprising:

[0048] An execution status monitoring module, used to monitor the execution status of all storage nodes in the target cluster, wherein the storage nodes are used to store data called by the target service according to the service request;

[0049] A data scheduling restriction generating module is used to send a first scheduling command to a management node of the current storage node to generate a data scheduling restriction if it is detected that the execution state corresponding to the current storage node is a call failure state;

[0050] A latest available data acquisition module, used to identify the invalid data in the current storage node and acquire the latest available data from the data source of the invalid data;

[0051] A spare node screening module, used for screening the spare node of the current storage node according to the storage mapping relationship between all storage nodes and the data source end, wherein the spare node contains the latest available data transmitted by the data source end;

[0052] A target service remediation execution module, used for jumping to the standby node to complete the data called by the target service when a standby node is screened;

[0053] A current storage node repair module, used to repair the current storage node and adjust the execution state corresponding to the current storage node to a fault repair state after the repair;

[0054] The data scheduling restriction lifting module is used to send a second scheduling command to the management node of the current storage node to lift the data scheduling restriction if it monitors that the execution state corresponding to the current storage node is adjusted from the call fault state to the fault repair state.

[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0056] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned data scheduling control method when executing the computer-readable instructions.

[0057] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0058] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data scheduling control method as described above.

[0059] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0060] The data scheduling control method described in the embodiment of the present application can quickly and flexibly perform data scheduling control when a storage node fails, avoid transferring normal data to the faulty node, facilitate timely data scheduling adjustment, and promptly delete or repair the faulty storage node, and try to avoid generating a large number of invalid or erroneous calls. Especially in the financial business data processing scenario with complex business data and large data volume, the data scheduling control method provided by the present application can be used to quickly and flexibly combine with the management node to perform data scheduling control, avoid transferring normal data to the faulty node, and delay the service life of the database and processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0063] Figure 2 is a flow chart of an embodiment of a data scheduling control method according to the present application;

[0064] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0065] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;

[0066] Figure 5 is a structural schematic diagram of an embodiment of a data scheduling control device according to the present application;

[0067] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0069] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0070] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0071] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0072] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0073] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0074] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0075] It should be noted that the data scheduling control method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the data scheduling control device is generally arranged in the terminal device.

[0076] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0077] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data scheduling control method according to the present application. The data scheduling control method comprises the following steps:

[0078] Step 201 : monitoring the execution status of all storage nodes in the target cluster, wherein the storage nodes are used to store data called by the target service according to the service request.

[0079] Start a preset execution status monitoring component, wherein the execution status monitoring component is used to perform execution status monitoring on all storage nodes in the target cluster, the nodes of the target cluster include management nodes and storage nodes, the management node is used to manage the storage nodes within the designated area of ​​the target cluster, the data stored in the storage node is used to be called by the target service according to the service request, and the data stored in different storage nodes under the same management node are completely different.

[0080] In this embodiment, different storage nodes under the same management node can perform balanced data transfer with each other in order to achieve balanced data storage. For example, if management node A includes three storage nodes a1, a2 and a3, a data balance rule is predefined among the three nodes. That is, when a1 has more data, the data cached in a1 can be adjusted to the caches of a2 and a3 according to the set data balance speed, thereby avoiding a single storage node or a single management node managing too much data. The data balance speed is only for adjusting the amount of storage data of different storage nodes under the same management node, ensuring that the amount of data of multiple storage nodes is not much different, and playing a role in balancing the amount of data between each storage node.

[0081] Specifically, the target cluster, such as a Ti DB data storage cluster, each storage node in the data storage cluster stores a Ti KV, each storage node stores a key-value type data, and the Ti DB data storage cluster is an open source distributed relational database. By presetting the execution status monitoring component, it is possible to timely perform data scheduling adjustments when a storage node fails, and timely delete or repair the failed storage node, so as to avoid generating a large number of invalid or erroneous calls. The Ti DB data storage cluster is applied to financial business data processing scenarios where business data is relatively complex and the data volume is large. In financial business data processing scenarios where business data is relatively complex and the data volume is large, the data scheduling control method provided by the present application can be used to quickly and flexibly combine the management node to perform data scheduling control, avoid transferring normal data to the failed node, and delay the service life of the database and processing system.

[0082] In this embodiment, the step of monitoring the execution status of all storage nodes in the target cluster specifically includes: real-time parsing of the current task execution log to obtain the latest execution result; performing a key field filtering operation on the latest execution result to obtain a filtering result; if the key field in the filtering result is an indication field when an operation error is reported, the storage node corresponding to the current task is in a call failure state, for example: the filtered key field is ERROR; if the key field in the filtering result is an indication field when the operation is successful, the storage node corresponding to the current task is in a normal call state, for example: the filtered key field is SUCCESS.

[0083] By real-time analysis of the current task execution log and screening of key fields, it is possible to identify whether the data storage node corresponding to the current task execution is in a faulty state, which facilitates further data scheduling and control processing.

[0084] A preset scheduling control component is started, wherein the scheduling control component is used to perform scheduling control on all management nodes in the target cluster.

[0085] In this embodiment, before executing the step of monitoring the execution status of all storage nodes in the target cluster, the method also includes: constructing a mapping relationship between management nodes and storage nodes; creating a scheduling control component, and deploying the mapping relationship between management nodes and storage nodes as reference knowledge into the scheduling control component; setting all management nodes in the target cluster nodes as scheduling control objects of the scheduling control component; according to the mapping relationship between management nodes and storage nodes, setting different storage nodes under the same management node as scheduling control objects of the same management node; setting the scheduling status and scheduling weight of all storage nodes and management nodes, wherein the scheduling status includes a schedulable status and a prohibited scheduling status, and the scheduling weight includes a first weight value and a second weight value, the schedulable status corresponds to the first weight value, and the prohibited scheduling status corresponds to the second weight value; initializing the initial scheduling status of all nodes to the schedulable status, and initializing the initial scheduling weight of all nodes to the first weight value; setting the initial data balancing speed of different storage nodes under the same management node in the scheduling control component, wherein the data balancing speed includes the data balancing speed corresponding to the schedulable status and the prohibited scheduling status respectively.

[0086] Specifically, the data balancing speed corresponding to the schedulable state is greater than the data balancing speed corresponding to the prohibited scheduling state. The data balancing speed corresponding to the schedulable state is a non-zero value, and the data balancing speed corresponding to the prohibited scheduling state can be zero. The initial data balancing speed corresponding to the schedulable state is any non-zero value, and the initial data balancing speed corresponding to the prohibited scheduling state is zero.

[0087] By presetting the scheduling control component and setting the initial scheduling status and initial scheduling weight of all nodes, setting the initial data balancing speed of all storage nodes, setting the management node as the scheduling control object of the scheduling control component, and setting different storage nodes under the same management node as the scheduling control objects of the same management node according to the mapping relationship between the management node and the storage node, it is achieved that during the task running process, the changes in data can be scheduled and controlled according to the changes in the status of the storage node.

[0088] Step 202: If it is monitored that the execution state corresponding to the current storage node is a call failure state, a first scheduling command is sent to the management node of the current storage node to generate a data scheduling restriction.

[0089] Specifically, if it is monitored that the execution state corresponding to the current storage node is a call failure state, a first scheduling command is sent to the management node of the current storage node through the scheduling control component to generate a data scheduling restriction.

[0090] The data scheduling restriction means that before the current storage node is repaired or a replacement node is available, the current storage node is prohibited from performing data migration with other storage nodes under the same management node.

[0091] In this embodiment, when it is monitored that the execution state corresponding to the current storage node is a call failure state, the method also includes: setting the scheduling state of the current storage node to a prohibited scheduling state; setting the scheduling weight of the current storage node to a second weight value; setting the data balancing speed of the current storage node to the data balancing speed corresponding to the prohibited scheduling state.

[0092] Specifically, when it is monitored that the execution status corresponding to the current storage node is a call failure status, that is, the data stored in the current storage node cannot be called, the scheduling status of the current storage node is set to a prohibited scheduling status; the scheduling weight of the current storage node is set to a second weight value; the data balancing speed of the current storage node is set to the data balancing speed corresponding to the prohibited scheduling status, which can effectively avoid the generation of a large amount of failed call log information, thereby waiting for the next step of processing of the system, wherein the next step of processing includes repairing or deleting the storage node.

[0093] In fact, before executing the steps of setting the scheduling state of the current storage node to a prohibited scheduling state; setting the scheduling weight of the current storage node to a second weight value; and setting the data balancing speed of the current storage node to a data balancing speed corresponding to the prohibited scheduling state, the method also includes: pre-identifying the current scheduling state, current scheduling weight and current data balancing speed of the current storage node; identifying whether reset is required based on the current scheduling state, current scheduling weight and current data balancing speed of the current storage node; if reset is required, executing the reset step; if reset is not required, executing subsequent management node related settings.

[0094] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:

[0095] Step 301, determining the management node corresponding to the current storage node and determining other storage nodes managed by the management node according to a preset mapping relationship between the management node and the storage node;

[0096] Step 302: Send the first scheduling command to the management node to instruct the management node to filter out a preset first adjustment strategy according to the first scheduling command, wherein the first adjustment strategy includes a scheduling state, a scheduling weight, and a data balancing speed corresponding to when generating a data scheduling restriction;

[0097] Step 303: setting the scheduling state of the management node to a schedulable state according to the first adjustment strategy;

[0098] Step 304: setting the scheduling weight of the management node to a first weight value according to the first adjustment strategy;

[0099] Step 305: according to the first adjustment strategy, the data balancing speed of other storage nodes managed by the management node is set to the data balancing speed corresponding to the scheduling disabled state.

[0100] Specifically, under normal circumstances, other storage nodes managed by the management node are in a non-call fault state. At this time, it is necessary to set the scheduling state of the management node to a schedulable state, and the scheduling weight to a first weight value, so as to facilitate the scheduling and control of all storage nodes under management by the management node. Specifically, the data balancing speed of other storage nodes managed by the management node is set to the data balancing speed corresponding to the prohibited scheduling state, that is, if there are storage nodes in the same group in a call fault state, the data balancing flow between them is prohibited at this time to avoid the normal data flow to the faulty node.

[0101] In this embodiment, after executing the steps of determining the management node corresponding to the current storage node according to the preset mapping relationship between the management node and the storage node, and determining other storage nodes managed by the management node, the method further includes: obtaining the execution status corresponding to the other storage nodes according to the execution status monitoring component; if the execution status corresponding to the other storage nodes are all call failure states, setting the scheduling status of the management node to a prohibited scheduling state, and setting the scheduling weight of the management node to a second weight value.

[0102] Specifically, when all storage nodes under the management node are in a call failure state, the scheduling state of the management node will be set to a scheduling prohibited state, and the scheduling weight of the management node will be set to a second weight value, so as to facilitate the transfer of all storage data managed by the management node and to facilitate the repair operations on the storage nodes under the management node.

[0103] Similarly, before setting the scheduling status and scheduling weight of the management node corresponding to the current storage node, the method also includes: pre-identifying the current scheduling status and current scheduling weight of the management node; identifying whether reset is required based on the current scheduling status and current scheduling weight of the management node; if reset is required, executing the reset step; if reset is not required, executing the step 305.

[0104] Step 203: Identify the invalid data in the current storage node, and obtain the latest available data from the data source of the invalid data.

[0105] Specifically, it is monitored that the execution state corresponding to the current storage node is a call failure state, that is, the data stored in the current storage node is invalid data and is no longer applicable to the target service. At this time, the invalid data may have been updated at the data source end. Therefore, the latest available data is obtained from the data source end of the invalid data to ensure the screening and synchronization operation of the latest available data.

[0106] Step 204 , screening a backup node of the current storage node according to the storage mapping relationship between all storage nodes and the data source, wherein the backup node contains the latest available data transmitted by the data source.

[0107] Specifically, the standby node, for example, is a standby node of the current storage node pre-set under other management nodes in the Ti DB data storage cluster, or a standby node independently set according to the data source end outside the Ti DB data storage cluster. The standby node of the current storage node is selected according to the storage mapping relationship between all storage nodes and the data source end, thereby avoiding the target service from being stopped and failing to execute the target service in time, and minimizing and reducing the bad experience caused to the user.

[0108] Step 205: when a standby node is selected, jump to the standby node to complete the data called by the target service.

[0109] In this embodiment, when a backup node is screened, the node jumps to the backup node to complete the data called by the target service. Specifically, the acquisition address of the called data is updated from the current storage node to the backup node, so as to obtain the data called by the target service from the backup node. The called data is the latest available data.

[0110] Step 206: Repair the current storage node, and adjust the execution state corresponding to the current storage node to a fault repair state after the repair.

[0111] In this embodiment, by presetting the execution status monitoring component, it is possible to timely perform data scheduling adjustments and timely delete or repair the failed storage node when a storage node fails, that is, the step of repairing the current storage node, specifically including: deleting the invalid data, storing the latest available data in the current storage node, or deleting the current storage node, recreating a new storage node, importing the data update in the backup node into the new storage node, and replacing the name of the new storage node with the name of the storage node in the failed state, thereby realizing both the deletion of the failed storage node and the repair of the failed storage node.

[0112] Step 207: If it is monitored that the execution state corresponding to the current storage node is adjusted from the call failure state to the failure repair state, a second scheduling command is sent to the management node of the current storage node to release the data scheduling restriction.

[0113] Specifically, if it is monitored that the execution state corresponding to the current storage node is adjusted from the call fault state to the fault repair state, a second scheduling command is sent to the management node of the current storage node through the scheduling control component to release the data scheduling restriction.

[0114] The lifting of data scheduling restrictions means that after the current storage node is repaired or a replacement node is available, data migration between the repaired current storage node and other storage nodes under the same management node is allowed.

[0115] In this embodiment, when it is monitored that the execution state corresponding to the current storage node is adjusted from the calling fault state to the fault repair state, the method also includes: adjusting the scheduling state of the current storage node from the prohibited scheduling state to the schedulable state; adjusting the scheduling weight of the current storage node from the second weight value to the first weight value; adjusting the data balancing speed of the current storage node from the data balancing speed corresponding to the prohibited scheduling state to the data balancing speed corresponding to the schedulable state.

[0116] Specifically, when it is monitored that the execution state corresponding to the current storage node is adjusted from the calling fault state to the fault repair state, that is, when the faulty node is repaired, at this time, the scheduling state, scheduling weight and data balancing speed of the current storage node are restored, and the current storage node continues to be put into normal use.

[0117] Continue to refer Figure 4 , Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes:

[0118] Step 401, determining the management node corresponding to the current storage node and determining other storage nodes managed by the management node according to the mapping relationship between the management node and the storage node;

[0119] Step 402: Send the second scheduling command to the management node to instruct the management node to filter out a preset second adjustment strategy according to the second scheduling command, wherein the second adjustment strategy includes a scheduling state, a scheduling weight, and a data balancing speed corresponding to lifting the data scheduling restriction;

[0120] Step 403: setting the scheduling state of the management node to a scheduling disabled state according to the second adjustment strategy;

[0121] Step 404: setting the scheduling weight of the management node to a second weight value according to the second adjustment strategy;

[0122] Step 405: according to the second adjustment strategy, the data balancing speed of other storage nodes managed by the management node is set to the data balancing speed corresponding to the schedulable state.

[0123] Specifically, under normal circumstances, the other storage nodes managed by the management node are in a non-call fault state, and the current storage node has also been repaired from a fault state. At this time, it is necessary to set the scheduling state of the management node to a prohibited scheduling state, and the scheduling weight is a second weight value. At this time, the management node is prohibited from scheduling and controlling all storage nodes under its management; the data balancing speed of other storage nodes managed by the management node is set to the data balancing speed corresponding to the schedulable state, that is, when all storage nodes under the same management node are in a normal state, it is convenient to automatically flow data among all storage nodes.

[0124] In this embodiment, after executing the steps of determining the management node corresponding to the current storage node according to the mapping relationship between the management node and the storage node in the scheduling control component, and determining other storage nodes managed by the management node, the method further includes: obtaining the execution status corresponding to the other storage nodes according to the execution status monitoring component; if the execution status corresponding to the other storage nodes has a call failure state, setting the scheduling state of the management node to a schedulable state, and setting the scheduling weight of the management node to a first weight value.

[0125] Specifically, when all storage nodes under the management node are in a call failure state, the scheduling state of the management node will be set to a schedulable state, and the scheduling weight of the management node will be set to a first weight value, so as to facilitate timely repair operations on the faulty storage nodes under the management node. The first weight value and the second weight value are set for use in conjunction with the call state respectively.

[0126] In this embodiment, after executing the steps of setting the scheduling state of the management node to a schedulable state and setting the scheduling weight of the management node to a first weight value, the method further includes: each time the scheduling state of the management node is set to a schedulable state, counting the total number of times all management nodes are set to a schedulable state, and counting the number of times the current management node is set to a schedulable state; performing a ratio operation based on the number of times the current management node is set to a schedulable state and the total number of times to obtain a ratio operation result; and updating the ratio operation result to the first weight value. Specifically, the first weight value can be dynamically updated in actual adjustments, and the frequency of storage nodes under each management node being called can be intuitively reflected based on the size of the first weight value.

[0127] In this embodiment, if a storage node whose failure frequency exceeds a preset frequency threshold is identified by the execution status monitoring component, the call status of the storage node is set to a prohibited call status, the call weight is set to a second call weight, and after all the data stored in the storage node is transferred, the storage node is deleted and a new storage node is created under the management node corresponding to the storage node.

[0128] By means of the execution status monitoring component and the scheduling control component, and by setting the node scheduling status, scheduling weight and data balancing speed in the scheduling control component, it is possible to quickly and flexibly combine the management node to perform data scheduling control when a storage node fails, so as to avoid transferring normal data to the faulty node, facilitate timely data scheduling adjustment, and delete or repair the faulty storage node in time, so as to avoid generating a large number of invalid or erroneous calls as much as possible. Especially in the financial business data processing scenario where the business data is relatively complex and the data volume is large, the data scheduling control method provided by the present application can be used to quickly and flexibly combine the management node to perform data scheduling control, so as to avoid transferring normal data to the faulty node and delay the service life of the database and processing system.

[0129] The present application uses the execution status monitoring component and the scheduling control component, and sets the node scheduling status, scheduling weight and data balancing speed in the scheduling control component. When a storage node fails, the present application can quickly and flexibly combine with the management node to perform data scheduling control, avoid transferring normal data to the faulty node, facilitate timely data scheduling adjustment, and delete or repair the faulty storage node in time, and try to avoid generating a large number of invalid or erroneous calls. Especially in the financial business data processing scenario where the business data is relatively complex and the data volume is large, the data scheduling control method provided by the present application can be used to quickly and flexibly combine with the management node to perform data scheduling control, avoid transferring normal data to the faulty node, and delay the service life of the database and processing system.

[0130] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0131] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0132] In an embodiment of the present application, through the execution status monitoring component and the scheduling control component, and setting the node scheduling status, scheduling weight and data balancing speed in the scheduling control component, it is possible to quickly and flexibly combine the management node to perform data scheduling control when a storage node fails, avoid transferring normal data to the faulty node, facilitate timely data scheduling adjustments, and delete or repair the faulty storage node in time, and try to avoid generating a large number of invalid or erroneous calls. Especially in the financial business data processing scenario where the business data is relatively complex and the data volume is large, the data scheduling control method provided by the present application can be used to quickly and flexibly combine the management node to perform data scheduling control, avoid transferring normal data to the faulty node, and delay the service life of the database and processing system.

[0133] Further references Figure 5 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data scheduling control device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0134] like Figure 5 As shown, the data scheduling control device 500 described in this embodiment includes: an execution status monitoring module 501, a data scheduling restriction generating module 502, a latest available data obtaining module 503, a spare node screening module 504, a target service remediation executing module 505, a current storage node repairing module 506 and a data scheduling restriction releasing module 507. Among them:

[0135] An execution status monitoring module 501 is used to monitor the execution status of all storage nodes in the target cluster, wherein the storage nodes are used to store data called by the target service according to the service request;

[0136] The data scheduling restriction generating module 502 is used to send a first scheduling command to the management node of the current storage node to generate a data scheduling restriction if it is detected that the execution state corresponding to the current storage node is a call failure state;

[0137] The latest available data acquisition module 503 is used to identify the invalid data in the current storage node and acquire the latest available data from the data source of the invalid data;

[0138] A spare node screening module 504 is used to screen the spare nodes of the current storage node according to the storage mapping relationship between all storage nodes and the data source end, wherein the spare nodes contain the latest available data transmitted by the data source end;

[0139] The target service remediation execution module 505 is used to jump to the backup node to complete the data called by the target service when the backup node is screened;

[0140] The current storage node repair module 506 is used to repair the current storage node and adjust the execution state corresponding to the current storage node to a fault repair state after the repair;

[0141] The data scheduling restriction lifting module 507 is used to send a second scheduling command to the management node of the current storage node to lift the data scheduling restriction if it is monitored that the execution state corresponding to the current storage node is adjusted from the call failure state to the failure repair state.

[0142] The data scheduling control device proposed in the present application belongs to the technical field of cluster scheduling optimization, and is applied to cluster storage node calling scenarios. Through the execution status monitoring component and the scheduling control component, as well as setting the node scheduling status, scheduling weight and data balancing speed in the scheduling control component, when a storage node fails, it can quickly and flexibly combine with the management node to perform data scheduling control, avoid transferring normal data to the faulty node, facilitate timely data scheduling adjustment, and delete or repair the faulty storage node in time, and try to avoid generating a large number of invalid or erroneous calls. Especially in the financial business data processing scenario with complex business data and large data volume, the data scheduling control method provided by the present application can be used to quickly and flexibly combine with the management node to perform data scheduling control, avoid transferring normal data to the faulty node, and delay the service life of the database and processing system.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0144] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0145] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0146] The computer device 6 includes a memory 6a, a processor 6b, and a network interface 6c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 having components 6a-6c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0147] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0148] The memory 6a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 6a can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 6a can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 6a can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 6a is generally used to store the operating system and various application software installed on the computer device 6, such as a computer-readable instruction of a data scheduling control method, etc. In addition, the memory 6a can also be used to temporarily store various types of data that have been output or are to be output.

[0149] The processor 6b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 6b is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 6b is used to run the computer-readable instructions stored in the memory 6a or process data, such as running the computer-readable instructions of the data scheduling control method.

[0150] The network interface 6c may include a wireless network interface or a wired network interface. The network interface 6c is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0151] The computer device proposed in this embodiment belongs to the field of cluster scheduling optimization technology and is applied to cluster storage node call scenarios. This application uses the execution status monitoring component and the scheduling control component, and sets the node scheduling status, scheduling weight and data balancing speed in the scheduling control component. When a storage node fails, the application can quickly and flexibly combine the management node to perform data scheduling control, avoid transferring normal data to the faulty node, facilitate timely data scheduling adjustment, and delete or repair the faulty storage node in time, and try to avoid generating a large number of invalid or erroneous calls. Especially in the financial business data processing scenario where the business data is relatively complex and the data volume is large, the data scheduling control method provided by this application can be used to quickly and flexibly combine the management node to perform data scheduling control, avoid transferring normal data to the faulty node, and delay the service life of the database and processing system.

[0152] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the data scheduling control method as described above.

[0153] The computer-readable storage medium proposed in this embodiment belongs to the field of cluster scheduling optimization technology and is applied to cluster storage node call scenarios. This application uses the execution status monitoring component and the scheduling control component, and sets the node scheduling status, scheduling weight and data balancing speed in the scheduling control component. When a storage node fails, the application can quickly and flexibly combine the management node to perform data scheduling control, avoid transferring normal data to the faulty node, facilitate timely data scheduling adjustment, and delete or repair the faulty storage node in time, and try to avoid generating a large number of invalid or erroneous calls. Especially in the financial business data processing scenario where the business data is relatively complex and the data volume is large, the data scheduling control method provided by this application can be used to quickly and flexibly combine the management node to perform data scheduling control, avoid transferring normal data to the faulty node, and delay the service life of the database and processing system.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0155] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data scheduling control method, characterized in that: The steps include: Monitoring the execution status of all storage nodes in the target cluster, wherein the storage nodes are used to store data called by the target service according to the service request; If it is monitored that the execution state corresponding to the current storage node is a call failure state, a first scheduling command is sent to the management node of the current storage node to generate a data scheduling restriction; Identify the invalid data in the current storage node, and obtain the latest available data from the data source of the invalid data; Filtering a backup node of the current storage node according to the storage mapping relationship between all storage nodes and the data source, wherein the backup node contains the latest available data transmitted by the data source; When a standby node is selected, jump to the standby node to complete the data called by the target service; Repairing the current storage node, and adjusting the execution state corresponding to the current storage node to a fault repair state after the repair; If it is monitored that the execution state corresponding to the current storage node is adjusted from the call fault state to the fault repair state, a second scheduling command is sent to the management node of the current storage node to release the data scheduling restriction.

2. The data scheduling control method according to claim 1, characterized in that: The step of monitoring the execution status of all storage nodes in the target cluster specifically includes: Analyze the current task execution log in real time to obtain the latest execution results; Perform a key field screening operation on the latest execution result to obtain a screening result; If the key field in the screening result is an indication field when a running error is reported, the storage node corresponding to the current task is in a call failure state; If the key field in the screening result is an indication field for successful operation, the storage node corresponding to the current task is in a normal calling state.

3. The data scheduling control method according to claim 1, characterized in that: Before executing the step of monitoring the execution status of all storage nodes in the target cluster, the method further includes: Construct a mapping relationship between management nodes and storage nodes; Setting all management nodes in the target cluster node as scheduling control objects; According to the mapping relationship between the management node and the storage node, different storage nodes under the same management node are set as scheduling control objects of the same management node; Set the scheduling state and scheduling weight of all storage nodes and management nodes, wherein the scheduling state includes a schedulable state and a prohibited scheduling state, and the scheduling weight includes a first weight value and a second weight value, the schedulable state corresponds to the first weight value, and the prohibited scheduling state corresponds to the second weight value; Initializing the initial scheduling state of all nodes to the schedulable state, and initializing the initial scheduling weights of all nodes to the first weight value; The initial data balancing speed of different storage nodes under the same management node is set, wherein the data balancing speed includes the data balancing speeds corresponding to the schedulable state and the prohibited scheduling state respectively.

4. The data scheduling control method according to claim 3, characterized in that: When it is detected that the execution state corresponding to the current storage node is a call failure state, the method further includes: Setting the scheduling state of the current storage node to a scheduling disabled state; Setting the scheduling weight of the current storage node to a second weight value; The data balancing speed of the current storage node is set to the data balancing speed corresponding to the scheduling disabled state.

5. The data scheduling control method according to claim 3, characterized in that: The step of sending a first scheduling command to the management node of the current storage node to generate a data scheduling restriction specifically includes: According to a preset mapping relationship between management nodes and storage nodes, determine the management node corresponding to the current storage node, and determine other storage nodes managed by the management node; Sending the first scheduling command to the management node to instruct the management node to filter out a preset first adjustment strategy according to the first scheduling command, wherein the first adjustment strategy includes a scheduling state, a scheduling weight, and a data balancing speed corresponding to when generating a data scheduling restriction; Setting the scheduling state of the management node to a schedulable state according to the first adjustment strategy; Setting the scheduling weight of the management node to a first weight value according to the first adjustment strategy; According to the first adjustment strategy, the data balancing speed of other storage nodes managed by the management node is set to the data balancing speed corresponding to the scheduling prohibited state.

6. The data scheduling control method according to claim 3, characterized in that: When it is monitored that the execution state corresponding to the current storage node is adjusted from the call failure state to the failure repair state, the method further includes: Adjusting the scheduling state of the current storage node from a scheduling-disabled state to a scheduling-enabled state; Adjusting the scheduling weight of the current storage node from the second weight value to the first weight value; The data balancing speed of the current storage node is adjusted from the data balancing speed corresponding to the scheduling disabled state to the data balancing speed corresponding to the scheduling enabled state.

7. The data scheduling control method according to claim 5, characterized in that: The step of sending a second scheduling command to the management node of the current storage node to release the data scheduling restriction specifically includes: Determine the management node corresponding to the current storage node and other storage nodes managed by the management node according to the mapping relationship between the management node and the storage node; Sending the second scheduling command to the management node to instruct the management node to filter out a preset second adjustment strategy according to the second scheduling command, wherein the second adjustment strategy includes a scheduling state, a scheduling weight, and a data balancing speed corresponding to lifting the data scheduling restriction; Setting the scheduling state of the management node to a scheduling prohibited state according to the second adjustment strategy; Setting the scheduling weight of the management node to a second weight value according to the second adjustment strategy; According to the second adjustment strategy, the data balancing speed of other storage nodes managed by the management node is set to the data balancing speed corresponding to the schedulable state.

8. A data scheduling control device, characterized in that: include: An execution status monitoring module, used to monitor the execution status of all storage nodes in the target cluster, wherein the storage nodes are used to store data called by the target service according to the service request; A data scheduling restriction generating module is used to send a first scheduling command to a management node of the current storage node to generate a data scheduling restriction if it is detected that the execution state corresponding to the current storage node is a call failure state; A latest available data acquisition module, used to identify the invalid data in the current storage node and acquire the latest available data from the data source of the invalid data; A spare node screening module, used for screening the spare node of the current storage node according to the storage mapping relationship between all storage nodes and the data source end, wherein the spare node contains the latest available data transmitted by the data source end; A target service remediation execution module, used for jumping to the standby node to complete the data called by the target service when a standby node is screened; A current storage node repair module, used to repair the current storage node and adjust the execution state corresponding to the current storage node to a fault repair state after the repair; The data scheduling restriction lifting module is used to send a second scheduling command to the management node of the current storage node to lift the data scheduling restriction if it monitors that the execution state corresponding to the current storage node is adjusted from the call fault state to the fault repair state.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data scheduling control method as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data scheduling control method according to any one of claims 1 to 7 are implemented.

Images