A cluster processing method and device, electronic equipment and computer readable medium

Abstract

The application discloses a cluster processing method and device, electronic equipment and computer readable medium, relates to the technical field of big data data preprocessing, and a specific embodiment comprises receiving a cluster processing request, obtaining a corresponding role identifier and cluster identifier; based on the cluster identifier, the corresponding cluster startup sequence is determined; according to the role identifier and the cluster identifier, the corresponding startup command is called, the corresponding startup command is executed based on the cluster startup sequence, after a preset time, it is checked whether the process corresponding to the corresponding startup command is started, the inspection result data is obtained; the availability identifier of each node in the cluster corresponding to the cluster identifier is obtained, the target node is determined based on the inspection result data and the availability identifier; the job task is submitted to the target node, and in response to the successful execution of the job task, the cluster startup sequence is determined based on the cluster startup sequence, and the corresponding cluster shutdown command is executed based on the cluster shutdown sequence. The cluster processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of big data preprocessing technology, and in particular to a cluster processing method, apparatus, electronic device and computer-readable medium. Background Technology

[0002] Currently, parallel transcoding systems use multiple servers and various open-source software. As a big data processing platform, parallel transcoding systems directly use open-source products and multiple machines, which cannot provide user-friendly support for program startup and post-startup monitoring. This requires a lot of manual operation and results in low cluster processing efficiency. Summary of the Invention

[0003] In view of this, embodiments of this application provide a cluster processing method, apparatus, electronic device, and computer-readable medium that can solve the problem of low efficiency in existing cluster processing.

[0004] To achieve the above objectives, according to one aspect of the embodiments of this application, a cluster processing method is provided, comprising:

[0005] Receive cluster processing requests and obtain the corresponding role identifier and cluster identifier;

[0006] Determine the corresponding cluster startup order based on the cluster identifier;

[0007] Based on the role identifier and cluster identifier, the corresponding startup command is invoked and executed according to the cluster startup order. After a preset time, the process corresponding to the startup command is checked to see if it has started and the check result data is obtained.

[0008] Obtain the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and availability identifiers;

[0009] Submit the job to the target node. In response to the successful execution of the job, determine the cluster shutdown order based on the cluster startup order, and execute the corresponding cluster shutdown command based on the cluster shutdown order.

[0010] Optionally, the corresponding cluster startup order is determined, including:

[0011] The cluster type is determined based on the cluster identifier;

[0012] Based on the cluster type, determine the cluster hierarchy, and then determine the cluster startup order based on the cluster hierarchy.

[0013] Optionally, before invoking the corresponding startup command, the method further includes:

[0014] Based on the role identifier, determine the corresponding list of startup commands;

[0015] Based on the cluster identifier, the corresponding startup command is determined from the startup command list.

[0016] Optionally, the target node is determined, including:

[0017] In response to the check results data, if the corresponding processes for the corresponding startup commands are all started, the available nodes in each slave node of the cluster are identified as the target nodes.

[0018] Optionally, submit the job task to the target node, including:

[0019] The number of job tasks is determined based on the corresponding startup command;

[0020] Submit job tasks to the target node based on the number of job tasks.

[0021] Optionally, after submitting the job task to the target node, the method further includes:

[0022] The checker is invoked to check whether the submitted job task has been accepted and is available by the target node. In response to the job task being accepted and available by the target node, the job task is determined to have been submitted successfully.

[0023] Optionally, the cluster shutdown order is determined based on the cluster startup order, including:

[0024] The reverse order of the cluster startup order is used to determine the cluster shutdown order.

[0025] In addition, this application also provides a cluster processing apparatus, including:

[0026] The receiving unit is configured to receive cluster processing requests and obtain the corresponding role identifier and cluster identifier.

[0027] The sequence determination unit is configured to determine the corresponding cluster startup order based on the cluster identifier;

[0028] The process startup unit is configured to call the corresponding startup command based on the role identifier and cluster identifier, execute the corresponding startup command according to the cluster startup order, and check whether the process corresponding to the corresponding startup command has started after a preset time, and obtain the check result data.

[0029] The target node determination unit is configured to obtain the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and the availability identifiers;

[0030] The execution unit is configured to submit job tasks to the target node, and in response to the successful execution of the job task, determine the cluster shutdown order based on the cluster startup order, and execute the corresponding cluster shutdown command based on the cluster shutdown order.

[0031] Optionally, the sequence determination unit is further configured to:

[0032] The cluster type is determined based on the cluster identifier;

[0033] Based on the cluster type, determine the cluster hierarchy, and then determine the cluster startup order based on the cluster hierarchy.

[0034] Optionally, the process initiation unit is further configured to:

[0035] Based on the role identifier, determine the corresponding list of startup commands;

[0036] Based on the cluster identifier, the corresponding startup command is determined from the startup command list.

[0037] Optionally, the target node determination unit is further configured to:

[0038] In response to the check results data, if the corresponding processes for the corresponding startup commands are all started, the available nodes in each slave node of the cluster are identified as the target nodes.

[0039] Optionally, the execution unit is further configured to:

[0040] The number of job tasks is determined based on the corresponding startup command;

[0041] Submit job tasks to the target node based on the number of job tasks.

[0042] Optionally, the execution unit is further configured to:

[0043] The checker is invoked to check whether the submitted job task has been accepted and is available by the target node. In response to the job task being accepted and available by the target node, the job task is determined to have been submitted successfully.

[0044] Optionally, the execution unit is further configured to:

[0045] The reverse order of the cluster startup order is used to determine the cluster shutdown order.

[0046] In addition, this application also provides a cluster processing electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the cluster processing method described above.

[0047] In addition, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the cluster processing method described above.

[0048] To achieve the above objectives, according to another aspect of the embodiments of this application, a computer program product is provided.

[0049] A computer program product according to an embodiment of this application includes a computer program that, when executed by a processor, implements the cluster processing method provided in an embodiment of this application.

[0050] One embodiment of the above invention has the following advantages or beneficial effects: This application receives a cluster processing request, obtains the corresponding role identifier and cluster identifier; determines the corresponding cluster startup order based on the cluster identifier; calls the corresponding startup command according to the role identifier and cluster identifier, executes the corresponding startup command according to the cluster startup order, checks whether the process corresponding to the startup command has started after a preset time, and obtains the check result data; obtains the availability identifier of each node in the cluster corresponding to the cluster identifier, determines the target node based on the check result data and availability identifier; submits the job task to the target node, and in response to the successful execution of the job task, determines the cluster shutdown order based on the cluster startup order, and executes the corresponding cluster shutdown command based on the cluster shutdown order. For the daily startup and shutdown of parallel transcoding clusters, the corresponding startup command reduces the complex manual startup, shutdown, and check work, improving cluster processing efficiency.

[0051] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0052] The accompanying drawings are provided to better understand this application and do not constitute an undue limitation thereof. Wherein:

[0053] Figure 1 This is a schematic diagram of the main flow of a cluster processing method according to an embodiment of this application;

[0054] Figure 2 This is a schematic diagram of the main flow of a cluster processing method according to an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the main flow of a cluster processing method according to an embodiment of this application;

[0056] Figure 4 This is a schematic diagram of the main units of a cluster processing apparatus according to an embodiment of this application;

[0057] Figure 5 This is an exemplary system architecture diagram to which embodiments of this application can be applied;

[0058] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0059] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the collection, analysis, use, transmission, and storage of user personal information involved in the technical solutions of this application comply with relevant laws and regulations, are used for legitimate and reasonable purposes, are not shared, disclosed, or sold outside of these legitimate uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to such personal information data, ensure that personnel authorized to access personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once this user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0060] When used, including in certain relevant applications, data is deidentified to protect user privacy, for example by removing specific identifiers, controlling the amount or specificity of stored data, controlling how data is stored, and / or other methods.

[0061] Figure 1 This is a schematic diagram of the main flow of a cluster processing method according to an embodiment of this application, as shown below. Figure 1 As shown, the cluster processing methods include:

[0062] Step S101: Receive cluster processing request and obtain the corresponding role identifier and cluster identifier.

[0063] In this embodiment, the execution entity of the cluster processing method (e.g., a server) can receive cluster processing requests via wired or wireless connections. Specifically, the cluster processing request can be a request to control the start and stop of the cluster. This embodiment does not specifically limit the content of the cluster processing request. After receiving the cluster processing request, the execution entity can obtain the role identifier and cluster identifier carried in the request. In a master-slave mode, the role identifier can be used to identify whether the service provider is the master or slave machine. This embodiment may involve ZooKeeper clusters or Hadoop clusters. The cluster identifier can be used to identify the cluster required by the cluster processing method in this embodiment, such as a ZooKeeper cluster or a Hadoop cluster. Both ZooKeeper and Hadoop clusters are clusters that use a parallel transcoding master-slave mode for service output. Specifically, parallel transcoding means using the "Spark on YARN" mode to transcode files, converting the encoding from EBCDIC to UTF-8. Spark on YARN runs Spark applications on a YARN cluster without requiring Spark to start any processes or services. Simply select a node, install Spark as a client, and submit Spark jobs to the YARN cluster for execution. The master / workers architecture embodies the divide-and-conquer principle in computer systems. The master machine maintains cluster metadata and performs scheduling based on this metadata, while the workers perform read / write operations on specific data slices (storage system) or act as execution units for subtasks (computation system).

[0064] Step S102: Determine the corresponding cluster startup order based on the cluster identifier.

[0065] For example, ZooKeeper was developed as a sub-project of Hadoop. Therefore, if a cluster identifier corresponds to both a ZooKeeper cluster and a Hadoop cluster, the startup order can be determined as: ZooKeeper cluster - Hadoop cluster. The Hadoop cluster needs to be started after the ZooKeeper cluster has started.

[0066] Step S103: Based on the role identifier and cluster identifier, call the corresponding startup command, execute the corresponding startup command according to the cluster startup order, and after a preset time, check whether the process corresponding to the startup command has started and obtain the check result data.

[0067] When the role identifier corresponds to the slave machine and the cluster identifier corresponds to the ZooKeeper cluster, the corresponding startup command could be, for example, `app_start.sh zookeeper`. This means: Log in to worker01, worker02, and worker03, start the ZooKeeper program using the startup command; after 30 seconds, check if the distributed application coordination service software ZooKeeper exists and if the node status is normal.

[0068] When the role identifier corresponds to the master and slave machines, and the cluster identifier corresponds to the Hadoop cluster, the corresponding startup command could be, for example, `app_start.sh hadoop`. This means: Log in to the master01 machine, first start the HDFS subsystem program of the Hadoop cluster; wait 30 seconds, check on the master node for the presence of the NameNode (the metadata node manages the file system namespace, storing metadata for all files and folders in a file system tree) and the DFSZKFailoverController; check on worker01-worker03 for the following processes starting: DataNode and JournalNode; check on other worker nodes for the presence of DataNode processes starting; then start the YARN subsystem program of the Hadoop cluster, wait 30 seconds, check on the master node for the ResourceManager process starting, and check on the worker nodes for the NodeManager process starting. Finally, use the `YARN node-list` command to check if the worker nodes are available. YARN is a resource manager, i.e., a general resource management system.

[0069] The corresponding startup command is executed based on the cluster startup sequence. After a preset time (e.g., 30 seconds), it is checked whether the process corresponding to the startup command has started and the check result data is obtained.

[0070] Step S104: Obtain the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and availability identifiers.

[0071] Specifically, determining the target node includes: in response to the check result data, if the corresponding processes of the corresponding startup commands are all started, for example, if the corresponding processes of the NameNode (metadata node), DFSZKFailoverController (failover controller), ResourceManager (resource manager), and NodeManager (node ​​agent) of the app_start.sh hadoop are all started, then the available nodes in each slave node of the cluster can be determined as the target nodes.

[0072] For example, check if the worker slave nodes in the cluster corresponding to the cluster identifier are available. Identify the available worker slave nodes as the target nodes.

[0073] Step S105: Submit the job task to the target node. In response to the successful execution of the job task, determine the cluster shutdown order based on the cluster startup order, and execute the corresponding cluster shutdown command based on the cluster shutdown order.

[0074] Specifically, submitting job tasks to the target node includes: determining the number of job tasks based on the corresponding start command; and submitting the job tasks to the target node based on the number of job tasks.

[0075] For example, the corresponding startup command could be: `app_start.sh converter N`, where N represents the number of job tasks. The startup command can be input by the user, and N in the startup command can be set by the user. This embodiment of the application does not specifically limit the number of tasks N in the startup command. The meaning of the startup command `app_start.sh converter N` can be: submit Spark (with its own cluster management and a separate scheduler) job tasks for the converter, where N is the number of job tasks. After startup, it checks whether the job tasks have been accepted and are available by the Hadoop (distributed system infrastructure) cluster (this can be checked using the `yarn application-list` command).

[0076] This embodiment receives cluster processing requests and obtains the corresponding role identifier and cluster identifier. Based on the cluster identifier, it determines the corresponding cluster startup order. According to the role identifier and cluster identifier, it invokes the corresponding startup command and executes it according to the cluster startup order. After a preset time, it checks whether the process corresponding to the startup command has started and obtains the check result data. It obtains the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determines the target node based on the check result data and availability identifiers. It submits the job task to the target node. In response to the successful execution of the job task, it determines the cluster shutdown order based on the cluster startup order and executes the corresponding cluster shutdown command according to the cluster shutdown order. For the daily startup and shutdown of parallel transcoding clusters, the corresponding startup command reduces the complex manual startup, shutdown, and checking work, improving cluster processing efficiency.

[0077] Figure 2 This is a schematic diagram of the main flow of a cluster processing method according to an embodiment of this application, as follows: Figure 2 As shown, the cluster processing methods include:

[0078] Step S201: Receive cluster processing request and obtain the corresponding role identifier and cluster identifier.

[0079] Role identifiers can be used to identify whether the machine providing the service is a master or a slave machine. Cluster identifiers can be used to identify the cluster used in the cluster processing method of this application embodiment, such as a ZooKeeper cluster and a Hadoop cluster.

[0080] Step S202: Determine the corresponding cluster startup order based on the cluster identifier.

[0081] In this embodiment of the application, the cluster identifier may also include a flag representing the cluster startup order. For example, the cluster identifier may be zookeeper-1 or hadoop-2, where 1 and 2 can represent the cluster startup order. By obtaining the numbers or letters in the cluster identifier, the corresponding cluster startup order can be determined based on the obtained data or letters (e.g., a and b).

[0082] Step S203: Determine the corresponding cluster type based on the cluster identifier.

[0083] Cluster identifiers can also contain flags indicating the type of cluster. For example, in the cluster identifier zookeeper-1, zookeeper represents a zookeeper cluster, and in hadoop-2, hadoop represents a hadoop cluster.

[0084] Step S204: Determine the cluster hierarchy based on the cluster type, and then determine the cluster startup order based on the cluster hierarchy.

[0085] Based on the determined cluster types and the predefined correspondence between cluster types and cluster subordination, the cluster subordination can be determined as follows: the ZooKeeper cluster is subordinate to the Hadoop cluster. The cluster with the lower subordination position will be started earlier. For example, the ZooKeeper cluster is in a lower subordination position compared to the Hadoop cluster, and the ZooKeeper cluster starts earlier than the Hadoop cluster. That is, the cluster startup order is: ZooKeeper cluster - Hadoop cluster.

[0086] Step S205: Based on the role identifier and cluster identifier, call the corresponding startup command, execute the corresponding startup command according to the cluster startup order, and after a preset time, check whether the process corresponding to the corresponding startup command has started and obtain the check result data.

[0087] Each role identifier and cluster identifier has a corresponding startup command. The executing entity can call a mapping table consisting of role identifiers, cluster identifiers, and startup commands to determine and call the corresponding startup command based on the role identifiers and cluster identifiers in this embodiment. The corresponding startup command is executed based on the determined cluster startup order, and after a preset time, such as 30 seconds, it checks whether each process corresponding to the startup command has started normally, obtaining the check result data for whether each process has started normally.

[0088] Step S206: Obtain the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and availability identifiers.

[0089] Availability identifiers, such as Y and N, can be used, where Y represents that the node is available and N represents that the node is unavailable. Nodes with an availability identifier of Y are identified as target nodes.

[0090] Step S207: Submit the job task to the target node. In response to the successful execution of the job task, determine the cluster shutdown order based on the cluster startup order, and execute the corresponding cluster shutdown command based on the cluster shutdown order.

[0091] The execution entity can execute the `app_start.sh allcvt N` command to sequentially start the ZooKeeper cluster and the Hadoop cluster, and submit the converter job task. Then, it can execute the script `jspall.sh` to log in to the cluster's master and worker nodes and display the Java programs running on those nodes.

[0092] Specifically, after submitting the job task to the target node, the method also includes: calling a checker to check whether the submitted job task has been accepted and is available by the target node, and in response to the job task being accepted and available by the target node, determining that the job task has been successfully submitted.

[0093] For example, after submitting the job to the target node, execute the script app_stop.sh with parameters similar to app_start.sh, execute the stop command, check the relevant programs on the machine, and force-kill any programs that are still running after 30 seconds.

[0094] Figure 3 This is a schematic diagram of the main flow of a cluster processing method according to an embodiment of this application, as follows: Figure 3 As shown, the cluster processing methods include:

[0095] Step S301: Receive cluster processing request and obtain the corresponding role identifier and cluster identifier.

[0096] Step S302: Determine the corresponding cluster startup order based on the cluster identifier.

[0097] Step S303: Determine the corresponding list of startup commands based on the role identifier.

[0098] For example, when the role identifier corresponds to a slave machine, the list of startup commands involved in the slave machine can be: app_start.sh zookeeper, app_start.sh hadoop, app_start.sh converter N, app_start.sh allcvt N, jspall.sh, app_stop.sh.

[0099] Step S304: Based on the cluster identifier, determine the corresponding startup command from the startup command list.

[0100] After determining the startup command list (app_start.sh zookeeper, app_start.sh hadoop, app_start.sh converter N, app_start.sh allcvt N), the execution entity can then determine the startup command for zookeeper (app_start.sh zookeeper) and the startup commands for Hadoop (app_start.sh hadoop, app_start.sh converter N, app_start.sh allcvt N) based on the cluster identifier (e.g., zookeeper and hadoop). After the startup commands are executed, the scripts jspall.sh and app_stop.sh are executed to display the Java programs started on the nodes and finally, the stop command is executed to terminate the corresponding processes.

[0101] Step S305: Invoke the corresponding startup command and execute the corresponding startup command based on the cluster startup order. After a preset time, check whether the process corresponding to the startup command has started and obtain the check result data.

[0102] After determining the startup commands corresponding to the role identifier and cluster identifier, the executing entity can execute the startup commands based on the cluster startup order. For example, it can first execute the startup command corresponding to the ZooKeeper cluster, and then execute the startup command corresponding to the Hadoop cluster after the ZooKeeper cluster has started. After a preset time (e.g., 30 seconds), the executing entity checks whether all processes corresponding to the executed startup commands have started and obtains the check result data. The check result data may include whether all processes have started and whether some processes have not started. This embodiment of the application does not specifically limit the check result data.

[0103] Step S306: Obtain the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and availability identifiers.

[0104] Step S307: Submit the job task to the target node. In response to the successful execution of the job task, determine the cluster shutdown order based on the cluster startup order, and execute the corresponding cluster shutdown command based on the cluster shutdown order.

[0105] Specifically, the cluster shutdown order is determined based on the cluster startup order, including reversing the startup order to determine the shutdown order. For example, if the cluster startup order is ZooKeeper cluster - Hadoop cluster - Converter application, then the cluster shutdown order could be Converter application - Hadoop cluster - ZooKeeper cluster. The execution entity can execute the `app_stop.sh allcvt` command to shut down the Converter application, Hadoop cluster, and ZooKeeper cluster in that order.

[0106] This application provides a script for one-click startup and shutdown of a Hadoop application cluster, along with a script to check the startup and shutdown status. It also provides a script to display Java programs on the cluster machines, facilitating daily maintenance of parallel transcoding systems. This application provides a simple script for starting and stopping parallel transcoding clusters, reducing complex manual startup, shutdown, and checking work, and improving the ability to resolve major problems in production and testing environments (such as large-scale outages). The cluster processing method of this application can be extended to other business components using Hadoop clusters and ZooKeeper clusters, improving the level of automation for other system components.

[0107] Figure 4 This is a schematic diagram of the main units of a cluster processing apparatus according to an embodiment of this application. Figure 4 As shown, the cluster processing device 400 includes a receiving unit 401, an order determination unit 402, a process initiation unit 403, a target node determination unit 404, and an execution unit 405.

[0108] The receiving unit 401 is configured to receive cluster processing requests and obtain the corresponding role identifier and cluster identifier.

[0109] Sequence determination unit 402 is configured to determine the corresponding cluster startup order based on the cluster identifier.

[0110] The process startup unit 403 is configured to call the corresponding startup command based on the role identifier and cluster identifier, execute the corresponding startup command based on the cluster startup order, and check whether the process corresponding to the startup command has started after a preset time, and obtain the check result data.

[0111] The target node determination unit 404 is configured to obtain the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and the availability identifiers.

[0112] Execution unit 405 is configured to submit job tasks to the target node, and in response to successful job task execution, determine the cluster shutdown order based on the cluster startup order, and execute the corresponding cluster shutdown command based on the cluster shutdown order.

[0113] In some embodiments, the sequence determination unit 402 is further configured to: determine the corresponding cluster type according to the cluster identifier; determine the cluster subordination relationship according to the cluster type; and then determine the cluster startup order according to the cluster subordination relationship.

[0114] In some embodiments, the process startup unit 403 is further configured to: determine the corresponding startup command list based on the role identifier; and determine the corresponding startup command from the startup command list based on the cluster identifier.

[0115] In some embodiments, the target node determination unit 404 is further configured to: in response to the fact that the processes corresponding to the corresponding start commands are all started according to the check result data, determine the available nodes in each slave node of the cluster as target nodes.

[0116] In some embodiments, the execution unit 405 is further configured to: determine the number of job tasks according to the corresponding start command; and submit the job tasks to the target node based on the number of job tasks.

[0117] In some embodiments, the execution unit 405 is further configured to: invoke a checker to check whether the submitted job task has been accepted and is available by the target node, and in response to the job task being accepted and available by the target node, determine that the job task has been successfully submitted.

[0118] In some embodiments, the execution unit 405 is further configured to determine the reverse order of the cluster startup order as the cluster shutdown order.

[0119] It should be noted that the cluster processing method and cluster processing device in this application are related in terms of specific implementation content, so repeated content will not be described again.

[0120] Figure 5 An exemplary system architecture 500 is shown that can be applied to the cluster processing method or cluster processing apparatus of the embodiments of this application.

[0121] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0122] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0123] Terminal devices 501, 502, and 503 can be various electronic devices with cluster processing screens and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0124] Server 505 can be a server providing various services, such as a backend management server supporting cluster processing requests submitted by users using terminal devices 501, 502, and 503 (for example only). The backend management server can receive cluster processing requests, obtain the corresponding role identifier and cluster identifier; determine the corresponding cluster startup order based on the cluster identifier; call the corresponding startup command according to the role identifier and cluster identifier, execute the corresponding startup command according to the cluster startup order, check whether the process corresponding to the startup command has started after a preset time, and obtain the check result data; obtain the availability identifier of each node in the cluster corresponding to the cluster identifier, determine the target node based on the check result data and availability identifier; submit the job task to the target node, and in response to the successful execution of the job task, determine the cluster shutdown order based on the cluster startup order, and execute the corresponding cluster shutdown command according to the cluster shutdown order. For the daily startup and shutdown of parallel transcoding clusters, the corresponding startup commands reduce the complex manual startup, shutdown, and checking work, improving cluster processing efficiency.

[0125] It should be noted that the cluster processing method provided in this application embodiment is generally executed by server 505, and correspondingly, the cluster processing device is generally set in server 505.

[0126] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0127] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing a terminal device according to the embodiments of this application. Figure 6 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0128] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0129] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0130] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this application.

[0131] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including a receiving unit, an order determination unit, a process initiation unit, a target node determination unit, and an execution unit. The names of these units do not necessarily limit the specific unit itself.

[0134] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the device, the device receives a cluster processing request, obtains the corresponding role identifier and cluster identifier; determines the corresponding cluster startup order based on the cluster identifier; calls the corresponding startup command according to the role identifier and cluster identifier, executes the corresponding startup command according to the cluster startup order, checks whether the process corresponding to the startup command has started after a preset time, and obtains the check result data; obtains the availability identifier of each node in the cluster corresponding to the cluster identifier, determines the target node based on the check result data and availability identifier; submits a job task to the target node, and in response to the successful execution of the job task, determines the cluster shutdown order based on the cluster startup order, and executes the corresponding cluster shutdown command based on the cluster shutdown order.

[0135] The computer program product of this application includes a computer program that, when executed by a processor, implements the cluster processing method in the embodiments of this application.

[0136] According to the technical solution of this application embodiment, for the start-up and shutdown of daily parallel transcoding clusters, the corresponding start command reduces the manual and complex start-up, shutdown and inspection work, thereby improving the cluster processing efficiency.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A cluster processing method, characterized in that, include: Receive cluster processing requests and obtain the corresponding role identifier and cluster identifier; Based on the cluster identifier, the corresponding cluster startup order is determined; Based on the role identifier and the cluster identifier, the corresponding startup command is invoked, and the corresponding startup command is executed according to the cluster startup order. After a preset time, it is checked whether the process corresponding to the corresponding startup command has started, and the check result data is obtained. Obtain the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and the availability identifiers; The job task is submitted to the target node. In response to the successful execution of the job task, the cluster shutdown order is determined based on the cluster startup order, and the corresponding cluster shutdown command is executed based on the cluster shutdown order.

2. The method according to claim 1, characterized in that, Determining the corresponding cluster startup order includes: Based on the cluster identifier, the corresponding cluster type is determined; Based on the cluster type, the cluster hierarchy is determined, and then the cluster startup order is determined based on the cluster hierarchy.

3. The method according to claim 1, characterized in that, Before invoking the corresponding startup command, the method further includes: Based on the role identifier, determine the corresponding list of startup commands; Based on the cluster identifier, the corresponding startup command is determined from the startup command list.

4. The method according to claim 1, characterized in that, The determination of the target node includes: In response to the fact that the processes corresponding to the corresponding startup commands are all started according to the inspection result data, the available nodes in each slave node of the cluster are identified as target nodes.

5. The method according to claim 1, characterized in that, The submission of the job task to the target node includes: The number of job tasks is determined based on the corresponding start command; The job tasks are submitted to the target node based on the number of job tasks.

6. The method according to claim 1, characterized in that, After submitting the job task to the target node, the method further includes: The checker is invoked to check whether the submitted job task has been accepted and is available by the target node. In response to the job task being accepted and available by the target node, the job task is determined to have been successfully submitted.

7. The method according to claim 1, characterized in that, The process of determining the cluster shutdown order based on the cluster startup order includes: The reverse order of the cluster startup sequence is used to determine the cluster shutdown sequence.

8. A cluster processing device, characterized in that, include: The receiving unit is configured to receive cluster processing requests and obtain the corresponding role identifier and cluster identifier. The sequence determination unit is configured to determine the corresponding cluster startup order based on the cluster identifier; The process startup unit is configured to call the corresponding startup command according to the role identifier and the cluster identifier, execute the corresponding startup command according to the cluster startup order, and check whether the process corresponding to the corresponding startup command has started after a preset time, and obtain the check result data. The target node determination unit is configured to obtain the availability identifiers of each node in the cluster corresponding to the cluster identifier, and determine the target node based on the inspection result data and the availability identifiers; The execution unit is configured to submit job tasks to the target node, and in response to the successful execution of the job task, determine the cluster shutdown order based on the cluster startup order, and execute the corresponding cluster shutdown command based on the cluster shutdown order.

9. The apparatus according to claim 8, characterized in that, The sequence determination unit is further configured to: Based on the cluster identifier, the corresponding cluster type is determined; Based on the cluster type, the cluster hierarchy is determined, and then the cluster startup order is determined based on the cluster hierarchy.

10. The apparatus according to claim 8, characterized in that, The process initiation unit is further configured to: Based on the role identifier, determine the corresponding list of startup commands; Based on the cluster identifier, the corresponding startup command is determined from the startup command list.

11. The apparatus according to claim 8, characterized in that, The target node determination unit is further configured to: In response to the fact that the processes corresponding to the corresponding startup commands are all started according to the inspection result data, the available nodes in each slave node of the cluster are identified as target nodes.

12. The apparatus according to claim 8, characterized in that, The execution unit is further configured to: The number of job tasks is determined based on the corresponding start command; The job tasks are submitted to the target node based on the number of job tasks.

13. The apparatus according to claim 8, characterized in that, The execution unit is further configured to: The checker is invoked to check whether the submitted job task has been accepted and is available by the target node. In response to the job task being accepted and available by the target node, the job task is determined to have been successfully submitted.

14. A cluster processing electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

15. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images