Cluster file updating method and device, equipment and storage medium

Abstract

The present application belongs to the technical field of computer, and discloses a cluster file updating method, device, equipment and storage medium. When detecting that a file distribution task is triggered, the present application determines a file access path of a to-be-distributed file according to task information corresponding to the file distribution task; finds a to-be-updated node corresponding to the to-be-distributed file in a target cluster; pushes the file access path to the to-be-updated node, so that the to-be-updated node acquires the to-be-distributed file according to the file access path and performs file updating according to the to-be-distributed file. Since the file access path of the to-be-distributed file can be determined according to the task information of the triggered file distribution task, the file access path is pushed to the to-be-updated node in the target cluster, so that the to-be-updated node performs file updating according to the file access path, and the whole process does not need manual participation, and the management personnel only need to set the to-be-distributed task to realize cluster file updating, thereby simplifying the operation difficulty of cluster file updating.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for updating clustered files. Background Technology

[0002] In real-world applications, search engines need to process massive amounts of web pages. The specific processing steps can be divided into multiple steps such as crawling, filtering, building inverted indexes, recalling, sorting, and displaying. Each step is completed on a distributed cluster. During the processing, the distributed cluster requires a large number of data files such as rules or models to complete its internal processing logic. During the operation of the search engine, the data files used by the distributed cluster also need to be updated. Therefore, distributing files to the distributed cluster and updating the distributed cluster files is a common requirement. However, currently, updating cluster files is generally done manually. Distributed clusters contain a large number of machines, which, depending on the size of the enterprise, may even contain tens of thousands of machines. Maintenance personnel need to operate a large number of machines, making the overall process complex and tedious. Moreover, manual operation is prone to oversights, and some machines may be missed during the update process.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a cluster file update method, apparatus, device, and storage medium, aiming to solve the technical problem of complex manual operation when updating files in a cluster in the prior art.

[0005] To achieve the above objectives, the present invention provides a cluster file update method, the method comprising the following steps:

[0006] When a file distribution task is detected to be triggered, the file access path of the file to be distributed is determined according to the task information corresponding to the file distribution task.

[0007] Locate the node in the target cluster that corresponds to the file to be distributed and needs to be updated.

[0008] The file access path is pushed to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file according to the file to be distributed.

[0009] Optionally, before the step of determining the file access path of the file to be distributed based on the task information corresponding to the file distribution task when the file distribution task is detected to be triggered, the method further includes:

[0010] Real-time monitoring of message changes in the task execution queue;

[0011] If a new queue message is detected in the task execution queue, it is determined that a file distribution task has been triggered.

[0012] Accordingly, the step of determining the file access path of the file to be distributed based on the task information corresponding to the file distribution task when the file distribution task is detected to be triggered includes:

[0013] When a file distribution task is detected to be triggered, the task message corresponding to the file distribution task is read from the task execution queue;

[0014] The file access path of the file to be distributed is determined based on the task message.

[0015] Optionally, the step of determining the file access path of the file to be distributed based on the task information corresponding to the file distribution task includes:

[0016] The user-specified file and the identifier of the file to be updated are determined based on the task information corresponding to the file distribution task.

[0017] Obtain the file digest information corresponding to the user-specified file;

[0018] Based on the file summary information and the file identifier to be updated, determine whether the user-specified file is a new file;

[0019] If the user-specified file is a new file, then the user-specified file will be used as the file to be distributed.

[0020] The file identifier to be updated and the file to be distributed are sent to a preset file system for storage, and the file access path of the file to be distributed in the preset file system is obtained.

[0021] Optionally, the step of detecting whether the user-specified file is a new file based on the file digest information and the file identifier to be updated includes:

[0022] Locate the historical distribution file corresponding to the file identifier to be updated;

[0023] Obtain the file digest information corresponding to the historical distributed files to obtain historical file digest information;

[0024] Compare the document summary information with the historical document summary information;

[0025] If the file summary information is inconsistent with the historical file summary information, then the user-specified file is determined to be a new file.

[0026] Optionally, the step of using the user-specified file as a file to be distributed if the user-specified file is a new file includes:

[0027] If the user-specified file is a new file, the file update type is determined based on the task information corresponding to the file distribution task;

[0028] If the file update type is a full update, then the user-specified file will be used as the file to be distributed.

[0029] Optionally, after the step of determining the file update type based on the task information corresponding to the file distribution task if the user-specified file is a new file, the method further includes:

[0030] If the file update type is an incremental update type, then the comparison file is determined according to the file identifier to be updated; the comparison file is compared with the user-specified file to determine the incremental content of the file;

[0031] A file to be distributed is generated based on the incremental content of the file.

[0032] Optionally, after the step of detecting whether the user-specified file is a new file based on the file identifier to be updated, the method further includes:

[0033] If the user-specified file is not a new file, the file access path of the file to be distributed is searched in the preset file system based on the file summary information and the file identifier to be updated.

[0034] Optionally, the step of finding the node to be updated corresponding to the file to be distributed in the target cluster includes:

[0035] Obtain the node registry center of the target cluster;

[0036] The file identifier to be updated is determined based on the task information corresponding to the file distribution task.

[0037] Search the node registration center for the subscription node information corresponding to the file identifier to be updated;

[0038] The update node corresponding to the file to be distributed is determined based on the subscription node information.

[0039] Optionally, before the step of determining the update node corresponding to the file to be distributed based on the subscription node information, the method further includes:

[0040] The file update mode is determined based on the task information corresponding to the file distribution task.

[0041] Accordingly, the step of determining the update node corresponding to the file to be distributed based on the subscription node information includes:

[0042] The update node corresponding to the file to be distributed is determined based on the file update mode and the subscription node information.

[0043] Optionally, after the step of pushing the file access path to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file according to the file to be distributed, the method further includes:

[0044] Obtain the node registration center of the target cluster;

[0045] Receive the file update result from the node to be updated;

[0046] The node information table in the node registration center is updated based on the file update results.

[0047] Optionally, after the step of updating the node information table in the node registration center based on the file update result, the method further includes:

[0048] Obtain the push time when the file access path is pushed to the node to be updated;

[0049] The time to be detected is determined based on the push time and the preset detection duration;

[0050] When the current system time reaches the time to be detected, check whether there are any nodes to be updated that have not returned file update results;

[0051] If there are nodes that have not yet returned file update results, then these nodes will be treated as nodes to be processed.

[0052] Remove the node to be processed from the service list of the target cluster, and push the file access path to the node to be processed so that the node to be processed can update the file according to the file access path.

[0053] Furthermore, to achieve the above objectives, the present invention also proposes a cluster file update device, which includes the following modules:

[0054] The task management module is used to determine the file access path of the file to be distributed based on the task information corresponding to the file distribution task when a file distribution task is detected to be triggered.

[0055] The node lookup module is used to locate the node to be updated corresponding to the file to be distributed in the target cluster;

[0056] The file update module is used to push the file access path to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file according to the file to be distributed.

[0057] Optionally, the task management module is also used to monitor changes in messages in the task execution queue in real time; if a new queue message is detected in the task execution queue, it is determined that a file distribution task has been triggered.

[0058] The task management module is also used to read the task message corresponding to the file distribution task from the task execution queue when a file distribution task is detected to be triggered; and to determine the file access path of the file to be distributed based on the task message.

[0059] Optionally, the task management module is further configured to: determine the user-specified file and the identifier of the file to be updated based on the task information corresponding to the file distribution task; obtain the file summary information corresponding to the user-specified file; detect whether the user-specified file is a new file based on the file summary information and the identifier of the file to be updated; if the user-specified file is a new file, then use the user-specified file as the file to be distributed; send the identifier of the file to be updated and the file to be distributed to a preset file system for storage, and obtain the file access path of the file to be distributed in the preset file system.

[0060] Optionally, the task management module is further configured to: locate the historical distribution file corresponding to the file identifier to be updated; obtain the file summary information corresponding to the historical distribution file; compare the file summary information with the historical file summary information; and if the file summary information is inconsistent with the historical file summary information, determine that the user-specified file is a new file.

[0061] Optionally, the task management module is further configured to determine the file update type based on the task information corresponding to the file distribution task if the user-specified file is a new file; and to use the user-specified file as the file to be distributed if the file update type is a full update type.

[0062] Optionally, the task management module is further configured to, if the file update type is an incremental update type, determine a comparison file based on the file identifier to be updated; compare the comparison file with the user-specified file to determine the incremental content of the file; and generate a file to be distributed based on the incremental content of the file.

[0063] Optionally, the task management module is further configured to, if the user-specified file is not a new file, search for the file access path of the file to be distributed in a preset file system based on the file summary information and the identifier of the file to be updated.

[0064] Furthermore, to achieve the above objectives, the present invention also proposes a cluster file update device, which includes: a processor, a memory, and a cluster file update program stored in the memory and executable on the processor. When the cluster file update program is executed by the processor, it implements the steps of the cluster file update method described above.

[0065] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a cluster file update program, wherein the cluster file update program, when executed, implements the steps of the cluster file update method described above.

[0066] This invention determines the file access path of the file to be distributed based on the task information corresponding to the file distribution task when a file distribution task is triggered; locates the node to be updated corresponding to the file to be distributed in the target cluster; and pushes the file access path to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file accordingly. Since the file access path of the file to be distributed can be determined based on the task information of the triggered file distribution task, and the file access path is pushed to the node to be updated in the target cluster, the node to be updated can update the file according to the file access path. The entire process requires no manual intervention; administrators only need to set up the task to be distributed to achieve cluster file updates, simplifying the operation of cluster file updates. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the structure of an electronic device in the hardware operating environment involved in the embodiments of the present invention;

[0068] Figure 2 This is a flowchart illustrating the first embodiment of the cluster file update method of the present invention;

[0069] Figure 3 This is a flowchart illustrating the second embodiment of the cluster file update method of the present invention;

[0070] Figure 4 This is a flowchart illustrating the third embodiment of the cluster file update method of the present invention;

[0071] Figure 5 This is a structural block diagram of the first embodiment of the cluster file update device of the present invention.

[0072] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0073] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0074] Reference Figure 1 , Figure 1 This is a schematic diagram of the cluster file update device structure of the hardware operating environment involved in the embodiments of the present invention.

[0075] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0076] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0077] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a cluster file update program.

[0078] exist Figure 1In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the cluster file update device. The electronic device calls the cluster file update program stored in the memory 1005 through the processor 1001 and executes the cluster file update method provided in the embodiment of the present invention.

[0079] This invention provides a cluster file update method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a cluster file update method according to the present invention.

[0080] In this embodiment, the cluster file update method includes the following steps:

[0081] Step S10: When a file distribution task is detected to be triggered, determine the file access path of the file to be distributed based on the task information corresponding to the file distribution task.

[0082] It should be noted that the execution subject of this embodiment can be the cluster file update device, which can be a personal computer, server or other electronic device, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the cluster file update device is used as an example to illustrate the cluster file update method of the present invention.

[0083] It should be noted that file distribution tasks can be pre-set by administrators of the cluster file update devices. If a file distribution task is triggered, it indicates that file updates need to be performed on machines in the distributed cluster. Task information may include task triggering conditions, task name, user-specified file path, and the identifier of the file to be updated. The task triggering conditions can be set using time expressions (cron expressions).

[0084] In practical use, determining the file access path of the file to be distributed based on the task information corresponding to the file distribution task can be done by reading the task information corresponding to the file distribution task, determining the user-specified file path based on the task information, storing the user-specified file in a preset file system based on the user-specified file path, and then using the stored user-specified file as the file to be distributed, and obtaining the access path of the file to be distributed in the preset file system. The preset file system can be a distributed storage system, such as Ceph or a similar distributed storage system.

[0085] In a practical implementation, the cluster file update device can be equipped with a scheduler module and an executor module. The administrators of the cluster file update device, i.e. the operations and maintenance personnel, can set up file distribution tasks through the scheduler module, while the executor module will read the task information corresponding to the file distribution task and execute subsequent steps when it detects that the file distribution task has been triggered.

[0086] In practical applications, to ensure high availability of cluster file update devices, multiple cluster file update devices can be deployed to form a master-slave backup cluster. However, to avoid duplicate execution of file distribution tasks, the master-slave backup cluster needs to ensure that only one cluster file update device provides service at any given time. This means ensuring that only one scheduling module and task execution module provide service at any given time. For example, a cluster file update device can be selected as the master device in the master-slave backup cluster using the master election mode of the cluster registry center, ZooKeeper, and then the master device can provide service.

[0087] Furthermore, in real-world scenarios, the number of machines in a distributed cluster that needs to be managed simultaneously is extremely large, and the data files corresponding to each machine are different depending on the business type. Therefore, the number of file distribution tasks existing simultaneously may also be large. Real-time monitoring of whether file distribution tasks have been triggered also consumes a lot of performance. If the timed scheduling module is also set in the cluster file update device, it may cause insufficient performance of the cluster file update device, resulting in lag or low performance. To avoid this, the timed scheduling module can be set in a separate timed scheduling device. In order to facilitate the interaction between the timed scheduling device and the cluster file update device, so that the cluster file update device can detect the triggering of file distribution tasks in a timely manner, a message queue can be set as a task execution queue. When the timed scheduling device detects that a task to be distributed has been triggered, it stores the task information of the task to be distributed in the task execution queue. The cluster file update device monitors the message changes in the task execution queue in real time to determine whether a file distribution task has been triggered. At this time, before step S10, the following may be included:

[0088] Real-time monitoring of message changes in the task execution queue;

[0089] If a new queue message is detected in the task execution queue, it is determined that a file distribution task has been triggered.

[0090] Accordingly, step S10 may include:

[0091] When a file distribution task is detected to be triggered, the task message corresponding to the file distribution task is read from the task execution queue;

[0092] The file access path of the file to be distributed is determined based on the task message.

[0093] It should be noted that consumers corresponding to task execution queues can be created in the cluster file update device. Consumers monitor changes in messages in the task execution queue. When a new queue message is detected in the task execution queue, it is determined that a file distribution task has been triggered. At this time, the consumer can read the new queue message from the task execution queue and use the read queue message as the task message corresponding to the file distribution task. Then, the task message is parsed to determine the file access path of the file to be distributed.

[0094] In practical applications, to ensure high availability of the scheduling devices, multiple scheduling devices can be deployed to form a master-slave backup cluster. However, to avoid message duplication in the task execution queue, the master-slave backup cluster needs to ensure that only one scheduling device provides service at any given time. For example, the master-slave backup cluster can be selected as the master device through the master election mode of the cluster registry center ZooKeeper, and the master device can then provide service.

[0095] Step S20: Locate the node to be updated corresponding to the file to be distributed in the target cluster.

[0096] It should be noted that the target cluster can be a distributed cluster that needs to update files, and the node to be updated can be a cluster node in the target cluster. A cluster node can be a machine in the target cluster or a virtual machine on a machine. Finding the node to be updated corresponding to the file to be distributed in the target cluster can be done by searching for cluster nodes in the target cluster that are associated with the file to be distributed, and then using the found cluster nodes as the nodes to be updated.

[0097] Furthermore, in order to quickly determine the node to be updated, step S20 in this embodiment may include:

[0098] Obtain the node registry center of the target cluster;

[0099] The file identifier to be updated is determined based on the task information corresponding to the file distribution task.

[0100] Search the node registration center for the subscription node information corresponding to the file identifier to be updated;

[0101] The update node corresponding to the file to be distributed is determined based on the subscription node information.

[0102] It should be noted that the node registry center can be a ZooKeeper registry center, which can register the subscription node information corresponding to each cluster node in the target cluster. This subscription node information can include node name, node identifier, subscription file identifier, etc. Determining the file identifier to be updated based on the task information corresponding to the file distribution task can be achieved by parsing the task information corresponding to the file distribution task, thereby obtaining the file identifier to be updated.

[0103] In practical use, finding the subscription node information corresponding to the file identifier to be updated in the node registry can be done by searching for subscription node information in the node registry that matches the file identifier to be updated. Of course, if there are many data files, generating one subscription node information for each subscription file identifier may result in a large number of subscription node information entries, which is inconvenient to manage. Therefore, multiple subscription file identifiers corresponding to cluster nodes can be concatenated to obtain a file identifier string, and then the file identifier string can be stored in the subscription node information. This ensures that one cluster node corresponds to one subscription node information entry. In this case, finding the subscription node information corresponding to the file identifier to be updated in the node registry can be done by searching for subscription node information in the node registry whose corresponding file identifier string contains the file identifier to be updated.

[0104] In practical use, determining the node to be updated corresponding to the file to be distributed based on the subscription node information can be done by parsing the subscription node information to obtain the node name and node identifier, and then determining the node to be updated based on the node name and node identifier.

[0105] It is understandable that different cluster nodes may correspond to different business operations in a distributed cluster, and therefore the data files corresponding to different cluster nodes may also be different. In order to quickly determine the node to be updated, the cluster node can be registered in the node registry center, thereby ensuring that when it is necessary to determine the node to be updated, the node to be updated can be quickly found based on the subscription node information generated during registration.

[0106] Furthermore, in practical applications, file updates may encounter defects (bugs), meaning the program may malfunction. Updating directly on all machines could lead to difficulties in recovering from bugs. To address potential bugs, a canary update approach may be used when updating cluster files. This involves performing small-batch updates first, followed by a full update only after confirming accuracy. To better meet user needs, this embodiment includes the following step before determining the node to be updated corresponding to the file to be distributed based on the subscription node information:

[0107] The file update mode is determined based on the task information corresponding to the file distribution task.

[0108] Accordingly, the step of determining the update node corresponding to the file to be distributed based on the subscription node information includes:

[0109] The update node corresponding to the file to be distributed is determined based on the file update mode and the subscription node information.

[0110] It should be noted that file update modes can be divided into canary update mode and full update mode. Determining the update nodes corresponding to the file to be distributed based on the file update mode and subscription node information can be done by determining the cluster nodes corresponding to the file to be distributed based on the subscription node information. If the file update mode is full update mode, all cluster nodes corresponding to the file to be updated are considered update nodes. If the file update mode is canary update mode, a preset canary ratio can be obtained, and the number of nodes to be updated can be determined based on the number of cluster nodes corresponding to the file to be distributed and the preset canary ratio. The number of update nodes is then selected from the cluster nodes corresponding to the file to be distributed, matching the number of nodes to be updated.

[0111] Step S30: Push the file access path to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file according to the file to be distributed.

[0112] It is understandable that by pushing the file access path to the node to be updated, the node can obtain the file to be distributed from the preset file system based on the received file access path, and then update the local files in the node to be updated based on the file to be distributed.

[0113] In practical use, the node to be updated can also monitor the file access paths corresponding to the data file identifiers in the node registry center of the target cluster in real time. The cluster file update device can update the file access paths corresponding to the file identifiers to be updated in the node registry center. When the node to be updated detects that the file access path corresponding to the file identifier to be updated has been updated, it will read the file access path from the node registry center, obtain the file to be distributed from the preset file system according to the file access path, and then update the local file in the node to be updated according to the file to be distributed.

[0114] In the implementation, each cluster node can be configured with an agent, and the cluster file update device can be configured with a node scheduling (Master) module. The cluster file update device can send the file access path to the node to be updated through the node scheduling module, thereby waking up the agent in the node to be updated, so that the agent downloads the file to be distributed and updates the file in the cluster node. As needed, the device can also control the service startup, reload, or restart of the cluster node. To ensure that the agent of the node to be updated is not woken up repeatedly at the same time, it is necessary to ensure that only one node scheduling module provides services at any given time.

[0115] This embodiment determines the file access path of the file to be distributed based on the task information corresponding to the file distribution task when a file distribution task is triggered; it then locates the node to be updated corresponding to the file to be distributed in the target cluster; and finally pushes the file access path to the node to be updated, enabling the node to retrieve the file to be distributed based on the file access path and update the file accordingly. Since the file access path of the file to be distributed can be determined based on the task information of the triggered file distribution task, and the file access path is pushed to the node to be updated in the target cluster, allowing the node to update the file based on the file access path, the entire process requires no manual intervention. Administrators only need to set up the task to be distributed to achieve cluster file updates, simplifying the operation of cluster file updates.

[0116] refer to Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of a cluster file update method according to the present invention.

[0117] Based on the first embodiment described above, step S10 of the cluster file update method in this embodiment includes:

[0118] Step S101: When a file distribution task is detected to be triggered, determine the user-specified file and the identifier of the file to be updated based on the task information corresponding to the file distribution task.

[0119] It should be noted that determining the user-specified file and the file identifier to be updated based on the task message corresponding to the file distribution task can be achieved by parsing the task message to obtain the user-specified file path and the file identifier to be updated, and then loading the user-specified file according to the user-specified file path. The file identifier to be updated can be the business identifier of the data file that needs to be updated in the distributed cluster. This business identifier can be pre-set by the distributed cluster's operations and maintenance personnel based on the data file's business type and function. For example, assuming the business identifier of the data file that needs to be updated in the distributed cluster is `modelA-weight`, it indicates that the data file is the weight file of model A in the distributed cluster.

[0120] Step S102: Obtain the file digest information corresponding to the user-specified file.

[0121] It should be noted that the file digest information can be the digest information obtained by performing a digest operation on the user-specified file using a preset digest algorithm. The preset digest algorithm can be the MD5 algorithm or a similar algorithm, and this embodiment does not limit it.

[0122] Step S103: Detect whether the user-specified file is a new file based on the file summary information and the file identifier to be updated.

[0123] It should be noted that detecting whether a user-specified file is a new file based on file summary information and the file identifier to be updated can be done by searching for the data file corresponding to the file identifier to be updated in the target cluster, comparing the summary information of that data file with the file summary information, and if the two match, then the user-specified file is determined to be a new file.

[0124] In practical use, since the data files containing the same business identifier in the cluster nodes of the target cluster should be consistent, searching for the data file corresponding to the file identifier to be updated in the target cluster can be done by searching for the data file corresponding to the file identifier to be updated in any cluster node in the target cluster that contains the file identifier to be updated.

[0125] Furthermore, after determining that the user-specified file is a new file, it is stored as a file to be distributed in the preset file system. If only the user-specified file is compared with the data file corresponding to the file identifier to be updated in the target cluster, in a real-world scenario, rollback and other phenomena might lead to a large number of identical files being stored in the preset file system. To avoid this phenomenon and save storage space in the preset file system, step S103 in this embodiment may include:

[0126] Locate the historical distribution file corresponding to the file identifier to be updated;

[0127] Obtain the file digest information corresponding to the historical distributed files to obtain historical file digest information;

[0128] Compare the document summary information with the historical document summary information;

[0129] If the file summary information is inconsistent with the historical file summary information, then the user-specified file is determined to be a new file.

[0130] It should be noted that finding the historical distribution file corresponding to the file identifier to be updated can be done by searching for the file corresponding to the file identifier in the preset file system and using the found file as the historical distribution file. Obtaining the file digest information corresponding to the historical distribution file can be done by performing a digest operation on the historical distribution file using a preset digest algorithm and using the resulting digest information as the historical file digest information. Alternatively, the preset file system may already store file digest information; in this case, obtaining the file digest information corresponding to the historical distribution file can be done by searching for the file digest information corresponding to the historical distribution file in the preset file system and using the found file digest information as the historical file digest information.

[0131] Understandably, if the file summary information is inconsistent with the historical file summary information, it means that the user-specified file has never been stored in the preset file system. Therefore, the user-specified file can be determined to be a new file.

[0132] Step S104: If the user-specified file is a new file, then the user-specified file is used as the file to be distributed.

[0133] It is understandable that if the user-specified file is a new file, it means that the user-specified file is not the same as the data file corresponding to the file identifier to be updated in the current distributed cluster. Therefore, the user-specified file can be used as the file to be distributed.

[0134] Furthermore, in practical scenarios, to improve the execution efficiency of cluster file updates, incremental updates can be performed only on the data files in the cluster, that is, only a portion of the content of the data files in the cluster is updated. If this update method is applied, the file update type will be specified when setting the file distribution task. In order to make the cluster file update method of the present invention adaptable to this update method, step S104 in this embodiment may include:

[0135] If the user-specified file is a new file, the file update type is determined based on the task information corresponding to the file distribution task;

[0136] If the file update type is a full update, then the user-specified file will be used as the file to be distributed.

[0137] It should be noted that determining the file update type based on the task information corresponding to the file distribution task can be achieved by parsing the task information corresponding to the file distribution task to obtain the update type parameter, and then determining the file update type based on the update type parameter. For example, if the update type parameter is All, it means that the file update type is a full update type; if the update type parameter is Part, it means that the file update type is an incremental update type.

[0138] In practical use, if the user-specified file is a new file, after determining the file update type based on the task information corresponding to the file distribution task, the following steps may also be included:

[0139] If the file update type is an incremental update type, then the comparison file is determined according to the file identifier to be updated;

[0140] The comparison file is compared with the user-specified file to determine the incremental content of the file;

[0141] A file to be distributed is generated based on the incremental content of the file.

[0142] It should be noted that determining the comparison file based on the identifier of the file to be updated can be done by searching for the corresponding data file in the target cluster based on the identifier of the file to be updated, and using the found data file as the comparison file. The comparison file is then compared with a user-specified file to determine the incremental content of the file. This can be done by using a text comparison tool (diff tool) to identify the differences between the files, thereby determining the incremental content of the file.

[0143] It is understandable that if the file update type is incremental update, it means that only part of the content needs to be updated when updating the cluster file. Therefore, the comparison file can be compared with the user-specified file to determine the difference between the two, thereby obtaining the incremental content of the file. Then, the file to be distributed can be generated based on the incremental content of the file, which can save a lot of the unmodified parts, reduce the size of the file to be distributed, and thus speed up the efficiency of updating the cluster file.

[0144] Step S105: Send the file identifier to be updated and the file to be distributed to a preset file system for storage, and obtain the file access path of the file to be distributed in the preset file system.

[0145] It should be noted that when the file identifier to be updated and the file to be distributed are sent to the preset file system, the preset file system will bind and store the file to be distributed with the file identifier to be updated. After storage is completed, it will generate a file storage result based on the file access path of the file to be distributed in the preset file system and feed it back to the cluster file update device. At this time, the cluster file update device can parse the file storage result to obtain the file access path of the file to be distributed in the preset file system.

[0146] In practical use, since the user-specified file is always a new file, it will be sent to the preset file system for storage along with the file identifier to be updated. Therefore, if the user-specified file is not a new file, it means that the user-specified file has already been stored in the preset file system. At this time, the file access path of the file to be distributed can be found in the preset file system based on the file summary information and the file identifier to be updated.

[0147] In a specific implementation, finding the file access path of the file to be distributed in the preset file system based on the file summary information and the file identifier to be updated can be done by finding historical distributed files in the preset file system based on the file identifier to be updated, comparing the summary information of the historical distributed files with the file summary information, taking the historical distributed files whose summary information matches the file summary information as the file to be distributed, and obtaining the file access path of the file to be distributed in the preset file system.

[0148] Of course, to avoid unnecessary performance loss due to multiple calculations of summary information, the summary information of the files to be stored can also be stored in the preset file system when storing the files. In this case, the file access path of the file to be distributed in the preset file system can be found by using the file summary information and the file identifier to be updated as query parameters, searching for the corresponding file in the preset file system, and using the found file as the file to be distributed, and then obtaining the file access path of the file to be distributed in the preset file system.

[0149] This embodiment determines the user-specified file and the file identifier to be updated based on the task information corresponding to the file distribution task; obtains the file digest information corresponding to the user-specified file; checks whether the user-specified file is a new file based on the file digest information and the file identifier to be updated; if the user-specified file is a new file, it is designated as the file to be distributed; the file identifier to be updated and the file to be distributed are sent to a preset file system for storage, and the file access path of the file to be distributed in the preset file system is obtained. Because the user-specified file is checked for newness based on the file digest information and the file identifier to be updated before being stored in the preset file system, and is only designated as the file to be distributed if it is a new file, cluster file updates are avoided due to identical files, reducing unnecessary performance consumption.

[0150] refer to Figure 4 , Figure 4 This is a flowchart illustrating a third embodiment of a cluster file update method according to the present invention.

[0151] Based on the first embodiment described above, after step S30 of the cluster file update method in this embodiment, the method further includes:

[0152] Step S40: Obtain the node registration center of the target cluster.

[0153] It should be noted that the node registration center may contain a node information table, which may contain node information of each cluster node in the target cluster. The node information in the node information table may be generated when the cluster node joins the target cluster. This node information may include node name, node network address (IP address), node data file identifier, node update status, and other information.

[0154] Step S50: Receive the file update result from the node to be updated.

[0155] It should be noted that, to facilitate the management of the update status of nodes to be updated, these nodes can be instructed to report the file update result to the cluster file update device after performing the file update. The file update result can include information such as node identifier, node name, file identifier to be updated, and file update status. The file update status can include either "update successful" or "update failed."

[0156] Step S60: Update the node information table in the node registration center according to the file update result.

[0157] It should be noted that updating the node information table in the node registration center based on the file update results can be done by parsing the file update results to obtain the node identifier, node name, file identifier to be updated, and file update status; searching for the corresponding node information in the node information table based on the node identifier, node name, and file identifier to be updated; and updating the node update status in the node information table based on the file update status.

[0158] Understandably, if the file update status in the file update result is "update failed", then the file access path can be pushed back to the node to be updated, allowing it to retry the file update.

[0159] Furthermore, to ensure that the node to be updated can perform file updates normally as much as possible, this embodiment may further include the following after step S60:

[0160] Obtain the push time when the file access path is pushed to the node to be updated;

[0161] The time to be detected is determined based on the push time and the preset detection duration;

[0162] When the current system time reaches the time to be detected, check whether there are any nodes to be updated that have not returned file update results;

[0163] If there are nodes that have not yet returned file update results, then these nodes will be treated as nodes to be processed.

[0164] Remove the node to be processed from the service list of the target cluster, and push the file access path to the node to be processed so that the node to be processed can update the file according to the file access path.

[0165] It should be noted that determining the detection time based on the push time and the preset detection duration can be achieved by adding the push time and the preset detection duration together. For example, if the push time is 9:00 and the preset detection duration is 1 minute, then the detection time is 9:01. The current system time can be the current system time of the cluster file update device.

[0166] Understandably, if the current system time reaches the detection time and there are still nodes that have not returned file update results, it means that the nodes that have not returned file update results may not have received the file access path. In this case, the data file of the node is inconsistent with the data files of other clusters in the target cluster. If the node continues to provide services, it may cause data discrepancies. Therefore, the nodes that have not returned file update results can be designated as pending nodes. Then, the pending nodes can be removed from the service list of the target cluster, the service of the pending nodes can be suspended, and the file access path can be pushed to the pending nodes again, so that the pending nodes can retry the file update according to the file access path.

[0167] In practical use, if the node to be processed successfully updates the file again, it can be added back to the service list of the target cluster so that it can provide services normally.

[0168] This embodiment obtains the node registration center of the target cluster; receives the file update result from the node to be updated; and updates the node information table in the node registration center according to the file update result. Since after sending the file access path to the node to be updated and performing the file update, the method also obtains the file update result from the node to be updated and updates the node information table in the node registration center corresponding to the target cluster, the cluster file update device can quickly detect cluster nodes that have failed to update and perform corresponding processing, thus improving the reliability of the cluster file update method of this invention.

[0169] Furthermore, this embodiment of the invention also proposes a storage medium storing a cluster file update program, which, when executed by a processor, implements the steps of the cluster file update method described above.

[0170] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the cluster file update device of the present invention.

[0171] like Figure 5 As shown, the cluster file update device proposed in this embodiment of the invention includes:

[0172] The task management module 10 is used to determine the file access path of the file to be distributed based on the task information corresponding to the file distribution task when a file distribution task is detected to be triggered.

[0173] The node lookup module 20 is used to find the node to be updated corresponding to the file to be distributed in the target cluster;

[0174] The file update module 30 is used to push the file access path to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file according to the file to be distributed.

[0175] This embodiment determines the file access path of the file to be distributed based on the task information corresponding to the file distribution task when a file distribution task is triggered; it then locates the node to be updated corresponding to the file to be distributed in the target cluster; and finally pushes the file access path to the node to be updated, enabling the node to retrieve the file to be distributed based on the file access path and update the file accordingly. Since the file access path of the file to be distributed can be determined based on the task information of the triggered file distribution task, and the file access path is pushed to the node to be updated in the target cluster, allowing the node to update the file based on the file access path, the entire process requires no manual intervention. Administrators only need to set up the task to be distributed to achieve cluster file updates, simplifying the operation of cluster file updates.

[0176] Furthermore, the task management module 10 is also used to monitor changes in messages in the task execution queue in real time; if a new queue message is detected in the task execution queue, it is determined that a file distribution task has been triggered.

[0177] The task management module is also used to read the task message corresponding to the file distribution task from the task execution queue when a file distribution task is detected to be triggered; and to determine the file access path of the file to be distributed based on the task message.

[0178] Furthermore, the task management module 10 is also used to determine the user-specified file and the identifier of the file to be updated based on the task information corresponding to the file distribution task; obtain the file summary information corresponding to the user-specified file; detect whether the user-specified file is a new file based on the file summary information and the identifier of the file to be updated; if the user-specified file is a new file, then the user-specified file is used as the file to be distributed; send the identifier of the file to be updated and the file to be distributed to a preset file system for storage, and obtain the file access path of the file to be distributed in the preset file system.

[0179] Furthermore, the task management module 10 is also used to find the historical distribution file corresponding to the file identifier to be updated; obtain the file summary information corresponding to the historical distribution file, and obtain historical file summary information; compare the file summary information with the historical file summary information; if the file summary information is inconsistent with the historical file summary information, then determine that the user-specified file is a new file.

[0180] Furthermore, the task management module 10 is also used to determine the file update type based on the task information corresponding to the file distribution task if the user-specified file is a new file; and to use the user-specified file as the file to be distributed if the file update type is a full update type.

[0181] Furthermore, the task management module 10 is also used to determine a comparison file based on the file identifier to be updated if the file update type is an incremental update type; compare the comparison file with the user-specified file to determine the incremental content of the file; and generate a file to be distributed based on the incremental content of the file.

[0182] Furthermore, the task management module 10 is also used to search for the file access path of the file to be distributed in the preset file system based on the file summary information and the identifier of the file to be updated if the user-specified file is not a new file.

[0183] Furthermore, the node lookup module 20 is also used to obtain the node registration center of the target cluster; determine the file identifier to be updated based on the task information corresponding to the file distribution task; search for the subscription node information corresponding to the file identifier to be updated in the node registration center; and determine the node to be updated corresponding to the file to be distributed based on the subscription node information.

[0184] Furthermore, the node lookup module 20 is also used to determine the file update mode based on the task information corresponding to the file distribution task;

[0185] The node lookup module 20 is also used to determine the node to be updated corresponding to the file to be distributed based on the file update mode and the subscription node information.

[0186] Furthermore, the file update module 30 is also used to obtain the node registration center of the target cluster; receive the file update result fed back by the node to be updated; and update the node information table in the node registration center according to the file update result.

[0187] Furthermore, the file update module 30 is also used to obtain the push time when the file access path is pushed to the node to be updated; determine the detection time according to the push time and the preset detection duration; when the current system time reaches the detection time, detect whether there is a node to be updated that has not returned a file update result; if there is a node to be updated that has not returned a file update result, then the node to be updated that has not returned a file update result is designated as a node to be processed; remove the node to be processed from the service list of the target cluster, and push the file access path to the node to be processed, so that the node to be processed can perform file update according to the file access path.

[0188] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0189] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0190] In addition, for technical details not described in detail in this embodiment, please refer to the cluster file update method provided in any embodiment of the present invention, which will not be repeated here.

[0191] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0192] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0194] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A cluster file update method, characterized in that, The cluster file update method includes the following steps: When a file distribution task is detected to be triggered, the file access path of the file to be distributed is determined according to the task information corresponding to the file distribution task. Locate the node in the target cluster that corresponds to the file to be distributed and needs to be updated. The file access path is pushed to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file according to the file to be distributed; The step of finding the node to be updated corresponding to the file to be distributed in the target cluster includes: Obtain the node registry center of the target cluster; The file identifier to be updated is determined based on the task information corresponding to the file distribution task. Obtain the subscription node information corresponding to each cluster node in the node registration center, wherein one cluster node in the target cluster corresponds to one subscription node information; Extract the file identifier string from the subscription node information. The file identifier string is constructed by concatenating multiple subscription file identifiers corresponding to the cluster nodes. The subscription node information containing the file identifier to be updated in the corresponding file identifier string shall be used as the subscription node information corresponding to the file identifier to be updated; The update node corresponding to the file to be distributed is determined based on the subscription node information corresponding to the file identifier to be updated.

2. The cluster file update method as described in claim 1, characterized in that, Before the step of determining the file access path of the file to be distributed based on the task information corresponding to the file distribution task when the file distribution task is detected to be triggered, the method further includes: Real-time monitoring of message changes in the task execution queue; If a new queue message is detected in the task execution queue, it is determined that a file distribution task has been triggered. Accordingly, the step of determining the file access path of the file to be distributed based on the task information corresponding to the file distribution task when the file distribution task is detected to be triggered includes: When a file distribution task is detected to be triggered, the task message corresponding to the file distribution task is read from the task execution queue; The file access path of the file to be distributed is determined based on the task message.

3. The cluster file update method as described in claim 1, characterized in that, The step of determining the file access path of the file to be distributed based on the task information corresponding to the file distribution task includes: The user-specified file and the identifier of the file to be updated are determined based on the task information corresponding to the file distribution task. Obtain the file digest information corresponding to the user-specified file; Based on the file summary information and the file identifier to be updated, determine whether the user-specified file is a new file; If the user-specified file is a new file, then the user-specified file will be used as the file to be distributed. The file identifier to be updated and the file to be distributed are sent to a preset file system for storage, and the file access path of the file to be distributed in the preset file system is obtained.

4. The cluster file update method as described in claim 3, characterized in that, The step of detecting whether the user-specified file is a new file based on the file digest information and the file identifier to be updated includes: Locate the historical distribution file corresponding to the file identifier to be updated; Obtain the file digest information corresponding to the historical distributed files to obtain historical file digest information; Compare the document summary information with the historical document summary information; If the file summary information is inconsistent with the historical file summary information, then the user-specified file is determined to be a new file.

5. The cluster file update method as described in claim 3, characterized in that, The step of using the user-specified file as a file to be distributed if the user-specified file is a new file includes: If the user-specified file is a new file, the file update type is determined based on the task information corresponding to the file distribution task; If the file update type is a full update, then the user-specified file will be used as the file to be distributed.

6. The cluster file update method as described in claim 5, characterized in that, The step of determining the file update type based on the task information corresponding to the file distribution task if the user-specified file is a new file further includes: If the file update type is an incremental update type, then the comparison file is determined according to the file identifier to be updated; The comparison file is compared with the user-specified file to determine the incremental content of the file; A file to be distributed is generated based on the incremental content of the file.

7. The cluster file update method as described in claim 3, characterized in that, After the step of detecting whether the user-specified file is a new file based on the file digest information and the file identifier to be updated, the method further includes: If the user-specified file is not a new file, the file access path of the file to be distributed is searched in the preset file system based on the file summary information and the file identifier to be updated.

8. The cluster file update method as described in claim 1, characterized in that, Before the step of determining the update node corresponding to the file to be distributed based on the subscription node information corresponding to the file identifier to be updated, the method further includes: The file update mode is determined based on the task information corresponding to the file distribution task. Accordingly, the step of determining the update node corresponding to the file to be distributed based on the subscription node information corresponding to the file identifier to be updated includes: The update node corresponding to the file to be distributed is determined based on the file update mode and the subscription node information corresponding to the file identifier to be updated.

9. The cluster file update method according to any one of claims 1-8, characterized in that, After the step of pushing the file access path to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file according to the file to be distributed, the method further includes: Obtain the node registration center of the target cluster; Receive the file update result from the node to be updated; The node information table in the node registration center is updated based on the file update results.

10. The cluster file update method as described in claim 9, characterized in that, After the step of updating the node information table in the node registration center based on the file update result, the method further includes: Obtain the push time when the file access path is pushed to the node to be updated; The time to be detected is determined based on the push time and the preset detection duration; When the current system time reaches the time to be detected, check whether there are any nodes to be updated that have not returned file update results; If there are nodes that have not yet returned file update results, then these nodes will be treated as nodes to be processed. Remove the node to be processed from the service list of the target cluster, and push the file access path to the node to be processed so that the node to be processed can update the file according to the file access path.

11. A cluster file update device, characterized in that, The cluster file update device includes the following modules: The task management module is used to determine the file access path of the file to be distributed based on the task information corresponding to the file distribution task when a file distribution task is detected to be triggered. The node lookup module is used to locate the node to be updated corresponding to the file to be distributed in the target cluster; The file update module is used to push the file access path to the node to be updated, so that the node to be updated can obtain the file to be distributed according to the file access path and update the file according to the file to be distributed; The node lookup module is further configured to: obtain the node registration center of the target cluster; determine the file identifier to be updated based on the task information corresponding to the file distribution task; obtain the subscription node information corresponding to each cluster node in the node registration center, wherein one cluster node in the target cluster corresponds to one subscription node information; extract a file identifier string from the subscription node information, wherein the file identifier string is constructed by concatenating multiple subscription file identifiers corresponding to the cluster node; use the subscription node information containing the file identifier to be updated in the corresponding file identifier string as the subscription node information corresponding to the file identifier to be updated; and determine the node to be updated corresponding to the file to be distributed based on the subscription node information corresponding to the file identifier to be updated.

12. The cluster file update apparatus as described in claim 11, characterized in that, The task management module is also used to monitor changes in messages in the task execution queue in real time; if a new queue message is detected in the task execution queue, it is determined that a file distribution task has been triggered. The task management module is also used to read the task message corresponding to the file distribution task from the task execution queue when a file distribution task is detected to be triggered; and to determine the file access path of the file to be distributed based on the task message.

13. The cluster file update apparatus as described in claim 11, characterized in that, The task management module is further configured to: determine the user-specified file and the identifier of the file to be updated based on the task information corresponding to the file distribution task; obtain the file summary information corresponding to the user-specified file; detect whether the user-specified file is a new file based on the file summary information and the identifier of the file to be updated; if the user-specified file is a new file, then use the user-specified file as the file to be distributed; send the identifier of the file to be updated and the file to be distributed to a preset file system for storage, and obtain the file access path of the file to be distributed in the preset file system.

14. The cluster file update apparatus as described in claim 13, characterized in that, The task management module is also used to find the historical distribution file corresponding to the file identifier to be updated; obtain the file summary information corresponding to the historical distribution file, and obtain historical file summary information; compare the file summary information with the historical file summary information; if the file summary information is inconsistent with the historical file summary information, then determine that the user-specified file is a new file.

15. The cluster file update apparatus as described in claim 14, characterized in that, The task management module is further configured to determine the file update type based on the task information corresponding to the file distribution task if the user-specified file is a new file; and to use the user-specified file as the file to be distributed if the file update type is a full update type.

16. The cluster file update apparatus as described in claim 15, characterized in that, The task management module is further configured to, if the file update type is an incremental update type, determine a comparison file based on the file identifier to be updated; compare the comparison file with the user-specified file to determine the incremental content of the file; and generate a file to be distributed based on the incremental content of the file.

17. The cluster file update apparatus as described in claim 13, characterized in that, The task management module is also used to search for the file access path of the file to be distributed in the preset file system based on the file summary information and the identifier of the file to be updated if the user-specified file is not a new file.

18. A cluster file update device, characterized in that, The cluster file update device includes: a processor, a memory, and a cluster file update program stored in the memory and executable on the processor. When the cluster file update program is executed by the processor, it implements the steps of the cluster file update method as described in any one of claims 1-10.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a cluster file update program, which, when executed, implements the steps of the cluster file update method as described in any one of claims 1-10.

Images