Concurrent file collection method, device, equipment and storage medium

Abstract

The application discloses a concurrent file collection method, device and equipment and a storage medium. The method comprises the following steps: when it is detected that the start is in a starting state, if it is determined that the master node competition is successful according to the master node competition result, a collection file list is acquired; according to the collection file list, a plurality of file collection tasks are acquired, and each file collection task is added to a preset to-be-processed queue; according to the collection task acquisition request sent by the slave node, a current file collection task is acquired from the preset to-be-processed queue, the current file collection task is sent to the slave node, and the collection file corresponding to the current file collection task sent by the slave node is received. According to the technical scheme, each node competes to become a master node, the master node is used for acquiring and distributing file collection tasks, each slave node is used for executing file collection tasks, concurrent file collection can be realized, load balancing of file collection tasks can be realized, and the file collection efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of telecommunications service support, and in particular to a concurrent file collection method, device, equipment and storage medium. BACKGROUND

[0002] In the telecommunications industry, a large amount of data needs to be exchanged between various systems, and the real-time requirement for data processing is high. In a specific scenario, a large number of data processing tasks are simultaneously issued by an upstream system to a downstream system, and the downstream system is required to collect and complete processing within a short period of time. Therefore, efficient file data collection is particularly important for supporting the implementation of telecommunications services.

[0003] Currently, the existing file collection method usually uses a separate load balancing product to distribute and process data in a distributed file system. However, for this special scenario of file collection, the existing load balancing product usually cannot achieve compatibility with different systems, so it cannot effectively improve the efficiency of file collection and easily increase the complexity of the system. SUMMARY

[0004] The present application provides a concurrent file collection method, device, equipment and storage medium, which can realize concurrent file collection, load balancing of file collection tasks, improve file collection efficiency and avoid increasing system complexity.

[0005] According to an aspect of the present application, a concurrent file collection method is provided, comprising:

[0006] When it is detected that the start is in the starting state, the master node competition result is obtained, and if it is determined that the master node competition is successful according to the master node competition result, the collection file list is obtained;

[0007] According to the collection file list, at least one file collection task is obtained, and each file collection task is added to a preset pending queue;

[0008] According to the collection task acquisition request sent by the slave node, the current file collection task is obtained from the preset pending queue, and the current file collection task is sent to the slave node, so that the slave node performs file collection according to the current file collection task;

[0009] The collection file corresponding to the current file collection task sent by the slave node is received.

[0010] According to another aspect of the present application, a concurrent file collection device is provided, comprising:

[0011] The collection file list acquisition module is configured to acquire a master node competition result when it is detected that the starting state is in a starting state, and acquire a collection file list if it is determined that the master node competition is successful according to the master node competition result.

[0012] The file collection task acquisition module is configured to acquire at least one file collection task according to the collection file list, and add each file collection task to a preset to-be-processed queue.

[0013] The file collection task sending module is configured to acquire a current file collection task from the preset to-be-processed queue according to a collection task acquisition request sent by a slave node, and send the current file collection task to the slave node, so that the slave node performs file collection according to the current file collection task.

[0014] The collection file receiving module is configured to receive a collection file corresponding to the current file collection task sent by the slave node.

[0015] According to another aspect of the present application, an electronic device is provided, which comprises:

[0016] at least one processor; and

[0017] a memory connected to the at least one processor in communication; wherein

[0018] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the concurrent collection method of the file according to any one of the embodiments of the present application.

[0019] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the concurrent collection method of the file according to any one of the embodiments of the present application when executed by the processor.

[0020] The technical scheme of the embodiment of the application, when detecting that the start is in the starting state, acquires the master node competition result, if it is determined that the master node competition is successful according to the master node competition result, acquires the collection file list; then, according to the collection file list, acquires multiple file collection tasks, and adds each file collection task to the preset to-be-processed queue; secondly, according to the collection task acquisition request sent by the slave node, acquires the current file collection task from the preset to-be-processed queue, and sends the current file collection task to the slave node, and finally receives the collection file corresponding to the current file collection task sent by the slave node, through the competition of each node to become the master node, and through the master node to acquire and distribute the file collection task, and through each slave node to execute the file collection task, concurrent collection of files can be realized, load balancing of the file collection task can be realized, and the file collection efficiency can be improved.

[0021] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1A is a flowchart of a concurrent collection method of a file according to the first embodiment of the application;

[0024] Figure 1B is a flowchart of a concurrent collection method of a file according to the first embodiment of the application;

[0025] Figure 1C is a flowchart of a concurrent collection method of a file according to the first embodiment of the application;

[0026] Figure 2A is a flowchart of a concurrent collection method of a file according to the second embodiment of the application;

[0027] Figure 2B is a flowchart of a concurrent collection method of a file according to the second embodiment of the application;

[0028] Figure 3 is a flowchart of a concurrent collection method of a file according to the third embodiment of the application;

[0029] Figure 4Fig. 1 is a structural schematic diagram of an electronic device for realizing a concurrent collection method of a file according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the personnel in the technical field better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should belong to the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", "target" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment one

[0033] Figure 1A A flowchart of a concurrent collection method of a file is provided for the first embodiment of the present application. The present embodiment can be applied to the concurrent collection of files in the telecommunications industry. The method can be performed by a concurrent collection device of a file, which can be realized in the form of hardware and / or software. The concurrent collection device of a file can be configured in an electronic device. Typically, the electronic device can be a computer device or a server. As shown in Figure 1A The method comprises the following steps.

[0034] In S110, when it is detected that the start is in a starting state, a master node competition result is acquired. If it is determined that the competition of the master node is successful according to the master node competition result, a collection file list is acquired.

[0035] In the embodiment, the file collection can be performed by a distributed system composed of multiple nodes, wherein the nodes can be servers. Each node can include one master node and multiple slave nodes. The node identities of different nodes can be specified by a preset database based on pre-set identity rules, for example, the node with the minimum corresponding node identifier (such as the minimum numerical identifier, the frontmost alphabetical identifier, etc.) is taken as the master node, and the other nodes are taken as the slave nodes.

[0036] In the embodiment, the master node is responsible for obtaining the file collection task and distributing the file collection task (related information of the file to be collected), can be configured with load balancing (to balance the distribution of the file collection task among the slave nodes), and can publish the Internet Protocol (IP) address and port number to be listened to, so as to listen to the corresponding port. Correspondingly, the slave node is responsible for collecting files from the target device (for example, a remote host), and completes the action of collecting files. Secondly, when the master node is abnormal, each slave node can become a new master node through competition, thereby realizing the abnormal takeover of the master node.

[0037] It should be noted that the same configuration file can be set for the processes of each node by configuring a distributed scheduling engine, for example, a table collect_SCHEDULE_TABLE recording the information of the master processing node. Typically, the data records in the preset configuration file can be as shown in Table 1.

[0038] Table 1: Data record format in preset configuration file

[0039] Field Name Data Type Description CLUSTER VARCHAR(32) NOT NULL Cluster name (primary key) SOURCE_NAME Varchar(64) NOT NULL Data source name (primary key) BLOCK_ID VARCHAR(64) NOT NULL block id SERVER_IP VARCHAR(64) NOT NULL IP address SERVER_PORT NUMERIC(12) NOT NULL Listening port RESV VARCHAR(255) Remark

[0040] Among them, one task point corresponds to one data record. In the embodiment, one task point can start multiple processes, each process can correspond to one node, and each process reads the same configuration file. Thus, each process can obtain the same SOURCE_NAME and CLUSTER, and different blockid. At this time, after each node is started, each node tries to compete to become the master node of task processing.

[0041] In a specific example, after each node is started, the operation of inserting the node information of the node into the information publishing table of the preset database (for example, DMDB, Oracle, Mysql, etc.) is performed simultaneously to compete for the master node. The preset database can determine the master node among the nodes according to the node information and the pre-set node identity allocation rule, perform the operation of writing the node information of the master node into the information publishing table, and feed back a message of successful execution to the master node. As for the other slave nodes, a message of failed execution is directly fed back. Thus, each node can determine the master node competition result according to the message fed back by the preset database.

[0042] At this time, if the node receives an execution success message of the preset database feedback, it indicates that the competition master node is successful, and the corresponding node identity can be determined as the master node. Then, based on the pre-configured task point information (to-be-collected data source information), a request for obtaining a collection file list is sent to the device of the to-be-collected file, and the collection file list feedback by the device is received. The collection file list can be a list of all files of the to-be-collected file device.

[0043] S120, according to the collection file list, at least one file collection task is obtained, and each file collection task is added to a preset to-be-processed queue.

[0044] Specifically, after obtaining the collection file list, the collection file list can be parsed to obtain file information that needs to be collected. Each file can be a file collection task, and each file collection task is sequentially added to a pre-set to-be-processed queue. The preset to-be-processed queue can output each file collection task stored in the queue according to the first-in first-out rule.

[0045] It should be noted that in some special cases, the content of the collection file list is empty, that is, there is no file to be collected, that is, no file collection task is generated; at this time, a waiting operation can be performed to obtain the next collection file list.

[0046] S130, according to the collection task acquisition request sent by the slave node, the current file collection task is obtained from the preset to-be-processed queue, and the current file collection task is sent to the slave node, so that the slave node performs file collection according to the current file collection task.

[0047] In this embodiment, after determining the node identity of each slave node, the master node information can be obtained from the preset database, and based on the master node information, the slave node can send a collection task acquisition request to the master node. After receiving the collection task acquisition request, the master node can obtain the file collection task with the earliest addition time from the preset to-be-processed queue as the current file collection task, and send the current file collection task to the slave node. After receiving the current file collection task, the slave node can parse the file collection task to obtain the file information that needs to be collected, so as to complete the acquisition of the collection file, and send the collection file to the master node.

[0048] S140, receiving the collection file corresponding to the current file collection task sent by the slave node.

[0049] Specifically, the master node can receive the collection files sent by each slave node, and store the collection files for a long time to complete collection of all files of the current task point (data source). The collection files can be files of switch data, base station data, or user call data (authorized by the user), and other telecommunication service data. Further, based on the obtained collection files, telecommunication security monitoring can be performed, or based on the obtained collection files, regional network or call quality improvement can be performed.

[0050] The technical scheme of the embodiment of the application, when it is detected that the start is in the starting state, acquires the master node competition result, and if it is determined that the master node competition is successful according to the master node competition result, acquires a collection file list; then, acquires multiple file collection tasks according to the collection file list, and adds each file collection task to a preset to-be-processed queue; next, acquires a current file collection task from the preset to-be-processed queue according to a collection task acquisition request sent by a slave node, and sends the current file collection task to the slave node; finally, receives a collection file corresponding to the current file collection task sent by the slave node, and through competition of each node to become a master node, and through the master node to acquire and distribute file collection tasks, and through each slave node to execute file collection tasks, concurrent collection of files can be implemented, load balancing of file collection tasks can be implemented, and file collection efficiency can be improved.

[0051] In an optional implementation of the embodiment, acquiring the master node competition result can include:

[0052] generating a first node information insertion request according to the node information, and sending the first node information insertion request to a preset database;

[0053] if it is detected that an execution success message corresponding to the first node information insertion request sent by the preset database is acquired, it is determined that the master node competition result is that the master node competition is successful;

[0054] if it is detected that an execution failure message corresponding to the first node information insertion request sent by the preset database is acquired, it is determined that the master node competition result is that the master node competition is failed;

[0055] The preset database is configured to determine node identities corresponding to each node according to node information insertion requests sent by each node, and generate corresponding execution result information according to the node identities corresponding to each node, and send the execution result information to each node.

[0056] In a specific example, the master node competition process can be as follows Figure 1BAs shown. Among them, taking node A and node B as an example. Specifically, node A and node B perform the operation of inserting the node information of the node into the information publishing table of the preset database DMDB after starting, and if the table already has the information of the node, the update operation will be performed. After receiving the node information insertion request of the two nodes, DMDB can obtain the node identifiers of the two nodes based on the request; in one case, based on the pre-set node identity allocation rule and the node identifiers corresponding to nodes A and B respectively, node A is specified as the master node, so the feedback execution success message is fed back to node A, and the feedback execution failure message is fed back to node B. Thus, node A becomes the master node, and node B becomes the slave node.

[0057] Then, node A can send a request information for obtaining a file list to a remote host (a device to be collected file), and receive a file collection list fed back by the remote host. Further, node A can parse the file collection list to obtain file information to be collected, and add a piece of file information as a file collection task to the to-be-processed queue in sequence. Node B can send a master node information query request to DMDB, and receive the node information (such as IP address and port number) of node A fed back by DMDB.

[0058] In another optional embodiment of the embodiment, obtaining the collection file list comprises:

[0059] According to a preset configuration file, obtaining file collection device information;

[0060] Generating a file list obtaining request, and sending the file list obtaining request to a file collection device according to the file collection device information;

[0061] Receiving a collection file list corresponding to the file list obtaining request sent by the file collection device.

[0062] In a specific example, the master node can parse the preset configuration file to obtain remote host information of the file to be collected, that is, file collection device information. Among them, the file collection device information can include at least one of the file collection device name, the file collection device cluster name, the Internet protocol address and the port number. It can be understood that the file collection device name and the cluster name to which it belongs can also be the corresponding identifier.

[0063] Then, the master node can generate a file list obtaining request, and send the file list obtaining request to the corresponding file collection device based on the obtained file collection device information. After receiving the file list obtaining request, the file collection device can send its own file list to the master node. The master node can receive the collection file list sent by the file collection device.

[0064] In one specific implementation of the embodiment, the flow of the concurrent file collection method can be as shown in Figure 1C Specifically, the master node can store the file list information in a to-be-processed queue in one file one file collection task after obtaining the collection file list. Secondly, the slave node can send a collection task acquisition request to the master node. The master node can output a file collection task from the to-be-processed queue to the slave node after receiving the collection task acquisition request.

[0065] After that, the slave node can collect the remote file based on the file collection task after obtaining the file collection task, and send the file collection result (contained in a new collection task acquisition request) to the master node. The master node can assign another file collection task to the slave node after receiving the file collection result. The slave node can continue to collect files from the remote host after obtaining the task. Thus, through the continuous interaction of the master node and each slave node, the concurrent collection of all files of the remote host can be realized.

[0066] Embodiment Two

[0067] Figure 2A A flowchart of a concurrent file collection method provided for the second embodiment of the application, which is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with one or more of the above embodiments. As shown in Figure 2A The method comprises:

[0068] S210, when it is detected that the start is in the starting state, obtaining the master node competition result, if it is determined that the master node competition fails according to the master node competition result, obtaining the master node information, and generating a collection task acquisition request.

[0069] In this embodiment, when the node fails to compete for the master node and becomes a slave node, the master node information can be obtained from the preset database, and a request information for collecting file collection tasks is generated.

[0070] In one optional implementation of the embodiment, obtaining the master node information can comprise:

[0071] Generating a master node information acquisition request and sending the master node information acquisition request to the preset database; when receiving the feedback information corresponding to the master node information acquisition request sent by the preset database, obtaining the master node information according to the feedback information.

[0072] In one specific example, the slave node can first generate a master node information acquisition request and send it to the preset database; the preset database can acquire the master node information (e.g., node name, identification, IP address, and port number, etc.) based on the node information insertion request of the master node, and can feed back the master node information to the slave node. The slave node can determine the master node information according to the feedback information of the preset database.

[0073] S220, according to the master node information, sending the acquisition task acquisition request to the master node, and receiving the file acquisition task corresponding to the acquisition task acquisition request sent by the master node.

[0074] Specifically, after determining the master node information, the slave node can generate an acquisition task acquisition request and send it to the master node to request to acquire the current file acquisition task. After receiving the acquisition task acquisition request, the master node can output the current file acquisition task from the preset to-be-processed queue, and feed back the file acquisition task to the slave node.

[0075] S230, according to the file acquisition task, acquiring the acquisition file, and sending the acquisition file to the master node.

[0076] Specifically, after acquiring the file acquisition task, the slave node can parse the file acquisition task to acquire the information of the file to be acquired, such as the file name, address, etc., and start the file acquisition operation to acquire the acquisition file. Finally, the slave node can send the acquired acquisition file to the master node.

[0077] S240, if it is detected that there is a communication abnormality with the master node, a second node information insertion request is generated according to the node information, and the second node information insertion request is sent to the preset database.

[0078] It can be understood that in some special cases, the master node may appear abnormal, at which time the node as a slave node can re-competes for the master node to determine a new master node. Specifically, when sending the acquisition file to the master node, if it is found that the communication is interrupted; or after successfully sending the acquisition file, the feedback information of the master node cannot be received, it can be determined that there is a communication abnormality with the master node. At this time, the node can re-generate a second node information insertion request to try to insert the node information into the information publishing table of the preset database.

[0079] Secondly, after receiving the second node information insertion request of each slave node, the preset database can screen a new master node from each slave node, and perform the operation of adding the corresponding master node information to the information publishing table, and feed back the execution success message to the new master node. For other slave nodes that fail to compete, an execution failure message is fed back.

[0080] S250, if the second node information insertion request sent by the preset database is detected to correspond to an execution success message, it is determined that the competition for the master node is successful, and the node identity corresponding to the current node is updated to the master node.

[0081] In one case, if the current node successfully receives the execution success message sent by the preset database, it indicates that the competition for the master node is successful, and the current node can be the new master node to obtain and distribute the file collection task.

[0082] The technical scheme of the embodiment of the application, if the competition for the master node is determined to fail according to the competition result of the master node, the master node information is obtained, and a collection task obtaining request is generated; then, according to the master node information, the collection task obtaining request is sent to the master node, and the file collection task corresponding to the collection task obtaining request sent by the master node is received; secondly, according to the file collection task, the collection file is obtained, and the collection file is sent to the master node; further, if it is detected that there is a communication abnormality between the current node and the master node, a second node information insertion request is generated according to the node information of the current node, and the second node information insertion request is sent to the preset database; finally, if the second node information insertion request sent by the preset database is detected to correspond to an execution success message, it is determined that the competition for the master node is successful, and the node identity corresponding to the current node is updated to the master node; by using the mechanism that the slave node becomes the new master node by competition when the master node is abnormal, the abnormal takeover of the master node can be realized, and the stability of the concurrent collection of files can be improved.

[0083] In one specific embodiment of the present embodiment, the switching process of the master-slave node can be as shown in Figure 2B . Among them, node A is taken as the master node, and nodes B and C are taken as the slave nodes. Specifically, first, nodes B and C send a collection task obtaining request to node A to obtain the current file collection task, and according to the file collection task, the collection file is obtained and sent to node A. At this time, if nodes B and C detect that there is a communication abnormality between node A or a collection task response abnormality, a node information insertion request can be generated and sent to the preset database DMDB to compete for the master node.

[0084] In one case, if the node B succeeds in the competition for the master node, the node B becomes the new master node, and the node C can continue to be a slave node and obtain the node information of the node B from the DMDB. Secondly, after becoming the new master node, the node B can obtain the collection file list from the remote host, and obtain each file collection task based on the collection file list, and add each file collection task to the preset to-be-processed queue of the node B. Then, the node C can send a collection task obtaining request to the node B, and the node B can respond to the node C with the current file collection task. The node C can obtain the collection file from the remote host based on the obtained file collection task, and send the collection file to the node B.

[0085] In the embodiment, through the master-slave mode, multiple collection processes can be concurrently processed, the throughput of the system can be more than 10 times of that of the single process (affected by the network and the remote host), and in the case of a large number of files appearing instantaneously, the real-time performance of file collection can be greatly improved.

[0086] Embodiment Three

[0087] Figure 3 A structure diagram of a concurrent file collection device according to Embodiment Three of the present application is provided. As shown in the figure, the device can include a collection file list obtaining module 310, a file collection task obtaining module 320, a file collection task sending module 330, and a collection file receiving module 340. Figure 3

[0088] The collection file list obtaining module 310 is configured to, when detecting that the start is in the starting state, obtain the master node competition result, and if it is determined according to the master node competition result that the competition for the master node succeeds, obtain the collection file list.

[0089] The file collection task obtaining module 320 is configured to obtain at least one file collection task according to the collection file list, and add each file collection task to the preset to-be-processed queue.

[0090] The file collection task sending module 330 is configured to obtain the current file collection task from the preset to-be-processed queue according to the collection task obtaining request sent by the slave node, and send the current file collection task to the slave node, so that the slave node performs file collection according to the current file collection task.

[0091] The collection file receiving module 340 is configured to receive the collection file corresponding to the current file collection task sent by the slave node.

[0092] ​The technical scheme of the embodiment of the application, when detecting that the starting state is started, acquires the master node competition result, if it is determined that the master node competition is successful according to the master node competition result, acquires the collection file list; then, according to the collection file list, acquires multiple file collection tasks, and adds each file collection task to a preset to-be-processed queue; secondly, according to the collection task acquisition request sent by the slave node, acquires the current file collection task from the preset to-be-processed queue, and sends the current file collection task to the slave node, and finally receives the collection file corresponding to the current file collection task sent by the slave node, through the competition of each node to become the master node, and through the master node to acquire and distribute the file collection task, and through each slave node to execute the file collection task, concurrent collection of files can be realized, load balancing of the file collection task can be realized, and the file collection efficiency can be improved.

[0093] Optionally, the collection file list acquisition module 310 comprises:

[0094] The first node information insertion request generation unit is configured to generate a first node information insertion request according to the node information of the current node, and send the first node information insertion request to the preset database.

[0095] The first master node competition result determination unit is configured to determine that the master node competition result is that the competition for the master node is successful, if it is detected that the execution success message corresponding to the first node information insertion request sent by the preset database is successful.

[0096] The second master node competition result determination unit is configured to determine that the master node competition result is that the competition for the master node is unsuccessful, if it is detected that the execution failure message corresponding to the first node information insertion request sent by the preset database is unsuccessful.

[0097] The preset database is configured to determine the node identity corresponding to each node according to the node information insertion request sent by each node, and send the corresponding execution result information to each node according to the node identity corresponding to each node.

[0098] Optionally, the concurrent file collection device further comprises:

[0099] The master node information acquisition module is configured to acquire the master node information and generate a collection task acquisition request, if it is determined that the competition for the master node is unsuccessful according to the master node competition result.

[0100] The file collection task receiving module is configured to send the collection task acquisition request to the master node according to the master node information, and receive the file collection task corresponding to the collection task acquisition request sent by the master node.

[0101] The file collection sending module is configured to acquire a collection file according to the file collection task, and send the collection file to the master node.

[0102] Optionally, the master node information obtaining module is configured to generate a master node information obtaining request, and send the master node information obtaining request to the preset database.

[0103] When the feedback information corresponding to the master node information obtaining request sent by the preset database is received, the master node information is obtained according to the feedback information.

[0104] Optionally, the concurrent file collection device further comprises:

[0105] The second node information insertion request generating module is configured to generate a second node information insertion request according to the node information of the current node, and send the second node information insertion request to the preset database if it is detected that there is a communication abnormality with the master node.

[0106] The node identity updating module is configured to update the node identity of the current node to the master node if it is detected that the execution success message corresponding to the second node information insertion request sent by the preset database is sent.

[0107] Optionally, the collection file list obtaining module 310 comprises:

[0108] The file collection device information obtaining unit is configured to obtain file collection device information according to a preset configuration file.

[0109] The file list obtaining request sending unit is configured to generate a file list obtaining request, and send the file list obtaining request to the file collection device according to the file collection device information.

[0110] The collection file list receiving unit is configured to receive the collection file list corresponding to the file list obtaining request sent by the file collection device.

[0111] Optionally, the file collection device information comprises at least one of a file collection device name, a cluster name to which the file collection device belongs, an Internet protocol address, and a port number.

[0112] The concurrent file collection device provided in the embodiment of the application can execute the concurrent file collection method provided in any embodiment of the application, and has the corresponding function modules and beneficial effects of the execution method.

[0113] It should be noted that, in the technical solution of the embodiment, the acquisition, storage and application of the user personal information comply with relevant laws and regulations, and do not violate public order and good customs.

[0114] Example Four

[0115] Figure 4 A structural diagram of an electronic device 40 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0116] As shown in Figure 4 The electronic device 40 includes at least one processor 41, and memory, such as read-only memory (ROM) 42, random access memory (RAM) 43, etc., communicatively connected to the at least one processor 41, where the memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 42 or loaded into the random access memory (RAM) 43 from the storage unit 48. Various programs and data required for the operation of the electronic device 40 can also be stored in the RAM 43. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0117] Various components in the electronic device 40 are connected to the I / O interface 45, including an input unit 46, such as a keyboard, a mouse, etc., an output unit 47, such as various types of displays, speakers, etc., a storage unit 48, such as a magnetic disk, an optical disk, etc., and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0118] The processor 41 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 41 performs various methods and processes described above, such as the concurrent file acquisition method.

[0119] In some embodiments, the concurrent collection method of a file can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 48. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 40 via, e.g., ROM 42 and / or communication unit 49. When the computer program is loaded onto RAM 43 and executed by processor 41, one or more steps of the concurrent collection method of a file as described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the concurrent collection method of a file by other means, e.g., with the aid of firmware.

[0120] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0121] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as part of a standalone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0122] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0124] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0125] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0126] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0127] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for concurrent file acquisition, characterized in that, include: When the startup state is detected, the master node competition result is obtained. If it is determined that the master node competition was successful based on the master node competition result, the list of collected files is obtained. Based on the list of collected files, obtain at least one file collection task and add each of the file collection tasks to a preset queue to be processed. Based on the acquisition task request sent by the slave node, the current file acquisition task is retrieved from the preset pending queue and sent to the slave node so that the slave node can perform file acquisition according to the current file acquisition task; Receive the file to be collected corresponding to the current file collection task sent by the slave node; Among them, obtaining the master node competition results includes: A first node information insertion request is generated based on the node information, and the first node information insertion request is sent to the preset database; If a success message corresponding to the first node information insertion request sent by the preset database is detected, then the result of the master node competition is determined to be a successful master node competition. If an execution failure message corresponding to the first node information insertion request sent by the preset database is detected, then the result of the master node competition is determined to be a failure to compete for the master node. The preset database is used to determine the node identity of each node based on the node information insertion request sent by each node, and to generate corresponding execution result information and send it to each node based on the node identity of each node.

2. The method according to claim 1, characterized in that, After obtaining the master node contention results, the following is also included: If the competition for the master node is determined to have failed based on the master node competition results, then the master node information is obtained and a data acquisition task request is generated. Based on the master node information, the acquisition task request is sent to the master node, and the file acquisition task corresponding to the acquisition task request sent by the master node is received. According to the file collection task, the collected file is obtained and sent to the master node.

3. The method according to claim 2, characterized in that, Obtain master node information, including: Generate a master node information retrieval request and send the master node information retrieval request to the preset database; When the feedback information corresponding to the master node information acquisition request sent by the preset database is received, the master node information is acquired according to the feedback information.

4. The method according to claim 2, characterized in that, After sending the collected file to the master node, the process also includes: If a communication anomaly is detected with the master node, a second node information insertion request is generated based on the node information and sent to the preset database. If a success message corresponding to the second node information insertion request sent by the preset database is detected, it is determined that the competition for the master node was successful, and the node identity corresponding to this node is updated to the master node.

5. The method according to claim 1, characterized in that, Get the list of collected files, including: According to the preset configuration file, obtain the file acquisition device information; Generate a file list retrieval request, and send the file list retrieval request to the file acquisition device according to the file acquisition device information; Receive the file list corresponding to the file list acquisition request sent by the file acquisition device.

6. The method according to claim 5, characterized in that, The file acquisition device information includes at least one of the following: file acquisition device name, cluster name to which the file acquisition device belongs, Internet Protocol address, and port number.

7. A concurrent file acquisition device, characterized in that, include: The file collection list acquisition module is used to acquire the master node competition result when the startup state is detected. If it is determined that the master node competition is successful based on the master node competition result, the file collection list is acquired. The file acquisition task acquisition module is used to acquire at least one file acquisition task based on the list of acquisition files, and add each of the file acquisition tasks to a preset pending queue. The file acquisition task sending module is used to retrieve the current file acquisition task from the preset pending queue according to the acquisition task acquisition request sent by the slave node, and send the current file acquisition task to the slave node so that the slave node can perform file acquisition according to the current file acquisition task. The file acquisition receiving module is used to receive the acquisition file corresponding to the current file acquisition task sent by the slave node; The file list acquisition module includes: The first node information insertion request generation unit is used to generate a first node information insertion request based on the node information and send the first node information insertion request to a preset database. The first master node contention result determination unit is used to determine the master node contention result as successful if a success message corresponding to the first node information insertion request sent by the preset database is detected. The second master node contention result determination unit is used to determine the master node contention result as a failure to compete for the master node if it detects an execution failure message corresponding to the first node information insertion request sent by the preset database. The preset database is used to determine the node identity of each node based on the node information insertion request sent by each node, and to generate corresponding execution result information and send it to each node based on the node identity of each node.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the concurrent acquisition method of the file according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the concurrent file acquisition method according to any one of claims 1-6.

Images