Peripheral file processing method and device, electronic equipment and storage medium

Abstract

The invention provides a peripheral file processing method and device, electronic equipment and a storage medium, and the method comprises the steps: scanning a target peripheral directory through a scanning node, and generating a scanning message corresponding to each target file in all target files obtained through scanning; based on the scanning message, index registration is conducted on all the target files through a plurality of registration nodes, the storage position of each target file is determined, the registration nodes are nodes deployed with index registration services, each registration node is used for conducting index registration on at least one target file, and the storage position of each target file is determined; each target file is subjected to index registration by only one registration node; and storing each target file according to the storage position of each target file. Through the file processing method and device, the problem that the efficiency is low due to the fact that files are accumulated and cannot be quickly processed in a file processing scheme for files issued by the periphery in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a peripheral file processing method and device, an electronic device and a storage medium. Background Art

[0002] In the process of the host system (such as an enterprise-level host system) sending peripheral files to the distributed system in batches, since the host migration is carried out in stages, there may be scenarios where the host system and the distributed system exist at the same time, which involves file interaction between the host system and the distributed system (the distributed system will use the files in the host system, and the host system will also use the files in the distributed system). Considering the possible instability of the distributed system nodes (for example, downtime), the cluster of the distributed system mounts the same NAS file directory (used to indicate the host system files required for distribution), and the files sent by the host are sent to this directory or a sub-file directory of the directory. After recording, the distributed batch processing system processes the files in the directory, which is mainly divided into four stages: the first stage: scanning the file directory (the directory where the host sends the file); the second stage: file index registration, recording the relationship between the file and the storage path according to the file name according to certain rules (such as date, certain characters in the file name), and recording this relationship; the third stage: putting the file in the specified location (NAS or HDFS), according to the relationship between the file and the path in the second stage, put the file in the corresponding path; the fourth stage: triggering corresponding operations based on the file. This stage mainly refers to the subsequent operations after the completion of the third stage, for example: notifying the system that the file is complete and can be used.

[0003] In the related art, a Shell script on the Linux system is used to periodically scan folders and files, and then index and register, move to a specified directory, and perform subsequent processing. This processing is a serial operation (i.e., one by one operation). However, although multiple nodes perform file scanning, since they all read folders, multiple nodes do repetitive tasks (i.e., process the same file). Therefore, only one node is actually working. It can be seen that the above-mentioned file processing solution for files sent from the periphery has the problem of file accumulation and inability to process quickly, which affects the operation of the overall batch processing job and leads to low efficiency.

[0004] Therefore, in the related art, there is a problem that the file processing solution for the files sent from the outside is that the files are piled up and cannot be processed quickly, which affects the operation of the overall batch processing job and thus has low efficiency. Summary of the invention

[0005] The present application provides a peripheral file processing method and device, an electronic device and a storage medium, so as to at least solve the problem in the related art that the file processing scheme for files sent from the periphery has file accumulation and cannot be processed quickly, which affects the operation of the overall batch processing job and thus has low efficiency.

[0006] According to one aspect of an embodiment of the present application, a peripheral file processing method is provided, including:

[0007] Scanning the target peripheral directory through a scanning node, generating a scanning message corresponding to each target file among all the target files obtained by the scan, wherein the scanning message is used to indicate that the corresponding target file has been scanned;

[0008] Based on the scan message, all the target files are indexed and registered through multiple registration nodes to determine the storage location of each target file, wherein the registration node is a node deployed with an index registration service, each of the registration nodes is used to index and register at least one of the target files, and each of the target files is indexed and registered by only one registration node;

[0009] Each target file is stored according to the storage location of each target file.

[0010] Optionally, as in the aforementioned method, the scanning of the target peripheral directory by the scanning node to generate a scanning message corresponding to each target file in all the target files obtained by the scanning includes:

[0011] Performing a scanning operation on the target peripheral directory through the scanning node, and determining all the target files scanned by the scanning operation;

[0012] Performing an identification operation on each target file, wherein the identification operation is used to identify each target file as having been scanned; and generating the scanning message corresponding to each target file based on the file information obtained by scanning each target file.

[0013] Optionally, as in the aforementioned method, before indexing and registering all the target files based on the scanning message and determining the storage location of each target file through at least one registration node, the method further includes:

[0014] Writing the scanned message into a message system;

[0015] The step of indexing and registering each target file based on the scan message and determining the storage location of each target file through at least one registration node includes:

[0016] The registration node consumes the scan message in the message system, determines the scanned target file, and removes the scan message in the message system;

[0017] The registration node indexes and registers the target file according to the file feature information of the target file, and determines the storage location where the target file needs to be stored.

[0018] Optionally, as in the aforementioned method, storing each target file according to the storage location of each target file includes:

[0019] Storing the target file in a storage location corresponding to the target file;

[0020] Delete the target file in the target peripheral directory.

[0021] Optionally, as in the aforementioned method, after scanning the target peripheral directory by the scanning node, the method further includes:

[0022] Obtaining a target record of the target file that has completed scanning, wherein each target file has a corresponding record;

[0023] The target record is written into the database.

[0024] Optionally, as in the aforementioned method, before scanning the target peripheral directory by the scanning node, the method further includes:

[0025] Among all candidate nodes, the first candidate node competing for the distributed lock is determined as the scanning node;

[0026] The distributed lock is allocated to the scanning node so that the target peripheral directory accepts scanning from the scanning node.

[0027] Optionally, as in the aforementioned method, after storing each target file according to the storage location of each target file, the method further includes:

[0028] Determining a processing rule for the target file;

[0029] In response to the target file being stored, a processing operation corresponding to the target file is performed according to the processing rule.

[0030] According to another aspect of the embodiment of the present application, there is also provided a peripheral file processing device, including:

[0031] A scanning module, used for scanning a target peripheral directory through a scanning node, and generating a scanning message corresponding to each target file among all target files obtained by the scanning, wherein the scanning message is used for indicating that the corresponding target file has been scanned;

[0032] An index registration module, configured to index and register all the target files based on the scan message through a plurality of registration nodes, and determine the storage location of each target file, wherein the registration node is a node on which an index registration service is deployed, each registration node is configured to index and register at least one target file, and each target file is indexed and registered by only one registration node;

[0033] The storage module is used to store each target file according to the storage location of each target file.

[0034] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps in any of the above embodiments by running the computer program stored in the memory.

[0035] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the method steps in any of the above embodiments when executed.

[0036] In an embodiment of the present application, by decoupling the scanning of target peripheral files and the determination of index registration of each target file, the scanning and index registration of target files do not need to be performed through one node; in addition, all target files are indexed and registered through multiple registration nodes to determine the storage location of each target file, so that all target files can be indexed and registered concurrently, thereby greatly improving the efficiency of index registration of target files, thereby effectively overcoming the technical problem in the related art that the file processing solution for files issued from the periphery has file accumulation and cannot be processed quickly, affecting the operation of the overall batch processing job and thus low efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 is a schematic diagram illustrating a process of an optional peripheral file processing method according to an embodiment of the present application;

[0040] Figure 2 is a flowchart of an optional peripheral file processing method according to another embodiment of the present application;

[0041] Figure 3 is a flowchart of an optional peripheral file processing method according to another embodiment of the present application;

[0042] Figure 4 It is a flowchart of an optional peripheral file processing method according to an application example of the present application;

[0043] Figure 5 is a structural block diagram of an optional peripheral file processing device according to an embodiment of the present application;

[0044] Figure 6 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] According to one aspect of an embodiment of the present application, a peripheral file processing method is provided. Optionally, in this embodiment, the above-mentioned peripheral file processing method can be applied to a hardware environment composed of a terminal and a server. The server is connected to the terminal via a network, and can be used to provide services (such as data processing services, data storage services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server.

[0048] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0049] The peripheral file processing method of the embodiment of the present application can be executed by a server, a terminal, or a server and a terminal together. The terminal can also execute the peripheral file processing method of the embodiment of the present application by a client installed thereon.

[0050] To move a large number of files in a file directory to a specified location, the traditional processing method is to read and process the files in the directory one by one. However, this solution has the disadvantages of slow processing speed and poor efficiency. Since multiple hosts are moving the same files, even if the number of hosts is increased, the processing efficiency is difficult to effectively improve, and the advantages of distributed systems cannot be fully utilized.

[0051] In order to overcome the above technical problems, taking the peripheral file processing method in this embodiment executed by the server as an example, Figure 1 A peripheral file processing method provided in an embodiment of the present application includes the following steps:

[0052] Step S101 , scanning a target peripheral directory through a scanning node, and generating a scanning message corresponding to each target file among all target files obtained by the scanning, wherein the scanning message is used to indicate that the corresponding target file has been scanned.

[0053] The peripheral file processing method in this embodiment can be applied to scenarios where it is necessary to store files sent from a host system, for example: scenarios where enterprise-level host systems are moved down to distributed systems in batches, and where other host systems are moved down to other types of systems. Unless there is a contradiction, the above-mentioned peripheral file processing method is also applicable.

[0054] After the host system has sent the file to the target peripheral directory, there are already multiple files in the target peripheral directory, and the files sent by the host system to the target peripheral directory are scattered.

[0055] The scanning node may be a node in a message subsystem of a distributed system, and the scanning node may scan a target peripheral directory to determine all subdirectories (if including subdirectories) under the target peripheral directory, and all files under the target peripheral directory.

[0056] After scanning the target peripheral directory once or performing a period of scanning (which may include multiple scans), the scanning node may scan and obtain one or more target files existing in the target peripheral directory.

[0057] After scanning the target file, a scanning message for the target file can be obtained. The scanning message can be used to remind the subsequent registration node for index registration to index and register the target file. The scanning message may include but is not limited to: 1. Flow records, that is, processing traces, used to indicate the stage and status information of the described file processing; 2. Sending messages to the message system (such as Redis, Kafka, etc.) to facilitate subsequent consumption of the message and processing, the corresponding target file name, file feature information, etc.

[0058] Step S102, based on the scan message, all target files are indexed and registered through multiple registration nodes to determine the storage location of each target file, wherein the registration node is a node deployed with an index registration service, each registration node is used to index and register at least one target file, and each target file is indexed and registered by only one registration node.

[0059] The registration node can determine the scanned target files based on the scanning message. After obtaining the scanning message of the target file, the target file can be indexed and registered based on the file name and / or file feature information of the target file in the scanning message to determine the storage location of each target file.

[0060] Furthermore, in this embodiment, the index registration of one target file is performed only through one registration node, and multiple registration nodes can concurrently perform index registration on multiple target files among all target files.

[0061] Index registration means putting scattered files in the periphery into designated locations according to certain rules.

[0062] In this embodiment, the efficiency is improved by decoupling the scanning operation and the index registration operation through the message system. The efficiency of the index registration stage can be improved by concurrent operations (including multiple programs performing index registration and batch registration of indexes for independent index systems).

[0063] Step S103: storing each target file according to the storage location of each target file.

[0064] For each target file, after the storage location of the target file is determined, the target file can be moved from the target peripheral directory to a storage location in a distributed system (eg, NAS or a distributed file system such as HDFS) for storage.

[0065] Through the method in this embodiment, the scanning of target peripheral files and the determination of index registration of each target file are decoupled, so that the scanning and index registration of target files do not need to be performed through one node; in addition, all target files are indexed and registered through multiple registration nodes to determine the storage location of each target file, so that all target files can be indexed and registered concurrently, thereby greatly improving the efficiency of index registration of target files, thereby effectively overcoming the technical problem that the file processing solution for files issued from the periphery in the related art has file accumulation and cannot be processed quickly, affecting the operation of the overall batch processing job and thus low efficiency.

[0066] As an optional embodiment, as in the aforementioned method, step S101 scans the target peripheral directory through a scanning node to generate a scanning message corresponding to each target file in all the scanned target files, including the following steps:

[0067] Step S201 : performing a scanning operation on a target peripheral directory through a scanning node, and determining all target files scanned by the scanning operation.

[0068] The scanning node performs a scanning operation on the target peripheral directory. One scanning operation can scan multiple target files and determine the information of each target file including but not limited to: the location of the target file in the target peripheral directory, the file name of the target file, and the file feature information of the target file.

[0069] Step S202: performing a marking operation on each target file, wherein the marking operation is used to mark each target file as having been scanned.

[0070] After the target files are determined, a marking operation for modifying the file name can be performed on each target file, for example, the file is renamed by adding a character "_" to the front of the file name. In addition, the marking operation can also be performed by adding other characters.

[0071] Step S203: Generate a scanning message corresponding to each target file based on the file information obtained by scanning each target file.

[0072] After the target file is scanned, the file information of the target file can be obtained, and the file information includes but is not limited to: the location of the target file in the target peripheral directory, the file name of the target file, and the file feature information of the target file.

[0073] Based on the file information, a scan message uniquely corresponding to the target file can be generated. The scan message can include all or part of the information in the file information, as long as it can uniquely point to the corresponding target file. After obtaining the scan message, the registration node performing index registration can consume the scan message to determine that the target file is included in the peripheral file system, thereby decoupling the file scan and index registration.

[0074] Optionally, if the latest scanning information is not obtained within a first preset time period (for example, 10s, 20s, 1min, etc.), the latest scanning node is re-determined, and the target peripheral directory is scanned through the latest scanning node to generate a scanning message corresponding to each target file in all target files scanned.

[0075] As an optional embodiment, as in the above method, before the step S102 is based on scanning messages and indexing and registering all target files through at least one registration node to determine the storage location of each target file, the method further includes the following steps:

[0076] Step P101, write the scanned message into the message system.

[0077] After the scan message is obtained, the scan message can be written into the message system, and all scan messages can be written into the message system in a concurrent manner.

[0078] The message system can be any of redis, MQ or kafka.

[0079] like Figure 2 As shown, the step S102 is based on scanning messages and indexing and registering each target file through at least one registration node to determine the storage location of each target file, including the following steps:

[0080] Step S301: The registration node consumes the scan message in the message system, determines the scanned target file, and removes the scan message in the message system.

[0081] For any registered node, the registered node can obtain certain data (i.e., one or more scanning messages) from the message system, and consume any scanning message in the certain data, and then determine the target file that has been scanned in the target peripheral directory. In this embodiment, the target file can be determined by scanning information such as the file name or file address in the message.

[0082] After determining the scanned target file, the scan message can be removed from the message system to prevent other registered nodes from repeatedly indexing and registering the same target file.

[0083] Step S302: The registration node indexes and registers the target file according to the file characteristic information of the target file, and determines the storage location where the target file needs to be stored.

[0084] After the target file is determined, the address where the target file needs to be stored can be determined based on the file feature information of the target file (including one or more of the following: file name, file type, etc.) and in accordance with preset indexing rules (i.e., limiting the location where each type of file feature information needs to be stored), and then the target file is indexed and registered based on the file feature information of the target file to determine the storage location where the target file needs to be stored.

[0085] Optionally, if the storage location where the specified file (any node among all target nodes) needs to be stored is not determined within a second preset time period (for example, 10s, 20s, 1min, etc.) after receiving the scanning message of the target file, the specified file is re-indexed and registered through a new registration node to determine the storage location of the specified file.

[0086] Furthermore, 20 may be inserted into the pipeline state of the target file in the database as a second record that the target file has completed index registration. As an optional embodiment, as in the aforementioned method, the step S103 stores each target file according to the storage location of each target file, including the following steps:

[0087] Step S401, storing the target file to a storage location corresponding to the target file;

[0088] Step S402, deleting the target file in the target peripheral directory.

[0089] After completing the index registration, the storage location corresponding to the target file has been determined. Therefore, the target file in the target peripheral directory can be stored in the storage location corresponding to the target file, for example, the target file is placed in NAS or a distributed file system such as HDFS.

[0090] After the target file is stored in the corresponding storage location, the target file in the target peripheral directory is also deleted.

[0091] Furthermore, in the database, in the information corresponding to the target file, completion information indicating that the target file has been stored in the corresponding storage location and deleted from the target peripheral directory may be written.

[0092] Optionally, if the target file is not stored in the storage location within a third preset time period (for example, 10s, 20s, 1min, etc.) after the storage location of the target file has been determined, the storage operation on the target file is restarted to perform the operation of storing the target file in the storage location corresponding to the target file again through the storage operation.

[0093] Furthermore, 30 may be inserted into the pipeline status of the target file in the database as a third record that the storage process of the target file has been completed.

[0094] Through the method in this embodiment, the storage of files can be decoupled from index registration, and deleting the target files in the target peripheral directory can free up new storage space in time, thereby effectively improving the utilization rate of the storage space.

[0095] like Figure 3 As shown, as an optional embodiment, as in the above method, after the target peripheral directory is scanned by the scanning node in step S101, the method further includes the following steps:

[0096] Step S501, obtaining target records of target files that have been scanned, wherein each target file has a corresponding record.

[0097] After the target file is scanned, in order to prevent other scanning nodes or the current scanning node from scanning the target file again, and to clearly inform the scanning node which stage the target peripheral directory has been scanned to, a target record of the target file being scanned can be obtained. For example, 10 is inserted into the pipeline state of the target file as the first record of the target file being scanned.

[0098] Step S502: write the target record into the database.

[0099] After obtaining the target record, in order to facilitate other scanning nodes to obtain the target record, the target record can be written into the database. All nodes in the distributed system can read and write the information in the database.

[0100] Through the method in this embodiment, all scanning nodes that are performing scanning can clearly know which stage the target peripheral directory has been scanned to, and can effectively avoid the occurrence of repeated scanning.

[0101] As an optional embodiment, as in the above method, before the step S101 scans the target peripheral directory by the scanning node, the method further includes the following steps:

[0102] Step S601: among all candidate nodes, determine the first candidate node that competes for acquiring the distributed lock as a scanning node.

[0103] In this embodiment, all candidate nodes can compete to become scanning nodes by competing to obtain the distributed lock of redis. The candidate node can be any node in the distributed system.

[0104] Furthermore, in this embodiment, the first candidate node competing for acquiring the distributed lock is determined as the scanning node.

[0105] Step S602: Allocate a distributed lock to a scanning node so that the target peripheral directory can be scanned by the scanning node.

[0106] Optionally, after the first scanning node competing for the distributed lock is determined, the distributed lock can be allocated to the scanning node, so that the target peripheral directory only accepts scans from the scanning node.

[0107] Furthermore, when a scanning node crashes (for example, the target peripheral directory has not been scanned for a long time (longer than a preset time (for example, 10s, 20s, etc.)), or a new record has not been written to the database for a long time), the distributed lock of the scanning node can be released, so that it can accept competition from other candidate nodes and re-determine the latest scanning node that can operate normally.

[0108] As an optional embodiment, as in the above method, after storing each target file according to the storage location of each target file in step S103, the method further includes the following steps:

[0109] Step S701, determining a processing rule for a target file.

[0110] After different files are stored, different subsequent processing needs to be performed.

[0111] The processing rules may be the interaction between the file and the original host system after the file is stored, such as notifying the host system that the file has arrived and that the next operation can be performed.

[0112] In this embodiment, the processing rules corresponding to each file can be obtained in advance, and then the target file amount processing rules can be matched.

[0113] Step S702: In response to the target file being stored, a processing operation corresponding to the target file is executed according to the processing rule.

[0114] After the target file has been stored, the processing operation corresponding to the target file can be executed according to the processing rules. For example, if it is a conditional trigger, it is triggered; if it is a job flow trigger, it is triggered, for example: telling the next job of the host system that the current target file has been uploaded, so as to inform the host system that subsequent actions can be taken; if it is a simulation notification file, it is inserted into redis.

[0115] Furthermore, a number 40 may be inserted into the pipeline status of the target file in the database as a fourth record that the target file has completed subsequent file processing.

[0116] like Figure 4 As shown, an application example of applying any of the above embodiments is provided:

[0117] The process of moving a large number of files in a file directory to a specified location is processed in stages, and the entire processing process is divided into four stages, splitting long steps into short steps, increasing concurrency, and processing quickly:

[0118] (1) The first stage is file scanning: Since the outermost directory to be scanned (i.e., the aforementioned target peripheral directory) is the same, if multiple nodes perform scanning at the same time, repeated operations will be wasted. Therefore, by using the distributed lock of redis, it is ensured that only one node in the distributed system obtains the distributed lock at the same time. This ensures that only one program can scan the outermost directory at the same time, reducing conflicts between each other. The scanned message is stored in the database, and the file name of the scanned file in the outermost directory is modified to indicate that the file has been processed and does not need to be processed again. The concurrent file is written to the message system (which can be redis, MQ or kafka). The split of this stage is mainly to make full use of the system characteristics and quickly scan. Other operations are handed over to other nodes, so that fast scanning can be achieved.

[0119] (2) The second stage is index registration: Index registration is to determine the rules for moving files to the specified location and record them, so that subsequent queries and searches can be performed based on the file name. Each node in the distributed system can obtain certain data (i.e., a certain number of scan messages, and remove the scan messages from the message system after obtaining the scan messages) from the message system for index registration. Since the index operation exists on other systems outside the file scanning system, the interaction between systems will increase network time consumption. This method uses batch operations for index registration, which can reduce network overhead and be more efficient.

[0120] (3) The third stage is to put the file in the designated location: After the second stage is completed, the third stage begins, and the file is put in the designated location (NAS or distributed file system such as HDFS) according to the designated location generated by the index;

[0121] (4) The fourth stage is the subsequent processing of files: After the third stage is completed, the fourth stage begins. The fourth stage mainly performs special processing on files according to the rules. Special processing refers to the interaction between the original system after the file operation is completed, such as notifying the system that the file has arrived and can proceed to the next step of operation;

[0122] 2. Add inspection actions:

[0123] Retry the records that have not been processed for a long time at each stage, which can completely avoid manual intervention.

[0124] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0126] According to another aspect of an embodiment of the present application, a peripheral file processing device for implementing the above peripheral file processing method is also provided. Figure 5 is a structural block diagram of an optional peripheral file processing device according to an embodiment of the present application, such as Figure 5 As shown, the device may include:

[0127] Scanning module 1, used for scanning the target peripheral directory through the scanning node, and generating a scanning message corresponding to each target file in all the target files obtained by the scanning, wherein the scanning message is used to indicate that the corresponding target file has been scanned;

[0128] Index registration module 2, used to index and register all target files through multiple registration nodes based on the scan message, and determine the storage location of each target file, wherein the registration node is a node deployed with an index registration service, each registration node is used to index and register at least one target file, and each target file is indexed and registered by only one registration node;

[0129] The storage module 3 is used to store each target file according to the storage location of each target file.

[0130] It should be noted that the scanning module 1 in this embodiment can be used to execute the above step S101, the index registration module 2 in this embodiment can be used to execute the above step S102, and the storage module 3 in this embodiment can be used to execute the above step S103.

[0131] The device in this embodiment, in addition to the above modules, may also include a module for executing any method in any of the above-mentioned peripheral file processing method embodiments.

[0132] It should be noted that the above modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can be run in the implementation of Figure 1 The hardware environment of the method shown may be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0133] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned peripheral file processing method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0134] According to another embodiment of the present application, there is also provided an electronic device, including: Figure 6 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0135] Memory 1503, used for storing computer programs;

[0136] The processor 1501 is used to implement the following steps when executing the program stored in the memory 1503:

[0137] Step S101 , scanning a target peripheral directory through a scanning node, and generating a scanning message corresponding to each target file among all target files obtained by the scanning, wherein the scanning message is used to indicate that the corresponding target file has been scanned.

[0138] Step S102, based on the scan message, all target files are indexed and registered through multiple registration nodes to determine the storage location of each target file, wherein the registration node is a node deployed with an index registration service, each registration node is used to index and register at least one target file, and each target file is indexed and registered by only one registration node.

[0139] Step S103: storing each target file according to the storage location of each target file.

[0140] Optionally, in this embodiment, the above-mentioned communication bus can be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0141] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0142] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0143] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0144] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0145] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0146] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0147] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A peripheral file processing method, characterized in that: include: Scanning the target peripheral directory through a scanning node, generating a scanning message corresponding to each target file among all the target files obtained by the scan, wherein the scanning message is used to indicate that the corresponding target file has been scanned; Based on the scan message, all the target files are indexed and registered through multiple registration nodes to determine the storage location of each target file, wherein the registration node is a node deployed with an index registration service, each of the registration nodes is used to index and register at least one of the target files, and each of the target files is indexed and registered by only one registration node; Each target file is stored according to the storage location of each target file.

2. The method according to claim 1, characterized in that The scanning node scans the target peripheral directory and generates a scanning message corresponding to each target file in all the target files obtained by the scanning, including: Performing a scanning operation on the target peripheral directory through the scanning node, and determining all the target files scanned by the scanning operation; Performing an identification operation on each target file, wherein the identification operation is used to identify each target file as having been scanned; and generating the scanning message corresponding to each target file based on the file information obtained by scanning each target file.

3. The method according to claim 1, characterized in that Before indexing and registering all the target files based on the scanning message and determining the storage location of each target file through at least one registration node, the method further includes: Writing the scanned message into a message system; The step of indexing and registering each target file based on the scan message and determining the storage location of each target file through at least one registration node includes: The registration node consumes the scan message in the message system, determines the scanned target file, and removes the scan message in the message system; The registration node indexes and registers the target file according to the file feature information of the target file, and determines the storage location where the target file needs to be stored.

4. The method according to claim 1, characterized in that: The storing of each target file according to the storage location of each target file includes: Storing the target file in a storage location corresponding to the target file; Delete the target file in the target peripheral directory.

5. The method according to claim 1, characterized in that After scanning the target peripheral directory by the scanning node, the method further includes: Obtaining a target record of the target file that has completed scanning, wherein each target file has a corresponding record; The target record is written into the database.

6. The method according to claim 1, characterized in that Before scanning the target peripheral directory by the scanning node, the method further includes: Among all candidate nodes, the first candidate node competing for the distributed lock is determined as the scanning node; The distributed lock is allocated to the scanning node so that the target peripheral directory accepts scanning from the scanning node.

7. The method according to any one of claims 1 to 6, characterized in that After storing each target file according to the storage location of each target file, the method further includes: Determining a processing rule for the target file; In response to the target file being stored, a processing operation corresponding to the target file is performed according to the processing rule.

8. A peripheral file processing device, characterized in that: include: A scanning module, used for scanning the target peripheral directory through a scanning node, and generating a scanning message corresponding to each target file among all the target files obtained by the scanning, wherein the scanning message is used for indicating that the corresponding target file has been scanned; An index registration module, configured to index and register all the target files based on the scan message through a plurality of registration nodes, and determine the storage location of each target file, wherein the registration node is a node on which an index registration service is deployed, each registration node is configured to index and register at least one target file, and each target file is indexed and registered by only one registration node; The storage module is used to store each target file according to the storage location of each target file.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is configured to execute the method steps of any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method steps described in any one of claims 1 to 7 when run.

Images