Directory file processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a directory file processing method and device, electronic equipment and a storage medium, and relates to the Internet and big data technology, and the method comprises the steps that the number of files in a target directory is obtained, and the target directory is any file directory which is provided for a target user by a platform and used for storing files; determining a task demand number according to the file number and a file processing threshold value of the task; determining an initial task number according to the total task resource number and the total directory number; determining the state of a target directory according to the task demand number and the initial task number; distributing the tasks according to the state of the target directory; and executing the allocated task according to the allocation result, and transmitting the file of the target user in the target directory to a corresponding directory of the opposite-end user. The number of tasks executing file transmission in the target directory is matched with the number of files in the target directory, dynamic adjustment of the number of tasks is achieved, and the file transmission efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology and big data technology, and in particular to a directory file processing method, device, electronic device and storage medium. Background Art

[0002] With the development of the Internet and big data technology, an open platform has emerged to provide data transmission functions for application parties and service parties, thereby realizing data interaction between application parties and service parties.

[0003] When the open platform is transmitting data, it scans the directory regularly and configures a fixed number of tasks for each directory to transfer the files in the directory.

[0004] However, when the files stored in the directory are concentrated and there are a large number of files to be transferred in the directory, using a fixed number of tasks cannot transfer the files in the directory in time. How to improve the transfer efficiency of files in the directory becomes an urgent problem to be solved. Summary of the invention

[0005] The present invention provides a directory file processing method, device, electronic device and storage medium to solve the problem of low efficiency of file transmission between an application party and a service party in the current open platform.

[0006] According to one aspect of the present invention, a directory file processing method is provided, comprising:

[0007] Get the number of files in the target directory, where the target directory is any file directory provided by the platform for the target user to store files;

[0008] Determine the number of task requirements based on the number of files and the file processing threshold of the task; determine the initial number of tasks based on the total number of task resources and the total number of directories;

[0009] Determine the state of the target directory according to the required number of tasks and the initial number of tasks;

[0010] The tasks are allocated according to the status of the target directory; the allocated tasks are executed according to the allocation result, and the files of the target user in the target directory are transferred to the corresponding directory of the opposite end user.

[0011] According to another aspect of the present invention, there is provided a directory file processing device, comprising:

[0012] A file quantity acquisition module, used to acquire the number of files in a target directory, where the target directory is any file directory provided by the platform for the target user to store files;

[0013] A task parameter determination module is used to determine the number of task requirements according to the number of files and the file processing threshold of the task; and to determine the initial number of tasks according to the total number of task resources and the total number of directories;

[0014] A state determination module, used for determining the state of the target directory according to the required number of tasks and the initial number of tasks;

[0015] The task allocation module is used to allocate tasks according to the status of the target directory; execute the allocated tasks according to the allocation results, and transfer the files of the target user in the target directory to the corresponding directory of the opposite end user.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the directory file processing method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the directory file processing method described in any embodiment of the present invention when executed.

[0021] The technical solution provided by the embodiment of the present invention obtains the number of files in the target directory, wherein the target directory is any file directory provided by the platform for the target user to store files; determines the number of task requirements according to the number of files and the file processing threshold of the task; determines the initial number of tasks according to the total number of task resources and the total number of directories; determines the state of the target directory according to the number of task requirements and the initial number of tasks; allocates tasks according to the state of the target directory; executes the allocated tasks according to the allocation results, and transfers the files of the target user in the target directory to the corresponding directory of the opposite end user. Compared with the current use of a fixed number of tasks for each directory for data transmission, the technical solution provided by the embodiment of the present invention can determine the number of task requirements according to the number of files and the file processing threshold of the task, and determine the initial number of tasks according to the total number of task resources and the total number of directories. Determine the state of the target directory according to the number of task requirements and the initial number of tasks. The state represents the relationship between the initial number of tasks allocated to the target directory and the number of task requirements required by the target directory after the system resources are evenly allocated. According to the state of the target directory, the tasks matching the target directory are allocated, so that the number of tasks for executing file transfer in the target directory matches the number of files in the target directory, and the dynamic adjustment of the number of tasks is realized. Since the target directory is any directory, the number of tasks finally executed can be adjusted for any directory provided by the platform in the above manner to improve the file transfer efficiency.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a schematic diagram of a system architecture applicable to a directory file processing method provided by an embodiment of the present invention;

[0025] Figure 2 It is a flowchart of a directory file processing method provided by an embodiment of the present invention;

[0026] Figure 3 It is a flowchart of another directory file processing method provided by an embodiment of the present invention;

[0027] Figure 4It is a structural schematic diagram of a directory file processing device provided by an embodiment of the present invention;

[0028] Figure 5 It is a structural schematic diagram of an electronic device for implementing the directory file processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] With the development of the Internet and big data technology, an open platform has emerged to provide data transmission functions for application parties and service parties, thereby realizing data interaction between application parties and service parties.

[0032] When the open platform is transmitting data, it scans the directory regularly and configures a fixed number of tasks for each directory to transfer the files in the directory.

[0033] However, when the files stored in the directory are concentrated and there are a large number of files to be transferred in the directory, using a fixed number of tasks cannot transfer the files in the directory in time. How to improve the transfer efficiency of files in the directory becomes an urgent problem to be solved.

[0034] Figure 1A schematic diagram of a system architecture applicable to the directory file processing method provided in an embodiment of the present invention includes an application party device, a service party device and an open platform. The open platform configures an application party storage space for the application party and a service party storage space for the service party. The platform can provide data transmission, security verification, permission verification and other functions for multiple application parties and multiple service parties. An application party can have one or more service parties as end users, and a service party can also have one or more application parties as end users. Exemplarily, the application party can be an organization that calls the open platform application programming interface (Application Programming Interface, API) service. The service party can be a third-party organization that develops and designs the application program interface according to requirements, registers the API service on the open platform, and publishes the API service.

[0035] The application party writes a data file to a file directory in the application storage space of the open platform or to a file directory in multiple file directories allocated to the application party through the application party device. The data file is transmitted to the corresponding directory of the opposite user through the technical solution provided by the embodiment of the present invention.

[0036] The embodiment of the present invention can be applied to the application scenarios where file reconciliation exists in bank-to-bank, bank-to-enterprise, micro-gold, etc. The application party and the service party are the two ends in the above scenarios.

[0037] Figure 2 This is a flowchart of a directory file processing method provided by an embodiment of the present invention. This embodiment is applicable to the case where an open platform transmits data between an application party and a service party. The method can be executed by a directory file processing device. The directory file processing device can be implemented in the form of hardware and / or software. The directory file processing device can be configured in electronic devices such as personal computers and servers. Figure 1 As shown, the method includes:

[0038] Step S101: Obtain the number of files in a target directory, where the target directory is any file directory provided by the platform for a target user to store files.

[0039] Optionally, the target user is an application party, and the peer user is a service party. Optionally, or, the target user is a service party, and the peer user is an application party.

[0040] The target user can send the file to the target directory of the open platform through a secure file transfer protocol. The secure file transfer protocol can be a secure file transfer protocol (Secure File Transfer Protocol, SFTP).

[0041] Optionally, get the number of files in the target directory, which can be implemented as:

[0042] Monitor the file status in the target directory; count the number of files according to the monitoring result to obtain the number of files.

[0043] The read and write status of the files in the target directory can be obtained in real time by monitoring. The number of files in the target directory is counted according to the read and write status. When a file is written, the count result is increased by one. When a file is deleted, the count result is decreased by one. If a file is transferred from the target directory to the corresponding directory of the other end, the task that transfers the file deletes the file from the target directory.

[0044] The above implementation method obtains the number of files in the target directory in real time by monitoring, so that the number of files is real-time and effective, improving the timeliness of the value, thereby improving the accuracy of the task requirement number calculated in subsequent steps and improving the accuracy of task allocation.

[0045] Step S102: determining the required number of tasks according to the number of files and the file processing threshold of the task; and determining the initial number of tasks according to the total number of task resources and the total number of directories.

[0046] The total number of directories may be the number of all directories provided by the open platform for target users.

[0047] Optionally, determining the required number of tasks according to the number of files and the file processing threshold of the task may be implemented as follows:

[0048] The task requirement number is calculated based on the number of files and the file processing threshold of the task, and the task requirement number represents the number of tasks required for all files contained in the transmission target directory.

[0049] Exemplarily, the number of files is divided by the file processing threshold of the task to obtain the task requirement number.

[0050] Optionally, the initial number of tasks is determined based on the total number of task resources and the total number of directories, which can be implemented as follows:

[0051] The initial number of tasks is calculated from the total number of task resources and the total number of directories, and the initial number of tasks represents the number of tasks initially allocated to the target directory.

[0052] Exemplarily, the total number of task resources is divided by the total number of directories to obtain the initial number of tasks.

[0053] It should be noted that if the number of task requirements is not an integer, it is rounded up to obtain an integer number of task requirements. For example, the calculated number of task requirements is 6.34, which is rounded up to 7.

[0054] If the initial number of tasks is not an integer, it is rounded up to an integer. For example, if the calculated initial number of tasks is 5.6, it is rounded up to 6.

[0055] The above implementation method can accurately calculate the number of task requirements and the initial number of tasks, so that the status of the target directory can be judged more accurately later, thereby improving the accuracy of task allocation.

[0056] Furthermore, before calculating the initial number of tasks from the total number of task resources and the total number of directories, the following is further included:

[0057] The total number of task resources is determined according to the number of processing nodes and the allocatable threads of each processing node, and the total number of task resources represents the number of all tasks that can be provided by the platform.

[0058] The total number of task resources is obtained by multiplying the number of processing nodes by the allocatable threads of each processing node. Each thread is used to execute one task. Optionally, tasks and threads are carried one-to-one, and the number of allocatable threads provided by all processing nodes is the total number of task resources.

[0059] The above implementation can accurately determine the total number of task resources according to the number of processing nodes and the assignable threads of each processing node, thereby improving the accuracy of the initial number of tasks calculated in subsequent steps and improving the accuracy of task allocation.

[0060] Step S103: determining the state of the target directory according to the required number of tasks and the initial number of tasks.

[0061] If the number of tasks required by the target directory is less than the initial number of tasks in the directory, it means that the target directory is idle and the idle tasks can be assigned to other directories. If the number of tasks required by the target directory is greater than the initial number of tasks in the directory, it means that the directory is not idle.

[0062] For example, it is assumed that the total number of task resources is the number of nodes multiplied by the CPU allocatable threads = 3*16 = 48. The application has 2 directories in total. Each directory has 50 files, and the file processing threshold of the task is 10.

[0063] The calculation process of the required number of tasks is 50 / 10=5, which means that 5 tasks are needed to complete the transfer of 50 files.

[0064] The calculation process of the initial number of tasks is 48 / 2=24, which means that each directory can be assigned 24 tasks.

[0065] In the above example, the required number of tasks is less than the initial number of tasks, and the status of the target directory is idle.

[0066] For example, assume that the total number of task resources is the number of nodes multiplied by the CPU allocatable threads = 3*16 = 48. The application has 2 directories in total. Each directory has 1000 files, and the file processing threshold of the task is 10.

[0067] The calculation process of the required number of tasks is 1000 / 10=100, which means that 100 tasks are needed to complete the transfer of 1000 files.

[0068] The calculation process of the initial number of tasks is 48 / 2=24, which means that each directory can be assigned 24 tasks.

[0069] In the above example, the number of required tasks is greater than the initial number of tasks, and the status of the target directory is non-idle.

[0070] S104, assigning tasks according to the status of the target directory; executing the assigned tasks according to the assignment results, and transferring the files of the target user in the target directory to the corresponding directory of the peer user.

[0071] Tasks can be assigned according to the state of the target directory. The load balancing of tasks can be performed according to the idle tasks corresponding to the directory. On the basis of meeting the needs of the directory itself, the idle tasks can be assigned to large directories to improve the transmission efficiency of large directories. After the above load balancing, the assignment result is obtained, and the thread is started to execute the task according to the assignment result. The appropriate execution unit can be selected according to different operating systems. For example, the thread is used as the execution unit, and the thread corresponds to the task one by one.

[0072] Optionally, tasks are assigned based on the status of the target directory, which can be implemented as follows:

[0073] Determining whether the target directory is a large directory according to the number of files;

[0074] If the target directory is a large directory, tasks are allocated to the target directory according to the initial task number, and when other directories are idle, idle tasks of other directories are allocated to the target directory, wherein the other directories are directories other than the target directory.

[0075] If the target directory is not a large directory and is idle, tasks are assigned to the target directory according to the task requirement number, idle tasks are determined according to the difference between the initial task number and the task requirement number, and the idle tasks are assigned to large directories in other directories.

[0076] If the target directory is not a large directory and the target directory is in a non-idle state, tasks are allocated to the target directory according to the initial number of tasks.

[0077] For example, assume that there are directories a, b, and c, where a is a large directory, and b and c are not large directories. The required number of tasks for directory b is 5, and the initial number of tasks is 8, so there are 3 idle tasks in directory b. The required number of tasks for directory c is 6, and the initial number of tasks is 8, so there are 2 idle tasks in directory c. The required number of tasks for directory a is 15, and the initial number of tasks is 8, so there are no idle tasks.

[0078] When directory b is the target directory, directory a and directory c are other directories, and directory b allocates 3 idle tasks to directory a as a large directory, so that directory a is allocated 8+3=11 tasks. On this basis, when directory c is the target directory, directory a and directory b are other directories, and directory c allocates 2 idle tasks to directory a as a large directory, so that directory a is allocated 11+2=13 tasks.

[0079] The above implementation method can allocate idle tasks in other directories to the large directory after determining the large directory, thereby ensuring that idle tasks are added to the large directory after the required tasks are allocated to the idle directories that are not large directories, thereby improving the data transmission efficiency of the large directory.

[0080] Optionally, determining whether the target directory is a large directory according to the number of files may be implemented as follows:

[0081] If the number of files is greater than a preset multiple of the file processing threshold of the task, it is determined that the target directory is a large directory.

[0082] Alternatively, if the number of files in the target directory is greater than the file processing threshold of the task after a batch of files are transferred, the target directory is determined to be a large directory. Otherwise, the target directory is not a large directory.

[0083] Alternatively, if the number of files is greater than the file processing threshold of the task, the number of files is less than a preset multiple of the file processing threshold of the task, and the target directory is a directory with a time limit, then the target directory is determined to be a large directory. For example, a transfer business is required to be completed within 30 minutes, which is a case with a time limit, so the target directory storing the business is determined to be a large directory. Otherwise, the target directory is not a large directory.

[0084] The above implementation can determine whether the target directory is a large directory in three ways, thereby improving the accuracy of determining the large directory.

[0085] The transfer of the target user's files in the target directory to the corresponding directory of the peer user can be implemented in the following manner:

[0086] Calculate the hash value of the target file's file name; determine the file name modulus according to the hash value and the modulus threshold; if the file name modulus matches the task modulus, transfer the target user's file in the target directory to the corresponding directory of the peer user.

[0087] In a distributed environment, task scheduling is random. To avoid duplicate file processing when multiple tasks are generated in a directory, different tasks process the same file. When a task is issued, it carries the total number of tasks and the task modulus. Different tasks have the same total number of tasks but different task moduli. Before a task processes a file, the file name is converted into a hash value, and the hash value is divided by the total number of tasks to get the file name modulus. If the file name modulus is the same as the task modulus, the file can be processed. If they are different, the next file will be skipped. In this way, when different tasks process the same file, only one task can process the file, achieving file dislocation processing and task isolation.

[0088] The above implementation method can enable the task to identify matching files for transmission through the file name modifier and the task modifier. Different tasks can transmit files with different file name modifiers, thereby improving the refinement of transmission and the transmission efficiency.

[0089] Furthermore, when the number of callable threads of the processing node is 0 and there are no idle tasks in the directories in the idle state, it means that the threads are fully loaded and all threads are involved in data transmission. At this time, the processor unit needs to be expanded to increase the number of callable threads.

[0090] The directory file processing method of the embodiment of the present invention obtains the number of files in the target directory, wherein the target directory is any file directory for storing files provided by the platform for the target user; determines the number of task requirements according to the number of files and the file processing threshold of the task; determines the initial number of tasks according to the total number of task resources and the total number of directories; determines the state of the target directory according to the number of task requirements and the initial number of tasks; allocates tasks according to the state of the target directory; executes the allocated tasks according to the allocation results, and transfers the files of the target user in the target directory to the corresponding directory of the opposite end user. Compared with the current use of a fixed number of tasks for each directory for data transmission, the directory file processing method provided by the embodiment of the present invention can determine the number of task requirements according to the number of files and the file processing threshold of the task, and determine the initial number of tasks according to the total number of task resources and the total number of directories. Determine the state of the target directory according to the number of task requirements and the initial number of tasks. The state represents the relationship between the initial number of tasks allocated to the target directory and the number of task requirements required by the target directory after the system resources are evenly allocated. According to the state of the target directory, the tasks matching the target directory are allocated, so that the number of tasks for executing file transfer in the target directory matches the number of files in the target directory, and the dynamic adjustment of the number of tasks is realized. Since the target directory is any directory, the number of tasks finally executed can be adjusted for any directory provided by the platform in the above manner to improve the file transfer efficiency.

[0091] Figure 3 1 is a flow chart of a directory file processing method provided by an embodiment of the present invention, the method comprising:

[0092] Step S201: monitor the file status in the target directory; count the number of files according to the monitoring result to obtain the number of files.

[0093] The target directory is any file directory provided by the platform for the target user to store files.

[0094] Step S202, determine the total number of task resources according to the number of processing nodes and the allocatable threads of each processing node; divide the number of files by the file processing threshold of the task to obtain the number of task requirements, and divide the total number of task resources by the total number of directories to obtain the initial number of tasks.

[0095] The total number of task resources indicates the number of all tasks that the platform can provide. The required number of tasks indicates the number of tasks required to transfer all files contained in the target directory. The initial number of tasks indicates the number of tasks initially allocated to the target directory.

[0096] Step S203: Determine whether the target directory is a large directory based on the number of files.

[0097] Optionally, if the number of files is greater than a preset multiple of the file processing threshold of the task, the target directory is determined to be a large directory. Alternatively, if after a batch of files are transferred, the number of files in the target directory is greater than the file processing threshold of the task, the target directory is determined to be a large directory. Alternatively, if the number of files is greater than the file processing threshold of the task, the number of files is less than a preset multiple of the file processing threshold of the task, and the target directory is a directory with a time limit, the target directory is determined to be a large directory.

[0098] Step S204: if the target directory is a large directory, tasks are assigned to the target directory according to the initial number of tasks, and when other directories are idle, idle tasks of other directories are assigned to the target directory, wherein the other directories are directories other than the target directory.

[0099] Step S205: If the target directory is not a large directory and the target directory is in an idle state, tasks are allocated to the target directory according to the task requirement number, idle tasks are determined according to the difference between the initial task number and the task requirement number, and the idle tasks are allocated to large directories in other directories. If the target directory is not a large directory and the target directory is in a non-idle state, tasks are allocated to the target directory according to the initial task number.

[0100] Step S206, calculate the hash value of the target file's file name; determine the file name modulus according to the hash value and the modulus threshold. If the file name modulus matches the task modulus, transfer the target user's file in the target directory to the corresponding directory of the peer user.

[0101] Figure 4 1 is a schematic diagram of a schematic diagram of a directory file processing device provided by an embodiment of the present invention, and the device is suitable for transmitting data between an open platform and an application and a service. Figure 4 As shown, the device includes: a file quantity acquisition module 31, a task parameter determination module 32, a state determination module 33 and a task allocation module 34.

[0102] A file quantity acquisition module 31, used to acquire the number of files in a target directory, where the target directory is any file directory provided by the platform for the target user to store files;

[0103] The task parameter determination module 32 is used to determine the number of task requirements according to the number of files and the file processing threshold of the task; and to determine the initial number of tasks according to the total number of task resources and the total number of directories;

[0104] A state determination module 33, used to determine the state of the target directory according to the required number of tasks and the initial number of tasks;

[0105] The task allocation module 34 is used to allocate tasks according to the status of the target directory; execute the allocated tasks according to the allocation results, and transfer the files of the target user in the target directory to the corresponding directory of the opposite user.

[0106] Based on the above embodiment, optionally, the file quantity acquisition module 31 is used to:

[0107] Monitor the file status in the target directory;

[0108] The number of files is counted according to the monitoring result to obtain the number of files.

[0109] Based on the above embodiment, optionally, the task parameter determination module 32 is used to:

[0110] The task requirement number is calculated based on the number of files and the file processing threshold of the task, and the task requirement number represents the number of tasks required to transfer all the files contained in the target directory;

[0111] The initial number of tasks is calculated from the total number of task resources and the total number of directories, and the initial number of tasks represents the number of tasks initially allocated to the target directory.

[0112] Based on the above embodiment, optionally, the task parameter determination module 32 is further used to:

[0113] Before calculating the initial number of tasks from the total number of task resources and the total number of directories, the total number of task resources is determined according to the number of processing nodes and the allocatable threads of each processing node, and the total number of task resources represents the number of all tasks that the platform can provide.

[0114] Based on the above embodiment, optionally, the task allocation module 34 is used to:

[0115] Determining whether the target directory is a large directory according to the number of files;

[0116] If the target directory is a large directory, tasks are allocated to the target directory according to the initial number of tasks, and when other directories are idle, idle tasks of other directories are allocated to the target directory, wherein the other directories are directories other than the target directory;

[0117] If the target directory is not a large directory and the target directory is in an idle state, tasks are allocated to the target directory according to the task requirement number, idle tasks are determined according to the difference between the initial task number and the task requirement number, and the idle tasks are allocated to large directories in other directories;

[0118] If the target directory is not a large directory and the target directory is in a non-idle state, tasks are allocated to the target directory according to the initial number of tasks.

[0119] Based on the above embodiment, optionally, the task allocation module 34 is used to determine whether the target directory is a large directory according to the number of files, including:

[0120] If the number of files is greater than a preset multiple of the file processing threshold of the task, the target directory is determined to be a large directory; or,

[0121] If, after a batch of files are transferred, the number of files in the target directory is greater than the file processing threshold of the task, the target directory is determined to be a large directory; or,

[0122] If the number of files is greater than the file processing threshold of the task, the number of files is less than a preset multiple of the file processing threshold of the task, and the target directory is a directory with a timeliness condition, then the target directory is determined to be a large directory.

[0123] Based on the above embodiment, optionally, the task assignment module 34 is used to transfer the files of the target user in the target directory to the corresponding directory of the peer user, including:

[0124] Calculate the hash value of the target file's file name;

[0125] Determine a file name modulus value according to the hash value and the modulus threshold;

[0126] If the file name mod value and the task mod value match, the file of the target user in the target directory is transferred to the corresponding directory of the peer user.

[0127] The directory file processing device provided by the embodiment of the present invention comprises a file number acquisition module 31, which is used to acquire the number of files in the target directory, wherein the target directory is any file directory provided by the platform for the target user to store files; a task parameter determination module 32, which is used to determine the number of task requirements according to the number of files and the file processing threshold of the task; and to determine the initial number of tasks according to the total number of task resources and the total number of directories; a state determination module 33, which is used to determine the state of the target directory according to the number of task requirements and the initial number of tasks; and a task allocation module 34, which is used to allocate tasks according to the state of the target directory; and to execute the allocated tasks according to the allocation results, and to transfer the files of the target user in the target directory to the corresponding directory of the opposite end user. Compared with the current practice of using a fixed number of tasks for data transmission in each directory, the directory file processing device provided by the embodiment of the present invention can determine the number of task requirements according to the number of files and the file processing threshold of the task, and determine the initial number of tasks according to the total number of task resources and the total number of directories. The state of the target directory is determined according to the number of task requirements and the initial number of tasks. The state represents the relationship between the initial number of tasks allocated to the target directory and the number of task requirements required by the target directory after the system resources are evenly allocated. According to the status of the target directory, tasks matching the target directory are allocated so that the number of tasks executing file transfers in the target directory matches the number of files in the target directory, thus achieving dynamic adjustment of the number of tasks. Since the target directory is any directory, the number of tasks that are ultimately executed can be adjusted for any directory provided by the platform in the above manner, thus improving the efficiency of file transfer.

[0128] The directory file processing device provided in the embodiment of the present invention can execute the directory file processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0129] Figure 5 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0130] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0131] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0132] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a directory file processing method.

[0133] In some embodiments, the directory file processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the directory file processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the directory file processing method in any other appropriate manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The computer program for implementing the directory file processing method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed completely on the machine, partially on the machine, partially on the machine and partially on a remote machine as an independent software package, or completely on a remote machine or server.

[0136] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a directory file processing method, the method comprising:

[0137] Get the number of files in the target directory, where the target directory is any file directory provided by the platform for the target user to store files;

[0138] Determine the number of task requirements based on the number of files and the file processing threshold of the task; determine the initial number of tasks based on the total number of task resources and the total number of directories;

[0139] Determine the state of the target directory according to the required number of tasks and the initial number of tasks;

[0140] The tasks are assigned according to the status of the target directory; the assigned tasks are executed according to the assignment result, and the files of the target user in the target directory are transferred to the corresponding directory of the opposite end user.

[0141] Based on the above embodiment, optionally, obtaining the number of files in the target directory includes:

[0142] Monitor the file status in the target directory;

[0143] The number of files is counted according to the monitoring result to obtain the number of files.

[0144] Based on the above embodiment, optionally, determining the number of task requirements according to the number of files and the file processing threshold of the task includes:

[0145] The task requirement number is calculated based on the number of files and the file processing threshold of the task, and the task requirement number represents the number of tasks required to transfer all the files contained in the target directory;

[0146] Accordingly, the initial number of tasks is determined based on the total number of task resources and the total number of directories, including:

[0147] The initial number of tasks is calculated from the total number of task resources and the total number of directories, and the initial number of tasks represents the number of tasks initially allocated to the target directory.

[0148] Based on the above embodiment, optionally, before calculating the initial number of tasks from the total number of task resources and the total number of directories, the method further includes:

[0149] The total number of task resources is determined according to the number of processing nodes and the allocatable threads of each processing node, and the total number of task resources represents the number of all tasks that can be provided by the platform.

[0150] Based on the above embodiment, optionally, tasks are allocated according to the state of the target directory, including:

[0151] Determining whether the target directory is a large directory according to the number of files;

[0152] If the target directory is a large directory, tasks are allocated to the target directory according to the initial number of tasks, and when other directories are idle, idle tasks of other directories are allocated to the target directory, wherein the other directories are directories other than the target directory;

[0153] If the target directory is not a large directory and the target directory is in an idle state, tasks are allocated to the target directory according to the task requirement number, idle tasks are determined according to the difference between the initial task number and the task requirement number, and the idle tasks are allocated to large directories in other directories;

[0154] If the target directory is not a large directory and the target directory is in a non-idle state, tasks are allocated to the target directory according to the initial number of tasks.

[0155] Based on the above embodiment, optionally, determining whether the target directory is a large directory according to the number of files includes:

[0156] If the number of files is greater than a preset multiple of the file processing threshold of the task, the target directory is determined to be a large directory; or,

[0157] If, after a batch of files are transferred, the number of files in the target directory is greater than the file processing threshold of the task, the target directory is determined to be a large directory; or,

[0158] If the number of files is greater than the file processing threshold of the task, the number of files is less than a preset multiple of the file processing threshold of the task, and the target directory is a directory with a timeliness condition, then the target directory is determined to be a large directory.

[0159] Based on the above embodiment, optionally, transferring the files of the target user in the target directory to the corresponding directory of the peer user includes:

[0160] Calculate the hash value of the target file's file name;

[0161] Determine a file name modulus value according to the hash value and the modulus threshold;

[0162] If the file name mod value and the task mod value match, the file of the target user in the target directory is transferred to the corresponding directory of the peer user.

[0163] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0165] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0166] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0167] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A directory file processing method, characterized in that: include: Get the number of files in the target directory, where the target directory is any file directory provided by the platform for the target user to store files; Determine the number of task requirements according to the number of files and the file processing threshold of the task; Determine the initial number of tasks based on the total number of task resources and the total number of directories; Determine the state of the target directory according to the required number of tasks and the initial number of tasks; Assign tasks based on the status of the target directory; The assigned task is executed according to the assignment result, and the files of the target user in the target directory are transferred to the corresponding directory of the peer user.

2. The method according to claim 1, characterized in that Get the number of files in the target directory, including: Monitor the file status in the target directory; The number of files is counted according to the monitoring result to obtain the number of files.

3. The method according to claim 1, characterized in that Determining the task requirement number according to the number of files and the file processing threshold of the task includes: The task requirement number is calculated based on the number of files and the file processing threshold of the task, and the task requirement number represents the number of tasks required to transfer all the files contained in the target directory; Accordingly, the initial number of tasks is determined based on the total number of task resources and the total number of directories, including: The initial number of tasks is calculated from the total number of task resources and the total number of directories, and the initial number of tasks represents the number of tasks initially allocated to the target directory.

4. The method according to claim 3, characterized in that Before the initial number of tasks is calculated from the total number of task resources and the total number of directories, it also includes: The total number of task resources is determined according to the number of processing nodes and the allocatable threads of each processing node, and the total number of task resources represents the number of all tasks that can be provided by the platform.

5. The method according to claim 1, characterized in that Assign tasks based on the status of the target directory, including: Determining whether the target directory is a large directory according to the number of files; If the target directory is a large directory, tasks are allocated to the target directory according to the initial number of tasks, and when other directories are idle, idle tasks of other directories are allocated to the target directory, wherein the other directories are directories other than the target directory; If the target directory is not a large directory and the target directory is in an idle state, tasks are allocated to the target directory according to the task requirement number, idle tasks are determined according to the difference between the initial task number and the task requirement number, and the idle tasks are allocated to large directories in other directories; If the target directory is not a large directory and the target directory is in a non-idle state, tasks are allocated to the target directory according to the initial number of tasks.

6. The method according to claim 5, characterized in that Determining whether the target directory is a large directory based on the number of files includes: If the number of files is greater than a preset multiple of the file processing threshold of the task, the target directory is determined to be a large directory; or, If, after a batch of files are transferred, the number of files in the target directory is greater than the file processing threshold of the task, the target directory is determined to be a large directory; or, If the number of files is greater than the file processing threshold of the task, the number of files is less than a preset multiple of the file processing threshold of the task, and the target directory is a directory with a timeliness condition, then the target directory is determined to be a large directory.

7. The method according to claim 1, characterized in that The file of the target user in the target directory is transferred to the corresponding directory of the peer user, including: Calculate the hash value of the target file's file name; Determine a file name modulus value according to the hash value and the modulus threshold; If the file name mod value and the task mod value match, the file of the target user in the target directory is transferred to the corresponding directory of the peer user.

8. A directory file processing device, characterized in that: include: A file quantity acquisition module, used to acquire the number of files in a target directory, where the target directory is any file directory provided by the platform for the target user to store files; A task parameter determination module, used to determine the task requirement number according to the number of files and the file processing threshold of the task; Determine the initial number of tasks based on the total number of task resources and the total number of directories; A state determination module, used for determining the state of the target directory according to the required number of tasks and the initial number of tasks; A task allocation module is used to allocate tasks according to the status of the target directory; The assigned task is executed according to the assignment result, and the files of the target user in the target directory are transferred to the corresponding directory of the peer user.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the directory file processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the directory file processing method according to any one of claims 1 to 7 when executed.

Images