File merging method and device, electronic equipment and storage medium

Abstract

The invention provides a file merging method and device, electronic equipment and a storage medium. The method comprises the steps that first related features corresponding to at least one folder in a file system at the current moment are obtained; determining at least one first to-be-merged folder from the at least one folder based on the first related features corresponding to the folders; utilizing a file merging scheduler and the initial execution quantity to initially merge files in the first to-be-merged folders, and determining a merging result; wherein the initial execution number is the number of the merging tasks executed by concurrently starting the first to-be-merged folder; based on the combination result, continuously adjusting the initial execution number to obtain a target execution number; and merging the files in the second to-be-merged folder by using the file merging scheduler and the target execution quantity. And by dynamically adjusting the execution quantity, the success rate and efficiency of folder merging are improved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a file merging method, device, electronic equipment and storage medium. Background Art

[0002] When processing big data in a distributed file system, small file problems are often encountered, which will affect storage efficiency and computing performance. Therefore, after monitoring that the number of small files reaches a certain level, a merge task needs to be started.

[0003] In the related art, the file merging method usually uses a pre-trained neural network model to determine whether folders need to be merged, and then merges the files in the folders that need to be merged. However, when executing the merge task, the number of folders that are set to start the merge task concurrently is consistent with the number of folders that need to be merged. During the merging process, folder merging may fail, and the number of merges cannot be adjusted dynamically, thereby reducing the success rate and efficiency of folder merging. Summary of the invention

[0004] The present invention provides a file merging method, device, electronic device and storage medium, which are used to solve the problem in the related art that when folder merging fails, the merging quantity cannot be dynamically adjusted, thereby reducing the success rate and efficiency of the folder merging.

[0005] The present invention provides a file merging method, comprising the following steps: Obtaining first related features corresponding to at least one folder in the file system at the current moment; Based on the first related features corresponding to the folders, determining at least one first folder to be merged from at least one folder; Using the file merge scheduler and the initial execution quantity, the files in each first folder to be merged are initially merged to determine the merge result; wherein the initial execution quantity is the number of concurrently started first folders to be merged to execute the merge task; Based on the merge results, the initial execution quantity is continuously adjusted to obtain the target execution quantity; The files in the second folder to be merged are merged using the file merge scheduler and the target execution quantity.

[0006] According to a file merging method provided by the present invention, based on the merging result, the initial execution quantity is continuously adjusted to obtain the target execution quantity, including: when there is a failure in merging the first folder to be merged in the merging result, the number of successful mergers of the first folder to be merged in the merging result is determined as the target execution quantity; when there is no failure in merging the first folder to be merged in the merging result, the initial execution quantity is increased by using the file merger scheduler to obtain the target execution quantity.

[0007] According to a file merging method provided by the present invention, based on the first relevant features corresponding to each folder, at least one first folder to be merged is determined from at least one folder, including: inputting the first relevant features corresponding to each folder into a pre-trained calculation model to obtain the merging probability corresponding to each folder; wherein the pre-trained calculation model is trained based on the relevant features of sample folders and the merging probability of the sample folders; based on the merging probability, at least one first folder to be merged is determined from at least one folder.

[0008] According to a file merging method provided by the present invention, before merging the files in the second folder to be merged using a file merging scheduler and a target execution number, the method also includes: if there is a merging failure of the first folder to be merged in the merging result, the first folder to be merged that initially failed to merge is used as the second folder to be merged; if there is no merging failure of the first folder to be merged in the merging result, the folder to be merged at the next moment is determined as the second folder to be merged.

[0009] According to a file merging method provided by the present invention, a method for determining a folder to be merged at the next moment includes: obtaining a second related feature corresponding to at least one folder in the file system at the next moment; wherein the interval between the current moment and the next moment is a preset time period; and determining the folder to be merged at the next moment based on the second related feature.

[0010] According to a file merging method provided by the present invention, the training process of the pre-trained computing model includes: obtaining relevant features of a sample folder and the merging probability of the sample folder; inputting the relevant features of the sample folder into the computing model to be trained to obtain the predicted sample merging probability; determining loss information based on the predicted sample merging probability and the merging probability of the sample folder; based on the loss information, continuously optimizing the computing model to be trained to obtain the pre-trained computing model.

[0011] According to a file merging method provided by the present invention, based on loss information, the computing model to be trained is continuously optimized to obtain a pre-trained computing model, including: obtaining historical folders to be merged and historical merging results corresponding to the historical folders to be merged; the historical folders to be merged are folders that have already executed the merging task; based on the loss information, the historical folders to be merged and the historical merging results, the computing model to be trained is continuously optimized to obtain a pre-trained computing model.

[0012] The present invention also provides a file merging device, comprising the following modules: An acquisition module, used for acquiring first related features corresponding to at least one folder in the file system at the current moment; A first determining module, configured to determine at least one first folder to be merged from at least one folder based on first related features corresponding to each folder; The second determination module is used to use the file merging scheduler and the initial execution quantity to initially merge the files in each first folder to be merged and determine the merging result; wherein the initial execution quantity is the number of concurrently started folders to be merged to execute the merging task; An adjustment module is used to continuously adjust the initial execution quantity based on the merging result to obtain the target execution quantity; The merging module is used to merge the files in the second to-be-merged folder by using the file merging scheduler and the target execution quantity.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned file merging methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the file merging method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned file merging methods.

[0016] The present invention provides a file merging method, device, electronic device and storage medium. The method obtains the first relevant features corresponding to each of at least one folder in the file system at the current moment; determines at least one first folder to be merged from at least one folder based on the first relevant features corresponding to each folder; uses a file merging scheduler and an initial execution number to initially merge the files in each first folder to be merged and determine the merging result; wherein the initial execution number is the number of concurrently started first folders to be merged to perform merging tasks; based on the merging result, the initial execution number is continuously adjusted to obtain a target execution number; and the files in the second folder to be merged are merged using the file merging scheduler and the target execution number. The technical solution of the present invention is, firstly, by obtaining the first relevant features corresponding to each folder in the file system at the current moment, the real-time performance of the first relevant features is improved; then, according to the first relevant features, the first folder to be merged is determined to improve the accuracy of the first folder to be merged; then, using the file merging scheduler and the initial execution number, the files in the first folder to be merged are initially merged to obtain the merging result, and then the initial execution number is continuously adjusted based on the merging result to obtain the target execution number, and the folders in the second folder to be merged are merged using the target execution number. In this way, the execution quantity is continuously optimized and adjusted through the merging results, so that the number of folders executing the merging tasks concurrently can be adjusted at any time, thereby improving the success rate and efficiency of the folder merging. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of merging multiple small files into a large file provided by the present invention.

[0019] Figure 2 This is one of the flowchart diagrams of the file merging method provided by the present invention.

[0020] Figure 3 This is the second flowchart of the file merging method provided by the present invention.

[0021] Figure 4 This is the third flow chart of the file merging method provided by the present invention.

[0022] Figure 5 This is the fourth flowchart of the file merging method provided by the present invention.

[0023] Figure 6 It is a schematic diagram of the application of the pre-training computing model provided by the present invention.

[0024] Figure 7 It is a schematic diagram of the file merging scheduler provided by the present invention dynamically adjusting the number of concurrently merged folders.

[0025] Figure 8 It is a structural schematic diagram of the file merging device provided by the present invention.

[0026] Fig. 9 A schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that the serial numbers assigned to the objects described in the present invention, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0029] First, take the Hive data warehouse system as an example to explain file merging. Hive is a data warehouse system based on Hadoop (distributed system infrastructure) for storing and processing large-scale data. However, due to the fact that the core component of Hadoop, the highly fault-tolerant system (Hadoop Distributed File System, HDFS), supports appending but not modification, and Hive uses MapReduce (map-reduce model) as a data processing programming model and distributed computing framework, it is very easy to cause the storage of data tables to be small files and fragmented, which greatly reduces the computing efficiency. Therefore, when the number of monitored small files reaches a certain level, it is necessary to start a merge task to merge many small files into a large file. However, in the folder merge process, computing resources will be occupied, the file system will be under great pressure, the merge task will fail, the merge task will increase the execution time of the business task, and the success rate of the business task execution will be reduced.

[0030] Figure 1It is a schematic diagram of merging multiple small files into a large file provided by the present invention, wherein there is a Hive directory in HDFS: / hive / warehouse / xx.db, and the Hive directory corresponds to multiple tables (tables are folders), such as table1, table2, table3, table4..., and each folder includes multiple data files, such as, Figure 1 Table4 includes multiple data files (i.e. small files), and multiple small files are merged into a single large data file.

[0031] If the folder merging process is not properly arranged, the following problems may occur: (1) Resource consumption: The small file merging operation itself is resource-intensive because it involves reading small files, writing the merged large file, and possible network transmission. If it is performed frequently, it may lead to excessive consumption of central processing unit (CPU) and input / output (I / O) resources. (2) Performance degradation: Small file merging involves a large number of disk I / O operations, which may reduce the responsiveness of the system, especially under high load or high concurrent access. (3) Write amplification: Small file merging may cause write amplification (WA), that is, the amount of data actually written to the storage system is much larger than the original amount of data. This is because during the merging process, small files are rewritten into larger files. (4) Increased space usage: During the merging process, more storage space may be temporarily occupied because the original small files and the merged large files may exist at the same time until the merge operation is completed and the old files are cleaned up. (5) Impact on business jobs: Since the small file merging operation consumes a lot of computing and storage resources, it may affect other running data processing jobs, causing the performance of these jobs to degrade. (6) Metadata pressure: In a distributed file system or database, frequent merging of small files will increase metadata pressure because each file requires metadata to manage.

[0032] In related technologies, there are two main methods for merging folders: Method 1 is based on a rule engine, for example, multiple judgment conditions are introduced based on a resource indicator system, such as CPU usage is less than 40% (indicator value 1), memory usage is less than 50% (indicator value 2), the total number of small files is greater than or equal to 10,000 (indicator value 3), the time is 1 a.m. (cycle value 1) ... ..., and file merging scheduling is initiated when some or all of the above conditions are met. Method 2 uses a pre-trained neural network model to determine whether a folder needs to be merged, and then merges the files in the folder that needs to be merged.

[0033] However, the main problems with method 1 include: (1) it cannot take into account the (performance) differences in the actual (hardware) environment; (2) the mechanical time period setting may cause parallel submission to cause excessive pressure and frequent failure of merge tasks, such as accidentally configuring too many merge tasks at the same time; (3) it cannot dynamically consider the actual data processing business tasks (adjustable at any time), resulting in business blockage. For example, if a user configures a core business task at 1 a.m., the rule engine cannot intelligently identify this configuration and adjust the rule threshold. The merge task is still started at 1 a.m., resulting in the core business task being blocked due to lack of resources; (4) as time goes by, the system's operating environment will undergo major changes. For example, as more and more business tasks take up more and more disk space, network throughput increases, and the system becomes slower and slower; a policy system that cannot learn autonomously requires constant manual adjustment of thresholds, which is time-consuming and labor-intensive and has poor results.

[0034] Although method 2 solves some of the problems brought by method 1, when executing the merge task, the number of folders set for concurrently starting the merge task must be consistent with the number of folders determined to be merged. During the merging process, folder merging may fail, and the number of merges cannot be adjusted dynamically, thereby reducing the success rate and efficiency of folder merging.

[0035] Based on the above-mentioned related problems, the present invention improves a file merging method. First, by obtaining the first related feature corresponding to each folder in the file system at the current moment, the real-time performance of the first related feature is improved; then, according to the first related feature, the first folder to be merged is determined to improve the accuracy of the first folder to be merged; then, using the file merge scheduler and the initial execution number, the files in the first file to be merged are initially merged to obtain the merge result, and then the initial execution number is continuously adjusted based on the merge result to obtain the target execution number, and the folders in the second folder to be merged are merged using the target execution number. In this way, the execution number is continuously optimized and adjusted through the merge result, so that the number of folders that concurrently execute the merge task can be adjusted at any time, thereby improving the success rate and efficiency of the folder merge.

[0036] Combine the following Figure 2-Figure 7 The file merging method provided by the present invention is described. The file merging method provided by the present invention is applicable to any operating system that needs to merge folders, including but not limited to Hive data warehouse system, Hadoop distributed file system, etc. The execution subject of the method can be an electronic device, or a file merging device set in the electronic device, and the file merging device can be implemented by software, hardware, or a combination of the two.

[0037] Figure 2 It is one of the flowcharts of the file merging method provided by the present invention, such as Figure 2As shown, the method includes the following steps 101 to 104: Step 101: Obtain a first related feature corresponding to at least one folder in the file system at the current moment.

[0038] The file system here can be any suitable file system, such as Hive data warehouse system, Hadoop distributed file system, etc.

[0039] Here, the relevant features corresponding to at least one folder in the file system at the current moment may be obtained once every preset time period.

[0040] The first related feature refers to the related feature of each folder in a certain file system, wherein the first related feature may be one-dimensional or multi-dimensional, and here, the related feature corresponds to the file system.

[0041] Specifically, taking the Hive data warehouse system and the corresponding related systems as an example, the related features of the Hive data warehouse system may include the following features: (1) System architecture related, for example, the computer language instructions executed by the microprocessor in the CPU architecture (x86) = 1, Advanced RISC Machine (ARM) = 2; the local solid-state drive (SSD) volume in the storage architecture = 1, the local mechanical hard disk volume = 2, the network attached storage (NAS) volume = 3; the network architecture (Gigabit network = 1, 10 Gigabit network = 2), the number of nodes of the container cloud platform (Kubernete, K8S), the number of nodes of the storage cluster, etc.

[0042] (2) Business system related, such as the query volume per hour (QPH) of data application programming interface (API), the amount of data transmitted per hour (Megabyte per Hour) of data integration writing, the amount of data written out (Megabyte per Hour), the amount of data governance tasks per hour (Task per Hour), etc.

[0043] (3) Resource usage related, such as the number of Yarn applications (Application, App), the number of Vcores used, the number of Mem used, the number of HDFS files, the amount of data transferred in and out of the network IO per second (Megabyte per Second), and the metadata system such as the CPU usage, memory usage, disk usage, and the number of transactions transferred per second (Transaction per Second) of the MySQL database management system (MySQL) server.

[0044] (4) Other customizable weight factors, such as major holiday protection (0 if no, 1 if yes), folder (table) classification (1 if starting with stg, 2 if starting with dwd, 3 if starting with dws, 4 if starting with amd, and 0 for others), whether the folder was last selected for execution (0 if no, 1 if yes), whether the execution was successful (0 if no, 1 if yes), etc.

[0045] Here, the file merging may be merging small files in a folder (eg, files occupying a space less than or equal to a threshold), or merging large files (eg, files occupying a space greater than a threshold).

[0046] Step 102: Determine at least one first folder to be merged from the at least one folder based on the first related features corresponding to the folders.

[0047] Here, the first folder to be merged refers to the folder that needs to perform a merge task at the current moment.

[0048] The method for determining the first folder to be merged may be any appropriate method, for example, determining the first folder to be merged based on the comparison result between the first relevant feature and the feature threshold; for another example, determining the first folder to be merged using a neural network.

[0049] Step 103: using a file merge scheduler and an initial execution quantity, initially merge the files in each of the first folders to be merged to determine a merge result; wherein the initial execution quantity is the number of concurrently started merge tasks for executing the first folders to be merged.

[0050] The file merge regulator is used to execute folder merge jobs and adjust the number of concurrently started merge tasks for the folders to be merged.

[0051] The initial execution number can be any set number, for example, infinity.

[0052] The merge result refers to the success or failure of each folder merge during the initial merge.

[0053] Step 104: Based on the merging result, continuously adjust the initial execution quantity to obtain a target execution quantity.

[0054] Here, the target execution quantity is the number of concurrently started folders to be merged to execute the merge task at the next moment. Adjusting the initial execution quantity may include increasing or decreasing the execution quantity.

[0055] Step 105: Merge the files in the second folder to be merged by using the file merge scheduler and the target execution quantity.

[0056] Here, the second folder to be merged may include a folder to be merged at the next moment or a first folder to be merged that failed to be merged last time, which is not limited in the present invention.

[0057] In one embodiment, after the first folder to be merged is merged, the target execution quantity is determined according to the merge result, and then the target execution quantity is used to execute the merge task of the second folder to be merged. The second folder to be merged may be all the folders to be merged in the file system at the next moment, and the second folder to be merged may overlap with the first folder to be merged.

[0058] In the embodiment of the present invention, first, by obtaining the first relevant feature corresponding to each folder in the file system at the current moment, the real-time performance of the first relevant feature is improved; then, according to the first relevant feature, the first folder to be merged is determined to improve the accuracy of the first folder to be merged; then, using the file merge scheduler and the initial execution number, the files in the first file to be merged are initially merged to obtain the merge result, and then the initial execution number is continuously adjusted based on the merge result to obtain the target execution number, and the folders in the second folder to be merged are merged using the target execution number. In this way, the execution number is continuously optimized and adjusted through the merge result, so that the number of folders that concurrently execute the merge task can be adjusted at any time, thereby improving the success rate and efficiency of the folder merge.

[0059] Figure 3 This is the second flow chart of the file merging method provided by the present invention. Figure 3 As shown, based on the merging result, the initial execution quantity is continuously adjusted to obtain the target execution quantity. The implementation method may include the following steps 1041 and 1042: Step 1041: When there are some first to-be-merged folders that fail to be merged in the merge result, determine the number of first to-be-merged folders that have been successfully merged in the merge result as the target execution number.

[0060] Specifically, after the initial merge is executed, if the execution result includes that m folders are merged successfully and s folders are merged unsuccessfully, the target execution quantity is adjusted to m.

[0061] Step 1042: When there is no merging failure of the first to-be-merged folder in the merging result, the initial execution quantity is increased by using the file merging scheduler to obtain the target execution quantity.

[0062] Here, the increased amount may be any appropriate amount, for example, adjusting the target execution amount to 1.2 times, 1.5 times, etc. of the initial execution amount.

[0063] In an embodiment of the present invention, based on the merging result, the number of folders to be merged that are concurrently started to perform merging tasks at the next moment is adaptively optimized and adjusted so that the number of folders for concurrent merging tasks can meet the actual situation, thereby improving the success rate and efficiency of merging the folders to be merged.

[0064] Figure 4 FIG. 3 is a flowchart of the file merging method provided by the present invention. Figure 4 As shown, based on the first related features corresponding to each of the folders, the implementation method of determining at least one first folder to be merged from the at least one folder includes the following steps 1021 and 1022: Step 1021: input the first relevant features corresponding to each of the folders into a pre-trained calculation model to obtain the merging probability corresponding to each of the folders; wherein the pre-trained calculation model is trained based on the relevant features of the sample folders and the merging probability of the sample folders.

[0065] Here, the pre-trained computing model may be a trained model obtained from a server, or may be a model trained using relevant feature data sets in a sample folder after the model is constructed, and the present invention is not limited to this.

[0066] Here, the merging probability refers to the probability that the folders need to be merged, wherein the greater the merging probability, the greater the possibility that the folders need to be merged; conversely, the smaller the possibility that the folders need to be merged.

[0067] The relevant features of the sample folder can be the line management features of the folder data set obtained from the Internet search; or they can be obtained based on the relevant features of the folder accumulated and recorded by many long-term data middle-end systems deployed by the company. The present invention is not limited to this.

[0068] Step 1022: Based on the merging probability, determine at least one first folder to be merged from the at least one folder.

[0069] Here, the method for determining the first folder to be merged based on the merge probability can be any suitable method, for example, comparing the merge probability with the probability threshold, and determining the folder corresponding to the merge probability greater than or equal to the probability threshold as the first folder to be merged. Here, the probability threshold can be any suitable value, for example, 0.5, 0.6, etc.; for another example, sorting the obtained merge probabilities from high to low, and taking the folders corresponding to the first N merge probabilities as the first folder to be merged.

[0070] In an embodiment of the present invention, first, a pre-trained computing model is used to quickly output the merging probability of each folder according to the first relevant feature, thereby improving the calculation speed of determining the folder merging probability; then, the first folder to be merged is determined through each merging probability, thereby improving the accuracy and speed of determining the first file to be merged, and at the same time, according to the first relevant feature at the current moment, it is determined whether the current moment is the merging time, thereby improving the accuracy of the folder merging time, and further improving the success rate and efficiency of the folder merging.

[0071] In an embodiment of the present invention, before merging the files in the second folder to be merged by using the file merge scheduler and the target execution number, the method also includes: when the merge result shows that the first folder to be merged failed to merge, using the first folder to be merged that failed to merge initially as the second folder to be merged; when the merge result shows that the merge result shows that the first folder to be merged failed to merge, determining the folder to be merged at the next moment as the second folder to be merged; when the merge result shows that the first folder to be merged failed to merge, recording the failed merged folder, and giving priority to executing the failed merged folder when the merge task is executed next time.

[0072] In the embodiment of the present invention, by determining the corresponding second folder to be merged according to different merging results, the accuracy of the second folder to be merged is improved, thereby improving the success rate and efficiency of the folder merging.

[0073] In one embodiment, a method for determining the folders to be merged at the next moment includes: obtaining a second related feature corresponding to at least one folder in the file system at the next moment; wherein the interval between the current moment and the next moment is a preset time period; and determining the folders to be merged at the next moment based on the second related feature.

[0074] Here, folders may be added or reduced in the file system at the next moment, or files may be added or deleted in the folders. The folders in the file system at the next moment may be different from the folders in the file system at the previous moment. The difference in folders will lead to different corresponding related features. Therefore, the second related feature may also be different from the first related feature.

[0075] The preset time period may be any appropriate time period, for example, obtaining relevant features every half an hour or one hour.

[0076] Here, according to the second related feature, the method for determining the folder to be merged at the next moment can refer to the above steps 1021 and 1022, and the present invention will not be repeated here.

[0077] In an embodiment of the present invention, relevant features corresponding to folders in the file system are collected once every preset time period, so that the current suitable folder to be merged is calculated in real time according to the changing relevant features, thereby improving the accuracy of the folder merging timing, further improving the utilization rate of system resources, reducing the impact on the business, and improving the success rate.

[0078] In an embodiment of the present invention, the training process of the pre-trained computing model includes: obtaining relevant features of the sample folder and the merging probability of the corresponding sample folder; inputting the relevant features of the sample folder into the computing model to be trained to obtain the predicted sample merging probability; determining loss information based on the predicted sample merging probability and the merging probability of the sample folder; based on the loss information, continuously optimizing the computing model to be trained to obtain the pre-trained computing model.

[0079] Here, the relevant features of the sample folder are the same as the first relevant features. Taking the Hive data warehouse system as an example, the relevant features of the sample folder may include the above-mentioned system architecture related, business system related, resource usage related and other customizable weight factors.

[0080] Here, the computational model to be trained may be a constructed feedforward neural network model (Feedforward Neural Network, FNN).

[0081] The feedforward neural network model includes an input layer, a hidden layer and an output layer, where the output layer is a single neuron node.

[0082] Convolution from input layer to hidden layer and As shown in the following formula (1): (1); in, is the number of input-dependent features, is the weight from the 𝑖th input-related feature to the 𝑗th hidden neuron, is the value of the 𝑖th input-related feature, It is the bias term of the 𝑗th hidden neuron.

[0083] The activation function from the input layer to the hidden layer includes but is not limited to a rectified linear unit (ReLU) activation function, a sigmoid activation function, and the like.

[0084] Convolution from hidden layer to output layer As shown in the following formula (2): (2); in, is the number of neurons in the hidden layer, is the weight from the 𝑖th input feature to the 𝑗th hidden neuron, is the output of the 𝑗th hidden neuron, is the bias term of the output layer.

[0085] Among them, in formula (2) As shown in the following formula (3): (3); in, Indicates the maximum value.

[0086] The Sigmoid activation function is used as the activation function from the hidden layer to the output layer.

[0087] Specifically, the output of the output layer As shown in the following formula (4): (4); in, is the input value, It is the output value of the Sigmoid function, that is, the predicted sample merging probability.

[0088] Here, the loss information may be determined based on a loss function, and is used to characterize the difference between the predicted sample merging probability and the merging probability of the sample folder. The loss function may be a cross entropy loss function or a square loss function.

[0089] It should be understood that the pre-trained computing model essentially updates the weight parameters of the computing model to be trained layer by layer through the back propagation algorithm based on the loss information, and uses the optimization algorithm (such as gradient descent method, Adam optimizer) to iteratively update the computing model to be trained until the loss information is minimized or the maximum number of iterations is reached.

[0090] In an embodiment of the present invention, the computational model to be trained is trained by using the multi-dimensional related features of the sample folders, continuously learning from the data and updating the parameters, thereby improving the generalization ability and robustness of the model, thereby improving the accuracy of the folder merging timing, as well as the success rate and efficiency of the folder merging.

[0091] In one embodiment, based on the loss information, the computing model to be trained is continuously optimized to obtain the pre-trained computing model, including: obtaining historical folders to be merged and historical merge results corresponding to the historical folders to be merged; the historical folders to be merged are folders on which a merge task has been performed; based on the loss information, the historical folders to be merged and the historical merge results, the computing model to be trained is continuously optimized to obtain the pre-trained computing model.

[0092] Here, after each merge task is executed, the merge result of each folder to be merged is recorded, and the folder that is successfully merged is marked as 1, and the folder that fails to merge is marked as 0.

[0093] Specifically, based on the historical merge results of the historical folders to be merged and the merge result labels of the historical folders to be merged, the loss result between the historical merge results and the merge result labels is determined, wherein the loss result is used to characterize the difference between the historical merge results and the merge result labels, and the larger the loss result, the larger the difference; otherwise, the smaller the difference.

[0094] The loss result can be calculated using the mean squared error (MSE) as shown in the following formula (5): (5)

[0095] in, is the number of historical folders to be merged, Indicates whether the merge task of the initiated historical folder to be merged is successful, where success is 1 and failure is 0; Indicates that the model determines that the merge task needs to be executed and believes that it will succeed. It is fixed at 1.

[0096] In an embodiment of the present invention, the loss result is determined based on the actual merging results of the historical folders to be merged, and the model of the neural network to be trained is automatically trained and updated based on the loss information and the loss result, so that the pre-trained computing model dynamically adapts to the continuous changes in the environment, improves the accuracy of the pre-trained computing model, and further improves the accuracy of the folder merging timing, as well as the success rate and efficiency of the merging.

[0097] The following is an application scenario embodiment of an embodiment of the present invention.

[0098] Figure 5 FIG. 4 is a flowchart of the file merging method provided by the present invention. Figure 5 As shown, the specific implementation includes steps 501 to 5010: Step 501: Obtain first relevant features corresponding to at least one folder in the file system at the current moment.

[0099] Step 502: Input each first related feature into the pre-trained computing model, and output the merging probability of each folder.

[0100] Figure 6 It is an application schematic diagram of the pre-trained computing model provided by the present invention, wherein a relevant feature data set 61 and a file table (i.e., folder) set 62 under the corresponding Hive directory are input into a pre-trained computing model 63, wherein the feature data set 61 includes business-related features, custom features, system architecture features, and resource-related features, etc. The pre-trained computing model 63 cyclically calculates the value of each folder in the folder set, and outputs a merging probability (Yn) set 64 of all folders, wherein the merging probability of folder 1 is 0.1, the merging probability of folder 2 is 0.8, the merging probability of folder 2... and so on.

[0101] Step 503: Input the merging probability into the file merging scheduler to determine the first folder to be merged.

[0102] Step 504: The file merge scheduler initializes the execution quantity to obtain an initial execution quantity.

[0103] Here, considering that the pre-trained computing model has fully considered all characteristic factors, the initial execution number is set to infinity to initiate as many tasks as possible.

[0104] Step 505: The file merge scheduler initiates a merge task with T=initial execution quantity.

[0105] Step 506 , determine whether all the folders to be merged are merged successfully, if so, continue to step 505 ; otherwise, execute step 507 .

[0106] Step 507: initiate a merge task with T = the number m of folders to be merged that have been successfully merged.

[0107] Step 508 , determine whether all the folders to be merged are merged successfully, if so, execute step 509 , otherwise, continue to execute step 507 .

[0108] Step 509: initiate a merge task of a number n where T is greater than m.

[0109] Step 5010, determine whether all the folders to be merged are merged successfully, if so, execute step 509; otherwise, execute step 507.

[0110] Figure 7It is a schematic diagram of the file merge scheduler provided by the present invention dynamically adjusting the number of concurrent merged folders, and inputting the merge probability (Yn) set 64 of all folders into the file merge scheduler 71; the file merge scheduler 71 determines the folders to be merged according to the probability threshold; then, the file merge scheduler calls the Hive data file merger 72 to initiate the execution of the file merge job, and inputs the execution (merge) result into the file merge scheduler 71. After the file merge scheduler 71 adjusts the initial execution quantity according to the merge result, it calls the Hive data file merger 72 to initiate the execution of the file merge job again, obtains the merge result, adjusts the execution quantity again according to the merge result, continuously optimizes and adjusts the execution quantity, and initiates the execution of the file merge job according to the adjusted execution quantity.

[0111] Compared with the related technologies, the technical solution provided by the present invention has the following advantages: (1) By using neural network technology, it is possible to learn and predict the execution status of the merge, and realize intelligent scheduling of merge tasks without human intervention; (2) It takes into account the characteristics of multiple dimensions to achieve more comprehensive decision support; (3) By training the neural network model and continuously optimizing it, it can adapt to the ever-changing system environment and business needs; and dynamically adjust the number of folders initiated for merge in parallel according to the merge results to improve the success rate and efficiency of folder merge.

[0112] The file merging device provided by the present invention is described below. The file merging device described below and the file merging method described above can be referred to each other.

[0113] Figure 8 is a schematic diagram of the structure of the file merging device provided by the present invention, referring to Figure 8 As shown, the file merging device 800 includes: an acquisition module 801, a first determination module 802, a second determination module 803, an adjustment module 804 and a merging module 805; wherein: An acquisition module 801 is used to acquire a first related feature corresponding to at least one folder in the file system at the current moment; A first determining module 802, configured to determine at least one first folder to be merged from the at least one folder based on first related features corresponding to each of the folders; The second determination module 803 is used to use the file merging scheduler and the initial execution quantity to perform an initial merge on the files in each of the first to-be-merged folders to determine the merge result; wherein the initial execution quantity is the number of concurrently started merge tasks for executing the first to-be-merged folders; An adjustment module 804, configured to continuously adjust the initial execution quantity based on the merging result to obtain a target execution quantity; The merging module 805 is used to merge the files in the second to-be-merged folder by using the file merging scheduler and the target execution quantity.

[0114] In an exemplary embodiment, the adjustment module 804 is specifically configured to: In the case where there are cases where the first to-be-merged folders fail to be merged in the merge result, determining the number of the first to-be-merged folders that have been successfully merged in the merge result as the target execution number; In the case that there is no merging failure of the first to-be-merged folder in the merging result, the initial execution quantity is increased by using the file merging scheduler to obtain the target execution quantity.

[0115] In an exemplary embodiment, the first determining module 802 is specifically configured to: Inputting the first relevant features corresponding to each of the folders into a pre-trained calculation model to obtain the merging probability corresponding to each of the folders; wherein the pre-trained calculation model is trained based on the relevant features of the sample folders and the merging probability of the sample folders; Based on the merging probability, at least one first folder to be merged is determined from the at least one folder.

[0116] In an exemplary embodiment, the device further includes: a third determination module. Before merging the files in the second to-be-merged folder using the file merging scheduler and the target execution quantity, the third determination module is used to: If the first to-be-merged folder fails to be merged in the merge result, the first to-be-merged folder that failed to be initially merged is used as the second to-be-merged folder; When there is no merging failure of the first to-be-merged folder in the merging result, the to-be-merged folder at the next moment is determined as the second to-be-merged folder.

[0117] In an exemplary embodiment, the apparatus further includes: a fourth determining module. The fourth determining module is configured to: Acquire a second related feature corresponding to at least one folder in the file system at a next moment; wherein the interval between the current moment and the next moment is a preset time period; Based on the second related feature, the folder to be merged at the next moment is determined.

[0118] In an exemplary embodiment, the apparatus further comprises: a training module. The training module is configured to: Obtain relevant features of the sample folder and the merging probability of the corresponding sample folder; Inputting relevant features of the sample folder into the computational model to be trained to obtain a predicted sample merging probability; Determining loss information based on the predicted sample merging probability and the merging probability of the sample folder; Based on the loss information, the to-be-trained computing model is continuously optimized to obtain the pre-trained computing model.

[0119] In an exemplary embodiment, the training module is further configured to: Obtaining a historical folder to be merged and a historical merge result corresponding to the historical folder to be merged; the historical folder to be merged is a folder for which a merge task has been executed; Based on the loss information, the historical folders to be merged and the historical merging results, the computing model to be trained is continuously optimized to obtain the pre-trained computing model.

[0120] The device of this embodiment can be used to execute the method of any one of the embodiments in the file merging method side embodiments. Its specific implementation process and technical effects are similar to those in the file merging method side embodiments. For details, please refer to the detailed description in the file merging method side embodiments, which will not be repeated here.

[0121] Fig. 9 A schematic diagram of the physical structure of an electronic device provided by the present invention is as follows: Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the file merging method, which includes: obtaining the first relevant features corresponding to each of at least one folder in the file system at the current moment; determining at least one first folder to be merged from the at least one folder based on the first relevant features corresponding to each of the folders; performing an initial merge on the files in each of the first folders to be merged using a file merge scheduler and an initial execution number to determine a merge result; wherein the initial execution number is the number of concurrently started merge tasks for the first folders to be merged; based on the merge result, the initial execution number is continuously adjusted to obtain a target execution number; and the files in the second folder to be merged are merged using the file merge scheduler and the target execution number.

[0122] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the file merging method provided by the above methods, which includes: obtaining the first relevant characteristics corresponding to at least one folder in the file system at the current moment; based on the first relevant characteristics corresponding to each of the folders, determining at least one first folder to be merged from the at least one folder; using a file merge scheduler and an initial execution number, initially merging the files in each of the first folders to be merged to determine the merging result; wherein the initial execution number is the number of concurrently started merge tasks for executing the merge task of the first folders to be merged; based on the merge result, continuously adjusting the initial execution number to obtain the target execution number; using the file merge scheduler and the target execution number, merging the files in the second folder to be merged.

[0124] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the file merging method provided by the above-mentioned methods, the method comprising: obtaining the first relevant features corresponding to each of the at least one folder in the file system at the current moment; determining at least one first folder to be merged from the at least one folder based on the first relevant features corresponding to each of the folders; performing an initial merge on the files in each of the first folders to be merged using a file merge scheduler and an initial execution number, and determining a merge result; wherein the initial execution number is the number of concurrently started merge tasks for executing the merge task for the first folders to be merged; based on the merge result, continuously adjusting the initial execution number to obtain a target execution number; and merging the files in the second folder to be merged using the file merge scheduler and the target execution number.

[0125] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file merging method, characterized in that: include: Obtaining first related features corresponding to at least one folder in the file system at the current moment; Based on the first related features corresponding to the folders, determining at least one first folder to be merged from the at least one folder; Using a file merge scheduler and an initial execution quantity, initially merge the files in each of the first folders to be merged to determine a merge result; wherein the initial execution quantity is the number of concurrently started merge tasks for executing the first folders to be merged; Based on the merging result, the initial execution quantity is continuously adjusted to obtain a target execution quantity; The files in the second folder to be merged are merged using the file merging scheduler and the target execution quantity.

2. The file merging method according to claim 1, characterized in that: The step of continuously adjusting the initial execution quantity based on the merging result to obtain the target execution quantity includes: In the case where there are cases where the first to-be-merged folders fail to be merged in the merge result, determining the number of the first to-be-merged folders that have been successfully merged in the merge result as the target execution number; In the case that there is no merging failure of the first to-be-merged folder in the merging result, the initial execution quantity is increased by using the file merging scheduler to obtain the target execution quantity.

3. The file merging method according to claim 1 or 2, characterized in that: The determining at least one first folder to be merged from the at least one folder based on the first related features corresponding to the folders includes: Inputting the first relevant features corresponding to each of the folders into a pre-trained calculation model to obtain the merging probability corresponding to each of the folders; wherein the pre-trained calculation model is trained based on the relevant features of the sample folders and the merging probability of the sample folders; Based on the merging probability, at least one first folder to be merged is determined from the at least one folder.

4. The method according to claim 3, characterized in that Before merging the files in the second to-be-merged folder by using the file merging scheduler and the target execution quantity, the method further includes: If the first to-be-merged folder fails to be merged in the merge result, the first to-be-merged folder that failed to be initially merged is used as the second to-be-merged folder; When there is no merging failure of the first to-be-merged folder in the merging result, the to-be-merged folder at the next moment is determined as the second to-be-merged folder.

5. The method according to claim 4, characterized in that The method for determining the folder to be merged at the next moment includes: Acquire a second related feature corresponding to at least one folder in the file system at a next moment; wherein the interval between the current moment and the next moment is a preset time period; Based on the second related feature, the folder to be merged at the next moment is determined.

6. The file merging method according to claim 3, characterized in that: The training process of the pre-trained computing model includes: Obtaining relevant features of the sample folder and the merging probability of the sample folder; Inputting relevant features of the sample folder into the computational model to be trained to obtain a predicted sample merging probability; Determining loss information based on the predicted sample merging probability and the merging probability of the sample folder; Based on the loss information, the to-be-trained computing model is continuously optimized to obtain the pre-trained computing model.

7. The file merging method according to claim 6, characterized in that: Based on the loss information, the to-be-trained computing model is continuously optimized to obtain the pre-trained computing model, including: Obtaining a historical folder to be merged and a historical merge result corresponding to the historical folder to be merged; the historical folder to be merged is a folder for which a merge task has been executed; Based on the loss information, the historical folders to be merged and the historical merging results, the computing model to be trained is continuously optimized to obtain the pre-trained computing model.

8. A file merging device, characterized in that: The device includes: An acquisition module, used for acquiring first related features corresponding to at least one folder in the file system at the current moment; A first determining module, configured to determine at least one first folder to be merged from the at least one folder based on first related features corresponding to each of the folders; A second determination module is used to use a file merging scheduler and an initial execution quantity to initially merge the files in each of the first folders to be merged, and determine a merging result; wherein the initial execution quantity is the number of concurrently started merging tasks for the first folders to be merged; An adjustment module, used for continuously adjusting the initial execution quantity based on the merging result to obtain a target execution quantity; The merging module is used to merge the files in the second folder to be merged by using the file merging scheduler and the target execution quantity.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the file merging method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the file merging method according to any one of claims 1 to 7 is implemented.

Images