File capacity adjustment method, device, electronic device and readable storage medium

Abstract

The embodiment of the present invention provides a method, device, electronic device and readable storage medium for adjusting file capacity. The method includes: in the embodiment of the present invention, in each time period, the operating parameters of the system are counted, and according to the operating parameters, the calculation coefficient corresponding to the time period is calculated, the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files, according to the calculation coefficient, and the corresponding relationship between the calculation coefficient and the file capacity, the target file capacity matching the time period is determined, the file capacity of the merged file within the time period is set as the target file capacity, and within the time period, the operation of merging small files in the system into the merged file is performed. It can avoid generating many merged files in a short time, and can improve the writing speed of the merged file when writing the distributed file system.

Description

Technical Field

[0001] The present invention belongs to the field of big data technology, and in particular relates to a method, device, electronic device and readable storage medium for adjusting file capacity. Background Art

[0002] A distributed file system is a distributed file system designed to run on general-purpose hardware. It has the characteristics of high fault tolerance and is used to store a small number of large files, that is, merge files, rather than storing a large number of small files.

[0003] The related technology usually combines small files into a combined file of fixed capacity, and then writes the combined file into a distributed file system.

[0004] However, during business peak hours, there are a large number of small files corresponding to business data. When writing small files into a fixed-capacity merged file in a short period of time, many merged files will be generated. Therefore, when writing these merged files into the distributed file system, the writing speed is slow. Summary of the invention

[0005] The present invention provides a method and device for adjusting file capacity, an electronic device and a readable storage medium, so as to solve the technical problem of slow writing speed of merging files in the related art.

[0006] In a first aspect, the present invention provides a method for adjusting file capacity, the method comprising:

[0007] In each time period, the operating parameters of the system are counted, and according to the operating parameters, a calculation coefficient corresponding to the time period is calculated, wherein the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the concurrent processing capability of the system for files;

[0008] Determining a target file capacity matching the time period according to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity;

[0009] Setting the file capacity of the merged file within the time period to the target file capacity;

[0010] During the time period, the operation of merging small files in the system into the merged file is performed.

[0011] Optionally, the operating parameters include: a query rate per second; and calculating the calculation coefficient corresponding to the time period according to the operating parameters includes:

[0012] If the query rate per second is less than or equal to a preset value, determining the calculation coefficient as a ratio of the preset value to the query rate per second;

[0013] If the query rate per second is greater than the preset value, the calculation coefficient is determined to be the inverse value of the ratio of the query rate per second to the preset value.

[0014] Optionally, before determining the target file capacity matching the time period according to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity, the method further includes:

[0015] Based on the variation curve of the calculation coefficient and the file capacity, the file capacity corresponding to the calculation coefficient is obtained, and a corresponding relationship between the calculation coefficient and the file capacity is generated; the variation curve is used to reflect the nonlinear relationship between the calculation coefficient and the file capacity.

[0016] Optionally, acquiring the file capacity corresponding to the calculation coefficient based on a curve graph of changes in the calculation coefficient and the file capacity includes:

[0017] The ordinate corresponding to the calculation coefficient in the change curve graph is determined as the file capacity corresponding to the calculation coefficient.

[0018] Optionally, the operation of merging small files in the system into the merged file within the time period includes:

[0019] If the capacity of the merged file is smaller than the capacity of the target file, then obtaining the data in the small file in the system;

[0020] The data of the small files in the system are added to the merged file until the capacity of the merged file reaches the target file capacity.

[0021] Optionally, the method further includes:

[0022] Get the target file capacity in the current time period and the target file capacity in the previous time period;

[0023] If the target file capacity in the current time period is smaller than the target file capacity in the previous time period, the merged file is deleted, and a new merged file is created with a capacity equal to the target file capacity in the current time period.

[0024] Optionally, after setting the file capacity of the merged file within the time period to the target file capacity, the method further includes:

[0025] Get the current timestamp and the capacity of the merged file;

[0026] Recording the current timestamp and the capacity of the merged file in a log file;

[0027] If the capacity of the merged file recorded in the log file is consistent with the capacity of the target file, a message indicating successful setting is issued;

[0028] If the capacity of the merged file recorded in the log file is inconsistent with the capacity of the target file, a message indicating setting failure is issued.

[0029] Optionally, the method further includes:

[0030] Within the time period, the merged file is written into the distributed file system.

[0031] In a second aspect, the present invention provides a device for adjusting file capacity, the device comprising:

[0032] A calculation module, used for collecting statistics of system operation parameters in each time period, and calculating a calculation coefficient corresponding to the time period according to the operation parameters, wherein the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files;

[0033] A determination module, configured to determine a target file capacity matching the time period according to the calculation coefficient and a corresponding relationship between the calculation coefficient and the file capacity;

[0034] A setting module, used for setting the file capacity of the merged file within the time period to the target file capacity;

[0035] The merging module is used to merge the small files in the system into the merged file within the time period.

[0036] In a third aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned file capacity adjustment method when executing the program.

[0037] In a fourth aspect, the present invention provides a readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned file capacity adjustment method.

[0038] In an embodiment of the present invention, in each time period, the operating parameters of the system are counted, and according to the operating parameters, the calculation coefficient corresponding to the time period is calculated, the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files. According to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity, the target file capacity matching the time period is determined, and the file capacity of the merged file within the time period is set as the target file capacity. Within the time period, the operation of merging small files in the system into the merged file is performed. In an embodiment of the present invention, by calculating the calculation coefficient corresponding to the time period according to the operating parameters of the system in each time period, and determining the target file capacity matching the time period according to the corresponding relationship between the calculation coefficient and the file capacity, the capacity of the merged file is dynamically adjusted according to the target file capacity, rather than setting the capacity of the merged file to a fixed value. Therefore, during the peak business period, the small files in the system can be merged into the merged file according to the target file capacity, so as to avoid generating many merged files in a short time, and improve the writing speed of the merged file when writing the distributed file system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a flowchart of a method for adjusting file capacity provided by an embodiment of the present invention;

[0041] Figure 2 is a flowchart of another method for adjusting file capacity provided by an embodiment of the present invention;

[0042] Figure 3 is a curve diagram of changes in calculation coefficient and file capacity provided by an embodiment of the present invention;

[0043] Figure 4 A device for adjusting file capacity provided by an embodiment of the present invention;

[0044] Figure 5 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Traditional file systems encounter storage bottlenecks and file upload and download efficiency issues. For storage, the first thing to consider is capacity expansion. There are two forms of capacity expansion: vertical expansion, which means adding disks and memory; and horizontal expansion, which means increasing the number of servers. This form of storage is the prototype of distributed file storage, which is achieved by expanding the scale.

[0047] Distributed file systems provide customers with distributed file system products that provide reliable storage and efficient processing capabilities for massive data, especially massive small file data scenarios. For example, Hadoop Distributed File System (HDFS) is independently developed based on the Hadoop ecosystem. It is an easily scalable distributed file system that runs on hundreds or thousands of low-cost machines. It has many similarities with existing distributed file systems. Both are system tools for storing data. However, HDFS is highly fault-tolerant and is designed to be deployed on low-cost machines. HDFS is mainly used to store and manage massive file information, that is, to solve the storage problem of large data files, such as terabytes or even picobytes.

[0048] In the case of massive amounts of tiny file data, a common practice of a distributed file system is to merge small files into a fixed-capacity merged file and then store the merged file in the distributed file system. Related technologies determine the fixed capacity of the merged file based on historical experience after the system has been running for a period of time.

[0049] However, during peak business access periods, there is a lot of small file data. If files are still merged according to fixed capacity, many merged files may be generated in a short period of time. When they are frequently written to the distributed file system, the writing speed is slow and the writing performance of the file system cannot be effectively utilized.

[0050] In an embodiment of the present invention, in each time period, the operating parameters of the system are counted, and according to the operating parameters, the calculation coefficient corresponding to the time period is calculated, the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files. According to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity, the target file capacity matching the time period is determined, and the file capacity of the merged file within the time period is set as the target file capacity. Within the time period, the operation of merging small files in the system into the merged file is performed. In an embodiment of the present invention, by calculating the calculation coefficient corresponding to the time period according to the operating parameters of the system in each time period, and determining the target file capacity matching the time period according to the corresponding relationship between the calculation coefficient and the file capacity, the capacity of the merged file is dynamically adjusted according to the target file capacity, rather than setting the capacity of the merged file to a fixed value. Therefore, during the peak business period, the small files in the system can be merged into the merged file according to the target file capacity, so as to avoid generating many merged files in a short time, and improve the writing speed of the merged file when writing the distributed file system.

[0051] Figure 1 is a flowchart of a method for adjusting file capacity provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0052] Step 101: In each time period, the operating parameters of the system are counted, and based on the operating parameters, a calculation coefficient corresponding to the time period is calculated, wherein the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files.

[0053] For example, the operating parameters of the system are used to reflect the status of business requests in the current time period. The operating parameters of the system may include: query rate per second. The query rate per second is used to indicate the average amount of request data for business requests per unit time. The query rate per second refers to the number of queries that a server can respond to per second, and is a measure of how many business requests a specific query server handles within a specified time. In large model scenarios, an increase in the query rate per second means an increase in the speed of model reasoning, which can provide services to more users or reduce the cost of model reasoning.

[0054] For example, the calculation coefficient is negatively correlated with the molecular potential energy of the system, that is, the smaller the molecular potential energy of the system, the larger the calculation coefficient. The larger the molecular potential energy of the system, the smaller the calculation coefficient. The molecular potential energy reflects the relationship between the molecules in the system. When the molecules are close to each other, the molecular potential energy increases, the pressure in the system will increase, and the calculation coefficient will decrease. When the molecules are far away from each other, the molecular potential energy decreases, the pressure in the system will decrease, and the calculation coefficient will increase.

[0055] For example, molecular potential energy is used to characterize the system's concurrent processing capability for files. The larger the molecular potential energy, the stronger the system's concurrent processing capability for files, and the smaller the molecular potential energy, the weaker the system's concurrent processing capability for files.

[0056] Step 102: Determine a target file capacity that matches the time period based on the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity.

[0057] For example, the correspondence between the calculation coefficient and the file capacity may be predetermined, for example, the correspondence between the calculation coefficient and the file capacity may be provided by a developer according to different business volumes, so as to improve the accuracy of the correspondence between the calculation coefficient and the file capacity.

[0058] For example, after calculating the calculation coefficient within the current time period, the target file capacity that matches the calculation coefficient of the current time period can be determined based on the corresponding relationship between the calculation coefficient and the file capacity, so that the capacity of the merged file can be dynamically adjusted according to the target file capacity, instead of setting the capacity of the merged file to a fixed value. Therefore, during business peak hours, small files in the system can be merged into a merged file according to the target file capacity, avoiding the generation of many merged files in a short period of time, and improving the writing speed of the merged file when writing the distributed file system.

[0059] Step 103: Set the file capacity of the merged file within the time period as the target file capacity.

[0060] For example, after obtaining the target file capacity that matches the query rate per second in the current time period, the file capacity of the merged file is set as the target file capacity, so that the merged file can be adjusted according to the target file capacity. When the amount of small file data increases sharply, the capacity of the merged file in the system can be expanded, and the merged file with larger capacity is aggregated and then written into the distributed file system, thereby reducing the number of writes and achieving the effect of improving write performance.

[0061] Step 104: within the time period, merge the small files in the system into the merged file.

[0062] For example, in the current time period, according to the target file capacity set for the merged file, the small files in the system are merged into the merged file, that is, the data in the small files in the system are written into the merged file. The present invention calculates the target file capacity of the merged file, and expands the capacity of the merged file to the target file capacity during the peak business period, thereby avoiding the generation of many merged files in a short period of time, and can improve the writing speed of the merged file when writing it into the distributed file system.

[0063] In an embodiment of the present invention, in each time period, the operating parameters of the system are counted, and according to the operating parameters, the calculation coefficient corresponding to the time period is calculated, the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files. According to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity, the target file capacity matching the time period is determined, and the file capacity of the merged file within the time period is set as the target file capacity. Within the time period, the operation of merging small files in the system into the merged file is performed. In an embodiment of the present invention, by calculating the calculation coefficient corresponding to the time period according to the operating parameters of the system in each time period, and determining the target file capacity matching the time period according to the corresponding relationship between the calculation coefficient and the file capacity, the capacity of the merged file is dynamically adjusted according to the target file capacity, rather than setting the capacity of the merged file to a fixed value. Therefore, during the peak business period, the small files in the system can be merged into the merged file according to the target file capacity, so as to avoid generating many merged files in a short time, and improve the writing speed of the merged file when writing the distributed file system.

[0064] Figure 2 FIG. 1 is a flowchart of another method for adjusting file capacity provided by an embodiment of the present invention. Figure 2 As shown, the method may include:

[0065] Step 201: In each time period, the operating parameters of the system are counted, and based on the operating parameters, a calculation coefficient corresponding to the time period is calculated, wherein the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files.

[0066] This step may specifically refer to the above step 101, which will not be described in detail here.

[0067] Optionally, the operating parameters include: query rate per second, and step 201 may specifically include:

[0068] Sub-step 2011: if the query rate per second is less than or equal to a preset value, determining the calculation coefficient as a ratio of the preset value to the query rate per second;

[0069] Sub-step 2012: If the query rate per second is greater than the preset value, determine the calculation coefficient as the inverse value of the ratio of the query rate per second to the preset value.

[0070] For sub-steps 2011-2012, the preset value is the query rate per second of the critical point of the system obtained after the performance stress test, and its value can be 1000. If the current query rate per second is less than or equal to 1000, 1000 is divided by the current query rate per second, and the division result is used as the calculation coefficient corresponding to the current time period. If the current query rate per second is greater than 1000, the current query rate per second is divided by 1000, and the opposite value of the division result is used as the calculation coefficient corresponding to the current time period. After calculating the calculation coefficient within the current time period, the target file capacity that matches the query rate per second of the current time period can be determined according to the corresponding relationship between the calculation coefficient and the file capacity, so that the capacity of the merged file can be dynamically adjusted according to the target file capacity, instead of setting the capacity of the merged file to a fixed value. Therefore, during the peak business period, the small files in the system can be merged into the merged file according to the target file capacity, avoiding the generation of many merged files in a short period of time, and improving the writing speed of the merged file when writing to the distributed file system.

[0071] Step 202: Determine the target file capacity that matches the time period based on the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity.

[0072] For example, the target file capacity corresponding to the query rate per second that matches the time period can be determined according to the correspondence between the query rate per second and the file capacity provided by the developer based on different business volumes. When the amount of small file data increases sharply, the capacity of the merged files in the system can be expanded, and the merged files with larger capacity can be aggregated and written to the distributed file system, thereby reducing the number of writes and improving the write performance.

[0073] Optionally, before step 202, the method further includes:

[0074] Step 206: Based on the variation curve of the calculation coefficient and the file capacity, the file capacity corresponding to the calculation coefficient is obtained, and a corresponding relationship between the calculation coefficient and the file capacity is generated.

[0075] For example, see Figure 3 , the change curve of the calculation coefficient and the file capacity is nonlinear. According to the change curve of the calculation coefficient and the file capacity, the file capacity corresponding to any calculation coefficient can be obtained, so that the corresponding relationship between the calculation coefficient and the file capacity can be generated.

[0076] For example, if the calculation coefficient is 50, the corresponding file capacity in the molecular potential energy diagram of the calculation coefficient and the file capacity is 1 Gigabyte. If the calculation coefficient is 100, the corresponding file capacity in the molecular potential energy diagram of the calculation coefficient and the file capacity is 3G, then the corresponding relationship between 50 and 1G, and 100 and 3G can be generated.

[0077] For example, when the query rate per second increases, the capacity of the merged files in the system can be expanded, and by aggregating the merged files with larger capacity and writing them to the distributed file system, the number of writes can be reduced, thereby improving the write performance. When the amount of small file data decreases, the capacity of the merged files in the system can be reduced. Through dynamic adaptation, the lag of manual adjustment can be reduced, and the requirements for professional knowledge and technical thresholds can be reduced.

[0078] Optionally, step 206 may specifically include:

[0079] Sub-step 2061: determine the vertical coordinate corresponding to the calculation coefficient in the change curve graph as the file capacity corresponding to the calculation coefficient.

[0080] For sub-step 2061, see, for example, Figure 3 When the molecules are close to each other, that is, when the horizontal coordinate changes from r0 to 0, the molecular potential energy increases, and the corresponding system pressure increases. At this time, the query rate per second increases, and the file capacity needs to be increased. The value of the vertical coordinate corresponding to the query rate per second is used as the file capacity corresponding to the query rate per second. The horizontal coordinate r1 is the lowest point of the molecular potential energy, and this is the minimum stage of system pressure. Between the horizontal coordinate r0 and the horizontal coordinate r1, the file capacity uses a fixed file block size, which can be preset according to the actual system conditions. In the process of the horizontal coordinate r1 changing to infinity, the system pressure gradually decreases, and the query rate per second at this time decreases, and the file capacity needs to be reduced. The vertical coordinate corresponding to the calculation coefficient in the molecular potential energy diagram is used as the file capacity corresponding to the query rate per second.

[0081] For example, when the horizontal coordinate r0 is close to the origin, according to the calculation coefficient, the vertical coordinate corresponding to the calculation coefficient in the change curve diagram is determined as the file capacity corresponding to the calculation coefficient. After the capacity of the merged file reaches the value of the vertical coordinate corresponding to the calculation coefficient in the molecular potential energy diagram, the merged file is written to the distributed file system, which can reduce the number of writes and achieve the effect of improving write performance. When r1 is close to infinity, according to the calculation coefficient, the vertical coordinate corresponding to the calculation coefficient in the molecular potential energy diagram is determined as the file capacity corresponding to the calculation coefficient. After the capacity of the merged file reaches the value of the vertical coordinate corresponding to the calculation coefficient in the molecular potential energy diagram, the merged file is written to the distributed file system, which can reduce the number of writes and achieve the effect of improving write performance.

[0082] For example, when the amount of small file data increases sharply, that is, when the query rate per second increases, the capacity of the merged files in the system can be expanded, and the merged files with larger capacity are aggregated and written to the distributed file system, thereby reducing the number of writes and improving the write performance. When the amount of small file data decreases, that is, when the query rate per second decreases, the capacity of the merged files in the system can be reduced. Through dynamic adaptation, the lag of manual adjustment is reduced, and the requirements for professional knowledge and technical thresholds are also reduced.

[0083] Step 203: Set the file capacity of the merged file within the time period as the target file capacity.

[0084] This step may be specifically referred to as the above step 103, which will not be described in detail here.

[0085] Optionally, after step 203, the method further includes:

[0086] Step B1, obtaining the current timestamp and the capacity of the merged file;

[0087] Step B2, recording the current timestamp and the capacity of the merged file in a log file;

[0088] Step B3: If the capacity of the merged file recorded in the log file is consistent with the capacity of the target file, a message indicating successful setting is issued;

[0089] Step B4: If the capacity of the merged file recorded in the log file is inconsistent with the capacity of the target file, a setting failure message is issued.

[0090] For example, for steps B1 to B4, after the file capacity of the merged file is set to the target file capacity within the time period, the current timestamp and the capacity of the merged file are obtained, and the current timestamp and the capacity of the merged file are recorded in the log file. If the capacity of the merged file recorded in the log file is consistent with the target file capacity, a message indicating that the setting is successful is issued. If the capacity of the merged file recorded in the log file is inconsistent with the target file capacity, a message indicating that the setting is failed is issued, so that it can be verified whether the setting is successful.

[0091] Step 204: If the capacity of the merged file is smaller than the capacity of the target file, then obtain the data in the small file in the system.

[0092] For example, the data in the small file may be information of a business request. When the capacity of the merged file is smaller than the capacity of the target file, the information of the business request in the small file in the system is obtained.

[0093] Step 205: Add the data of the small files in the system to the merged file until the capacity of the merged file reaches the target file capacity.

[0094] For example, the information of business requests in small files in the system is written into the merged file. In the process of writing the merged file, the capacity of the merged file is obtained until the capacity of the merged file reaches the target file capacity, then the writing of business request information in small files in the system into the merged file is stopped. When the amount of small file data increases sharply, the capacity of the merged file in the system can be expanded, and by aggregating merged files with larger capacity and then writing them into the distributed file system, the number of writes can be reduced, thereby achieving the effect of improving write performance. When the amount of small file data decreases, the capacity of the merged file in the system can be reduced. Through dynamic adaptation, the lag of manual adjustment can be reduced, and the requirements for professional knowledge and technical thresholds can also be reduced.

[0095] Optionally, the method further includes:

[0096] Step 206: Obtain the target file capacity in the current time period and the target file capacity in the previous time period;

[0097] Step 207: If the target file capacity in the current time period is smaller than the target file capacity in the previous time period, the merged file is deleted, and a new merged file with a capacity equal to the target file capacity in the current time period is created.

[0098] For step 206-step 207, if the target file capacity in the current time period is 500 megabytes (M) and the target file capacity in the previous time period is 2 gigabytes (G), the merged file with the current capacity of 2G is deleted, and a new merged file with a capacity of 500M is created.

[0099] Optionally, the method further includes:

[0100] Step 208: Write the merged file into the distributed file system within the time period.

[0101] For example, after the capacity of the merged file reaches the target file capacity, the merged file is written into the distributed file system.

[0102] In an embodiment of the present invention, in each time period, the operating parameters of the system are counted, and according to the operating parameters, the calculation coefficient corresponding to the time period is calculated, the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files. According to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity, the target file capacity matching the time period is determined, and the file capacity of the merged file within the time period is set as the target file capacity. Within the time period, the operation of merging small files in the system into the merged file is performed. In an embodiment of the present invention, by calculating the calculation coefficient corresponding to the time period according to the operating parameters of the system in each time period, and determining the target file capacity matching the time period according to the corresponding relationship between the calculation coefficient and the file capacity, the capacity of the merged file is dynamically adjusted according to the target file capacity, rather than setting the capacity of the merged file to a fixed value. Therefore, during the peak business period, the small files in the system can be merged into the merged file according to the target file capacity, so as to avoid generating many merged files in a short time, and improve the writing speed of the merged file when writing the distributed file system.

[0103] Figure 4 A file capacity adjustment device 40 provided by an embodiment of the present invention may include:

[0104] The calculation module 401 is used to collect statistics of the operating parameters of the system in each time period, and calculate the calculation coefficient corresponding to the time period according to the operating parameters, wherein the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the concurrent processing capability of the system for files;

[0105] A determination module 402, configured to determine a target file capacity matching the time period according to the calculation coefficient and a corresponding relationship between the calculation coefficient and the file capacity;

[0106] A setting module 403, configured to set the file capacity of the merged file within the time period to the target file capacity;

[0107] The merging module 404 is used to merge the small files in the system into the merged file within the time period.

[0108] Optionally, the calculation module includes:

[0109] A first determination submodule, configured to determine the calculation coefficient as a ratio of the preset value to the query rate per second if the query rate per second is less than or equal to a preset value;

[0110] The second determination submodule is used to determine the calculation coefficient as the inverse value of the ratio of the query rate per second to the preset value if the query rate per second is greater than the preset value.

[0111] Optionally, the device further comprises:

[0112] An acquisition module is used to acquire the file capacity corresponding to the calculation coefficient based on a change curve graph of the calculation coefficient and the file capacity, and generate a corresponding relationship between the calculation coefficient and the file capacity; the change curve graph is used to reflect the nonlinear relationship between the calculation coefficient and the file capacity.

[0113] Optionally, the acquisition module includes:

[0114] The determination submodule determines the ordinate corresponding to the calculation coefficient in the change curve diagram as the file capacity corresponding to the calculation coefficient.

[0115] Optionally, the merging module includes:

[0116] An acquisition submodule, used for acquiring data in a small file in the system if the capacity of the merged file is smaller than the capacity of the target file;

[0117] The adding submodule is used to add the data of the small files in the system to the merged file until the capacity of the merged file reaches the target file capacity.

[0118] Optionally, the device further comprises:

[0119] A first acquisition module is used to acquire the target file capacity in the current time period and the target file capacity in the previous time period;

[0120] The processing module is used to delete the merged file if the target file capacity in the current time period is smaller than the target file capacity in the previous time period, and create a new merged file with a capacity equal to the target file capacity in the current time period.

[0121] Optionally, the device further comprises:

[0122] The second acquisition module is used to obtain the current timestamp and the capacity of the merged file;

[0123] A recording module, used for recording the current timestamp and the capacity of the merged file in a log file;

[0124] A first sending module, configured to send a message indicating successful setting if the capacity of the merged file recorded in the log file is consistent with the capacity of the target file;

[0125] The second sending module is used to send a setting failure message if the capacity of the merged file recorded in the log file is inconsistent with the capacity of the target file.

[0126] Optionally, the device further comprises:

[0127] The writing module is used to write the merged file into the distributed file system within the time period.

[0128] In an embodiment of the present invention, in each time period, the operating parameters of the system are counted, and according to the operating parameters, the calculation coefficient corresponding to the time period is calculated, the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the system's concurrent processing capability for files. According to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity, the target file capacity matching the time period is determined, and the file capacity of the merged file within the time period is set as the target file capacity. Within the time period, the operation of merging small files in the system into the merged file is performed. In an embodiment of the present invention, by calculating the calculation coefficient corresponding to the time period according to the operating parameters of the system in each time period, and determining the target file capacity matching the time period according to the corresponding relationship between the calculation coefficient and the file capacity, the capacity of the merged file is dynamically adjusted according to the target file capacity, rather than setting the capacity of the merged file to a fixed value. Therefore, during the peak business period, the small files in the system can be merged into the merged file according to the target file capacity, so as to avoid generating many merged files in a short time, and improve the writing speed of the merged file when writing the distributed file system.

[0129] The present invention also provides an electronic device, see Figure 5 , including: a processor 501, a memory 502, and a computer program 5021 stored in the memory and executable on the processor, wherein when the processor executes the program, the file capacity adjustment method of the aforementioned embodiment is implemented.

[0130] The present invention also provides a readable storage medium, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the file capacity adjustment method of the above embodiment.

[0131] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0132] It should be noted that the various information and data obtained in the embodiments of the present invention are all obtained with the authorization of the information / data holder.

[0133] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming voice. It should be understood that various programming voices can be utilized to realize the content of the present invention described herein, and the description of the above specific voices is for the purpose of disclosing the best mode of the present invention.

[0134] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0135] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0136] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0137] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0138] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0139] The user information (including but not limited to the user's device information, user personal information, etc.) and related data involved in the present invention are all information authorized by the user or by all parties.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0142] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for adjusting file capacity, characterized in that: The method comprises: In each time period, the operating parameters of the system are counted, and according to the operating parameters, a calculation coefficient corresponding to the time period is calculated, wherein the calculation coefficient is negatively correlated with the molecular potential energy of the system, and the molecular potential energy is used to characterize the concurrent processing capability of the system for files; Determining a target file capacity matching the time period according to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity; Setting the file capacity of the merged file within the time period to the target file capacity; During the time period, the operation of merging small files in the system into the merged file is performed.

2. The method according to claim 1, characterized in that The operating parameters include: query rate per second; The calculating, according to the operating parameters, a calculation coefficient corresponding to the time period includes: If the query rate per second is less than or equal to a preset value, determining the calculation coefficient as a ratio of the preset value to the query rate per second; If the query rate per second is greater than the preset value, the calculation coefficient is determined to be the inverse value of the ratio of the query rate per second to the preset value.

3. The method according to claim 1, characterized in that Before determining the target file capacity matching the time period according to the calculation coefficient and the corresponding relationship between the calculation coefficient and the file capacity, the method further includes: Based on the variation curve of the calculation coefficient and the file capacity, the file capacity corresponding to the calculation coefficient is obtained, and a corresponding relationship between the calculation coefficient and the file capacity is generated; the variation curve is used to reflect the nonlinear relationship between the calculation coefficient and the file capacity.

4. The method according to claim 3, characterized in that The acquiring the file capacity corresponding to the calculation coefficient based on the change curve graph of the calculation coefficient and the file capacity includes: The ordinate corresponding to the calculation coefficient in the change curve graph is determined as the file capacity corresponding to the calculation coefficient.

5. The method according to claim 1, characterized in that The operation of merging the small files in the system into the merged file within the time period includes: If the capacity of the merged file is smaller than the capacity of the target file, then obtaining the data in the small file in the system; The data of the small files in the system are added to the merged file until the capacity of the merged file reaches the target file capacity.

6. The method according to claim 1, characterized in that The method further comprises: Get the target file capacity in the current time period and the target file capacity in the previous time period; If the target file capacity in the current time period is smaller than the target file capacity in the previous time period, the merged file is deleted, and a new merged file is created with a capacity equal to the target file capacity in the current time period.

7. The method according to claim 1, characterized in that After setting the file capacity of the merged file within the time period to the target file capacity, the method further includes: Get the current timestamp and the capacity of the merged file; Recording the current timestamp and the capacity of the merged file in a log file; If the capacity of the merged file recorded in the log file is consistent with the capacity of the target file, a message indicating successful setting is issued; If the capacity of the merged file recorded in the log file is inconsistent with the capacity of the target file, a message indicating setting failure is issued.

8. The method according to claim 1, characterized in that The method further comprises: Within the time period, the merged file is written into the distributed file system.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the program.

10. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods described in claims 1-8.

Images