Information processing program, information processing method, and information processing apparatus

Abstract

To provide an information processing program, an information processing method, and an information processing device that improve data processing efficiency.SOLUTION: The information processing program for managing a plurality of storage devices and a plurality of servers causes a computer to execute the following processing of: obtaining an actual compression ratio and a decompression speed of compressed data obtained by compressing data on the servers while varying compression ratio settings using a part of the data used by the plurality of servers; and determining a compression ratio setting to be used, using the actual compression ratio and the decompression speed for each of the obtained compression ratio settings on the basis of the maximum bandwidth of the total of the plurality of storage devices and the number of servers.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

Background Art

[0002] In recent years, with the evolution and spread of AI (Artificial Intelligence) technology, its applications in image recognition, voice translation, etc. have advanced, and the demand for processing large-capacity data has been increasing. In particular, in order to improve the inference accuracy, it is desirable to use more learning data. However, currently, the computing performance of computers does not sufficiently keep up with the amount of data to be processed. Therefore, techniques for expanding the computational amount without reducing the learning accuracy through parallel learning using multiple servers have been developed.

[0003] There are two methods of parallelization by dividing the learning process. One is a method called data parallelism, in which the learning data is divided and different data is used on each server for processing sharing. The other is model parallelism, in which the neural network is divided for each region and processing is performed for each region on each server.

[0004] In parallel learning using data parallelism, the divided learning data is often placed in a shared storage, and the servers for learning often start learning after performing staging to copy and acquire the learning data from the shared storage before learning. In this case, which data is transferred to which server varies depending on the number of computing nodes.

[0005] Here, since the staging time is considered to be a part of the overall learning time, it is preferably made as short as possible. One method for this is to reduce the learning data to be staged. For example, when the learning data is not compressed, the bandwidth of the shared storage may be saturated and staging may take a long time. In contrast, compressing the learning data can suppress the saturation of the bandwidth.

[0006] Furthermore, as a data compression technique, there is a method for improving parallel execution performance in a distributed environment of machine learning functions that use large amounts of data by calculating the most effective file size for data compression based on metadata, assigning a set of jobs to a distributed set, and improving parallel execution performance. In addition, there is a technique in which a DNN module compresses / decompresses startup data to reduce the use of memory bus bandwidth between neurons and memory. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] U.S. Patent Application Publication No. 2020 / 0250007 [Patent Document 2] U.S. Patent Application Publication No. 2018 / 0300606 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, while lossless compression can reduce the size of training data, there is a trade-off between the compression ratio, the decompression speed, and the CPU (Central Processing Unit) resources used. For example, when deciding on a lossless compression algorithm, it is possible to select compression methods such as lexicographical or decryption, as well as parameters such as the range and window size for searching for duplicate data. Generally, selecting a compression method or parameters with a high compression ratio increases the amount of computation required for both compression and decompression.

[0009] Therefore, if training data is compressed without any consideration, the decompression speed of the compressed data may become a bottleneck in the training process. In other words, for efficient staging, neither a high compression ratio nor a fast decompression speed is necessarily good. To put it another way, the compression ratio to choose will vary depending on whether the bottleneck in staging efficiency is the transfer bandwidth or the processing speed on the computer performing the decompression. Thus, since the optimal compression ratio setting depends on the bandwidth between the training server and the shared storage, the actual compression ratio, and the decompression speed, it is difficult to uniquely determine the optimal compression ratio with general data compression techniques. For this reason, shortening staging time through appropriate training data compression is not easy, and improving the efficiency of the training process is difficult.

[0010] Furthermore, in techniques that calculate the most effective file size for data compression based on metadata, it is difficult to balance the compression ratio, decompression cost, and bandwidth, making it challenging to set an appropriate compression ratio. Similarly, in techniques that reduce memory bus bandwidth usage by compressing / decompressing startup data using a DNN module, the handling of training data is not considered, making it difficult to set an appropriate compression ratio. Therefore, regardless of the technique used, improving the processing efficiency of data processing is difficult.

[0011] The disclosed technology was made in view of the above, and aims to provide an information processing program, an information processing method, and an information processing device that improve the processing efficiency of data processing. [Means for solving the problem]

[0012] In one embodiment of the information processing program, information processing method, and information processing apparatus disclosed in this application, an information processing program for managing a plurality of storage devices for storing data and a plurality of servers for decompressing and processing the data transferred from the storage devices, wherein, in the process of determining the compression ratio when compressing the data when transferring the data from the storage devices to the servers, the time required for the servers to decompress the compressed data compressed with a compression ratio setting using a portion of the data, and the data part The larger of the time required to transfer the data to the aforementioned server is calculated as the required time. Process The computer then performs a process to determine which compression ratio setting to use, by running the process for each of the multiple candidate compression ratio settings and selecting the first compression ratio setting that corresponds to the shortest required time among the multiple required times calculated. [Effects of the Invention]

[0013] In one respect, the present invention can improve the processing efficiency of data processing. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 is a system configuration diagram of a parallel learning system. [Figure 2] Figure 2 shows the flow of compression and decompression of training data. [Figure 3] Figure 3 is a block diagram of the storage device and learning server. [Figure 4] Figure 4 is a diagram illustrating the generation of the compression ratio setting correspondence table. [Figure 5] Figure 5 illustrates one limitation of the compression size. [Figure 6] Figure 6 shows an example of a table corresponding to the actual compression ratio. [Figure 7] Figure 7 shows an example of a transfer time correspondence table. [Figure 8]FIG. 8 is an overall flowchart of learning processing by the parallel learning system according to the embodiment. [Figure 9] FIG. 9 is a flowchart of learning data characteristic identification processing. [Figure 10] FIG. 10 is a flowchart of compression rate setting determination processing. [Figure 11] FIG. 11 is a flowchart of the process for creating the summarized actual compression rate correspondence table. [Figure 12] FIG. 12 is a flowchart of the process for selecting the compression size. [Figure 13] FIG. 13 is a diagram showing an example of a computer. [[ID=?]]

Mode for Carrying Out the Invention

[0015] Hereinafter, embodiments of the information processing program, information processing method, and information processing apparatus disclosed in the present application will be described in detail based on the drawings. Note that the information processing program, information processing method, and information processing apparatus disclosed in the present application are not limited by the following embodiments.

Embodiment

[0016] FIG. 1 is a system configuration diagram of a parallel learning system. The parallel learning system 1 includes a plurality of storage devices 10 and learning servers 20. Each storage device 10 and each learning server 20 are connected to each other via a switch 30. The total network bandwidth of the learning servers 20 is sufficiently larger than the overall throughput of the storage devices 1 which are shared storage. Also, there is no bias in the bandwidth used for communication for each of the learning servers 20, and there is no bias in the bandwidth used for communication for each of the storage devices 10.

[0017] It should be noted that there seems to be a typo in the original text where "the storage devices 1" is likely meant to be "the storage devices 10". This has been corrected in the translation where appropriate. Also, the tag [[ID=?]] in the original text seems to be an error, and it has been left as is in the translation.The storage device 10 stores the training data used for training by the training server 20. Furthermore, the storage device 10 uses a portion of the training data to determine the compression ratio and compression size, taking into account the balance between compression ratio, decompression cost, and bandwidth, and stores the training data in advance for actual training. The compression ratio setting is information indicating the compression ratio to be set for the compression algorithm used to compress the training data.

[0018] The compression ratio setting can take values ​​within a range such as 1 to 20 for a compression archiver, and does not take the form of a parameter representing the compression ratio for a specific data compression method. For example, a compression ratio setting of 1 is given in advance as the setting when using compression algorithm A and parameter N, and a compression ratio setting of 2 is given as the setting when using compression algorithm B and parameter M.

[0019] The compression size refers to the size of each compression unit when training data is compressed in batches. In other words, the compression size is a value that represents how many training data points are compressed together. For example, if 10 training data points are used as the compression unit, the compression size will be 10.

[0020] The learning server 20 acquires the compressed training data held by the storage device 10 through staging. Staging includes transferring and decompressing the compressed training data from the storage device 10. The learning server 20 immediately decompresses the training data transferred from the storage device 10. In other words, the learning server 20 acquires and decompresses the training data from the storage device 10 in parallel. Then, multiple learning servers 20 perform training using the training data that they have each collaboratively decompressed.

[0021] Figure 2 is a diagram illustrating the flow of compression and decompression of training data. Here, using Figure 2, we will explain the state of training data in the parallel learning system 1 according to this embodiment.

[0022] As shown in Figure 2, in the parallel learning system 1 according to this embodiment, some of the individual learning data 101 included in the overall learning data group 100 are combined into compressed learning data 102 and sent to the learning server 20. The compressed learning data 102 is then decompressed in the learning server 20 to become a learning data group 103 in which the learning data group is divided, containing multiple learning data 101.

[0023] Figure 3 is a block diagram of the storage device and the learning server. Next, the operation of the storage device 10 and the learning server 20 will be described in detail with reference to Figure 3.

[0024] As shown in Figure 2, the storage device 10 includes a data identification and measurement unit 11, a compression ratio setting and determination unit 12, a compression size determination unit 13, a compression unit 14, and a data storage unit 15.

[0025] The data storage unit 15 is a memory device. Multiple training data sets before compression are stored in the data storage unit 15. Subsequently, the compressed training data is stored in the data storage unit 15.

[0026] The data identification and measurement unit 11 extracts a portion of the training data as a sample and performs training data characteristic identification processing to determine the actual compression ratio and the decompression throughput of the training data for each compression ratio setting, while changing the compression ratio setting. The training data characteristic identification processing by the data identification and measurement unit 11 is described below.

[0027] The data characteristic measurement unit 11 extracts a portion of the training data stored in the data storage unit 15. Next, the data characteristic measurement unit 11 selects one compression ratio setting from among several available compression ratio settings. Then, using the selected compression ratio setting, the data characteristic measurement unit 11 compresses the extracted portion of training data to obtain the actual compression ratio. The actual compression ratio is the compression ratio obtained when the data is actually compressed under a specific compression ratio setting. For example, the data characteristic measurement unit 11 can determine the actual compression ratio to be 0.8 when the compression ratio setting is 1. This process by the data characteristic measurement unit 11 corresponds to the process in Figure 2 where several training data 101 are grouped together and compressed to generate compressed training data 102.

[0028] Next, the data characteristics measurement unit 11 compresses a portion of the extracted training data using a selected compression ratio setting and sends the compressed training data to one of the training servers 20. Subsequently, the data characteristics measurement unit 11 receives the measurement results of the decompression throughput when the transmitted compressed training data is decompressed from the training server 20. This process by the data characteristics measurement unit 11 corresponds to the process in Figure 2 in which the compressed training data 102 is decompressed by the training server 20 and the training data group 103 is generated.

[0029] The data characteristics measurement unit 11 selects each compression ratio setting one by one in sequence and obtains the actual compression ratio and decompression throughput for all compression ratio settings. Figure 4 is a diagram illustrating the generation of the compression ratio setting correspondence table. The data characteristics measurement unit 11 generates the actual compression ratio correspondence table 111 shown in Figure 4, which summarizes the correspondence between each compression ratio setting and the actual compression ratio obtained by actually compressing the data. The data characteristics measurement unit 11 also generates the decompression throughput correspondence table 112 shown in Figure 4, which summarizes the correspondence between each compression ratio setting and the decompression throughput information obtained from the learning server 20. Then, the data characteristics measurement unit 11 combines the actual compression ratio correspondence table 111 and the decompression throughput correspondence table 112 to generate the compression ratio setting correspondence table 113. After that, the data characteristics measurement unit 11 outputs the compression ratio setting correspondence table 113 to the compression ratio setting determination unit 12.

[0030] The compression ratio setting determination unit 12 receives input to the compression ratio setting correspondence table 113 from the data characteristic measurement unit 11. Then, the compression ratio setting determination unit 12 uses the compression ratio setting correspondence table 113 to perform a compression ratio setting selection process that selects the compression ratio setting that minimizes the time required from the transfer of the compressed learning data to the completion of decompression. The compression ratio setting selection process by the compression ratio setting determination unit 12 is described below.

[0031] The compression ratio setting determination unit 12 selects one row from the compression ratio setting correspondence table 113. Next, the compression ratio setting determination unit 12 calculates the time required for decompression processing on the learning server 20 when the compression ratio setting of the selected row is used. The time calculated here corresponds to the time required for the compressed learning data 102 to be decompressed and the learning data group 103 to be generated in Figure 2.

[0032] Specifically, the compression ratio setting determination unit 12 divides the total size of the training data, which is the training data size, by the number of training nodes, which is the number of training servers 20 that perform parallel training, to determine the size of the training data used by each training server 20. Next, the compression ratio setting determination unit 12 divides the determined size of the training data used by each training server 20 by the decompression throughput in the training server 20 to calculate the time required for decompression processing in the training server 20. Here, if the number of training nodes is N, the decompression throughput of the selected row is D, and the time required for decompression processing in the training server 20 is T1, the compression ratio setting determination unit 12 calculates (training data size / N) / D and sets it to T1.

[0033] Next, the compression ratio setting determination unit 12 calculates the time required to transfer and decompress the entire training data using the selected compression ratio setting, i.e., the staging time. The staging time calculated here corresponds to the time it takes from when the compressed training data 102 is sent to the training server 20 until it is decompressed at each training server 20, as shown in Figure 2.

[0034] Specifically, the compression ratio setting determination unit 12 multiplies the training data size by the actual compression ratio to obtain the compressed size of the entire training data. Next, the compression ratio setting determination unit 12 divides the compressed size of the entire training data by the number of training nodes to calculate the data size to be sent to each training server 20. Here, the smaller of the value obtained by dividing the total bandwidth of the storage device 10 by the number of training nodes and the connection bandwidth of the training server 20 becomes the bandwidth between the storage device 10 and the training server 20 when transferring the compressed training data. Therefore, the compression ratio setting determination unit 12 divides the data size to be sent to each training server 20 by the smaller of the value obtained by dividing the total bandwidth of the storage device 10 by the number of training nodes and the connection bandwidth of the training server 20 to calculate the time required to transfer the entire training data. Here, let the number of training nodes be N, the compression ratio setting and actual compression ratio of the selected row be C and P, respectively, and the time required to transfer the entire training data be T2. In this case, the compression ratio setting determination unit 12 calculates (learning data size × P / N) / min (total bandwidth of storage device 10 / N, connection bandwidth of learning server 20) and sets it to T2.

[0035] As described above, in the learning server 20, the transmission of learning data from the storage device 10 and the decompression of the acquired learning data are performed in parallel. That is, the staging time is the longer of the time it takes to transmit the learning data from the storage device 10 or the time it takes to decompress the learning data in the learning server 20. Therefore, the compression ratio setting determination unit 12 selects the larger of the time required for the decompression process in the learning server 20 and the time required to transfer the entire learning data when the compression ratio setting for the selected row is used, and sets this as the required time when the compression ratio setting for the selected row is used. Here, if the number of learning nodes is fixed at one value, one required time is determined, but if there are multiple values ​​that can be the number of learning nodes, the compression ratio setting determination unit 12 calculates the average of the required times for each case and sets this as the required time when the compression ratio setting is used.

[0036] The compression ratio setting determination unit 12 selects rows one by one from the compression ratio setting correspondence table 113 and calculates the time required when using all the compression ratio settings in each row. Then, the compression ratio setting determination unit 12 selects the compression ratio setting that minimizes the required time as the compression ratio setting to be used. After that, the compression ratio setting determination unit 12 outputs the information of the selected compression ratio setting and the compression ratio setting correspondence table 113 to the compression size determination unit 13.

[0037] The compression size determination unit 13 receives information on the selected compression ratio setting and input to the compression ratio setting correspondence table 113 from the compression ratio setting determination unit 12. The compression size determination unit 13 then performs the following compression size determination process to select the compression size that minimizes the staging time for the selected compression ratio setting. In the compression size determination process, the compression size determination unit 13 measures and collects the relationship between the amount of data and the compression ratio for the selected compression ratio setting using some sample data, because the compression ratio may increase when data is aggregated. Subsequently, the compression size determination unit 13 searches for the value that minimizes the staging time for each compression size, which is the size of data to be aggregated.

[0038] When searching for the compression size that minimizes staging time, it is advisable to pay attention to the following two points. One point to note is that increasing the compression size results in wasted data transfer. Figure 5 illustrates one limitation of the compression size. If the compression size is larger than the number of data used for training by each training server 20, which is calculated by dividing the total number of training data to be staged (number of training data) by the number of training nodes, then the data in the range 120 in Figure 5 will not be used for training by the training server 20 at the staging destination. In other words, sending the data in the range 120 to the training server 20 will result in wasted data and wasted data transfer.

[0039] Another point to note is that when the learning server 20 accesses the training data, limitations arise due to metadata access performance. The learning server 20 maintains information such as location, file name, and access rights for each file and directory; this is called metadata. Accessing this metadata is metadata access. Metadata access is performed when accessing data, but there are limits to the metadata access performance that can perform this metadata access. For example, let's consider metadata processing that obtains the identifier and data number of the learning server 20 that holds the actual data from a path name. In this case, the learning server 20 has a table that associates path names, server identifiers, and data numbers. By using this table, the learning server 20 can obtain a response to a query using a path name. When the learning server 20 accesses training data, it performs the operation of looking up this table and then accessing the actual data based on the result. The number of times the learning server 20 can perform this operation per second is the metadata access performance. Here, the total throughput between the learning server 20 and all storage devices 10 is defined as the total transfer throughput. If the total transfer throughput of the learning server 20 divided by the compressed size exceeds the meta-access performance, the transfer speed is limited to a value obtained by multiplying the decompressed size (calculated by dividing the compressed size by the compression ratio) by the meta-access performance. In staging, the data transferred is of this limited size, and since the data is not yet decompressed during transfer, the transfer throughput is limited to a value obtained by multiplying the compressed size by the meta-access performance.

[0040] The compression size determination process by the compression size determination unit 13, taking the above two points into consideration, will be described below.

[0041] The compression size determination unit 13 determines an upper limit for the compression size and calculates the actual compression ratio, which is the actual compression ratio when the training data of the compression size is compressed together by increasing the compression size by a predetermined amount between 1 and the upper limit. It is preferable to determine the upper limit within a range that is realistically feasible. Furthermore, if the predetermined amount to increase is such that the training data is increased one by one, it will take an enormous amount of time and calculation will become difficult if the total number of training data is large. Therefore, it is preferable to determine the amount to increase in a way that fits within a manageable amount of computation.

[0042] For example, the compression size determination unit 13 sets the upper limit of the compression size as the number of training data (the total number of training data to be staged) divided by the number of training nodes multiplied by 10. Then, the compression size determination unit 13 repeats the following process while repeatedly selecting and changing the compression size in increments of 1 to a power of 2 (1, 2, 4, 8, 16, ...) between 1 and the upper limit of the compression size. The compression size determination unit 13 acquires training data for the selected compression size, compresses it at the selected compression setting rate, and measures the actual overall compression rate. Then, the compression size determination unit 13 generates the overall actual compression rate correspondence table 130 shown in Figure 6 by registering the measurement results of each compression size and overall actual compression rate in a table. Figure 6 is a diagram showing an example of the overall actual compression rate correspondence table.

[0043] After generating the summary actual compression ratio table 130, the compression size determination unit 13 determines the upper limit of the compression size, increases it by a predetermined amount between 1 and the upper limit, and changes the compression size, executing the following transfer time calculation process for each compression size.

[0044] First, the compression size determination unit 13 determines whether the value obtained by dividing the learning server 20's total transfer throughput by the selected compression size is greater than the meta-access performance. If the value obtained by dividing the learning server 20's total transfer throughput by the selected compression size is greater than the meta-access performance, meta-access becomes a bottleneck, and the transfer speed is limited by the value obtained by multiplying the compression size by the meta-access performance. Therefore, the compression size determination unit 13 excludes the selected compression size from the list of candidate compression sizes to use and proceeds to the transfer time calculation process for the next compression size.

[0045] In response to this, if the value obtained by dividing the total transfer throughput of the learning server 20 by the selected compression size is less than or equal to the meta-access performance, the compression size determination unit 13 performs the following processing. The compression size determination unit 13 determines whether the selected compression size is greater than or equal to the number of learning data used by each learning server 20, which is calculated by dividing the number of learning data by the number of learning nodes.

[0046] If the compression size is less than the number of training data used by each training server 20, the compression size determination unit 13 calculates the size of the training data per training server 20 by dividing the total size of the training data (training data size) by the number of training nodes. Next, the compression size determination unit 13 obtains the corresponding aggregate compression ratio from the aggregate compression ratio correspondence table 130, multiplies the size of the training data per server by the selected compression size, and uses that value as the transfer size of the training data to be sent to one training server 20. Here, if the number of training nodes is N and the aggregate compression ratio is P', the compression size determination unit 13 calculates (training data size / N) × P' as the transfer size.

[0047] Furthermore, if the compression size is less than the number of training data used by each training server 20, all of the transmitted compressed training data will be used efficiently by each training server 20. Therefore, the compression size determination unit 13 sets the effective transfer volume coefficient, which represents the proportion of the transferred data that was actually used, to 1.

[0048] In contrast, if the compressed size is greater than or equal to the number of training data used by each training server 20, the compression size determination unit 13 calculates the transfer size of the training data to be sent to one training server 20 by multiplying the compressed size by the aggregated actual compression ratio. Here, if the compression ratio is T and the aggregated actual compression ratio is P', the compression size determination unit 13 calculates T × P' to obtain the transfer size.

[0049] Furthermore, if the compressed size is greater than or equal to the number of training data used by each training server 20, the amount of training data actually used by each training server 20 from the transmitted compressed training data is the training data size divided by the number of training nodes. Therefore, the compression size determination unit 13 uses the ratio calculated by dividing the value obtained by dividing the training data size by the number of nodes N by the compressed size obtained by dividing the compressed size by the corresponding aggregate actual compression ratio as the effective transfer volume coefficient, which represents the proportion of the transferred data that was actually used. For example, if the number of training nodes is N, the compressed size is T, and the aggregate actual compression ratio is P', the compression size determination unit 13 calculates (training data size / N) / (T×P') as the effective transfer volume coefficient.

[0050] The compression size determination unit 13 then calculates the throughput for each learning server 20 by dividing the total throughput of the storage device 10 by the number of learning nodes. Next, the compression size determination unit 13 divides the calculated transfer size by the throughput for each learning server 20 to calculate a provisional time to transfer all the learning data. Furthermore, the compression size determination unit 13 calculates the transfer time by dividing the calculated provisional time by the effective transfer coefficient. For example, if the total throughput of the storage device 10 is S and the number of learning nodes is N, the compression size determination unit 13 calculates (transfer size / (S / N)) / effective transfer coefficient to obtain the transfer time.

[0051] The compression size determination unit 13 calculates the transfer time for each compression size and generates a transfer time correspondence table 140, shown in Figure 7, which associates the compression size, the number of training nodes, and the transfer time. Figure 7 shows an example of a transfer time correspondence table.

[0052] Subsequently, if the number of learning nodes is 1, the compression size determination unit 13 refers to the transfer time correspondence table 140 and selects the compression size that minimizes the transfer time as the compression size to be used. If there are several numbers of learning nodes, the compression size determination unit 13 refers to the transfer time correspondence table 140, calculates the average transfer time for each compression size, and selects the compression size that minimizes the calculated average value as the compression size to be used. After that, the compression size determination unit 13 outputs the compression ratio setting selected by the compression ratio setting determination unit 12 and the compression size it has selected to the compression unit 14.

[0053] The compression unit 14 receives information from the compression size determination unit 13, including the compression ratio setting selected by the compression ratio setting determination unit 12 and the compression size selected by the compression size determination unit 13. The compression unit 14 then compresses the training data stored in the data storage unit 15 using the selected compression size and compression setting to generate compressed training data, which is then stored in the data storage unit 15.

[0054] As shown in Figure 3, the learning server 20 includes a data expansion speed measurement unit 21, a staging execution unit 22, a learning data holding unit 23, and a learning unit 24.

[0055] The data decompression speed measurement unit 21 receives compressed learning data, which is a portion of the learning data extracted in the data characteristics measurement process, from the data identification measurement unit 11 of the storage device 10 for each compression setting. Next, the data decompression speed measurement unit 21 decompresses the received compressed learning data and measures the decompression throughput for each compression setting. After that, the data decompression speed measurement unit 21 transmits the measurement results of the decompression throughput for each compression setting to the data characteristics measurement unit 11.

[0056] The staging execution unit 22 of each learning server 20 retrieves compressed learning data, including the learning data allocated to itself, from the data storage unit 15 of the storage device 10, which has been compressed by the storage device 10. The staging execution unit 22 then decompresses the retrieved compressed learning data while performing meta-access processing and stores it in the learning data holding unit 23.

[0057] Each learning server 20's learning unit 24 has the same learning model to be learned. For example, the learning model can be a DNN (Deep Neural Network). The learning unit 24 then uses the learning data stored in the learning data holding unit 23 to train the learning model and calculate the weight update amount. The learning unit 24 then communicates with the learning servers 20 to calculate the average of the weight update amounts calculated by each learning server 20 and reflects this in the learning model of each learning server 20's learning unit 24.

[0058] Figure 8 is a flowchart of the overall learning process by the parallel learning system according to the embodiment. Next, the overall flow of the learning process by the parallel learning system 1 according to the embodiment will be explained with reference to Figure 8.

[0059] The data characteristic measurement unit 11 of the storage device 10 uses the data expansion speed measurement unit 21 of the learning server 20 to perform learning data characteristic identification processing (step S1).

[0060] Next, the compression ratio setting determination unit 12 of the storage device 10 executes a compression ratio setting determination process using the compression ratio setting correspondence table 113 obtained in the training data characteristic identification process to determine the compression ratio setting to be used (step S2).

[0061] Next, the compression size determination unit 13 of the storage device 10 executes a compression size determination process using the compression ratio setting and compression ratio setting correspondence table 113 selected by the compression ratio setting determination unit 12 to determine the compression size to be used (step S3).

[0062] The compression unit 14 compresses the training data using the compression ratio setting selected by the compression ratio setting determination unit 12 and the compression size selected by the compression size determination unit 13, and stores the generated compressed image data in the data storage unit 15 (step S4).

[0063] The staging execution unit 22 of the learning server 20 retrieves the compressed learning data stored in the data storage unit 15 of the storage device 10, decompresses it, and stores it in the learning data holding unit 23 to perform staging (step S5).

[0064] The learning unit 24 of the learning server 20 acquires the learning data from the learning data storage unit 23. Then, the learning unit 24 uses the acquired learning data to train the learning model (step S6).

[0065] Figure 9 is a flowchart of the training data characteristic identification process. Next, we will explain the flow of the training data characteristic identification process with reference to Figure 9. Each process shown in the flowchart of Figure 9 is an example of the process executed in step S1 of Figure 8.

[0066] The data identification and measurement unit 11 extracts a portion of the training data as a sample (step S101).

[0067] Next, the data identification measurement unit 11 selects one compression ratio setting from among several existing compression ratio settings (step S102).

[0068] Next, the data identification and measurement unit 11 compresses a portion of the extracted training data using the selected compression ratio setting to obtain the actual compression ratio (step S103).

[0069] Next, the data characteristic measurement unit 11 compresses the extracted training data using the selected compression ratio setting and sends the compressed training data to the training server 20 (step S104).

[0070] The data decompression speed measurement unit 21 of the learning server 20 receives compressed learning data from the data characteristic measurement unit 11 of the storage device 10. The data decompression speed measurement unit 21 then decompresses the received compressed learning data and measures the decompression throughput (step S105).

[0071] The data expansion speed measurement unit 21 transmits the expansion throughput measurement result to the data characteristics measurement unit 11. The data characteristics measurement unit 11 acquires the expansion throughput measurement result transmitted from the data expansion speed measurement unit 21 (step S106).

[0072] Next, the data characteristics measurement unit 11 determines whether or not the actual compression ratio and decompression throughput have been obtained for all compression ratio settings (step S107). If there are any compression ratio settings for which the actual compression ratio and decompression throughput have not been obtained (step S107: negative), the data characteristics measurement unit 11 returns to step S102.

[0073] In response to this, if the actual compression ratio and decompression throughput are obtained for all compression ratio settings (step S107: affirmative), the data characteristic measurement unit 11 generates an actual compression ratio correspondence table 111 by summarizing the correspondence between each compression ratio setting and the actual compression ratio. The data characteristic measurement unit 11 also generates a decompression throughput correspondence table 112 by summarizing the correspondence between each compression ratio setting and the decompression throughput. Then, the data characteristic measurement unit 11 combines the actual compression ratio correspondence table 111 and the decompression throughput correspondence table 112 to generate a compression ratio setting correspondence table 113 (step S108). After that, the data characteristic measurement unit 11 outputs the compression ratio setting correspondence table 113 to the compression ratio setting determination unit 12.

[0074] Figure 10 is a flowchart of the compression ratio setting determination process. Next, the flow of the compression ratio setting determination process will be explained with reference to Figure 10. Each process shown in Figure 10 is an example of the process performed in step S2 of Figure 8.

[0075] The compression ratio setting determination unit 12 obtains the compression ratio setting correspondence table 113 from the data characteristic measurement unit 11. Next, the compression ratio setting determination unit 12 selects one row from the compression ratio setting correspondence table 113 (step S201).

[0076] Next, the compression ratio setting determination unit 12 calculates the time required for decompression processing on the learning server 20 when the compression ratio setting for the selected row is used. That is, if the number of learning nodes is N, the decompression throughput for the selected row is D, and the time required for decompression processing on the learning server 20 is T1, the compression ratio setting determination unit 12 calculates T1 = (learning data size / N) / D (step S202).

[0077] Next, the compression ratio setting determination unit 12 calculates the time required to transfer the entire training data, i.e., the staging time, when using the compression ratio setting for the selected row. Here, let N be the number of training nodes, P be the actual compression ratio for the selected row, and T2 be the time required to transfer the entire training data. In this case, the compression ratio setting determination unit 12 calculates T2 = (training data size × P / N) / min (total bandwidth of storage device 10 / N, connection bandwidth of training server 20) (step S203).

[0078] Next, the compression ratio setting determination unit 12 selects the larger of the time required for decompression processing on the learning server 20 and the time required to transfer the entire learning data when the compression ratio setting for the selected row is used, and sets this as the required time when the compression ratio setting for the selected row is used. That is, if the required time is Time, the time required for decompression processing on the learning server 20 is T1, and the time required to transfer the entire learning data is T2, the compression ratio setting determination unit 12 calculates Time = max(T1, T2) (step S204).

[0079] Next, the compression ratio setting determination unit 12, if the time required when using the compression ratio setting for the selected row is less than the minimum required time it holds, holds the time required when using the compression ratio setting for the selected row as the minimum maintenance time. Here, if the minimum required time is not held, the compression ratio setting determination unit 12 holds the time required when using the compression ratio setting for the selected row as the minimum maintenance time. That is, if the time required when using the compression ratio setting for the selected row is Time and the minimum required time is Tmin, the compression ratio setting determination unit 12 holds Tmin <Timeならば、Tmin=Timeとする(ステップS205)。

[0080] Subsequently, the compression ratio setting determination unit 12 determines whether or not the required time, or Time, has been calculated for all rows in the compression ratio setting corresponding table 113 (step S206). If there are rows for which the Time calculation has not been completed (step S206: negative), the compression ratio setting determination unit 12 returns to step S201.

[0081] In response to this, if the calculation of Time is completed for all rows (step S206: affirmative), the compression ratio setting determination unit 12 selects the compression ratio setting corresponding to Tmin, which is the minimum required time among the calculated required times, as the compression ratio setting to be used (step S207). Subsequently, the compression ratio setting determination unit 12 outputs the information of the selected compression ratio setting and the compression ratio setting correspondence table 113 to the compression size determination unit 13.

[0082] Figure 11 is a flowchart of the process for creating the summary actual compression ratio table. Figure 12 is a flowchart of the process for selecting the compression size. Figures 11 and 12 are examples of processes performed in the compression size determination process in step S3 of Figure 8. Referring to Figure 11, the flow of the process for creating the summary actual compression ratio table by the compression size determination unit 13 will be explained.

[0083] The compression size determination unit 13 determines the upper limit of the compression size. Furthermore, the compression size determination unit 13 sets the compression size T to 1 (step S301).

[0084] Next, the compression size determination unit 13 acquires training data of a size corresponding to the compression size T, performs compression using the compression setting determination unit 12, and measures the actual compression ratio, which is the overall actual compression ratio (step S302).

[0085] Then, the compression size determination unit 13 associates the compression size T with the measurement result of the combined actual compression ratio and registers it in the combined actual compression ratio correspondence table 130 (step S303).

[0086] Next, the compression size determination unit 13 increases the compression size T by a predetermined amount (step S304). For example, the compression size determination unit 13 increases the compression size in increments of 1 to a power of 2 (1, 2, 4, 8, 16, ...).

[0087] Next, the compression size determination unit 13 determines whether the compression size T exceeds the upper limit of the compression size. For example, if the upper limit is the value obtained by dividing the number of training data by 10 times the number of nodes N, the compression size determination unit 13 determines whether T > number of training data / (N × 10) (step S305). If T ≤ number of training data / (N × 10) (step S305: negation), the compression size determination unit 13 returns to step S301.

[0088] In contrast, if T > number of training data / (N / 10) (step S305: affirmative), the compression size determination unit 13 terminates the process of creating the summary actual compression ratio correspondence table 130.

[0089] Next, referring to Figure 12, the flow of the compression size selection process by the compression size determination unit 13 will be explained.

[0090] The compression size determination unit 13 sets the compression size T to 1 (step S311).

[0091] Next, the compression size determination unit 13 determines whether the value obtained by dividing the total transfer throughput of the learning server 20 by the selected compression size T is less than or equal to the meta-access performance value M. That is, the compression size determination unit 13 determines whether total transfer throughput / T ≤ M (step S312).

[0092] If the total transfer throughput / T > M (step S312: negative), the compression size determination unit 13 excludes the selected compression size T from the list of candidate compression sizes (step S313). The compression size determination unit 13 then proceeds to step S318.

[0093] In contrast, if the total transfer throughput / T ≤ M (step S312: affirmative), the compression size determination unit 13 determines whether the compression size T is greater than the number of training data per training server 20, which is calculated by dividing the number of training data by the number of nodes N. That is, the compression size determination unit 13 determines whether T > number of training data / N (step S314).

[0094] If T ≤ number of training data / N (step S314: negation), the compression size determination unit 13 calculates the size of the training data per training server 20 by dividing the total size of the training data, which is the training data size, by the number of training nodes. Next, the compression size determination unit 13 obtains the corresponding aggregate actual compression ratio from the aggregate actual compression ratio correspondence table 130, multiplies the size of the training data per server by the selected compression size, and uses that value as the transfer size. Here, if the number of training nodes is N and the aggregate actual compression ratio is P', the compression size determination unit 13 calculates (training data size / N) × P' as the transfer size. Furthermore, the compression size determination unit 13 sets the effective transfer amount coefficient, which represents the proportion of the transferred data that was actually used, to 1 (step S315).

[0095] In contrast, if T > number of training data / N (step S314: affirmative), the compression size determination unit 13 calculates the transfer size to one training server 20 by multiplying the compressed size by the actual aggregation compression ratio. That is, if the compression ratio is T and the actual aggregation compression ratio is P', the compression size determination unit 13 calculates T × P' to obtain the transfer size. Furthermore, the compression size determination unit 13 uses the ratio obtained by dividing the training data size by the number of nodes N by the compressed size calculated by dividing the compressed size by the corresponding actual aggregation compression ratio as the effective transfer volume coefficient, which represents the proportion of the transferred data that was actually used. That is, if the number of training nodes is N, the compressed size is T, and the actual aggregation compression ratio is P', the compression size determination unit 13 calculates (training data size / N) / (T × P') to obtain the effective transfer volume coefficient (step S316).

[0096] Next, the compression size determination unit 13 calculates the throughput for each learning server 20 by dividing the total throughput of the storage device 10 by the number of learning nodes. Then, the compression size determination unit 13 calculates a provisional time to transfer all the learning data by dividing the calculated transfer size by the throughput for each learning server 20. Furthermore, the compression size determination unit 13 calculates the transfer time by dividing the calculated provisional time by the effective transfer coefficient. That is, if the total throughput of the storage device 10 is S and the number of learning nodes is N, the compression size determination unit 13 calculates (transfer size / (S / N)) / effective transfer coefficient as the transfer time. Then, the compression size determination unit 13 registers the calculated transfer time in the transfer time correspondence table 140 (step S317).

[0097] The compression size determination unit 13 increases the compression size T by a predetermined amount (step S318). For example, the compression size determination unit 13 increases the compression size in increments of 1 to a power of 2 (1, 2, 4, 8, 16, ...).

[0098] Next, the compression size determination unit 13 determines whether the compression size T exceeds the upper limit of the compression size. For example, if the upper limit is the value obtained by dividing the number of training data by 10 times the number of nodes N, the compression size determination unit 13 determines whether T > number of training data / (N × 10) (step S319). If T ≤ number of training data / (N × 10) (step S319: negation), the compression size determination unit 13 returns to step S312.

[0099] In contrast, if T > number of training data / (N / 10) (step S319: affirmative), the compression size determination unit 13 refers to the transfer time correspondence table 140 and calculates the average transfer time for each compression size. Then, the compression size determination unit 13 selects T, which is the compression size that minimizes the calculated average value, as the compression size to be used (step S320). After that, the compression size determination unit 13 outputs the compression ratio setting selected by the compression ratio setting determination unit 12 and the compression size it has selected to the compression unit 14.

[0100] As described above, the storage device according to this embodiment obtains the actual compression ratio and decompression throughput of the training data for each compression ratio setting and selects the compression ratio setting that minimizes the staging time. Furthermore, the storage device selects the compression size that minimizes the staging time for the selected compression ratio setting, taking into account meta-access performance and the presence of unnecessary transfer data. The storage device then compresses and stores the training data using the selected compression ratio setting and compression size.

[0101] If each training data file is stored as a separate file in shared storage, metadata access to the shared storage may become a bottleneck, potentially slowing down the transfer speed. Therefore, instead of compressing each training data file individually, compressing a certain number of training data files together reduces the metadata access load on the shared storage server and improves compression efficiency. However, if too much data is grouped together, situations may arise where the entire compressed group is transferred to transfer only a portion of the data, so it is preferable to vary the number of data files grouped together depending on the situation. For example, the number of training nodes is not always constant for a given compressed training dataset, so it is preferable to select a compressed data size that is optimal for various expected numbers of training nodes. In this regard, the storage device according to this embodiment can select the optimal compressed data size by selecting a compression size that minimizes staging time while considering the existence of unnecessary data transfer.

[0102] In other words, by determining the compression ratio while considering three factors—the properties of the training data, such as its ease of compression; the decompression speed on the training server; and the bandwidth of the shared storage where the training data is stored—the time required for staging the training data can be minimized. Furthermore, by compressing the training data into the largest possible size, staging time can be minimized while considering the performance of the shared storage, such as metadata access, and the occurrence of unnecessary transfers. As a result, the training server can shorten staging time and improve the efficiency of data processing.

[0103] (Hardware configuration) Figure 13 shows an example of a computer. The storage device 10 and the learning server 20 can be implemented, for example, by the computer 90 shown in Figure 13.

[0104] As shown in Figure 13, the computer 90 includes a processor 91, memory 92, non-volatile memory 93, an I / O interface 94, and a network device 95. An SSD (Solid State Drive) 96, a SAS (Serial Attached SCSI (Small Computer System Interface))-HDD (Hard Disk Drive) 97, and other devices are connected to the I / O interface 94.

[0105] The processor 91 is connected via a bus to the memory 92, non-volatile memory 93, I / O interface 94, and network device 95. Furthermore, the processor 91 is connected to the SSD 96 and SAS-HDD 97 via the I / O interface 94.

[0106] The network device 95 is connected to the switch 30. The network device 95 is an interface for communication between, for example, the storage device 10 and the learning server 20, and for communication between the learning servers 20. The processor 91 communicates with external devices via the network device 95.

[0107] Memory 92 is the main memory. DRAM (Dynamic Random Access Memory) can be used for memory 92. Non-volatile memory 93 is the auxiliary memory. ROM (Read Only Memory) or flash memory can be used for non-volatile memory 93.

[0108] SSD96 and SAS-HDD97 are auxiliary storage devices. Storage device 10 holds a larger number of SSD96 and SAS-HDD97 compared to the learning server 20. SSD96 and SAS-HDD97 realize the functions of the data storage unit 15 and the learning data holding unit 23 as illustrated in Figure 3.

[0109] In the case of the storage device 10, the SSD 96 and SAS-HDD 97 store various programs, including information processing programs for realizing the functions of the data identification measurement unit 11, the compression ratio setting determination unit 12, the compression size determination unit 13, and the compression unit 14, as illustrated in Figure 3. In the case of the learning server 20, the SSD 96 and SAS-HDD 97, as illustrated in Figure 3, store various programs, including information processing programs for realizing the functions of the data decompression speed measurement unit 21, the staging execution unit 22, and the learning unit 24.

[0110] The processor 91 reads various programs stored in the SSD 96 and SAS-HDD 97, expands them into memory 92, and executes them. In the case of the storage device 10, the processor 91 implements the functions of the data identification and measurement unit 11, the compression ratio setting and determination unit 12, the compression size determination unit 13, and the compression unit 14. In the case of the learning server 20, the processor 91 implements the functions of the data expansion speed measurement unit 21, the staging execution unit 22, and the learning unit 24.

[0111] Furthermore, although this embodiment describes a case where the storage device 10 has a function to determine the compression ratio and compression size of the learning data, this function may be held by another information processing device that manages the storage device 10 and the learning server 20. In that case, the information processing device holds the data identification and measurement unit 11, the compression ratio setting determination unit 12, the compression size determination unit 13, and the compression unit 14 as illustrated in Figure 3. The information processing device then determines the compression ratio setting and compression size using the learning data held by the storage device 10, and notifies the storage device 10 of the determined compression ratio setting and compression size. [Explanation of symbols]

[0112] 1. Parallel Learning System 10 Storage devices 11 Data Identification Measurement Unit 12 Compression ratio setting determination unit 13 Compression Size Determination Section 14 Compression section 15 Data storage unit 20 Learning Servers 21 Data Expansion Speed ​​Measurement Unit 22 Staging Execution Department 23. Training data storage unit 24 Learning Department 111 Actual Compression Ratio Corresponding Table 112 Deployment throughput compatible tables 113 Compression Ratio Setting Support Table 130 Summary Table of Actual Compression Ratios 140 Transfer Time Correspondence Table

Claims

1. An information processing program for managing multiple storage devices that store data and multiple servers that expand and process the data transferred from the storage devices, When transferring the data from the storage device to the server, in the process of determining the compression ratio when compressing the data, the process of calculating the required time as the larger of the time required for the server to decompress the compressed data compressed with the compression ratio setting using a portion of the data and the time required to transfer the portion of the data to the server is performed for each of the candidate compression ratio settings. The compression ratio setting corresponding to the first required time, which is the shortest of the multiple required times calculated, is determined as the compression ratio setting to be used. An information processing program characterized by having a computer perform the processing.

2. The process includes causing the computer to further determine the compression size, which indicates the number of data items to be compressed. There are multiple possible numbers that can be the number of the aforementioned servers, The process for determining the compression size is as follows: The process involves compressing the training data equal to the compression size using the determined compression ratio setting, measuring the overall actual compression ratio, calculating the transfer size using the measured overall actual compression ratio, and calculating the transfer time by dividing the transfer size by the throughput of each server, and this process is performed for each of the multiple candidate compression sizes and for each of the possible number of servers. The average value of the transfer time for each of the multiple candidate compression sizes is calculated. This process determines the compression size to be used, which corresponds to the average value that minimizes the aforementioned average value. The information processing program according to feature 1.

3. In the process of determining the compression size, it is determined whether the value obtained by dividing the total transfer throughput of the server by a first compression size, which is one of the candidate compression sizes, is greater than the value of the meta-access performance. If the value is large, the first compression size will not be used as the compression size to be used. The information processing program according to feature 2.

4. An information processing method for managing a plurality of storage devices that store data and a plurality of servers that expand and process the data transferred from the storage devices, When transferring the data from the storage device to the server, in the process of determining the compression ratio when compressing the data, the process of calculating the required time as the larger of the time required for the server to decompress the compressed data compressed with the compression ratio setting using a portion of the data and the time required to transfer the portion of the data to the server is performed for each of the candidate compression ratio settings. The compression ratio setting corresponding to the first required time, which is the shortest of the multiple required times calculated, is determined as the compression ratio setting to be used. An information processing method characterized in that the processing is performed by a computer.

5. An information processing device that manages a plurality of storage devices for storing data and a plurality of servers for expanding and processing the data transferred from the storage devices, When transferring the data from the storage device to the server, in the process of determining the compression ratio when compressing the data, a data characteristics measurement unit performs a process for each of the candidate compression ratio settings to calculate the required time as the larger of the time required for the server to decompress the compressed data compressed with a portion of the data using the compression ratio setting and the time required to transfer the portion of the data to the server. A compression ratio setting determination unit determines the compression ratio setting to be used, which corresponds to the first required time, which is the minimum required time among the multiple required times calculated. An information processing device characterized by having the following features.

Images