Data access method, device, equipment and storage medium

Abstract

The present application discloses a data access method, device, equipment and storage medium, which obtains a training data set for training a target deep learning model, determines a target index file based on the training data set, and randomly accesses each training data in the training data set based on the index. On this basis, operations such as random sorting of training data are converted into operations on index information, that is, in each round of training of the target deep learning model, the order of index information in the target index file is randomly disrupted to obtain a target index file in which the index information is arranged in a second order, and the target training data corresponding to the target index information is read from the training data set according to the second order and input into the target deep learning model for the round of training. As a result, only a small amount of index information and a small amount of training data required for each step of training need to be stored in the memory, which significantly reduces memory usage and can ensure the high efficiency of training data reading.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data access method, apparatus, device and storage medium. Background Art

[0002] In recent years, deep learning technology has made great progress, achieving performance close to or even exceeding that of humans in many fields such as natural language processing, computer vision, and speech recognition. Deep learning models with high training accuracy and good results cannot be separated from massive amounts of training data.

[0003] At present, deep learning models are mainly trained based on the stochastic gradient descent algorithm. In most model training scenarios, the entire training data set needs to be read into memory first, and then the training data in the training data set is randomly shuffled before each round of training, and a small part of the training data in the shuffled training data set is read in sequence and input into the model.

[0004] However, as the data used in model training grows, this approach becomes increasingly unfeasible, because tens or even hundreds of GB of training data may not be fully loaded into memory. In addition, only a small part of the entire training data set is actually used in each step of model training, resulting in low memory utilization and a large amount of memory resources wasted. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a data access method, device, equipment and storage medium to ensure that part of the required data is read in each round of training, avoiding a large amount of memory resource waste caused by loading the entire training data set. By randomly disrupting the order of index information in the target index file in each round of training, random access to each training data in the training data set based on the index is achieved, and on this basis, operations such as random sorting of training data can be converted into operations on index information, so that only a small amount of index information and a small amount of training data required for each step in training need to be stored in the memory, which significantly reduces memory usage compared to loading the entire training data set, and can ensure the high efficiency of training data reading.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a data access method, the method comprising:

[0008] Obtain a training dataset for training the target deep learning model;

[0009] Determine a target index file according to the training data set, wherein the target index file records index information of each piece of training data in the training data set in a first order, wherein the index information indicates a position of the training data in the training data set;

[0010] In each round of training the target deep learning model, randomly disrupting the order of the index information in the target index file to obtain a target index file in which the index information is arranged in a second order, wherein the first order is different from the second order;

[0011] According to the second order, target index information belonging to the same read batch is determined from the target index file;

[0012] For each read batch, the target training data corresponding to the target index information is read from the training data set and input into the target deep learning model to perform this round of training on the target deep learning model.

[0013] In a second aspect, an embodiment of the present application provides a data access device, the device comprising an acquisition unit, a determination unit, a scrambling unit and a reading unit:

[0014] The acquisition unit is used to acquire a training data set for training a target deep learning model;

[0015] The determining unit is used to determine a target index file according to the training data set, wherein the target index file records index information of each training data in the training data set in a first order, and the index information indicates a position of the training data in the training data set;

[0016] The scrambling unit is used to randomly scramble the order of the index information in the target index file during each round of training the target deep learning model, so as to obtain a target index file in which the index information is arranged in a second order, wherein the first order is different from the second order;

[0017] The determining unit is further configured to determine target index information belonging to the same read batch from the target index file according to the second sequence;

[0018] The reading unit is used to read the target training data corresponding to the target index information from the training data set for each reading batch, and input the target training data into the target deep learning model to perform this round of training on the target deep learning model.

[0019] In a third aspect, an embodiment of the present application provides a device for data access, the device comprising a processor and a memory:

[0020] The memory is used to store program code and transmit the program code to the processor;

[0021] The processor is configured to execute the method described in the first aspect according to instructions in the program code.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect.

[0023] It can be seen from the above technical solution that in the process of deep learning, multiple rounds of training are usually performed on the entire training data set. The present application can propose a method for quickly accessing a training data set. Specifically, a training data set for training a target deep learning model is obtained, and a target index file is determined based on the training data set. The target index file records the index information of each training data in the training data set in a first order, and the index information indicates the position of the training data in the training data set. In each round of training of the target deep learning model, the order of the index information in the target index file is randomly disrupted to obtain a target index file in which the index information is arranged in a second order, wherein the first order is different from the second order. According to the second order, the target index information belonging to the same read batch is determined from the target index file, so that for each read batch, the target training data corresponding to the target index information is read from the training data set and input into the target deep learning model to perform the round of training on the target deep learning model, thereby ensuring that each round of training reads part of the required data, and avoiding a large amount of memory resource waste caused by loading the entire training data set. By randomly disrupting the order of index information in the target index file in each round of training, random access to each training data in the training data set can be achieved based on the index. On this basis, operations such as random sorting of training data can be converted into operations on index information, so that only smaller index information and a small amount of training data required for each step of training need to be stored in the memory. Compared with loading the entire training data set, memory usage is significantly reduced and the efficiency of training data reading can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 An algorithm flow chart of a data access method provided for related technologies;

[0026] Figure 2 A schematic diagram of an application scenario of a data access method provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of a data access method provided in an embodiment of the present application;

[0028] Figure 4 An algorithm flow chart of a data access method provided in an embodiment of the present application;

[0029] Figure 5 An overall framework diagram of a data access method provided in an embodiment of the present application;

[0030] Figure 6 An algorithm flow chart for reading training data through an index search module provided in an embodiment of the present application;

[0031] Figure 7 A flowchart of an algorithm for constructing a target index file provided in an embodiment of the present application;

[0032] Figure 8 A flowchart of a data access method provided in an embodiment of the present application;

[0033] Fig. 9 A structural diagram of a data access device provided in an embodiment of the present application;

[0034] Fig.10 A structural diagram of a terminal device provided in an embodiment of the present application;

[0035] Fig.11 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0037] In some related technologies, the entire training data set is read into memory, and then the training data in the training data set is randomly shuffled before each round of training, and a small portion of the training data in the shuffled training data set is read in turn and input into the model, see Figure 1 shown.

[0038] Wherein, e indicates that the current training round is the eth round, e starts counting from 0, when e=0, it indicates the start of the first round of training, and E indicates the total number of training rounds. After the training starts, that is, when e=0 (see S101), the entire training data set is read into the memory (see S102). If e<total number of training rounds E (see S103), it means that the training has not yet ended, so it is necessary to randomly shuffle the training data in the entire training data set (see S104), and read a small part of the training data in the shuffled training data set in turn and input it into the model (see S105), e+=1 (see S106), and re-execute S103.

[0039] However, as the data used in model training grows, the large amount of training data may not be fully loaded into the memory. In addition, only a small part of the entire training data set is actually used in each step of model training, resulting in low memory utilization and a large amount of memory resources wasted.

[0040] To this end, the embodiments of the present application provide a data access method, device, equipment and storage medium to ensure that each round of training reads part of the required data, avoiding a large amount of memory resource waste caused by loading the entire training data set. By randomly disrupting the order of index information in the target index file in each round of training, random access to each training data in the training data set based on the index is achieved, and on this basis, operations such as random sorting of training data can be converted into operations on index information, so that only a small amount of index information and a small amount of training data required for each step in training need to be stored in the memory, which significantly reduces memory usage compared to loading the entire training data set, and can ensure the high efficiency of training data reading.

[0041] It should be noted that the embodiments of the present application involve the field of artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0042] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0043] The data access method provided in the embodiments of the present application can be widely used in machine learning / deep learning scenarios. For example, the data access method provided in the present application can be used as a training acceleration component to improve the training speed of the deep model and save users' time costs.

[0044] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0045] The method provided in the embodiments of the present application may also involve blockchain technology, such as the data access method disclosed in the present application, in which the training data set used in the deep learning process can be stored on the blockchain.

[0046] It should be noted that the method can be applied to a data processing device, which may be a terminal device. The terminal device may be, for example, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0047] The data processing device may also be a server, which may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0048] Of course, the data processing device may be a terminal device and a server, that is, the two cooperate to execute, and the terminal device and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0049] To facilitate understanding of the technical solution of the present application, the data access method provided in the embodiment of the present application is introduced below by taking a terminal device as an example in combination with actual application scenarios.

[0050] See also Figure 2 , Figure 2 This is a schematic diagram of an application scenario of the data access method provided in the embodiment of the present application, and the application scenario may include a terminal device 201 and a server 202. The terminal device 201 may be used to execute the data access method provided in the embodiment of the present application, and the server 202 may store a training data set.

[0051] Various training data can be stored on the server 202. When the terminal device 201 needs to train a certain model, such as a target deep learning model, the terminal device 201 can obtain a training data set for training the target deep learning model from the server 202.

[0052] In the training process of the target deep learning model, it is usually necessary to train multiple rounds on the entire training data set, and the order of the training data read in each round of training should be different to ensure the accuracy of the model training. This application converts operations such as random sorting of training data into operations on index information. Therefore, in this application, the terminal device 201 needs to determine the target index file based on the training data set. Among them, the target index file records the index information of each training data in the training data set in the first order, and the index information indicates the position of the training data in the training data set.

[0053] Then, in each round of training of the target deep learning model, the terminal device 201 randomly shuffles the order of the index information in the target index file to obtain a target index file in which the index information is arranged in a second order, and the first order is different from the second order, so as to achieve the purpose of reading the training data in different orders in the process of reading the training data based on the index information by shuffling the order of the index information. In this way, the terminal device 201 determines the target index information belonging to the same reading batch from the target index file according to the second order, and can divide the index information into batches, and each time reads the target training data corresponding to the target index information from the training data set and inputs it into the target deep learning model to perform the round of training on the target deep learning model.

[0054] Next, taking the data processing device being a terminal device as an example, the data access method provided in the embodiment of the present application will be described in detail in conjunction with the accompanying drawings.

[0055] See also Figure 3 , Figure 3 A flow chart of a data access method is shown, the method comprising:

[0056] S301. Obtain a training data set for training a target deep learning model.

[0057] When a terminal device needs to train a certain model, such as a target deep learning model, the terminal device can obtain a training data set for training the target deep learning model from a server.

[0058] Among them, the training data set used to train the target deep learning model may include one or more.

[0059] S302: Determine a target index file according to the training data set.

[0060] In the training process of the target deep learning model, it is usually necessary to train multiple rounds on the entire training data set, and the order of the training data read in each round of training should be different to ensure the accuracy of the model training. This application converts operations such as random sorting of training data into operations on index information. Therefore, in this application, the terminal device needs to determine the target index file based on the training data set.

[0061] The target index file records the index information of each piece of training data in the training data set in a first order, and the index information indicates the position of the training data in the training data set.

[0062] In a possible implementation, the index information is represented by the byte offset of the training data in the training data set, and the byte offset may be the byte number of the first byte of a training data in the training data set. For example, the training data set includes multiple training data, and the byte length of the first training data is the 1st to 20th bytes in the training data set, then the first byte of the first training data is the 1st byte in the training data set, and its byte offset may be 1; the byte length of the second training data is the 21st to 50th bytes in the training data set, then the first byte of the second training data is the 21st byte in the training data set, and its byte offset is 21, and so on.

[0063] In order to ensure that a complete piece of training data can be read each time the training data is read according to the index information, the index information can also carry a line break character, or the training length of the corresponding training data, so that the end position of the training data in the training data set is known when reading the training data.

[0064] It should be noted that, in this embodiment, the target index file may already exist or may not yet be constructed. Based on this, there are multiple ways to determine the target index file. That is, in one possible implementation, the target index file may be determined based on the training data set by searching the target index file corresponding to the training data set in the index file storage area, that is, determining whether the target index file already exists (see Figure 4As shown in S402, e=0 shown in S401 may indicate that the training has just started, that is, the first round of training has just been carried out. If the target index file is not found in the index file storage area, a target index file corresponding to the training data set is constructed (see Figure 4 If the target index file is found in the index file storage area, the target index file is directly loaded (see S403). Figure 4 (as shown in S404).

[0065] It should be noted that, in the case where a target index file corresponding to a training data set needs to be constructed, the method provided in the embodiment of the present application may include two parts: a data preprocessing process and a model training process. Figure 5 As shown. The data preprocessing process can generate a target index file corresponding to the training data set through the index generation module 501, whose input is the training data set and output is the target index file corresponding to the training data set. The model training process can search for the target training data corresponding to the target index information determined subsequently through the index search module 502, and input them into the target deep learning model in sequence.

[0066] S303. In each round of training the target deep learning model, randomly disrupt the order of the index information in the target index file to obtain a target index file in which the index information is arranged in a second order.

[0067] In each round of training of the target deep learning model, the order of the index information in the target index file can be randomly disrupted to obtain a target index file in which the index information is arranged in a second order, thereby replacing the random sorting of the training data in the related art. The first order is different from the second order.

[0068] For example, the order of the index information in the target index file determined in S302 is index X, index Y, index Z, ... (first order), where index X is the index information corresponding to the first piece of training data, index Y is the index information corresponding to the second piece of training data, index Z is the index information corresponding to the third piece of training data, .... After randomly disrupting the order of the index information in the target index file, assuming that the obtained second order is index Z, index X, index Y, ..., it is related to changing the order of reading the training data when reading the training data.

[0069] It should be noted that in the actual algorithm, Figure 4Where e indicates that the current training round is the eth round, e starts counting from 0, when e=0 it indicates the start of the first round of training, and E indicates the total number of training rounds. After the training starts, that is, when e=0 (see S401), after obtaining the target index file based on S402 or S403, it can be determined whether e is less than the total number of training rounds E (see S405). If e is less than the total number of training rounds E, it means that the training has not yet ended, so the index information in the target index file needs to be randomly shuffled (see S406).

[0070] S304: Determine target index information belonging to the same read batch from the target index file according to the second order.

[0071] S305. For each read batch, read the target training data corresponding to the target index information from the training data set and input it into the target deep learning model to perform this round of training on the target deep learning model.

[0072] In each training step, since only a small part of the entire training data set is used, the target index information belonging to the same read batch can be determined from the target index file according to the needs of each training step. Belonging to the same read batch means that the training data corresponding to this read batch is needed in this training step. For example, several adjacent index information can be regarded as a batch. In this way, the corresponding training data can be read from the training data set according to the target index information and then input into the target deep learning model for training (see Figure 4 (as shown in S407).

[0073] Since the next round of training may be carried out, e+=1 (see Figure 4 As shown in S408, the current training round number is increased by 1 to enter the next round of training, and S405 is executed again. If the current training round number e=0, e+=1 and then e=1, e=1 is compared with the total training round number E in S405.

[0074] It should be noted that, in this embodiment, Figure 5 The index search module 502 shown inputs the scrambled index information into the index search module 502 in sequence, and the index search module 502 searches for the target training data record content corresponding to the target index in the training data set and returns it. The returned target training data record will be further input into the target deep learning model for model training.

[0075] For example, the determined target index information is index Z, index X, and index Y, then the target training data output by the index search module 502 are the third training data, the first training data, and the second training data..., respectively, and the third training data, the first training data, and the second training data... are input into the target deep learning model in sequence.

[0076] It can be seen from the above technical solution that in the process of deep learning, multiple rounds of training are usually performed on the entire training data set. The present application can propose a method for quickly accessing a training data set. Specifically, a training data set for training a target deep learning model is obtained, and a target index file is determined based on the training data set. The target index file records the index information of each training data in the training data set in a first order, and the index information indicates the position of the training data in the training data set. In each round of training of the target deep learning model, the order of the index information in the target index file is randomly disrupted to obtain a target index file in which the index information is arranged in a second order, wherein the first order is different from the second order. According to the second order, the target index information belonging to the same read batch is determined from the target index file, so that for each read batch, the target training data corresponding to the target index information is read from the training data set and input into the target deep learning model to perform the round of training on the target deep learning model, thereby ensuring that each round of training reads part of the required data, and avoiding a large amount of memory resource waste caused by loading the entire training data set. By randomly disrupting the order of index information in the target index file in each round of training, random access to each training data in the training data set can be achieved based on the index. On this basis, operations such as random sorting of training data can be converted into operations on index information, so that only smaller index information and a small amount of training data required for each step of training need to be stored in the memory. Compared with loading the entire training data set, memory usage is significantly reduced and the efficiency of training data reading can be guaranteed.

[0077] Compared with the related art that randomly divides the entire training data set into multiple small data sets and only loads one small data set into the memory each time, the method provided in the embodiment of the present application can not only solve the problems of large memory usage and low memory utilization to a certain extent, but also compared with the related art that can only achieve random shuffling within each small data set and between each small data set, but cannot achieve random shuffling of all data, the method provided in the embodiment of the present application achieves the purpose of randomly shuffling all training data in the training data set by randomly shuffling all index information in the target index file, thereby ensuring the performance of the trained model.

[0078] At the same time, compared with the related art in which the entire training data set is randomly shuffled and a copy of the training data set using a different shuffle order is created in each round, the method provided in the embodiment of the present application randomly shuffles the index information, and even if it is saved, a copy of the target index file is saved, which greatly reduces the file size. In addition, this method only needs to traverse the training data set when generating the target index file, and then the generated target index file can be saved to the disk. When the same training data set needs to be read in the future, only the generated target index file needs to be read, and there is no need to traverse the training data set again. The index file is very small compared to the entire training data set, so the reading speed of the training data set can be significantly improved.

[0079] Next, the specific method of creating the target index file when it is necessary to create the target index file will be introduced in detail.

[0080] In this embodiment, the training data set used to train the target deep learning model may include one or more training data sets. When the number of training data sets is different, the method of generating the target index file is slightly different.

[0081] If the training data set includes one, a possible implementation method for constructing a target index file corresponding to the training data set may be to traverse the training data set, record the position of each training data in the training data set, and construct the target index file according to the position.

[0082] Among them, since the index information can be represented by the byte offset of the training data in the training data set, that is, the position of each training data in the training data set can be represented by the byte offset of the training data in the training data set, the byte offsets of all training data in the training data set can constitute an array A[i], where i represents the i-th training data, that is, use an increasing integer variable i, initialized to 0, and then traverse each training data in the training data set, and assign A[i] to the byte offset of the training data in the training data set.

[0083] If the training data set includes multiple data, a possible implementation method for constructing the target index file corresponding to the training data set is to traverse the multiple training data sets respectively, record the position of each training data in the corresponding training data set, and construct a first array. Open each training data set in turn, and record the file handle corresponding to the training data set in a second array. Construct a third array according to the position subscript of the file handle corresponding to each training data set in the second array. Construct the target index file according to the first array, the second array, and the third array.

[0084] The first array is the array A[i] mentioned above. The byte offset recorded in the array A[i] is the byte offset of the i-th training data in a training data set in the corresponding training data set. The second array can be represented by the array C[i]. Assume that M represents the number of training data set files to be read, and N j represents the number of training data contained in the jth training data set (j∈{1,…,M}), N represents the total number of training data contained in all training data sets, Then the length of array C[i] is M, and the length of array A[i] is N. The third array can be represented by B[i]. B[i] is assigned the position subscript of the file handle of the current training data set in array C[i]. After traversing each training data, i is increased by 1. After traversing all training data sets, (A[i], B[i]) can be used as the index information of the i-th training data. Finally, array A[i], array B[i] and array C[i] are saved in the index file for subsequent reading. This process needs to traverse all training data records once, and the time complexity is O(N). O(N) is a function that characterizes the time complexity of the algorithm. N is the total number of training data contained in all the above training data sets, that is, the time complexity is related to the total number of training data.

[0085] In this case, since the training data sets include multiple ones, the first array records the byte offset of the i-th training data in a training data set in the corresponding training data set. Therefore, in order to be able to read the correct training data according to the first array representing the index information, it is necessary to first determine the training data set where the training data to be read is located from multiple training data sets, and then the training data can be read according to the byte offset of the training data to be read in the training data set. Therefore, in one possible implementation method, the method of reading the target training data corresponding to the target index information from the training data set and inputting it into the target deep learning model can be to determine the target training data set where the target training data is located from multiple training data sets according to the third array and the second array, and then read the target training data corresponding to the target index information from the target training data set according to the first array and input it into the target deep learning model.

[0086] The algorithm flow chart for reading training data through the index lookup module can be found in Figure 6 As shown, the index information (A[i], B[i]) is input to the index search module (see Figure 6 The index search module first finds the file handle C[B[i]] of the training data set where the training data to be read is located, that is, determines the target training data set where the target training data is located (see Figure 6Then, according to its byte offset A[i] in the target data set, the file pointer corresponding to the target data set is moved to the beginning of the target training data (for example, it can be the position indicated by the byte offset, see Figure 6 Then, a complete target training data is read from this position (see S603). Figure 6 ), and returns the target training data (see Figure 6 As shown in S605 in the figure, this process can be implemented by fseek(handle, A[i]). In the above process, the search operation and fseek operation on the file handle are both random access operations, and the time complexity is O(1). O(1) is a fixed constant, indicating that the time complexity is independent of the amount of training data and is a fixed constant.

[0087] For a scenario where there is only one training data set, that is, M=1, only the array A[i] can be used to directly operate on the file handle of the only training data set.

[0088] In a possible implementation, when constructing the target index file, a hash table can be constructed using the file path of each training data set as a key and the position subscript of the file handle corresponding to each training data set in the second array as a value. The hash table can be represented by an array H, and the hash table H stores the mapping from the file path to the position subscript of the file handle. Thus, the target index file can be constructed according to the first array, the second array, the third array and the hash table, that is, the array A[i], the array B[i] and the hash table H are saved in the target index file.

[0089] In this embodiment, after the target index file is constructed, if the training data needs to be re-read in the future, the constructed target index file can be used, and there is no need to re-determine the target index file. It is only necessary to read the constructed array A[i], array B[i] and hash table H. However, since the file handle stored in array C[i] may change, it is sufficient to reconstruct array C[i]. When reconstructing array C[i], for the file path of each training data set, it is necessary to ensure that its position in array C[i] is subscripted to the value recorded in hash table H. Therefore, when re-reading the training data to input the target deep learning model, the second array can be reconstructed according to the hash table.

[0090] If you need to build array A[j], array B[j], array C[j] and hash table H when building the target index file, assume that M represents the number of training data set files that need to be read, N jrepresents the number of training data contained in the jth training data set (j∈{1,…,M}), N represents the total number of training data contained in all training data sets, The algorithm flow for building the target index file can be found in Figure 7 As shown: initialize array A[j], array B[j], array C[j] and hash table H (see S701); input the path array D of the training data set and the number of training data sets M (see S702); starting from j=0 (see S703), determine whether j is less than M (see S704), if so, open the training data set, and record the file handle corresponding to the training data set in array C[j] (see S705), which can be implemented by C[j]=open(D[j]). Hash table H stores the mapping of file path to file handle position index, which can be implemented by H[D[j]]=j. Increase j by 1, that is, j+=1 (see S706). If the result of S704 is no, then i=0, j=0 (see S707) to determine array A[i] and array B[i]. Determine whether j is less than M (see S708). If so, determine whether array C[j] has been traversed (see S709). If not, determine the byte offset of the training data in the corresponding training data set (see S710). This process can be implemented by A[i]=ftell(C[j]). The ftell function is used to return the position of the file pointer in the current training data set. Assign B[i] to the position subscript of the file handle of the current training data set in array C[i] (see S711). Move the file pointer of C[j] to point to the end of the next training data (see S712). i+=1 (see S713), j+=1 (see S714), save array A[i], array B[i] and hash table H to the target index file (see S715). If the determination result of S708 is no, directly execute S715; if the determination result of S709 is no, directly execute S714.

[0091] It should be noted that, in this embodiment, the format of the target index file can be expanded or simplified according to the characteristics of the training data and the needs of the actual scenario.

[0092] For example, if you only need to read one training data set, you do not need to build and save array B[i], array C[i] and hash table H. You only need to build and save array A[i]. If you can ensure that the order of reading the training data set is the same each time, you do not need to build and save hash table H. You only need to build and save array A[i], B[i] and array C[i]. If you do not need to save the target index file for subsequent use and only need to use the target index file in a single training, you do not need to build array C[i] and hash table H. You can directly save the file handle of the training data set in array C[i]. In addition, you can also directly store the file path of the training data set in array B[i] instead of the position index, and use a hash table H to directly save the mapping from the file path of the training data set to the file handle of the training data set.

[0093] Next, the data access method provided by the embodiment of the present application is described in detail in combination with actual application scenarios. The application scenario is that in the relevant scenarios of deep learning, the method is used as a training acceleration component. When training a model trained based on deep learning, such as a target deep learning model, the training data required for training the target deep learning model can be read based on the method. For example, the training data set includes one, see Figure 8 , the method comprising:

[0094] S801. The terminal device obtains a training data set for training a target deep learning model.

[0095] S802, the terminal device determines whether the target index file corresponding to the training data set already exists, if so, execute S803, if not, execute S804.

[0096] S803: The terminal device loads the target index file.

[0097] S804: The terminal device calls an index generation module to create a target index file.

[0098] When creating a target index file, an array A[i] may be constructed and the array A[i] may be saved to the target index file, where i represents the i-th piece of training data.

[0099] S805. During each round of training of the target deep learning model, the terminal device randomly shuffles the order of index information in the target index file to obtain a target index file in which the index information is arranged in a second order.

[0100] S806: The terminal device selects a plurality of adjacent index information as target index information of the same read batch according to the second order.

[0101] S807. Read the target training data corresponding to the target index information from the training data set and input it into the target deep learning model to complete the training.

[0102] Based on the data access method provided in the above embodiment, the present application embodiment provides a data access device, see Fig. 9 As shown, the device includes an acquisition unit 901, a determination unit 902, a scrambling unit 903 and a reading unit 904:

[0103] The acquisition unit 901 is used to acquire a training data set for training a target deep learning model;

[0104] The determining unit 902 is used to determine a target index file according to the training data set, wherein the target index file records index information of each training data in the training data set in a first order, and the index information indicates a position of the training data in the training data set;

[0105] The scrambling unit 903 is used to randomly scramble the order of the index information in the target index file during each round of training the target deep learning model, so as to obtain a target index file in which the index information is arranged in a second order, wherein the first order is different from the second order;

[0106] The determining unit 902 is further configured to determine target index information belonging to the same read batch from the target index file according to the second sequence;

[0107] The reading unit 904 is used to read the target training data corresponding to the target index information from the training data set for each reading batch, and input the target training data into the target deep learning model to perform this round of training on the target deep learning model.

[0108] In one implementation, the determining unit 902 is configured to:

[0109] Searching for a target index file corresponding to the training data set in an index file storage area;

[0110] If the target index file is not found in the index file storage area, a target index file corresponding to the training data set is constructed.

[0111] In one implementation, if the training data set includes one, the determining unit 902 is configured to:

[0112] Traversing the training data set, and recording the position of each piece of training data in the training data set;

[0113] The target index file is constructed according to the position.

[0114] In one implementation, if the training data set includes multiple data, the determining unit 902 is used to:

[0115] Traversing the plurality of training data sets respectively, recording the position of each piece of training data in the corresponding training data set, and constructing a first array;

[0116] Open each of the training data sets in sequence, and record the file handle corresponding to the training data set in a second array;

[0117] Construct a third array according to the position subscript of the file handle corresponding to each of the training data sets in the second array;

[0118] The target index file is constructed according to the first array, the second array and the third array.

[0119] In one implementation, the reading unit 904 is used to:

[0120] Determine, according to the third array and the second array, a target training data set in which the target training data is located from the plurality of training data sets;

[0121] According to the first array, the target training data corresponding to the target index information is read from the target training data set and input into the target deep learning model.

[0122] In one implementation, the determining unit 902 is further configured to:

[0123] Constructing a hash table using the file path of each training data set as a key and the position subscript of the file handle corresponding to each training data set in the second array as a value;

[0124] The step of constructing the target index file according to the first array, the second array, and the third array includes:

[0125] The target index file is constructed according to the first array, the second array, the third array and the hash table.

[0126] In one implementation, the determining unit 902 is further configured to:

[0127] When the training data is re-read and input into the target deep learning model, the second array is reconstructed according to the hash table.

[0128] In one implementation, the reading unit 904 is further configured to:

[0129] If the target index file is found in the index file storage area, the target index file is loaded.

[0130] The embodiment of the present application further provides a device for data access, which may be a data processing device for executing a data access method, and which may be a terminal device. For example, the terminal device is a smart phone:

[0131] Fig.10 The block diagram shows a partial structure of a smart phone related to the terminal device provided in the embodiment of the present application. Fig.10 The smartphone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. The input unit 1030 may include a touch panel 1031 and other input devices 1032, the display unit 1040 may include a display panel 1041, and the audio circuit 1060 may include a speaker 1061 and a microphone 1062. Those skilled in the art will appreciate that Fig.10 The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0132] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0133] The processor 1080 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0134] In this embodiment, the processor 1080 in the terminal device may perform the following steps:

[0135] Obtain a training dataset for training the target deep learning model;

[0136] Determine a target index file according to the training data set, wherein the target index file records index information of each piece of training data in the training data set in a first order, wherein the index information indicates a position of the training data in the training data set;

[0137] In each round of training the target deep learning model, randomly disrupting the order of the index information in the target index file to obtain a target index file in which the index information is arranged in a second order, wherein the first order is different from the second order;

[0138] According to the second order, target index information belonging to the same read batch is determined from the target index file;

[0139] For each read batch, the target training data corresponding to the target index information is read from the training data set and input into the target deep learning model to perform this round of training on the target deep learning model.

[0140] The present application also provides a server. Fig.11 As shown, Fig.11The structural diagram of the server 1100 provided in the embodiment of the present application, the server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1122 (for example, one or more processors) and memory 1132, one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1122 can be configured to communicate with the storage medium 1130 and execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0141] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0142] In this embodiment, the steps implemented by the server can be based on Fig.11 The structure of the server is implemented.

[0143] According to one aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the data access method described in the above-mentioned embodiments.

[0144] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations of the above embodiments.

[0145] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (RandomAccess Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0150] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technical members in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data access method, characterized in that: The method comprises: Obtain a training dataset for training the target deep learning model; According to the training data set, a target index file is determined, the target index file records the index information of each training data in the training data set in a first order, the index information indicates the position of the training data in the training data set, wherein the method for creating the target index file comprises: if the training data set includes one, the training data set is traversed, the position of each training data in the training data set is recorded, and the target index file is constructed according to the position; if the training data set includes multiple, the multiple training data sets are traversed respectively, the position of each training data in the corresponding training data set is recorded, a first array is constructed, each of the training data sets is opened in turn, and the file handle corresponding to the training data set is recorded in a second array, a third array is constructed according to the position subscript of the file handle corresponding to each training data set in the second array, and the target index file is constructed according to the first array, the second array and the third array; In each round of training the target deep learning model, randomly disrupting the order of the index information in the target index file to obtain a target index file in which the index information is arranged in a second order, wherein the first order is different from the second order; According to the second order, target index information belonging to the same read batch is determined from the target index file; For each read batch, the target training data corresponding to the target index information is read from the training data set and input into the target deep learning model to perform this round of training on the target deep learning model.

2. The method according to claim 1, characterized in that: The step of determining a target index file according to the training data set includes: Searching for a target index file corresponding to the training data set in an index file storage area; If the target index file is not found in the index file storage area, a target index file corresponding to the training data set is constructed.

3. The method according to claim 1, characterized in that The step of reading target training data corresponding to the target index information from the training data set and inputting the target deep learning model includes: Determine, according to the third array and the second array, a target training data set in which the target training data is located from the plurality of training data sets; According to the first array, the target training data corresponding to the target index information is read from the target training data set and input into the target deep learning model.

4. The method according to claim 1, characterized in that The method further comprises: Constructing a hash table using the file path of each training data set as a key and the position subscript of the file handle corresponding to each training data set in the second array as a value; The step of constructing the target index file according to the first array, the second array, and the third array includes: The target index file is constructed according to the first array, the second array, the third array and the hash table.

5. The method according to claim 4, characterized in that The method further comprises: When the training data is re-read and input into the target deep learning model, the second array is reconstructed according to the hash table.

6. The method according to claim 2, characterized in that The method further comprises: If the target index file is found in the index file storage area, the target index file is loaded.

7. The method according to any one of claims 1 to 6, characterized in that: The index information is represented by the byte offset of the training data in the training data set.

8. A data access device, characterized in that: The device comprises an acquisition unit, a determination unit, a scrambling unit and a reading unit: The acquisition unit is used to acquire a training data set for training a target deep learning model; The determination unit is used to determine a target index file according to the training data set, wherein the target index file records index information of each training data in the training data set in a first order, and the index information indicates the position of the training data in the training data set, wherein the method for creating the target index file includes: if the training data set includes one, traversing the training data set, recording the position of each training data in the training data set, and constructing the target index file according to the position; if the training data set includes multiple, traversing the multiple training data sets respectively, recording the position of each training data in the corresponding training data set, constructing a first array, opening each of the training data sets in turn, and recording the file handle corresponding to the training data set in a second array, constructing a third array according to the position subscript of the file handle corresponding to each training data set in the second array, and constructing the target index file according to the first array, the second array, and the third array; The scrambling unit is used to randomly scramble the order of the index information in the target index file during each round of training the target deep learning model, so as to obtain a target index file in which the index information is arranged in a second order, wherein the first order is different from the second order; The determining unit is further configured to determine target index information belonging to the same read batch from the target index file according to the second sequence; The reading unit is used to read the target training data corresponding to the target index information from the training data set for each reading batch, and input the target training data into the target deep learning model to perform this round of training on the target deep learning model.

9. The device according to claim 8, characterized in that The determining unit is used for: Searching for a target index file corresponding to the training data set in an index file storage area; If the target index file is not found in the index file storage area, a target index file corresponding to the training data set is constructed.

10. The device according to claim 8, characterized in that The reading unit is used to Determine, according to the third array and the second array, a target training data set in which the target training data is located from the plurality of training data sets; According to the first array, the target training data corresponding to the target index information is read from the target training data set and input into the target deep learning model.

11. The device according to claim 8, characterized in that The determining unit is further configured to: Constructing a hash table using the file path of each training data set as a key and the position subscript of the file handle corresponding to each training data set in the second array as a value; The step of constructing the target index file according to the first array, the second array, and the third array includes: The target index file is constructed according to the first array, the second array, the third array and the hash table.

12. The device according to claim 11, characterized in that The determining unit is further configured to: When the training data is re-read and input into the target deep learning model, the second array is reconstructed according to the hash table.

13. The device according to claim 9, characterized in that The reading unit is further used for: If the target index file is found in the index file storage area, the target index file is loaded.

14. A device for data access, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1 to 7 according to the instructions in the program code.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the method according to any one of claims 1 to 7.

16. A computer program product, characterized in that The computer program product comprises computer instructions, and a processor of a computer device executes the computer instructions, so that the computer device performs the method according to any one of claims 1 to 7.

Images