Data loading method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data loading method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: in response to a received data acquisition instruction, determining a to-be-loaded data group matched with the data acquisition instruction; determining a first data packet from the plurality of to-be-loaded data packets according to the data packet identifier, reading the first data packet from a first preset cache region, and loading a second data packet adjacent to the first data packet to a second preset cache region; and under the condition that the data volume of the second data packet loaded to the second preset cache region is greater than a preset threshold value, loading a third data packet adjacent to the second data packet to the first preset cache region, and reading the second data packet from the second preset cache region until the reading of the to-be-loaded data group is completed. By adopting the method, the efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data loading method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence, large models have been widely used in question-answering dialogue, semantic understanding, text writing, etc. due to their strong emergence, generalization, multi-task, and migration capabilities. Usually, a large model trained with a large amount of data can achieve good performance.

[0003] In the related art, due to the large amount of data, the data used for training is generally stored in the hard disk. When training a large model, the data can be loaded into the memory for processing and model training, or the data can be directly loaded from the hard disk for model training. However, the amount of data that can be loaded in the former is limited by the memory capacity of the server and cannot be used for large-scale data training, while the latter is limited by the hard disk reading speed and the data loading efficiency is low. Summary of the invention

[0004] Based on this, it is necessary to provide a data loading method, apparatus, computer device, computer-readable storage medium and computer program product that can improve efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data loading method, comprising:

[0006] In response to receiving a data acquisition instruction, determining a data group to be loaded that matches the data acquisition instruction, wherein the data group to be loaded includes a plurality of data packets to be loaded, and each of the data packets to be loaded corresponds to a data packet identifier;

[0007] According to the data packet identifier, determine a first data packet from the multiple data packets to be loaded, read the first data packet from a first preset buffer area, and load a second data packet adjacent to the first data packet into a second preset buffer area;

[0008] When the amount of data loaded from the second data packet into the second preset cache area is greater than a preset threshold, a third data packet adjacent to the second data packet is loaded into the first preset cache area, and the second data packet is read from the second preset cache area until the reading of the data group to be loaded is completed, wherein the data group to be loaded is used to train to obtain a target model.

[0009] In one embodiment, reading the first data packet from the first preset buffer area includes:

[0010] If the first data packet is not loaded in the first preset buffer area, loading the first data packet into the first preset buffer area, and reading the first data packet from the first preset buffer area; and / or,

[0011] In the case where the first data packet has been loaded in the first preset buffer area, reading the first data packet from the first preset buffer area; and / or,

[0012] In the case where the first data packet is being loaded into the first preset buffer area, when the loading of the first data packet is completed, the first data packet is read from the first preset buffer area.

[0013] In one embodiment, determining the to-be-loaded data group matching the data acquisition instruction includes:

[0014] Determining a target data group identifier according to the data acquisition instruction;

[0015] The data group corresponding to the target data group identifier is determined as the data group to be loaded.

[0016] In one embodiment, when the amount of data of the second data packet loaded into the second preset buffer area is greater than a preset threshold, loading a third data packet adjacent to the second data packet into the first preset buffer area includes:

[0017] When the amount of data of the second data packet loaded into the second preset buffer area is greater than a preset threshold, determining a third data packet adjacent to the second data packet;

[0018] If the third data packet exists, the third data packet is loaded into the first preset buffer area.

[0019] In one embodiment, the method of determining the data group to be loaded includes:

[0020] Acquire original text data, wherein the original text data includes a plurality of characters;

[0021] Based on the correspondence between characters and token values, generating token sequence data according to the original text data;

[0022] The token sequence data is divided according to a preset data amount to generate a plurality of data packets to be loaded, thereby obtaining a data group to be loaded.

[0023] In one embodiment, generating token sequence data based on the correspondence between characters and token values ​​according to the original text data includes:

[0024] Preprocessing the original text data to obtain target text data;

[0025] Based on the correspondence between characters and token values, characters in the target text data are converted into token values ​​to obtain token sequence data.

[0026] In a second aspect, the present application also provides a data loading device, comprising:

[0027] A determination module, configured to determine, in response to receiving a data acquisition instruction, a data group to be loaded that matches the data acquisition instruction, wherein the data group to be loaded includes a plurality of data packets to be loaded, and each of the data packets to be loaded corresponds to a data packet identifier;

[0028] a loading module, configured to determine a first data packet from the plurality of data packets to be loaded according to the data packet identifier, read the first data packet from a first preset buffer area, and load a second data packet adjacent to the first data packet into a second preset buffer area;

[0029] A reading module is used to load a third data packet adjacent to the second data packet into the first preset cache area when the amount of data loaded from the second data packet into the second preset cache area is greater than a preset threshold, and read the second data packet from the second preset cache area until the reading of the data group to be loaded is completed, wherein the data group to be loaded is used to train to obtain a target model.

[0030] In one embodiment, the loading module is further used for:

[0031] If the first data packet is not loaded in the first preset buffer area, loading the first data packet into the first preset buffer area, and reading the first data packet from the first preset buffer area; and / or,

[0032] In the case where the first data packet has been loaded in the first preset buffer area, reading the first data packet from the first preset buffer area; and / or,

[0033] In the case where the first data packet is being loaded into the first preset buffer area, when the loading of the first data packet is completed, the first data packet is read from the first preset buffer area.

[0034] In one embodiment, the determining module is further configured to:

[0035] Determining a target data group identifier according to the data acquisition instruction;

[0036] The data group corresponding to the target data group identifier is determined as the data group to be loaded.

[0037] In one embodiment, the reading module is further used for:

[0038] When the amount of data of the second data packet loaded into the second preset buffer area is greater than a preset threshold, determining a third data packet adjacent to the second data packet;

[0039] If the third data packet exists, the third data packet is loaded into the first preset buffer area.

[0040] In one embodiment, the device further comprises a module for determining the data group to be loaded, and the module for determining the data group to be loaded is further configured to:

[0041] Acquire original text data, wherein the original text data includes a plurality of characters;

[0042] Based on the correspondence between characters and token values, generating token sequence data according to the original text data;

[0043] The token sequence data is divided according to a preset data amount to generate a plurality of data packets to be loaded, thereby obtaining a data group to be loaded.

[0044] In one embodiment, the module for determining the data group to be loaded is further used for:

[0045] Preprocessing the original text data to obtain target text data;

[0046] Based on the correspondence between characters and token values, characters in the target text data are converted into token values ​​to obtain token sequence data.

[0047] In a third aspect, the embodiments of the present disclosure further provide a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0048] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0049] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0050] The above-mentioned data loading method, device, computer equipment, computer-readable storage medium and computer program product, when loading data, receive a data acquisition instruction, determine a data group to be loaded that matches the data acquisition instruction, and determine a first data packet from multiple data packets to be loaded in the data group to be loaded according to the data packet identifier, read the first data packet from the first preset cache area, load the second data packet adjacent to the first data packet into the second preset cache area, and when the amount of data loaded from the second data packet to the second preset cache area is greater than the preset threshold, continue to load the third data packet adjacent to the second data packet into the first preset cache area, and read the second data packet until the multiple data packets to be loaded in the data group to be loaded are all read, and the reading of the data group to be loaded is completed, which can be used for training to obtain a target model. The data is divided into data packets to be loaded according to the preset size, and the first preset cache area and the second preset cache area are set. When the data group to be loaded is obtained, the data packets are alternately loaded through the two cache areas. In the case of limited cache area, the high-speed loading of data can also be maintained, avoiding the problem of low data loading efficiency caused by insufficient memory, so that the high-speed data loading process is not limited by the memory size, and is suitable for more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A schematic diagram of a data loading method in one embodiment;

[0053] Figure 2 is a schematic diagram of a method for determining a data group to be loaded in one embodiment;

[0054] Figure 3 is a flow chart of a data loading method in another embodiment;

[0055] Figure 4 is a flow chart of a data loading method in another embodiment;

[0056] Figure 5 is a flow chart of a data loading method in another embodiment;

[0057] Figure 6 is a structural block diagram of a data loading device in one embodiment;

[0058] Figure 7FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] In one embodiment, Figure 1 As shown, a data loading method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0061] Step S110, in response to receiving the data acquisition instruction, determining a data group to be loaded that matches the data acquisition instruction, wherein the data group to be loaded includes a plurality of data packets to be loaded, and each of the data packets to be loaded corresponds to a data packet identifier;

[0062] Exemplarily, when reading data, the data may be loaded from the hard disk into the memory for reading. In response to receiving the data acquisition instruction, the data group to be loaded is determined according to the data acquisition instruction. In one example, the data group to be loaded includes data stored in the hard disk.

[0063] Optionally, the data acquisition instruction may be an instruction for acquiring the data group to be loaded from the hard disk. In one example, the data acquisition instruction may carry identification information of the data group to be loaded, so that the corresponding data group to be loaded can be determined according to the data acquisition instruction. In one example, the data acquisition instruction may also carry data identification information in a data packet corresponding to the data group to be loaded, such as an index in the data packet. In a possible implementation, the data acquisition instruction may carry data identification information, and the corresponding data group to be loaded is determined according to the data identification information.

[0064] In one example, the data group to be loaded includes multiple data packets to be loaded, each of which corresponds to a data packet identifier, wherein the data packet identifiers of the multiple data packets to be loaded in the data group to be loaded are different, which can be specifically determined according to actual application scenarios.

[0065] In a possible implementation, a plurality of data packets to be loaded may correspond to a preset sequence, and the corresponding data packet identifiers may be determined according to the preset sequence.

[0066] Step S120, determining a first data packet from the plurality of data packets to be loaded according to the data packet identifier, reading the first data packet from a first preset buffer area, and loading a second data packet adjacent to the first data packet into a second preset buffer area;

[0067] Exemplarily, the first data packet is determined according to the data packet identifier. In one example, the first data packet may be the first data packet determined according to the order of the data packet identifiers.

[0068] Optionally, two preset buffer areas are set in the memory, and the sizes of the two preset buffer areas can be determined according to actual application scenarios, wherein the sizes of the two preset buffer areas are both larger than the size of a data packet to be cached.

[0069] The first data packet is read from the first preset cache area. In one example, two preset cache areas can be queried to determine that the preset cache area loaded with the first data packet is the first preset cache area; if the first data packet is not loaded, one of the two preset cache areas can be determined as the first preset cache area based on the actual application scenario, and the first data packet can be loaded.

[0070] In one example, the next action of reading the first data packet is to read the second data packet, so at this time, the second data packet adjacent to the first data packet is loaded into the second preset buffer area to achieve zero-wait data reading. Exemplarily, the second data packet is the next data packet adjacent to the first data packet among the multiple data packets to be loaded.

[0071] Step S130, when the amount of data of the second data packet loaded into the second preset cache area is greater than a preset threshold, a third data packet adjacent to the second data packet is loaded into the first preset cache area, and the second data packet is read from the second preset cache area until the reading of the data group to be loaded is completed, wherein the data group to be loaded is used to train to obtain a target model.

[0072] Exemplarily, when the amount of data loaded by the second data packet into the second preset buffer area is greater than a preset threshold, the next data packet, i.e., the third data packet, may be loaded. The preset threshold may be a value determined according to an actual application scenario, for example, half of the amount of data of a single data packet may be set as the preset threshold.

[0073] The third data packet may be the next data packet adjacent to the second data packet among the multiple data packets to be loaded.

[0074] Optionally, when the second data packet is loaded, the second data packet is read from the second preset buffer area.

[0075] In one example, when the amount of data loaded from the third data packet into the first preset cache area is greater than a preset threshold, the next data packet can be loaded into the second preset cache area, and the loading and reading can be repeated in this way until multiple data packets to be loaded are loaded and read, and it is determined that the reading of the data group to be loaded is completed.

[0076] In one example, after the data in the preset buffer area is read, the preset buffer area is idle and the next data packet can be loaded. In one example, the data group to be loaded is data for model training, and the target model can be trained using the data group to be loaded.

[0077] Exemplarily, the data group to be loaded may be data obtained after processing and can be directly used for model training.

[0078] In the disclosed embodiment, when data is loaded, a data acquisition instruction is received, a data group to be loaded that matches the data acquisition instruction is determined, and according to the data packet identifier, a first data packet is determined from multiple data packets to be loaded in the data group to be loaded, the first data packet is read from the first preset cache area, and the second data packet adjacent to the first data packet is loaded into the second preset cache area. When the amount of data loaded from the second data packet to the second preset cache area is greater than the preset threshold, the third data packet adjacent to the second data packet is continuously loaded into the first preset cache area, and the second data packet is read until multiple data packets to be loaded in the data group to be loaded are read, and the reading of the data group to be loaded is completed, which can be used for training to obtain the target model. The data is divided into data packets to be loaded according to the preset size, and the first preset cache area and the second preset cache area are set. When the data group to be loaded is obtained, the data packets are alternately loaded through the two cache areas. In the case of limited cache area, the high-speed loading of data can also be maintained, avoiding the problem of low data loading efficiency caused by insufficient memory, so that the high-speed data loading process is not limited by the memory size and is suitable for more application scenarios; thereby effectively ensuring the amount of data and data acquisition efficiency during subsequent model training, and improving the model training efficiency.

[0079] In one embodiment, the reading the first data packet from the first preset buffer area includes:

[0080] If the first data packet is not loaded in the first preset buffer area, loading the first data packet into the first preset buffer area, and reading the first data packet from the first preset buffer area; and / or,

[0081] In the case where the first data packet has been loaded in the first preset buffer area, reading the first data packet from the first preset buffer area; and / or,

[0082] In the case where the first data packet is being loaded into the first preset buffer area, when the loading of the first data packet is completed, the first data packet is read from the first preset buffer area.

[0083] Optionally, when reading the first data packet from the first preset buffer area, it is necessary to first determine the loading status of the first preset buffer area.

[0084] In one example, if the first data packet is not loaded in the first preset buffer area, the first data packet is loaded from the hard disk to the first preset buffer area, and the first data packet is read after the loading is completed.

[0085] In one example, if the first data packet has been loaded into the first preset buffer area, the first data packet is directly read from the first preset buffer area.

[0086] In one example, if the first data packet is being loaded into the first preset buffer area, the first data packet can be read when the loading of the first data packet is completed. In one example, when the amount of data loaded from the first data packet into the first preset buffer area is greater than a preset threshold, the second data packet is loaded into the second preset buffer area to achieve zero waiting for reading the second data packet and ensure data reading efficiency.

[0087] In the disclosed embodiment, when reading the first data packet from the first preset cache area, the data loading status in the first preset cache area is first determined; if the first data packet is not loaded in the first preset cache area, the first data packet is loaded into the first preset cache area and then read; if the first data packet has been loaded in the first preset cache area, the first data packet is directly read; if the first data packet is being loaded in the first preset cache area, the first data packet is read after the loading is completed, thereby being able to read the first data packet according to the actual data loading status in the first preset cache area.

[0088] In one embodiment, determining the to-be-loaded data group that matches the data acquisition instruction includes:

[0089] Determining a target data group identifier according to the data acquisition instruction;

[0090] The data group corresponding to the target data group identifier is determined as the data group to be loaded.

[0091] Exemplarily, the data acquisition instruction corresponds to a target data group identifier, which is an identifier corresponding to the to-be-loaded data group to be read. The data group identifier can be a character, a number, etc., which can be determined according to the actual application scenario.

[0092] Determine the data group corresponding to the target data group identifier as the data group to be loaded. In one example, the hard disk may store multiple data groups, and different data groups correspond to different data group identifiers. The data group to be loaded may be determined by searching the hard disk through the target data group identifier.

[0093] In the embodiment of the present disclosure, when determining the data group to be loaded, the target data group identifier is determined according to the data acquisition instruction, and the data group to be loaded is determined according to the data group identifier, so that the data groups can be distinguished and searched by the data group identifier. When data loading is required, the data group to be loaded can be determined quickly and accurately, further improving the efficiency of data loading and being suitable for more application scenarios.

[0094] In one embodiment, when the amount of data of the second data packet loaded into the second preset buffer area is greater than a preset threshold, loading a third data packet adjacent to the second data packet into the first preset buffer area includes:

[0095] When the amount of data of the second data packet loaded into the second preset buffer area is greater than a preset threshold, determining a third data packet adjacent to the second data packet;

[0096] If the third data packet exists, the third data packet is loaded into the first preset buffer area.

[0097] Exemplarily, when loading the next data packet in advance, it is necessary to first determine whether the next data packet exists.

[0098] Optionally, when the amount of data loaded from the second data packet into the second preset buffer area is greater than a preset threshold, a third data packet adjacent to the second data packet is determined. If the third data packet does not exist, there is no need to load the third data packet.

[0099] In the case where the third data packet exists, it can be considered that data loading and reading need to be continued at this time, and the third data packet is loaded into the first preset buffer area.

[0100] In the embodiment of the present disclosure, when the amount of loaded data in the second data packet is greater than a preset threshold, the next data packet is determined and loaded into another cache area, namely, the first preset cache area, thereby ensuring that data reading and data loading are performed alternately and orderly. When the second data packet is loaded to the preset threshold, the next data packet is loaded in advance, and after the reading of the second data packet is completed, the third data packet can be read with zero waiting time, thereby ensuring the efficiency and orderliness of data loading and reading.

[0101] In one embodiment, the method for determining the data group to be loaded includes:

[0102] Acquire original text data, wherein the original text data includes a plurality of characters;

[0103] Based on the correspondence between characters and token values, generating token sequence data according to the original text data;

[0104] The token sequence data is divided according to a preset data amount to generate a plurality of data packets to be loaded, thereby obtaining a data group to be loaded.

[0105] Exemplarily, the data group to be loaded is data obtained after processing the original text data. In one example, the original text data includes multiple characters, and the multiple characters usually have a front-to-back relationship. Based on the correspondence between the characters and the token values, the token sequence data corresponding to the original text data is generated.

[0106] In one example, the correspondence between characters and token values ​​can be determined in advance according to actual application scenarios, and different characters correspond to different token values. Exemplarily, the token sequence data is data that can be directly used for model training.

[0107] Optionally, the token sequence data is divided according to a preset data volume to obtain a plurality of data packets to be loaded. Exemplarily, the preset data volume may be the size of each data packet determined according to an actual application scenario. The sizes of the first preset buffer area and the second preset buffer area are both greater than or equal to the preset data volume.

[0108] In one example, there is a contextual relationship between characters, so the token values ​​in the token sequence data also have a contextual relationship, and after the token sequence data is divided, multiple data packets to be loaded are numbered according to the contextual relationship, and a data packet identifier corresponding to each data packet to be loaded is obtained, wherein the data packet identifier corresponds to the contextual relationship of the token sequence data.

[0109] In the embodiment of the present disclosure, when determining the data group to be loaded, the original text data is obtained, and token sequence data is generated according to the corresponding relationship between characters and token values ​​according to the original text data, and the token sequence data is divided according to a preset data amount to generate a data packet to be loaded, thereby obtaining the data group to be loaded. In this way, before the data is loaded, the original text data can be converted into token sequence data that can be processed by the model, and the data processing process can be pre-placed in the data packet generation process, thereby reducing repeated data processing processes, reducing the training time of large models, and being suitable for more application scenarios.

[0110] In one embodiment, the generating token sequence data based on the correspondence between characters and token values ​​according to the original text data includes:

[0111] Preprocessing the original text data to obtain target text data;

[0112] Based on the correspondence between characters and token values, characters in the target text data are converted into token values ​​to obtain token sequence data.

[0113] Exemplarily, the original text data may be preprocessed before token conversion. In one example, the preprocessing process may include but is not limited to word segmentation, completion, cleaning, etc., which may be determined based on actual application scenarios.

[0114] The target text data obtained after preprocessing the original text data includes a plurality of characters, and the plurality of characters have a contextual relationship. Optionally, according to the correspondence between the characters and the token values, the characters in the target text data are converted into token values ​​to obtain token sequence data. The arrangement order of the token values ​​in the token sequence data is the same as the arrangement order of the characters in the target text data.

[0115] Figure 2 is a schematic diagram showing a method for determining a data group to be loaded according to an exemplary embodiment, with reference to Figure 2 As shown, after pre-processing the original data (i.e., the original text data), token data (i.e., token sequence data) is obtained. The token data is segmented to obtain N data blocks (i.e., the data packets to be loaded), and each data block has a corresponding identifier (i.e., the data packet identifier). In one example, the data is segmented according to the same length (corresponding to the preset data volume), except that the size of the last data block may be smaller than the segmentation length, the sizes of other data blocks are the same, and these data blocks are saved. In one example, the size of the data block can be set in the range of [0.2GB, 2GB].

[0116] The obtained N data blocks are stored in a hard disk. In one example, a data index file may be created in the hard disk to record the corresponding relationship between the original data and the data in the data blocks.

[0117] In the disclosed embodiment, the original text data is preprocessed and then converted into token sequence data, so that the data to be loaded obtained by subsequent reading is the data after data preprocessing, thereby ensuring the reliability and validity of the data, and prepending the data preprocessing flow to the data packet production process. After the subsequent data is read, it can be directly used for model training, thereby shortening the model training cycle and further improving the model training efficiency.

[0118] In one embodiment, two cache areas (i.e., a first preset cache area and a second preset cache area) may be initialized in the memory. In one example, the size of each cache area is exactly the same as the size of the data block. The data index file is read. According to the data acquisition instruction, a new thread is started to load the first data block (i.e., the first data packet) in a certain cache area.

[0119] When the i-th data is needed in training, a data acquisition instruction can be sent to obtain the i-th data (i.e., the data group to be loaded) from the hard disk. The data block number k and the data block index i_k (i.e., the data identification information in the data packet) corresponding to the i-th data are obtained according to the index file. The following situations may occur: (1) If the data block k has been loaded in the cache area, the data in the cache area is directly read according to the data block index i_k, such as Figure 3 As shown; (2) If the cache area is loading data block k, wait for the data block to be loaded; (3) If the cache area is not loaded with data block k, start a new thread to load data block k in another cache area, as shown Figure 4 (4) If the index i_k in the data slice has been loaded to half the size of the data slice, and there is a data slice k+1, and the data slice k+1 has not been loaded into the cache area, then a new thread is started and the data slice k+1 is loaded into another cache area.

[0120] Figure 5 is a flow chart of a data loading method according to an exemplary embodiment, referring to Figure 5 As shown in the figure, the original data has 5 token data after preprocessing, and the size of the data block is 2. At this time, the token data is divided into 3 files, and the last file has only 1 data, which is less than the segmentation length. In addition, an index file is generated, for example, the second piece of original data corresponds to the second piece of data in data block 1.

[0121] During initialization, two cache areas are requested in the memory, the index file is read, and data block 1 is loaded into cache area 1. In one example, in order to avoid conflicts, in the subsequent process of loading data blocks, odd-numbered data blocks are loaded into cache area 1, and even-numbered data blocks are loaded into cache area 2.

[0122] When reading the first piece of data, according to the index file, the first piece of data in buffer area 1 is read to obtain data (0,1). Since the data index has reached half of the size of the data block, a new thread is started to load data block 2 in cache area 2. When reading the second piece of data, according to the index file, the second piece of data in buffer area 1 is read to obtain data (2,3). When reading the third piece of data, according to the index file, the first piece of data in buffer area 2 is read to obtain data (4,5). Since the data index has reached half of the size of the data block, a new thread is started to load data block 3 in cache area 1. When reading the fourth piece of data, according to the index file, the second piece of data in buffer area 2 is read to obtain data (6,7). When reading the fifth piece of data, according to the index file, the first piece of data in buffer area 1 is read to obtain data (8,9). Although the data index has reached half of the size of the data block, there is no data block 4, so the cache area has not changed. If the first data is read again, according to the index file, the first data in buffer area 1 should be read. Since there is no data block 1 in the cache area, a new thread is started to load data block 1 in cache area 1. After waiting for all the data to be loaded, the data (0,1) is obtained. Although the data index has reached half the size of the data block, data block 2 has been loaded in cache area 2, so a new data loading thread is not started.

[0123] In the disclosed embodiment, the data is evenly sliced, and only specific data slices are loaded into the memory, thereby realizing high-speed loading of large-scale data. Two cache areas are established in the memory, and high-speed loading of data can be maintained by alternately loading data slices. The data slices where the original data is located and the indexes in the slices can be found through the index file. By combining the cache technology with the index file, the correct data slices and the subscripts in the slices can be quickly and accurately queried even when the data reading index is not sequential. The preprocessing process of large model training is pre-placed in the production process of data slices, which can reduce repeated data preprocessing and reduce the training time of large models. Since the preprocessing processes such as word segmentation, scrambling, and filling in the large model training are carried out in advance and the data is written to the hard disk, these steps can be omitted during training, thereby reducing the training time. The traditional data loading method, whether it is all loaded into the memory or read directly from the hard disk, requires data preprocessing before each large model training. When batch training is required or a certain training interruption needs to be restarted, the preprocessing process needs to be repeated. This solution puts the preprocessing process in front and saves it as data blocks. The data read during each training is preprocessed data, thus reducing repeated data preprocessing.

[0124] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0125] Based on the same inventive concept, the embodiment of the present application also provides a data loading device for implementing the data loading method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data loading device embodiments provided below can refer to the limitations on the data loading method above, and will not be repeated here.

[0126] In an exemplary embodiment, Figure 6 As shown, a data loading device 600 is provided, comprising:

[0127] A determination module 610 is used to determine, in response to receiving a data acquisition instruction, a data group to be loaded that matches the data acquisition instruction, wherein the data group to be loaded includes a plurality of data packets to be loaded, and each of the data packets to be loaded corresponds to a data packet identifier;

[0128] A loading module 620, configured to determine a first data packet from the plurality of data packets to be loaded according to the data packet identifier, read the first data packet from a first preset buffer area, and load a second data packet adjacent to the first data packet into a second preset buffer area;

[0129] A reading module 630 is used to load a third data packet adjacent to the second data packet into the first preset cache area when the amount of data loaded from the second data packet into the second preset cache area is greater than a preset threshold, and read the second data packet from the second preset cache area until the reading of the data group to be loaded is completed, wherein the data group to be loaded is used to train to obtain a target model.

[0130] In one embodiment, the loading module is further used for:

[0131] If the first data packet is not loaded in the first preset buffer area, loading the first data packet into the first preset buffer area, and reading the first data packet from the first preset buffer area; and / or,

[0132] In the case where the first data packet has been loaded in the first preset buffer area, reading the first data packet from the first preset buffer area; and / or,

[0133] In the case where the first data packet is being loaded into the first preset buffer area, when the loading of the first data packet is completed, the first data packet is read from the first preset buffer area.

[0134] In one embodiment, the determining module is further configured to:

[0135] Determining a target data group identifier according to the data acquisition instruction;

[0136] The data group corresponding to the target data group identifier is determined as the data group to be loaded.

[0137] In one embodiment, the reading module is further used for:

[0138] When the amount of data of the second data packet loaded into the second preset buffer area is greater than a preset threshold, determining a third data packet adjacent to the second data packet;

[0139] If the third data packet exists, the third data packet is loaded into the first preset buffer area.

[0140] In one embodiment, the device further comprises a module for determining the data group to be loaded, and the module for determining the data group to be loaded is further configured to:

[0141] Acquire original text data, wherein the original text data includes a plurality of characters;

[0142] Based on the correspondence between characters and token values, generating token sequence data according to the original text data;

[0143] The token sequence data is divided according to a preset data amount to generate a plurality of data packets to be loaded, thereby obtaining a data group to be loaded.

[0144] In one embodiment, the module for determining the data group to be loaded is further used for:

[0145] Preprocessing the original text data to obtain target text data;

[0146] Based on the correspondence between characters and token values, characters in the target text data are converted into token values ​​to obtain token sequence data.

[0147] Each module in the above data loading device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0148] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the method described in this embodiment, such as a data group to be loaded. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data loading method is implemented.

[0149] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0152] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0154] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0155] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data loading method, characterized in that: The method comprises: In response to receiving a data acquisition instruction, determining a data group to be loaded that matches the data acquisition instruction, wherein the data group to be loaded includes a plurality of data packets to be loaded, and each of the data packets to be loaded corresponds to a data packet identifier; According to the data packet identifier, determine a first data packet from the multiple data packets to be loaded, read the first data packet from a first preset buffer area, and load a second data packet adjacent to the first data packet into a second preset buffer area; When the amount of data loaded from the second data packet into the second preset cache area is greater than a preset threshold, a third data packet adjacent to the second data packet is loaded into the first preset cache area, and the second data packet is read from the second preset cache area until the reading of the data group to be loaded is completed, wherein the data group to be loaded is used to train to obtain a target model.

2. The method according to claim 1, characterized in that The reading the first data packet from the first preset buffer area includes: If the first data packet is not loaded in the first preset buffer area, loading the first data packet into the first preset buffer area, and reading the first data packet from the first preset buffer area; and / or, In the case where the first data packet has been loaded in the first preset buffer area, reading the first data packet from the first preset buffer area; and / or, In the case where the first data packet is being loaded into the first preset buffer area, when the loading of the first data packet is completed, the first data packet is read from the first preset buffer area.

3. The method according to claim 1, characterized in that The step of determining a to-be-loaded data group that matches the data acquisition instruction includes: Determining a target data group identifier according to the data acquisition instruction; The data group corresponding to the target data group identifier is determined as the data group to be loaded.

4. The method according to claim 1, characterized in that: The method of loading a third data packet adjacent to the second data packet into the first preset buffer area when the amount of data of the second data packet loaded into the second preset buffer area is greater than a preset threshold value comprises: When the amount of data of the second data packet loaded into the second preset buffer area is greater than a preset threshold, determining a third data packet adjacent to the second data packet; If the third data packet exists, the third data packet is loaded into the first preset buffer area.

5. The method according to claim 1, characterized in that The method for determining the data group to be loaded includes: Acquire original text data, wherein the original text data includes a plurality of characters; Based on the correspondence between characters and token values, generating token sequence data according to the original text data; The token sequence data is divided according to a preset data amount to generate a plurality of data packets to be loaded, thereby obtaining a data group to be loaded.

6. The method according to claim 5, characterized in that The generating token sequence data based on the correspondence between characters and token values ​​according to the original text data includes: Preprocessing the original text data to obtain target text data; Based on the correspondence between characters and token values, characters in the target text data are converted into token values ​​to obtain token sequence data.

7. A data loading device, characterized in that: The device comprises: A determination module, configured to determine, in response to receiving a data acquisition instruction, a data group to be loaded that matches the data acquisition instruction, wherein the data group to be loaded includes a plurality of data packets to be loaded, and each of the data packets to be loaded corresponds to a data packet identifier; a loading module, configured to determine a first data packet from the plurality of data packets to be loaded according to the data packet identifier, read the first data packet from a first preset buffer area, and load a second data packet adjacent to the first data packet into a second preset buffer area; A reading module is used to load a third data packet adjacent to the second data packet into the first preset cache area when the amount of data loaded from the second data packet into the second preset cache area is greater than a preset threshold, and read the second data packet from the second preset cache area until the reading of the data group to be loaded is completed, wherein the data group to be loaded is used to train to obtain a target model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images