Data loading method and device and electronic equipment
By converting the dataset to binary format and utilizing the LMDB database, the problem of low data loading efficiency in deep learning frameworks is solved, achieving efficient data loading and hardware resource utilization, and improving the overall efficiency of large model training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UBTECH ROBOTICS CORP LTD
- Filing Date
- 2025-12-24
- Publication Date
- 2026-05-05
AI Technical Summary
Existing deep learning frameworks suffer from low data loading efficiency when loading large-scale datasets, especially due to the large I/O copy overhead, which causes the GPU to wait for data and reduces hardware utilization, making it unable to meet the training needs of large models.
By converting the dataset in the first format into binary data and storing it as a dataset in the second format, a high-performance key-value database such as LMDB is used to reduce I/O copy overhead. A third-party library is introduced into the target framework to implement data loading and support reading datasets in LMDB format.
This improved data loading efficiency, reduced data loading time, and increased hardware utilization and overall training throughput.
Smart Images
Figure CN121979964A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing technology, and in particular relates to data loading methods, apparatus, electronic devices, computer-readable storage media, and computer program products. Background Technology
[0002] As artificial intelligence technology penetrates into scenarios such as embodied intelligence, the requirements for data loading efficiency in deep learning models (such as natural language processing models, computer vision models, and multimodal interaction models) are increasing. The data loading stage, as a pre-training step, directly impacts the overall training cycle. This is especially true when dealing with large-scale datasets, such as multimodal data like images, videos, and text with tens of millions of samples. Inefficient data loading leads to computing resources (such as graphics processing units, GPUs) being idle, severely affecting training efficiency and creating the core bottleneck of "computing waiting for data."
[0003] In current model training practices, data is typically read using the built-in data loading pipeline of deep learning frameworks. However, when the storage format of the training data causes high input / output (I / O) copy overhead, it creates a data loading bottleneck, leading to reduced hardware utilization due to GPU / accelerator waiting for data, ultimately limiting the overall training throughput. For example, when handling large-scale datasets, the mainstream approach of Scalable Lightweight Infrastructure for Fine-Tuning (Swift) is to load text format files such as JSON Lines (JSONL). However, since JSONL parses text data line by line, data needs to be copied between disk and memory multiple times during the reading process, making it difficult to unleash the hardware I / O potential. Therefore, it can no longer meet the efficiency requirements of large models for loading large-scale datasets. Summary of the Invention
[0004] This application provides a data loading method, apparatus, and electronic device, which can solve the problem of low data loading efficiency in existing target frameworks when loading data.
[0005] In a first aspect, embodiments of this application provide a data loading method, including:
[0006] Read the file of the first dataset, and extract the text question-and-answer pairs, the source path of the visual data, and the meta-information of the visual data from the file of the first dataset. The first dataset is a dataset in a first format, and the first dataset includes visual data and text question-and-answer pairs corresponding to the visual data.
[0007] The text question-and-answer pairs are serialized into binary data to obtain text binary data;
[0008] Read the corresponding visual data from the source path, and convert the read visual data and the metadata of the visual data into binary data to obtain visual binary data;
[0009] The text binary data and the visual binary data are stored to obtain a second dataset. The second dataset is a dataset in a second format, and the copy overhead of the second format dataset for I / O is less than that of the first format dataset for I / O.
[0010] Read the text binary data and the visual binary data from the second dataset, deserialize the read text binary data into a string, and convert the string and the visual binary data into a content format supported by the target framework.
[0011] Secondly, embodiments of this application provide a data loading apparatus, including:
[0012] The first dataset file reading module is used to read the first dataset file and extract the text question-and-answer pairs, the source path of the visual data, and the meta-information of the visual data from the first dataset file. The first dataset is a dataset in a first format and includes visual data and text question-and-answer pairs corresponding to the visual data.
[0013] The text binary data determination module is used to serialize the text question-and-answer pair into binary data to obtain text binary data;
[0014] The visual binary data determination module is used to read the corresponding visual data from the source path and convert the read visual data and the metadata of the visual data into binary data to obtain visual binary data.
[0015] The second dataset determination module is used to store the text binary data and the visual binary data to obtain a second dataset. The second dataset is a dataset in a second format, and the copy overhead of the second format dataset to I / O is less than that of the first format dataset to I / O.
[0016] The file reading module for the second dataset is used to read the text binary data and the visual binary data in the second dataset, deserialize the read text binary data into a string, and convert the string and the visual binary data into a content format supported by the target framework.
[0017] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.
[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.
[0019] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect.
[0020] The beneficial effects of the embodiments in this application compared with the prior art are:
[0021] In this embodiment, text question-and-answer pairs, visual data, and metadata of the visual data are extracted from a file of a first-format dataset (i.e., the first dataset). The text question-and-answer pairs are converted into text binary data, and the visual data and its metadata are converted into visual binary data. The text binary data and visual binary data are stored to obtain a second-format dataset (i.e., the second dataset). Then, the binary data of the second dataset is read and converted into a content format supported by the target framework. Since reading the binary data of the second dataset and converting it into a content format supported by the target framework is equivalent to completing the adaptation of the first-format data with the native format of the target framework, that is, realizing the function of loading second-format data within the improved target framework. Furthermore, since the I / O copy overhead of the second-format dataset (i.e., the second dataset) is less than that of the first-format dataset, the data loading efficiency is improved when loading data through the improved target framework. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.
[0023] Figure 1 This is a flowchart illustrating a data loading method provided in an embodiment of this application;
[0024] Figure 2 This is a schematic diagram of the structure of a data loading device provided in an embodiment of this application;
[0025] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0030] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0031] During model training, if the data format supported by the deep learning framework requires significant I / O copy overhead, the data loading efficiency of that deep learning framework will be low.
[0032] Taking the Swift framework as an example, when dealing with large-scale datasets, the mainstream approach is to load text format files such as JSONL. However, due to the inherent defects of JSONL files, such as low I / O throughput, it can no longer meet the needs of large models for loading large-scale datasets.
[0033] Lightning Memory-Mapped Database (LMDB), a high-performance key-value database, has become an ideal alternative to JSONL format due to its "memory-mapped I / O" and "ultra-large-scale data storage" features. LMDB directly maps images, videos, and other files to the memory address space, drastically reducing the copy overhead of traditional I / O. It can stably store petabytes of multimodal data (such as images, text, and videos), perfectly meeting the needs of large-scale model training and becoming the mainstream storage solution for large datasets (such as pre-trained text corpora and multimodal sample libraries) in large-scale model training.
[0034] However, the Swift framework does not have built-in LMDB data processing capabilities; it only supports loading logic for text formats such as JSONL and lacks a dedicated processing branch for LMDB. This prevents the direct utilization of LMDB's performance characteristics, necessitating additional integration to leverage LMDB's high-performance storage advantages. Because LMDB's high-performance advantages cannot be implemented within the Swift framework, the data loading process becomes inefficient during large model training due to limitations in the data loading stage.
[0035] To improve the data loading efficiency of target frameworks (such as Swift framework, Meagtron-LM framework, hugging facetransformers, Deepspeed, etc.), this application provides a data loading method.
[0036] In this data loading method, text question-and-answer pairs, source paths of visual data, and metadata of visual data are extracted from the file of the dataset in the first format. The extracted content is then converted into corresponding binary data, and the converted binary data is stored as a dataset in the second format. Finally, the binary data of the dataset in the second format is read, and the read binary data is converted into a content format supported by the target framework.
[0037] The data loading method provided in the embodiments of this application is described below with reference to the accompanying drawings.
[0038] Figure 1 A flowchart illustrating a data loading method provided in an embodiment of this application is shown. This data loading method can be applied to electronic devices, and is described in detail below:
[0039] S11, Read the file of the first dataset, extract the text question-and-answer pairs, the source path of the visual data, and the metadata of the visual data from the file of the first dataset, wherein the first dataset is a dataset in a first format, and the first dataset includes visual data and text question-and-answer pairs corresponding to the visual data.
[0040] The first format mentioned above includes JSONL format, JSON format, etc., which are not limited here.
[0041] The first dataset mentioned above includes a training set and / or a validation set for training the model. This dataset includes text question-answer pairs, visual data, and the correspondence between the text question-answer pairs and the visual data.
[0042] In this embodiment of the application, visual data includes image data and / or video data. A text question-and-answer pair may correspond to one piece of visual data, a text question-and-answer pair may correspond to multiple pieces of visual data, and a piece of visual data may correspond to multiple text question-and-answer pairs.
[0043] The above text-based question and answer pair contains a description of the question and the content of the answer. Its format can be as follows: {"question":"What is JSONL?","answer":"JSONL is a data format in which one JSON object is per line"}.
[0044] The source path of the aforementioned visual data is the storage path of the image, and / or the storage path of the video.
[0045] The aforementioned metadata of visual data is used to describe the attributes of the visual data, including: when the visual data is an image, the image resolution, color space, file size, format, creation time, shooting parameters, etc.; when the visual data is a video, the format, video duration, file size, frame rate, encoding, etc.
[0046] S12, serialize the above text question-and-answer pairs into binary data to obtain text binary data.
[0047] Serialization refers to the process of converting data structures in memory into a format that can be stored or transmitted.
[0048] In this embodiment of the application, serializing the text question-and-answer pair into binary data involves converting the text question-and-answer pair (JSONL format) into a byte stream (i.e., a sequence of 0s and 1s) so that it can be efficiently stored in underlying storage engines such as LMDB.
[0049] S13, read the corresponding visual data from the above source path, and convert the read visual data and the metadata of the visual data into binary data to obtain visual binary data.
[0050] In this embodiment, visual data is read as binary data via a binary stream. Since the metadata of the visual data is textual description information, this metadata needs to be deserialized into binary data. Optionally, the binary data corresponding to the metadata can be used as the header information of the binary data corresponding to the visual data to obtain the aforementioned visual binary data. Of course, in practice, the binary data corresponding to the metadata can also be used as the tail information of the binary data corresponding to the visual data to obtain the aforementioned visual binary data; this is not limited here.
[0051] S14, store the above-mentioned text binary data and the above-mentioned visual binary data to obtain a second dataset, which is a dataset in a second format.
[0052] The second format mentioned above includes LMDB format, columnar storage format (Apache Parquet), in-memory columnar data format (Apache Arrow), etc., and is not limited here.
[0053] In some embodiments, a dataset in a second format can be transmitted to a designated device. Because the second format data is in binary format and utilizes memory mapping technology, reading and writing data in the second format is significantly faster than reading and writing data in the first format. Furthermore, in network transmission, since binary formats generally have lower copy overhead and higher efficiency than text formats, converting a dataset in a first format (such as JSONL) to a second format (such as LMDB) before transmission can improve data transmission speed.
[0054] S15, read the text binary data and visual binary data from the second dataset, deserialize the read text binary data into a string, and convert the string and visual binary data into a content format supported by the target framework.
[0055] In this embodiment, reading data in the second format can be achieved by introducing a third-party library. For example, when the second format is LMDB and the target framework is Swift, the native integration of LMDB data loading capability at the Swift framework's Loader layer can be achieved by introducing the third-party LMDB library (which refers to a Swift language wrapper library for LMDB databases). Specifically, the third-party LMDB library is introduced by configuring its library dependencies and environment initialization parameters. After the Swift framework introduces the third-party LMDB library, it has the function of loading datasets in LMDB format. The Swift framework can then identify whether the dataset is in LMDB format by recognizing the dataset's format identifier. Optionally, the dataset's format identifier can be represented by the file extension of the dataset or by the dataset's configuration parameters.
[0056] In this embodiment, a target framework that incorporates a third-party library can read the text binary data of the second dataset and deserialize the read text binary data into a corresponding string. This string, along with the visual binary data read from the second dataset, is then converted into a content format supported by the target framework. Optionally, when the target framework is a Swift framework, the content format supported by the Swift framework can be HFDataSet format. HFDataSet format typically refers to the dataset format used in the HuggingFace ecosystem, and its core is based on the standard structure defined by the Hugging Face Datasets library. This format is designed for efficient storage and processing of machine learning data, supports multiple underlying serialization methods, and provides a unified Application Programming Interface (API) for access.
[0057] Besides the HFDataSet format, the Swift framework also supports content formats such as the Messages structured format, ShareGPT format, and others, which will not be elaborated here. The Messages structured format is a standardized way to organize and transmit message data. Its core purpose is to define message content, roles, and metadata in a clear and parsable format, facilitating system processing and cross-component interaction. The ShareGPT format is a data format used for fine-tuning large language models, particularly suitable for multi-turn dialogue scenarios. It originated from the community's collection and sharing of real user-chat model dialogue records, aiming to simulate natural dialogue flow and help the model learn context preservation and response generation capabilities.
[0058] Optionally, considering that the content format supported by the target framework may differ from the information structuring mode required for model training, to ensure that the read data can be directly used in subsequent model training processes and improve model training efficiency, the content format supported by the target framework can be converted to the information structuring mode required for model training after converting the strings and visual binary data to the content format supported by the target framework. For example, if the content format supported by the Swift framework is HFDataSet format, and the information structuring mode required for model training is Messages structured format, then after converting the strings and visual binary data to HFDataSet format, the HFDataSet format can be converted to Messages structured format.
[0059] In this embodiment, text question-and-answer pairs, visual data, and metadata of the visual data are extracted from a file of a first-format dataset (i.e., the first dataset). The text question-and-answer pairs are converted into text binary data, and the visual data and its metadata are converted into visual binary data. The text binary data and visual binary data are stored to obtain a second-format dataset (i.e., the second dataset). Then, the binary data of the second dataset is read and converted into a content format supported by the target framework. Since reading the binary data of the second dataset and converting it into a content format supported by the target framework is equivalent to completing the adaptation of the first-format data with the native format of the target framework, that is, realizing the function of loading second-format data within the improved target framework. Furthermore, since the I / O copy overhead of the second-format dataset (i.e., the second dataset) is less than that of the first-format dataset, the data loading efficiency is improved when loading data through the improved target framework.
[0060] In some embodiments, considering that the dataset in the first format may have a one-to-many correspondence or a many-to-one relationship, for example, one text question-and-answer pair corresponds to multiple visual data, or multiple text pairs correspond to one visual data, if each correspondence stores the corresponding text question-and-answer pair and visual data, it will lead to the text question-and-answer pair being stored repeatedly, or the visual data being stored repeatedly. That is, the above storage method has the problem of storage redundancy. To solve the problem of storage redundancy, the corresponding data can be stored by establishing a mapping relationship between the unique identifiers of the text binary data and the visual data. Specifically, in S14 above, the above text binary data and the above visual binary data are stored to obtain a second dataset, including:
[0061] A1. Determine the unique identifier for the above visual data.
[0062] In this embodiment, the unique identifier of the visual data can be manually set or automatically generated through other methods. For example, the unique identifier of the visual data can be generated using a hash algorithm based on the storage path and timestamp of the visual data. Of course, in practice, other methods can also be used to determine the identifier, which will not be elaborated here.
[0063] A2. Store the text binary data and the unique identifier of the visual data in correspondence, and store the visual data and the unique identifier of the visual data in correspondence, to obtain the second dataset.
[0064] In this embodiment, a dual second-format storage mode is used to store text binary data and visual binary data. The aforementioned second dataset may include two sets: one may be a text second-format dataset, and the other may be a media second-format dataset. The text second-format dataset is used to store unique identifiers for the text binary data and visual data, while the media second-format dataset is used to store unique identifiers for the visual data and visual binary data. Since each piece of text binary data and each piece of visual binary data only needs to be stored once, eliminating the need for repeated storage, this storage method helps avoid data redundancy issues.
[0065] Correspondingly, reading the text binary data and the visual binary data from the second dataset includes:
[0066] B1. Read the unique identifiers of the aforementioned text binary data and visual data from the aforementioned second dataset.
[0067] B2. Read the aforementioned visual binary data from the second dataset based on the unique identifier of the aforementioned visual data.
[0068] In this embodiment, considering that unique visual data can be found based on its unique identifier, the text binary data and the corresponding unique identifier of the visual data can be read first, and then the corresponding visual data can be found based on the unique identifier of the read visual data. For example, assuming the second dataset includes a text LMDB and a media LMDB, the text LMDB file is read first to obtain the unique identifiers of the text binary data and the visual data, and then the associated visual binary data is retrieved from the media LMDB file using the unique identifier of the visual data.
[0069] In some embodiments, considering that the target framework (such as the Swift framework) does not natively support training with video binary data, the core layer logic of the target framework is also extended on the basis of the Loader layer of the target framework as described above, so as to form an extension and closed loop of the video data processing link. Specifically, (1) a video metadata output module is added. The video metadata output module is used to parse the header information of the video binary data of the second dataset (such as the media LMDB included in the second dataset) (if the metadata is stored in the header of the video binary data) to obtain the metadata of the video binary data. (2) a video binary data to tensor module is added. The video binary data to tensor module calls the decoding interface (such as the FFmpeg encapsulation interface) in the target framework based on the metadata output by the video metadata output module to decode the video binary stream into a frame sequence, and then generates a video tensor acceptable to the model through the tensor conversion function (the format of the video tensor can be [batch size, frame number, height, width, channel number]). Subsequently, the content format supported by the target framework corresponding to the video binary data can be obtained based on the video tensor.
[0070] Once the target framework has formed an extension and closed loop of the video data processing link, the read video binary data can be converted into a content format supported by the target framework. That is, when the visual data includes video data and the visual binary data includes video binary data, the above-mentioned S15, in converting the above-mentioned string and the above-mentioned visual binary data into a content format supported by the target framework, includes:
[0071] C1. Parse the metadata of the video data from the information in the specified part of the video binary data, where the specified part is the part where the metadata is stored.
[0072] The specified parts mentioned above may include the head and tail of the video binary data, and may also include other parts, which are not limited here.
[0073] For example, when the specified part is the header of the video binary data, the metadata of the video data is parsed from the information in the header of the video binary data.
[0074] In this embodiment, if the video data metadata includes frame rate (FPS), considering that the frame rate is essentially a rational number in fractional form rather than an integer, the video's FPS can be securely parsed using a precise numerical type (such as a fraction class) to reduce floating-point precision errors. When the video data metadata includes the total number of frames and resolution, static errors such as the integrity of video frames and resolution consistency can also be checked to output corresponding metadata. This metadata may include one or more of the following: FPS, resolution, total number of frames, video encoding, timestamp, and bitstream parameters.
[0075] C2. Decode the above video binary data into a frame sequence based on the metadata of the above video data.
[0076] In this embodiment, a decapsulator and decoder can be selected based on the metadata of the video data to correctly split the video binary data into corresponding frames. Each split frame is read by a frame reader, and the sampling and timing are controlled by the metadata of the video data to finally obtain the above-mentioned frame sequence.
[0077] C3. Convert the above frame sequence into a video tensor.
[0078] Specifically, a tensor transformation function can be used to generate video tensors from the frame sequence that are acceptable to the model to be trained later.
[0079] C4. Convert the above string and the above video tensor into a content format supported by the target frame.
[0080] In this embodiment, the target framework (such as the Swift framework) supports content formats that, for videos, include video tensors, and for images, include image binary data. Therefore, when visual data includes video data, the read video binary data needs to be converted into a video tensor, and then the video tensor and its corresponding string need to be converted into a content format supported by the target framework. Since the read video binary data and its corresponding string can be converted into a content format supported by the target framework, the improved target framework can support training on video binary data.
[0081] In some embodiments, if the data in the second format (such as LMDB) supports multi-threaded parallel reading, then in order to fully utilize the performance characteristics of the data in the second format, this application embodiment adds support for streaming processing. When reading data using a streaming processing method, in S15 above, the text binary data and the visual binary data in the second dataset are read, the read text binary data is deserialized into a string, and the string and the visual binary data are converted into a content format supported by the target framework, including:
[0082] D1. Using a streaming processing method, read the text binary data and visual binary data from the second dataset in batches, and temporarily store the read text binary data and visual binary data in a buffer.
[0083] In this embodiment, the amount of data (text binary data and visual binary data) read in each batch is related to the video memory of the electronic device where the target frame is located. For example, the larger the video memory, the larger the amount of data read in each batch, and vice versa.
[0084] D2. Read the text binary data from the buffer and deserialize it into a string, then convert the string and the visual binary data read from the buffer into a content format supported by the target frame.
[0085] Considering that the data in the first format (such as the traditional JSONL file format) is read in a non-streaming manner and the next processing action is not performed until all data has been read, when the dataset is large, the data reading method of the traditional JSONL file format will result in a longer waiting time. Therefore, in this embodiment, a streaming processing method is adopted to read data from the second dataset and convert it into a format supported by the target framework in order to improve the efficiency of coordination with I / O resources.
[0086] Optionally, the streaming processing here can be implemented using a producer-consumer pattern. Specifically, the logic for reading the second-format data is optimized as follows: the production group reads text binary data and visual binary data from the second dataset in batches, temporarily storing the read text binary data and visual binary data in a process-safe buffer without waiting for the full data to load; the consumer group retrieves data from the buffer in real time and synchronously performs preprocessing (which includes at least one of the following: text segmentation, image normalization, and video frame sampling) to convert the strings and the visual binary data read from the buffer into a content format supported by the target framework. The buffer is typically implemented using a queue structure to meet the first-in-first-out (FIFO) data order requirement; however, a stack can be used in special scenarios requiring last-in-first-out (LIFO) processing.
[0087] Optionally, the process of streaming data in the second format described above can be encapsulated using an iterator to support a pipelined execution of "loading, preprocessing, and training simultaneously" during model training. This encapsulation significantly reduces data waiting time, especially in scenarios with hundreds of millions of samples, reducing initial loading time from hours to minutes.
[0088] As described above, the improved target framework provided in this application embodiment can realize the reading and processing of data in a second format, such as reading and processing text data, image data, and video data in LMDB format (the processing here includes converting the read data into a content format supported by the target framework). During the data reading and processing, a single-threaded approach or a multi-threaded approach can be used. When using a multi-threaded approach, in step S15 above, the text binary data and visual binary data in the second dataset are read, the read text binary data is deserialized into a string, and the string and visual binary data are converted into a content format supported by the target framework, including:
[0089] E1. Divide the files in the second dataset into several contiguous blocks. The files in the second dataset include the aforementioned text binary data and the aforementioned visual binary data.
[0090] E2. For each of the above small blocks, a corresponding subprocess reads the above text binary data and the above visual binary data of the above small block, deserializes the read above text binary data into a string, and converts the above string and the above visual binary data into a content format supported by the target framework.
[0091] In this embodiment, the second dataset is processed using block sampling. For example, the files in the second dataset are divided into several contiguous blocks according to the data index, and then multiple child processes are scheduled through a process pool. Each scheduled child process is typically responsible for reading and processing only one small block of data at a time. The number of scheduled child processes is usually less than or equal to the number of small blocks. Because a multi-process approach is used to read and process the data, the memory overflow caused by a single process processing the entire dataset can be reduced.
[0092] Optionally, when reading and processing data through multiple processes, caching and memory release mechanisms can be added. Specifically, a buffer is set up. After a process reads and processes data, the processed data is stored in this buffer to aggregate the data processed by each process. Optionally, to prevent training batch synchronization blocking, the amount of data (or sample size) processed by each process is checked in real time when aggregating the data processed by each process. If inconsistencies are found, all processes are automatically unified to the minimum amount of data. For example, assuming three processes are used to process data, when aggregating the data processed by the three processes in the buffer, if process 1 processes 100 data items, process 2 processes 120 data items, and process 3 processes 110 data items, then during aggregation, 100 data items processed by each of the three processes are selected respectively. For example, all data processed by process 1 is selected, the first 100 data items processed by process 2 are selected, and the first 100 data items processed by process 3 are selected.
[0093] As described above, the first dataset of this application may include a training set in a first format and a validation set in a first format. Correspondingly, the second dataset may also include a training set in a second format and a validation set in a second format. Considering that the validation set typically accounts for 5% to 10% of the total dataset size (e.g., the sum of the training and validation sets), and that the difference between streaming and non-streaming processing methods is mainly a difference in processing efficiency, the appropriate processing method can be selected based on the amount of data in the validation set. For example, when the number of samples (or data volume) in the second-format validation set is large, streaming processing is selected; when the number of samples is small, non-streaming processing is selected.
[0094] That is, in some embodiments, when the second dataset includes a validation set in a second format, the above-mentioned process of reading the text binary data and the visual binary data in the second dataset in batches using a pipelined processing method, and temporarily storing the read text binary data and the visual binary data in a buffer, includes:
[0095] Determine the number of samples in the above validation set;
[0096] When the number of samples exceeds a preset threshold, the text binary data and visual binary data in the verification dataset are read in batches using a streaming processing method, and the read text binary data and visual binary data are temporarily stored in a buffer.
[0097] When the number of samples is not greater than a preset threshold, the text binary data and the second visual binary data are preprocessed in a non-streaming manner to obtain a serialized file. The serialized file includes content in the content format supported by the Swift framework.
[0098] The serialized file mentioned above can be a pickle file, which can contain text binary data corresponding to text question-and-answer pairs, binary data corresponding to images, or video tensors corresponding to videos. When stored in a dual second format storage mode, the pickle file can also include a mapping table of unique identifiers for visual data.
[0099] In this embodiment, when the number of samples in the validation set is small (e.g., <100,000), the second format data of the validation set is pre-generated as a pickle serialized file. Since the pickle serialized file includes content in formats supported by the target framework (e.g., video tensors), the video tensor conversion step is saved during model training, allowing direct reading of the pickle file, thereby improving data reading speed and validation efficiency. When the number of samples in the validation set is large, the text binary data and the aforementioned visual binary data in the validation set are read in batches using a streaming processing method, and the read text binary data and the aforementioned visual binary data are temporarily stored in a buffer. That is, for both the training and validation sets, the parenting process is similar when using a pipelined processing method. Since the validation set and the training set share the same loading logic, the consistency between the validation process and the training process is ensured, thereby avoiding validation deviations caused by differences in data loading methods.
[0100] In some embodiments, for ease of invocation, the processing logic for the second format data provided in the embodiments of this application can be encapsulated. The processing logic for the second format data includes converting the dataset in the first format to the dataset in the second format, reading the data from the dataset in the second format, converting the content format supported by the target framework, and streaming processing.
[0101] After encapsulation, a unified entry function (such as load_lmdb_dataset) can be used to provide external calls, thereby reducing code coupling. Since the processing logic for the second-format data is encapsulated, it can automatically adapt to the interface requirements of different versions of the target framework. Furthermore, even after the target framework is updated, there is no need to adjust the core processing logic for the second-format data, thus achieving version compatibility with "minimal modification cost".
[0102] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0103] Corresponding to the data loading method described in the above embodiments, Figure 2 This diagram illustrates a structural block diagram of a data loading apparatus provided in an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown.
[0104] Reference Figure 2 The data loading device 2 is applied to an electronic device. The data loading device 2 includes: a file reading module 21 for a first dataset, a text binary data determination module 22, a visual binary data determination module 23, a second dataset determination module 24, and a file reading module 25 for a second dataset.
[0105] in:
[0106] The first dataset file reading module 21 is used to read the first dataset file and extract the text question-and-answer pairs, the source path of the visual data, and the meta-information of the visual data from the first dataset file. The first dataset is a dataset in a first format and includes visual data and text question-and-answer pairs corresponding to the visual data.
[0107] The text binary data determination module 22 is used to serialize the above text question-and-answer pairs into binary data to obtain text binary data;
[0108] The visual binary data determination module 23 is used to read the corresponding visual data from the above source path, and convert the read visual data and the metadata of the visual data into binary data to obtain visual binary data.
[0109] The second dataset determination module 24 is used to store the above-mentioned text binary data and the above-mentioned visual binary data to obtain a second dataset. The second dataset is a dataset in a second format, and the copy overhead of the second format dataset to I / O is less than that of the first format dataset to I / O.
[0110] The file reading module 25 for the second dataset is used to read the text binary data and the visual binary data in the second dataset, deserialize the read text binary data into a string, and convert the string and the visual binary data into a content format supported by the target framework.
[0111] In this embodiment, text question-and-answer pairs, visual data, and metadata of the visual data are extracted from a file of a first-format dataset (i.e., the first dataset). The text question-and-answer pairs are converted into text binary data, and the visual data and its metadata are converted into visual binary data. The text binary data and visual binary data are stored to obtain a second-format dataset (i.e., the second dataset). Then, the binary data of the second dataset is read and converted into a content format supported by the target framework. Since reading the binary data of the second dataset and converting it into a content format supported by the target framework is equivalent to adapting the first-format data to the native format of the target framework, the function of loading second-format data within the improved target framework is realized. Furthermore, since the I / O copy overhead of the second-format dataset (i.e., the second dataset) is less than that of the first-format dataset, the data loading efficiency is improved when loading data through the improved target framework.
[0112] In some embodiments, such as the monkey dataset, the second dataset determination module 24 includes:
[0113] A unique identifier determination unit for visual data, used to determine the unique identifier of the aforementioned visual data;
[0114] A dual-storage mode storage unit is used to store the text binary data and the unique identifier of the visual data in correspondence, and to store the unique identifier of the visual data and the visual binary data in correspondence, to obtain the second dataset.
[0115] Correspondingly, when reading the text binary data and the visual binary data in the second dataset, the file reading module 25 specifically performs the following functions:
[0116] Read the unique identifiers of the aforementioned text binary data and visual data from the aforementioned second dataset;
[0117] Read the aforementioned visual binary data from the second dataset based on the unique identifier of the aforementioned visual data.
[0118] In some embodiments, the visual data includes video data, and the file reading module 25 of the second dataset, when deserializing the read text binary data into a string and converting the string and the visual binary data into a content format supported by the target framework, includes:
[0119] The video data metadata parsing unit is used to parse the metadata of the video data from information in a specified part of the video binary data, where the specified part is the part where the metadata is stored.
[0120] A frame sequence decoding unit is used to decode the above video binary data into a frame sequence based on the metadata of the above video data.
[0121] The video tensor determination unit is used to convert the above frame sequence into a video tensor.
[0122] The video tensor conversion unit is used to convert the above string and the above video tensor into a content format supported by the target frame.
[0123] In some embodiments, the file reading module 25 for the second dataset described above includes:
[0124] A batch data caching unit is used to read the text binary data and the visual binary data in the second dataset in batches in a streaming manner, and temporarily store the read text binary data and the visual binary data in a buffer.
[0125] The cached data reading unit is used to read the text binary data in the buffer and deserialize it into a string, and convert the string and the visual binary data read from the buffer into a content format supported by the target framework.
[0126] In some embodiments, the file reading module 25 for the second dataset described above includes:
[0127] The file partitioning unit is used to divide the files in the second dataset into several consecutive small blocks. The files in the second dataset include the text binary data and the visual binary data.
[0128] The multi-processing unit is used to read the text binary data and visual binary data of each of the above-mentioned small blocks using a corresponding sub-process, deserialize the read text binary data into a string, and convert the string and the visual binary data into a content format supported by the target framework.
[0129] In some embodiments, the second dataset includes a validation set in LMDB format, and the batch data caching unit is specifically used for:
[0130] Determine the number of samples in the above validation set;
[0131] When the number of samples exceeds a preset threshold, the text binary data and visual binary data in the verification set are read in batches using a streaming processing method, and the read text binary data and visual binary data are temporarily stored in a buffer.
[0132] In some embodiments, the file reading module 25 for the second dataset described above is specifically used for:
[0133] When the number of samples is not greater than a preset threshold, the text binary data and the visual binary data are preprocessed in a non-streaming manner to obtain a serialized file. The serialized file includes content in the content format supported by the target framework.
[0134] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0135] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: at least one processor 30 ( Figure 3 The diagram shows only one processor, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, wherein the processor 30 executes the computer program 32 to implement the steps in any of the above method embodiments.
[0136] The electronic device 3 can be a desktop computer, laptop, handheld computer, or cloud server, etc. This electronic device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.
[0137] The processor 30 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0138] In some embodiments, the memory 31 may be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. In other embodiments, the memory 31 may be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. Furthermore, the memory 31 may include both internal and external storage units of the electronic device 3. The memory 31 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 31 can also be used to temporarily store data that has been output or will be output.
[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0140] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.
[0141] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.
[0142] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.
[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0144] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0145] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0146] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
Claims
1. A data loading method, characterized in that, include: Read the file of the first dataset, and extract the text question-and-answer pairs, the source path of the visual data, and the meta-information of the visual data from the file of the first dataset. The first dataset is a dataset in a first format, and the first dataset includes visual data and text question-and-answer pairs corresponding to the visual data. The text question-and-answer pairs are serialized into binary data to obtain text binary data; Read the corresponding visual data from the source path, and convert the read visual data and the metadata of the visual data into binary data to obtain visual binary data; The text binary data and the visual binary data are stored to obtain a second dataset. The second dataset is a dataset in a second format, and the copy overhead of the second format dataset for I / O is less than that of the first format dataset for I / O. Read the text binary data and the visual binary data from the second dataset, deserialize the read text binary data into a string, and convert the string and the visual binary data into a content format supported by the target framework.
2. The data loading method as described in claim 1, characterized in that, The storage of the text binary data and the visual binary data yields a second dataset, including: Determine a unique identifier for the visual data; The text binary data is stored in correspondence with the unique identifier of the visual data, and the unique identifier of the visual data is stored in correspondence with the visual binary data to obtain the second dataset; The step of reading the text binary data and the visual binary data from the second dataset includes: Read the text binary data and the unique identifier of the visual data from the second dataset; The visual binary data in the second dataset is read based on the unique identifier of the visual data.
3. The data loading method as described in claim 1, characterized in that, The visual data includes video data. The step of deserializing the read text binary data into a string, and converting the string and the visual binary data into a content format supported by the target framework includes: Meta-information of the video data is parsed from information in a specified part of the video binary data, where the specified part is the part where the metadata is stored. The video binary data is decoded into a frame sequence based on the metadata of the video data; Convert the frame sequence into a video tensor; Convert the string and the video tensor into a content format supported by the target frame.
4. The data loading method as described in claim 1, characterized in that, The process of reading the text binary data and the visual binary data from the second dataset, deserializing the read text binary data into a string, and converting the string and the visual binary data into a content format supported by the target framework includes: The text binary data and the visual binary data in the second dataset are read in batches using a streaming processing method, and the read text binary data and the visual binary data are temporarily stored in a buffer. The text binary data in the buffer is read and deserialized into a string, and the string and the visual binary data read from the buffer are converted into a content format supported by the target framework.
5. The data loading method according to any one of claims 1 to 4, characterized in that, The process of reading the text binary data and the visual binary data from the second dataset, deserializing the read text binary data into a string, and converting the string and the visual binary data into a content format supported by the target framework includes: The files in the second dataset are divided into several consecutive small blocks, and the files in the second dataset include the text binary data and the visual binary data; For each of the aforementioned small blocks, a corresponding subprocess reads the text binary data and the visual binary data of the small block, deserializes the read text binary data into a string, and converts the string and the visual binary data into a content format supported by the target framework.
6. The data loading method as described in claim 4, characterized in that, The second dataset includes a validation set in a second format. The step of reading the text binary data and the visual binary data from the second dataset in batches using a pipelined processing method, and temporarily storing the read text binary data and the visual binary data in a buffer, includes: Determine the number of samples in the validation set; When the number of samples exceeds a preset threshold, the text binary data and visual binary data in the verification set are read in batches using a streaming processing method, and the read text binary data and visual binary data are temporarily stored in a buffer.
7. The data loading method as described in claim 6, characterized in that, The process of reading the text binary data and the visual binary data from the second dataset, deserializing the read text binary data into a string, and converting the string and the visual binary data into a content format supported by the target framework includes: When the number of samples is not greater than a preset threshold, the text binary data and the visual binary data are preprocessed in a non-streaming manner to obtain a serialized file, which includes content in the format supported by the target framework.
8. A data loading device, characterized in that, The data loading device includes: The first dataset file reading module is used to read the first dataset file and extract the text question-and-answer pairs, the source path of the visual data, and the meta-information of the visual data from the first dataset file. The first dataset is a dataset in a first format and includes visual data and text question-and-answer pairs corresponding to the visual data. The text binary data determination module is used to serialize the text question-and-answer pair into binary data to obtain text binary data; The visual binary data determination module is used to read the corresponding visual data from the source path and convert the read visual data and the metadata of the visual data into binary data to obtain visual binary data. The second dataset determination module is used to store the text binary data and the visual binary data to obtain a second dataset. The second dataset is a dataset in a second format, and the copy overhead of the second format dataset to I / O is less than that of the first format dataset to I / O. The file reading module for the second dataset is used to read the text binary data and the visual binary data in the second dataset, deserialize the read text binary data into a string, and convert the string and the visual binary data into a content format supported by the target framework.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.
11. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1 to 7 to be performed.