Data processing method and device, electronic equipment and storage medium
By migrating the preprocessing process to the data loading process and flexibly adjusting the number of processes, the problem of prolonged GPU waiting time caused by preprocessing in embodied intelligence model training is solved, thus improving the model training speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JD DIGITS HAIYI INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-02-14
- Publication Date
- 2026-06-26
AI Technical Summary
In the training of embodied intelligence models, as the scale of sample data expands, the amount of preprocessing computation increases, leading to preprocessing queuing and other issues, which prolongs the waiting time of the GPU and reduces the training speed of the model.
The preprocessing process is moved from the main process to the data loading process, and the number of data loading processes is flexibly adjusted. The first data loading process reads and preprocesses sample data from memory and provides it directly to the GPU for model training.
By controlling the number of data loading processes, preprocessing queuing is avoided, GPU waiting time is shortened, and model training speed is improved.
Smart Images
Figure CN122288967A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to data processing methods, apparatus, electronic devices and storage media. Background Technology
[0002] Currently, in the training of embodied intelligence models, the data loading process reads sample data from memory; the main process in the central processing unit (CPU) preprocesses the acquired sample data to obtain data that can be used for model training.
[0003] In the above schemes, preprocessing typically involves multimodal alignment and normalization, which are computationally intensive. As the sample data scales to millions, the computational load increases significantly, potentially leading to preprocessing queues, thus prolonging the waiting time of the graphics processing unit (GPU) and reducing the model training speed. Summary of the Invention
[0004] This application discloses a data processing method, apparatus, electronic device, and storage medium.
[0005] One embodiment of this application proposes a data processing method applied to a central processing unit (CPU) in a GPU server. The method includes: acquiring at least one preprocessed sample data through a first data loading process; the first data loading process being configured to read at least one target sample data from a memory and preprocess the target sample data to obtain the preprocessed sample data; and providing at least one of the preprocessed sample data to a graphics processing unit (GPU) for model training processing.
[0006] The data processing method of this application embodiment includes a central processing unit (CPU) in a GPU server acquiring at least one preprocessed sample data through a first data loading process. The first data loading process is used to read at least one target sample data from memory and preprocess the target sample data to obtain preprocessed sample data. The at least one preprocessed sample data is provided to the graphics processing unit (GPU) for model training. The preprocessing process is moved from the main process to the data loading process, and the number of data loading processes can be flexibly adjusted. By controlling the number of data loading processes, preprocessing queuing can be avoided, thereby shortening the GPU's waiting time and improving the model training speed.
[0007] Another embodiment of this application proposes a data processing apparatus applied to a central processing unit (CPU) in a GPU server. The apparatus includes: a first acquisition module, configured to acquire at least one preprocessed sample data through a first data loading process; the first data loading process, configured to read at least one target sample data from a memory and preprocess the target sample data to obtain the preprocessed sample data; and a providing module, configured to provide at least one of the preprocessed sample data to a GPU for model training processing.
[0008] The data processing apparatus of this application embodiment includes a central processing unit (CPU) in a GPU server. The CPU acquires at least one preprocessed sample data through a first data loading process. This first data loading process is used to read at least one target sample data from memory and to preprocess the target sample data to obtain preprocessed sample data. The at least one preprocessed sample data is then provided to the graphics processing unit (GPU) for model training. The preprocessing process is moved from the main process to the data loading process, and the number of data loading processes can be flexibly adjusted. This allows for control over the number of data loading processes, avoiding preprocessing queuing and thus shortening the GPU's waiting time and improving model training speed.
[0009] Another embodiment of this application proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method of the embodiments of this application.
[0010] Another embodiment of this application proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method of the embodiments of this application.
[0011] Another embodiment of this application proposes a computer program product that, when executed by an instruction processor in the computer program product, implements the data processing method of the embodiments of this application.
[0012] Other effects of the above-mentioned alternative methods will be described below in conjunction with specific embodiments. Attached Figure Description
[0013] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:
[0014] Figure 1 This is a schematic flowchart of a data processing method according to an embodiment of this application.
[0015] Figure 2 This is a schematic flowchart of a data processing method according to another embodiment of this application.
[0016] Figure 3 This is a schematic diagram of a data processing method.
[0017] Figure 4 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of this application.
[0018] Figure 5 This is a block diagram of an electronic device according to an embodiment of the present application. Detailed Implementation
[0019] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0020] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. The acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.
[0021] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.
[0022] The data processing method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.
[0023] Figure 1 This is a schematic flowchart of a data processing method according to an embodiment of this application. It should be noted that the execution entity of the data processing method provided in this embodiment is a data processing device, which can be implemented by software and / or hardware. In this embodiment, the data processing device can be an electronic device, or can be configured in an electronic device to enable the electronic device to have data processing capabilities.
[0024] In this embodiment, the electronic device may include, but is not limited to, terminal devices, servers, server clusters, platforms, etc., and this embodiment does not specifically limit the electronic device. In the following embodiments, the central processing unit (CPU) in a GPU server is used as an example for describing the data processing device.
[0025] like Figure 1 As shown, the data processing method may include: Step 101: Obtain at least one preprocessed sample data through a first data loading process; the first data loading process is used to read at least one target sample data from the memory and to preprocess the target sample data to obtain preprocessed sample data.
[0026] In this embodiment of the application, taking an embodied intelligent robot as an example in the field of robotics, the embodied intelligent robot is equipped with an embodied intelligent model. The sample data involved in training the embodied intelligent model may include at least one of the following: sensor data, language command data, and robot motion data. The sensor data may include at least one of the following: robot body state data and environmental state data; the environmental state data may include environmental image data.
[0027] Embodied intelligent models, also known as embodied models, refer to artificial intelligence models applied to physical devices such as robots. The inputs typically include visual images and the entity's state, and the output is control actions.
[0028] The sensors used to collect sensor data can include internal and external sensors. Internal sensors can be used to collect robot body state data, while external sensors can be used to collect environmental state data. Robot body state data includes, for example, joint angle data and end-effector coordinate data.
[0029] In the embodiments of this application, the preprocessing of the target sample data, such as the size adjustment of the environmental image data in the target sample data, the structured processing of the language instruction data, and the normalization processing of the robot body state data and robot motion data, is not specifically limited here and can be set according to actual needs.
[0030] In this embodiment of the application, the process of the CPU executing step 101 may, for example, be sending a data access request to the first data loading process, instructing the first data loading process to perform data reading and preprocessing according to the data access request, and obtaining at least one preprocessed sample data; receiving a data access response returned by the first data loading process; the data access response carries at least one preprocessed sample data.
[0031] The first data loading process can be, for example, the DataLoader process in the PyTorch deep learning framework. The DataLoader process can read the target sample data from memory by calling the getitem function in the HuggingFace Dataset library.
[0032] Among these, storage, such as disks and memory in a GPU server, is where the sample dataset is stored.
[0033] In this embodiment of the application, to further avoid preprocessing queuing, the number of data loading processes can be determined based on the number of sample data to be read, and a first data loading process can be selected. Correspondingly, before step 101, the CPU can also perform the following process: determine the number of data loading processes based on the number of sample data to be read; select multiple first data loading processes from multiple candidate data loading processes based on the number of data loading processes and the status of multiple candidate data loading processes.
[0034] The CPU can be a multi-core CPU. When the CPU is a multi-core CPU, it can select the first data loading process from the candidate data loading processes among the CPU cores, thereby improving the utilization of multi-core resources.
[0035] After determining the number of data loading processes, the number of processes can be set by configuring `num_workers`, thus enabling the selection of the first data loading process. The value of `num_workers` represents the number of data loading processes.
[0036] In this embodiment of the application, in order to ensure that the GPU can directly obtain the required data from at least one preprocessed sample data, improve the data reading speed, and thus further improve the model training speed, after step 101, the CPU can also perform the following process: perform format conversion processing on at least one preprocessed sample data according to the data format requirements of the GPU server.
[0037] Step 102: Provide at least one preprocessed sample data to the image processor GPU for model training processing.
[0038] The GPU can train the model by performing gradient calculations and parameter updates based on at least one preprocessed sample data.
[0039] The data processing method of this application embodiment includes a central processing unit (CPU) in a GPU server acquiring at least one preprocessed sample data through a first data loading process. The first data loading process is used to read at least one target sample data from memory and preprocess the target sample data to obtain preprocessed sample data. The at least one preprocessed sample data is provided to the graphics processing unit (GPU) for model training. The preprocessing process is moved from the main process to the data loading process, and the number of data loading processes can be flexibly adjusted. By controlling the number of data loading processes, preprocessing queuing can be avoided, thereby shortening the GPU's waiting time and improving the model training speed.
[0040] based on Figure 1 Based on the previous example, the preprocessing logic can be moved to the data loading process before loading the sample data. This allows for flexible control over the number of data loading processes, thereby improving the preprocessing speed of the sample data, shortening the GPU's waiting time, and further improving the model training speed. Figure 2 This is a schematic flowchart of a data processing method according to another embodiment of this application. It should be noted that this embodiment is... Figure 1 Further refinement or optimization of the embodiments.
[0041] like Figure 2 As shown, the data processing method may include: Step 201: Obtain the preprocessing logic and the second data loading process; the second data loading process is used to read at least one target sample data from the memory; the preprocessing logic is used to preprocess the target sample data.
[0042] In this embodiment of the application, the triggering condition for step 201 may include at least one of the following: no preprocessing logic exists in the data loading process, preprocessing logic exists in the CPU main process, the GPU waits for sample data for a duration greater than a preset duration threshold, etc., without specific limitations here.
[0043] In one example of this application embodiment, the process of the CPU obtaining preprocessing logic can be, for example, determining whether preprocessing logic exists in the CPU main process; and if preprocessing logic exists in the CPU main process, obtaining the preprocessing logic in the CPU main process.
[0044] To avoid the CPU repeatedly executing the preprocessing logic, the preprocessing logic can be integrated into the second data loading process. After obtaining the first data loading process, the preprocessing logic in the CPU main process can be deleted.
[0045] In another example, the process by which the CPU obtains the preprocessing logic can be as follows: determine whether preprocessing logic exists in the model training process; if preprocessing logic exists in the model training process, extract the preprocessing logic from the model training process.
[0046] To avoid the GPU repeatedly executing preprocessing logic during the model training process, the preprocessing logic can be integrated into the second data loading process. After the first data loading process is obtained, the GPU is prompted to delete the preprocessing logic in the model training process.
[0047] The second data loading process can be, for example, the DataLoader Worker process that can be called by the CPU main process.
[0048] Step 202: Integrate the preprocessing logic into the second data loading process to obtain the first data loading process.
[0049] In one example of this application embodiment, the process of the CPU executing step 202 may be as follows: obtain a custom function in the second data loading process; the custom function is empty; use preprocessing logic to replace the custom function in the second data loading process to obtain the first data loading process.
[0050] Among them, user-defined functions can be, for example, the collate_fn function.
[0051] Alternatively, as an alternative to the above example, the CPU can add preprocessing logic to a custom function in the second data loading process to obtain the first data loading process.
[0052] In this example, when there are multiple second data loading processes, preprocessing logic needs to be added for each second data loading process to obtain the first data loading process.
[0053] Alternatively, as an alternative to the above example, the CPU can add preprocessing logic to the `collate_fn` function in the HuggingFace Dataset library of the PyTorch deep learning framework to implement update processing of the `collate_fn` function in the HuggingFace Dataset library.
[0054] In this example, only the `collate_fn` function in the HuggingFace Dataset library needs to be updated. The second data loading process, during runtime, calls and executes the `collate_fn` function in the HuggingFace Dataset library to perform the preprocessing logic.
[0055] In another example, the process of the CPU executing step 202 can be as follows: obtaining the target function in the second data loading process; the target function reads at least one target sample data from memory during runtime; and adding preprocessing logic to the target function in the second data loading process to obtain the first data loading process.
[0056] The target function can be, for example, the getitem function.
[0057] In this example, when there are multiple second data loading processes, preprocessing logic needs to be added to the objective function for each second data loading process to obtain the first data loading process.
[0058] Alternatively, as an alternative to the above example, the CPU can add preprocessing logic to the objective function in the HuggingFace Dataset library within the PyTorch deep learning framework to implement update processing of the objective function in the HuggingFace Dataset library.
[0059] In this example, only the target function in the HuggingFace Dataset library needs to be updated. The second data loading process, at runtime, calls and executes the target function in the HuggingFace Dataset library to perform the preprocessing logic.
[0060] Step 203: Obtain at least one preprocessed sample data through the first data loading process; the first data loading process is used to read at least one target sample data from the memory and to preprocess the target sample data to obtain preprocessed sample data.
[0061] Step 204: Provide at least one preprocessed sample data to the image processor GPU for model training processing.
[0062] For a detailed explanation of steps 203 to 204, please refer to [link / reference needed]. Figure 1 The contents of steps 101 to 102 in the embodiment will not be described in detail here.
[0063] The data processing method of this application embodiment includes a central processing unit (CPU) in a GPU server. The CPU acquires preprocessing logic and a second data loading process. The second data loading process reads at least one target sample data from memory. The preprocessing logic preprocesses the target sample data. The preprocessing logic is integrated into the second data loading process to obtain a first data loading process. At least one preprocessed sample data is acquired through the first data loading process. The first data loading process reads at least one target sample data from memory and preprocesses the target sample data to obtain preprocessed sample data. At least one preprocessed sample data is provided to the GPU for model training. The integration of preprocessing logic into the second data loading process allows the first data loading process to load sample data and preprocess the loaded sample data, thereby avoiding preprocessing of the sample data by the CPU main process. This allows for flexible control over the number of data loading processes, avoiding preprocessing queuing and shortening GPU waiting time, thus improving model training speed.
[0064] To ensure a clear understanding of this application, the following will be combined with... Figure 3 The method of this embodiment is described by way of example.
[0065] Figure 3 This is a schematic diagram of a data processing method. Figure 3 This may include the following steps: Step 301: The parallel worker process performs raw reading and complex preprocessing.
[0066] In this process, multiple parallel worker processes read sample data from disk storage (i.e., the data in Episode Data) and preprocess the read sample data to obtain preprocessed sample data.
[0067] Step 302: The main process performs rapid aggregation on the read data.
[0068] The main process (i.e., the CPU main process) performs rapid aggregation processing on the preprocessed sample data provided by multiple parallel worker processes to obtain the aggregation result.
[0069] Step 303: The aggregation result obtained by fast aggregation is transmitted to the GPU via H2D for model training processing.
[0070] Corresponding to the data processing methods provided in the above embodiments, one embodiment of this application also provides a data processing apparatus. Since the data processing apparatus provided in this application corresponds to the data processing methods provided in the above embodiments, the implementation methods of the data processing methods are also applicable to the data processing apparatus provided in this embodiment, and will not be described in detail in this embodiment.
[0071] Figure 4 This is a schematic diagram of a data processing apparatus according to an embodiment of this application. It should be noted that the data processing apparatus can be implemented in software and / or hardware. In this embodiment, the data processing apparatus can be an electronic device, or can be configured within an electronic device. The electronic device in this embodiment may include, but is not limited to, terminal devices and servers, etc., and this embodiment does not specifically limit the type of electronic device.
[0072] like Figure 4 As shown, the data processing device 400 is applied to the central processing unit (CPU) in a GPU server and includes: a first acquisition module 401 and a providing module 402.
[0073] The first acquisition module 401 is used to acquire at least one preprocessed sample data through a first data loading process; the first data loading process is used to read at least one target sample data from the memory and preprocess the target sample data to obtain the preprocessed sample data; the providing module 402 is used to provide at least one of the preprocessed sample data to the image processor GPU for model training processing.
[0074] In one embodiment of this application, the apparatus further includes: a second acquisition module and an integration module; the second acquisition module is used to acquire preprocessing logic and a second data loading process; the second data loading process is used to read at least one of the target sample data from the memory; the preprocessing logic is used to preprocess the target sample data; the integration module is used to integrate the preprocessing logic into the second data loading process to obtain the first data loading process.
[0075] In one embodiment of this application, the second acquisition module is specifically used to determine whether the preprocessing logic exists in the CPU main process; and if the preprocessing logic exists in the CPU main process, to acquire the preprocessing logic in the CPU main process.
[0076] In one embodiment of this application, the apparatus further includes a deletion module for deleting the preprocessing logic in the CPU main process.
[0077] In one embodiment of this application, the second acquisition module is specifically used to determine whether the preprocessing logic exists in the model training process; and if the preprocessing logic exists in the model training process, to extract the preprocessing logic from the model training process.
[0078] In one embodiment of this application, the integration module is specifically used to: obtain a custom function in the second data loading process; the custom function is empty; and use the preprocessing logic to replace the custom function in the second data loading process to obtain the first data loading process.
[0079] In one embodiment of this application, the integration module is specifically used to: obtain the target function in the second data loading process; the target function reads at least one target sample data from the memory during runtime; and add the preprocessing logic to the target function in the second data loading process to obtain the first data loading process.
[0080] In one embodiment of this application, the number of the first data loading processes is multiple; the device further includes: a determining module and a selecting module; the determining module is used to determine the number of data loading processes based on the number of sample data to be read; the selecting module is used to select multiple first data loading processes from the multiple candidate data loading processes based on the number of data loading processes and the status of multiple candidate data loading processes.
[0081] In one embodiment of this application, the apparatus further includes: a conversion processing module, configured to perform format conversion processing on at least one of the preprocessed sample data according to the data format requirements of the GPU server.
[0082] The data processing apparatus of this application embodiment includes a central processing unit (CPU) in a GPU server. The CPU acquires at least one preprocessed sample data through a first data loading process. This first data loading process is used to read at least one target sample data from memory and to preprocess the target sample data to obtain preprocessed sample data. The at least one preprocessed sample data is then provided to the graphics processing unit (GPU) for model training. The preprocessing process is moved from the main process to the data loading process, and the number of data loading processes can be flexibly adjusted. This allows for control over the number of data loading processes, avoiding preprocessing queuing and thus shortening the GPU's waiting time and improving model training speed.
[0083] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.
[0084] Figure 5This is a block diagram of an electronic device according to an embodiment of the present application.
[0085] like Figure 5 As shown, the electronic device includes: The memory 501, the processor 502, and the computer instructions stored in the memory 501 and executable on the processor 502.
[0086] When processor 502 executes instructions, it implements the data processing method provided in the above embodiments.
[0087] Furthermore, electronic devices also include: Communication interface 503 is used for communication between memory 501 and processor 502.
[0088] Memory 501 is used to store computer instructions that can be executed on processor 502.
[0089] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0090] The processor 502 is used to implement the data processing method of the above embodiments when executing a program.
[0091] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0092] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.
[0093] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0094] This application also proposes a computer program product that implements the data processing method of the embodiments of this application when the instruction processor in the computer program product is executed.
[0095] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0097] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0098] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0099] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0100] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0102] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A data processing method, characterized in that, The method, applied to a central processing unit (CPU) in a GPU server, includes: At least one preprocessed sample data is obtained through a first data loading process; the first data loading process is used to read at least one target sample data from the memory and to preprocess the target sample data to obtain the preprocessed sample data; At least one of the preprocessed sample data is provided to the graphics processing unit (GPU) for model training processing.
2. The method according to claim 1, characterized in that, The method further includes: The system acquires preprocessing logic and a second data loading process; the second data loading process is used to read at least one of the target sample data from the memory; the preprocessing logic is used to preprocess the target sample data. The preprocessing logic is integrated into the second data loading process to obtain the first data loading process.
3. The method according to claim 2, characterized in that, The preprocessing logic includes: Determine whether the preprocessing logic exists in the CPU main process; If the preprocessing logic exists in the CPU main process, the preprocessing logic in the CPU main process is obtained.
4. The method according to claim 3, characterized in that, The method further includes: Delete the preprocessing logic in the CPU main process.
5. The method according to claim 2, characterized in that, The preprocessing logic includes: Determine whether the preprocessing logic exists in the model training process; If the preprocessing logic exists in the model training process, the preprocessing logic is extracted from the model training process.
6. The method according to claim 2, characterized in that, The step of integrating the preprocessing logic into the second data loading process to obtain the first data loading process includes: Retrieve the custom function in the second data loading process; the custom function is empty. The preprocessing logic is used to replace the custom function in the second data loading process to obtain the first data loading process.
7. The method according to claim 2, characterized in that, The step of integrating the preprocessing logic into the second data loading process to obtain the first data loading process includes: Obtain the target function in the second data loading process; the target function reads at least one target sample data from the memory during runtime; The preprocessing logic is added to the objective function in the second data loading process to obtain the first data loading process.
8. The method according to claim 1, characterized in that, The number of the first data loading processes is multiple; the method further includes: The number of data loading processes is determined based on the number of sample data to be read; Based on the number of data loading processes and the status of multiple candidate data loading processes, select multiple first data loading processes from the multiple candidate data loading processes.
9. The method according to claim 1, characterized in that, After acquiring at least one preprocessed sample data through the first data loading process, the method further includes: According to the data format requirements of the GPU server, at least one of the preprocessed sample data is subjected to format conversion processing.
10. A data processing apparatus, characterized in that, The device is used in a central processing unit (CPU) of a GPU server, and includes: The first acquisition module is configured to acquire at least one preprocessed sample data through a first data loading process; the first data loading process is configured to read at least one target sample data from a memory and preprocess the target sample data to obtain the preprocessed sample data. A module is provided for providing at least one of the preprocessed sample data to a graphics processing unit (GPU) for model training processing.
11. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the data processing method as described in any one of claims 1-9.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method as described in any one of claims 1-9.
13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-9.