Optimization method for deep learning, host device and PNM device

Abstract

The invention provides an optimization method for deep learning, host equipment and PNM equipment. The optimization method for deep learning comprises: in each period of deep learning, loading an nth batch of data from a PNM memory of a PNM device for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the nth batch of data, n being a positive integer greater than or equal to 1; at least one of the loss value and the confidence score corresponding to each data in the nth batch of data is transmitted to the PNM device, and the nth batch of data at least comprises data prefetched by the PNM memory from a PNM memory of the PNM device or a memory outside the PNM device.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence acceleration and computer storage technology, and more specifically, to an optimization method, a host device, and a PNM device for deep learning. Background Art

[0002] Deep learning (DL) training requires a large amount of training data (TB level), and it is impossible to store all the data in the memory of the host device (or host) or graphics processing unit (GPU). Therefore, the data can only be obtained in batches from storage (e.g., solid-state drive (SSD)). Loading the next batch of data to be trained from storage to memory in a timely manner is a key issue. The training process requires multiple iterations, so data is loaded repeatedly for training. In order to avoid learning the same data sequence, a batch of training data needs to be randomly selected in each iteration. Therefore, in each cycle (epoch), the host will first shuffle the ID sequence of the dataset, and then load each batch of data in the new shuffled order. Although all data are generally loaded once in each cycle during training, different data contributes differently to training. Training too much low-quality data cannot effectively reduce the loss value, thereby affecting the training efficiency. In addition, compared with the computing speed of GPU, the IO (input and output) performance of SSD is low, so the IO between memory and storage affects the efficiency of training.

[0003] To optimize IO performance, the DeepMemoryDL solution uses CXL (Compute Express Link) to provide additional memory and fast scratch space for DL ​​workloads to reduce overall training time. Manage the allocation of additional CXL-based memory, introduce a fast intermediate storage layer, and provide intelligent prefetching and caching mechanisms for DL ​​workloads. The prefetch mechanism ensures that each training batch is loaded into memory before the next training iteration is ready to execute, reducing IO time. However, although DeepMemoryDL can improve IO efficiency, it only uses CXL features to prefetch the next batch of data and load it into memory, which does not improve training performance or fundamentally reduce useless training data.

[0004] In addition, during the training process, there is a large loss between the output and the target of some samples (i.e., the loss value is large). These samples are called difficult samples (or difficult to learn samples, difficult samples). Difficult samples contribute more to improving the accuracy of deep learning models. In order to make better use of difficult samples, the OHEM (Online Hard Example Mining) scheme treats misclassified samples as difficult samples and trains them more times than simple samples (or easy to learn samples, ordinary samples), which is similar to spending more time practicing skills that have not yet been mastered. Training more difficult samples helps improve the accuracy of the model, but it will prolong the training time, and there is still an IO bottleneck when the amount of training data is large. Therefore, this method simply increases the number of times difficult samples are trained, and does not improve training efficiency. Summary of the invention

[0005] One of the objectives of exemplary embodiments of the present disclosure is to provide an optimization method capable of optimizing sample buffering.

[0006] One of the objectives of exemplary embodiments of the present disclosure is to provide an optimization method capable of reducing training time.

[0007] One of the purposes of the exemplary embodiments of the present disclosure is to provide an optimization method that can effectively improve the sample quality in the later stage of training and improve the training performance.

[0008] According to a first aspect of the present disclosure, there is provided an optimization method for deep learning, the optimization method comprising: in each cycle of deep learning, loading an nth batch of data from a PNM memory of a PNM device for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the nth batch of data, wherein n is a positive integer greater than or equal to 1; transmitting at least one of a loss value and a confidence score corresponding to each data in the nth batch of data to the PNM device, wherein the nth batch of data includes at least data pre-fetched by the PNM memory from the PNM memory of the PNM device or a memory outside the PNM device.

[0009] According to an embodiment of the present disclosure, the optimization method may further include: before loading the n+1th batch of data for deep learning training, initiating a request to the PNM device to generate a difficult sample set in the PNM memory or update the difficult sample set in the PNM memory, wherein the difficult sample set is generated or updated based on at least one of the loss value and the confidence score corresponding to each data in the nth batch of data.

[0010] According to an embodiment of the present disclosure, the optimization method may further include: loading the n+1th batch of data from the PNM memory that generates or updates the difficult sample set for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the n+1th batch of data; transmitting at least one of a loss value and a confidence score corresponding to each data in the n+1th batch of data to the PNM device, wherein the n+1th batch of data includes the difficult-to-learn sample data in the difficult sample set and the easy-to-learn sample data pre-fetched by the PNM memory from the PNM memory of the PNM device or the memory outside the PNM device.

[0011] According to an embodiment of the present disclosure, the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data can be determined based on at least one of the loss value and the confidence score corresponding to each data in the nth batch of data, the batch size of deep learning, and the total number of cycles of deep learning.

[0012] According to an embodiment of the present disclosure, the optimization method may further include: initiating a pre-fetch instruction to the PNM device at a preset time interval, the pre-fetch instruction being used to instruct the PNM memory to pre-fetch data from the PNM memory or a memory outside the PNM device.

[0013] According to a second aspect of the present disclosure, an optimization method for deep learning is provided, the optimization method comprising: prefetching data from a PNM memory of a PNM device or a memory outside the PNM device in response to a prefetch instruction request of a host device.

[0014] According to an embodiment of the present disclosure, the optimization method may further include: in response to a sample request from a host device, generating or updating a difficult sample set in a PNM memory according to at least one of a loss value and a confidence score corresponding to each data in the nth batch of data obtained by performing deep learning training on the nth batch of data, where n is a positive integer greater than or equal to 1.

[0015] According to an embodiment of the present disclosure, the optimization method may further include: determining the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained, wherein the easy-to-learn sample data in the n+1th batch of data to be trained comes from data in the pre-fetched data of the PNM memory except for data corresponding to the sample data determined to be difficult-to-learn, and the difficult-to-learn sample data in the n+1th batch of data to be trained comes from a difficult sample set.

[0016] According to an embodiment of the present disclosure, the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained can be determined based on at least one of the loss value and the confidence score corresponding to each data in the nth batch of data, the batch size of deep learning, and the total number of cycles of deep learning.

[0017] According to an embodiment of the present disclosure, the step of determining the proportion of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained may include: in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a first condition, reducing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and increasing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained; in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a second condition, increasing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and reducing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained, wherein the first condition includes that the loss value of each data in the nth batch of data is greater than a first threshold value or the confidence score of each data in the nth batch of data is less than a second threshold value, and the second condition includes that the loss value of each data in the nth batch of data is less than or equal to the first threshold value or the confidence score of each data in the nth batch of data is greater than or equal to the second threshold value.

[0018] According to an embodiment of the present disclosure, the optimization method may further include: before determining the n+1th batch of data to be trained, marking the pre-fetched data in the PNM memory corresponding to the sample data determined to be difficult to learn as cleared.

[0019] According to a third aspect of the present disclosure, an optimization method for deep learning is provided, and the optimization method may include: pre-fetching data from a PNM memory of a PNM device or a memory outside the PNM device; in each cycle of deep learning, loading the nth batch of data from the PNM memory for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the nth batch of data; generating a difficult sample set in the PNM memory or updating the difficult sample set in the PNM memory according to at least one of a loss value and a confidence score corresponding to each data in the nth batch of data obtained by performing deep learning training on the nth batch of data, wherein n is a positive integer greater than or equal to 1.

[0020] According to an embodiment of the present disclosure, the optimization method may further include: determining the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained, wherein the easy-to-learn sample data in the n+1th batch of data to be trained comes from data in the pre-fetched data of the PNM memory except for data corresponding to the sample data determined to be difficult-to-learn, and the difficult-to-learn sample data in the n+1th batch of data to be trained comes from a difficult sample set.

[0021] According to an embodiment of the present disclosure, the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained can be determined based on at least one of the loss value and the confidence score corresponding to each data in the nth batch of data, the batch size of deep learning, and the total number of cycles of deep learning.

[0022] According to an embodiment of the present disclosure, the step of determining the proportion of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained may include: in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a first condition, reducing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and increasing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained; in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a second condition, increasing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and reducing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained, wherein the first condition includes that the loss value of each data in the nth batch of data is greater than a first threshold value or the confidence score of each data in the nth batch of data is less than a second threshold value, and the second condition includes that the loss value of each data in the nth batch of data is less than or equal to the first threshold value or the confidence score of each data in the nth batch of data is greater than or equal to the second threshold value.

[0023] According to an embodiment of the present disclosure, the optimization method may further include: loading the n+1th batch of data from the PNM memory in which the difficult sample set is generated or updated for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the n+1th batch of data.

[0024] According to an embodiment of the present disclosure, the optimization method may further include: before determining the n+1th batch of data to be trained, marking the pre-fetched data in the PNM memory corresponding to the sample data determined to be difficult to learn as cleared.

[0025] According to a fourth aspect of the present disclosure, a host device for deep learning is provided, the host device comprising: a processing unit and a storage unit, the storage unit storing a program or an instruction, and when the program or the instruction is executed by the processing unit, the processing unit is prompted to execute the above-mentioned optimization method.

[0026] According to a fifth aspect of the present disclosure, a PNM device for deep learning is provided, the PNM device comprising: a PNM memory, pre-fetching data from the PNM memory or a memory outside the PNM device; a difficult sample update module, configured to: generate or update a difficult sample set in the PNM memory according to at least one of a loss value and a confidence score corresponding to each data in the nth batch of data obtained by deep learning training for the nth batch of data, wherein n is a positive integer greater than or equal to 1; an adaptive sample allocation module, determining a ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained, wherein the easy-to-learn sample data in the n+1th batch of data to be trained comes from data in the pre-fetched data of the PNM memory except for data corresponding to the sample data determined to be difficult to learn, and the difficult-to-learn sample data in the n+1th batch of data to be trained comes from the difficult sample set.

[0027] According to a sixth aspect of the present disclosure, a computer system is provided, the computer system comprising the above-mentioned host device and / or the above-mentioned PNM device.

[0028] The optimization method according to the embodiment of the present disclosure can effectively utilize PNM memory and computing cores to actively pre-fetch and cache training data to improve the performance of deep learning workloads. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects and features of the exemplary embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate the embodiments, in which: Figure 1 is a flow chart showing an optimization method according to a first embodiment of the present disclosure; Figure 2 is a flow chart showing an optimization method according to a first embodiment of the present disclosure; Figure 3 is a block diagram showing a data reading process of the prior art and a data loading process of the present disclosure.

[0030] Figure 4 is a block diagram illustrating a computer system of the present disclosure; Figure 5 is a flow chart showing an optimization method according to a second embodiment of the present disclosure; Figure 6 is a flow chart showing an optimization method according to a second embodiment of the present disclosure; Figure 7 is a flow chart showing an optimization method according to a second embodiment of the present disclosure; Figure 8 is a schematic diagram showing a method for selecting the n+1th batch of data; Fig. 9 is a flow chart showing an optimization method according to a third embodiment of the present disclosure; Fig.10 is a block diagram illustrating a host device and a PNM device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the drawings in order to explain the present disclosure.

[0032] The following detailed description is provided to help gain a comprehensive understanding of the methods, devices and / or systems described herein. However, the order of operations described herein is only an example and is not limited to those orders set forth herein, but may be equivalently replaced or changed except for operations that must occur or be performed in a specific order. In addition, for greater clarity and simplicity, the description of content known in the art will be omitted or simplified.

[0033] Unless otherwise specified, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments may be omitted if they appear again in later embodiments. In addition, the technical features described in different or the same embodiments may be combined in any manner, as long as the combined embodiments or technical solutions are complete and can solve the technical problems of the present application or achieve the technical effects described or not described in the present disclosure but can be determined based on the above complete technical solutions.

[0034] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as terms defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner. The terms used in the present disclosure will be briefly described below.

[0035] Near Memory Processing (PNM) is a technology that integrates memory and logic chips into advanced integrated circuit packages, reducing data movement between the CPU and memory by utilizing memory for data calculations. In the case of PNM, computing functions are performed closer to the memory, thereby reducing the bottleneck of data transfer between the CPU and memory.

[0036] The Compute Express Link (CXL) interface is an open standard for high-speed, high-capacity connections between processors and memory. CXL memory expansion technology allows servers to expand memory capacity to tens of TB while increasing bandwidth to hundreds of GB per second.

[0037] CXL-PNM is a cutting-edge technology that reduces data movement between the CPU and memory. By placing the processing unit closer to the memory, CXL-PNM technology loads AI models 2 times faster and increases capacity 4 times.

[0038] Deep learning training is I / O intensive because data samples need to be continuously fetched from remote storage (e.g., SSD). As accelerators become more powerful and more data hungry, I / O performance lags behind. In addition, research results show that not all data samples are equally important, and different data samples contribute differently to improving model accuracy. Therefore, proactively prefetching and caching training data is critical. In addition, a reasonable choice of the ratio of difficult and easy samples can help reduce the number of iterations of model training while improving the accuracy of deep learning models.

[0039] The present disclosure provides an optimization method for deep learning, specifically, multiple training data to be trained are placed in PNM memory (PNM Memory) in advance, so that the data can be loaded into the host at high speed using CXL.mem (CXL memory protocol). Secondly, the training efficiency is further improved by the reasonable use of difficult samples (or difficult-to-learn sample data), specifically, the difficult-to-learn sample data and the easy-to-learn sample data (or easy-to-learn samples, ordinary samples) are mixed to generate the next batch of data to be trained, so as to dynamically adjust the ratio of difficult-to-learn sample data and easy-to-learn sample data according to the training process of the host. The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0040] Figure 1 is a flow chart showing an optimization method according to a first embodiment of the present disclosure.

[0041] The optimization method according to the first embodiment of the present disclosure may be performed by a host or a host device. The host device may include a central processing unit (CPU), a GPU, etc. The optimization method according to the embodiment of the present disclosure may include steps S110 and S120.

[0042] In step S110, in each cycle (epoch) of deep learning, the nth batch of data is loaded from the PNM memory of the PNM device for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the nth batch of data, wherein n is a positive integer greater than or equal to 1. Deep learning may include various neural network models such as back propagation algorithm, convolutional neural network, recurrent neural network, generative adversarial network, etc. As long as a batch of data is loaded from the PNM device, the training time can be reduced, the IO bottleneck can be improved, etc.

[0043] The total number of cycles and batch size (i.e., the number or amount of data in each batch) of deep learning can be pre-set as needed. During deep learning training, data in a data set stored in a memory needs to be loaded in batches. The total batch can be determined based on the total amount of data in the data set and the batch size. As an example, n can be greater than or equal to 1 and less than or equal to the ratio of the total amount of data in the data set to the batch size.

[0044] At least one of a loss value and a confidence score corresponding to each data in each batch of data can be determined according to the value returned by the deep learning training. Specifically, the loss value and the confidence score can be obtained according to an existing calculation method.

[0045] In each cycle (epoch), the same batch size and the same loading method (loading from PNM memory) can be used for deep learning training. However, in different epochs, the data loaded in the same batch may be different, and the ratio of difficult-to-learn sample data to easy-to-learn sample data in the same batch may be different.

[0046] In step S120, at least one of the loss value and the confidence score corresponding to each data in the nth batch of data is transmitted to the PNM device. As an example, the update data agent in the host device (a software module in the CPU in the host device) can communicate with the compute core in the PNM device, thereby transmitting the loss value and the confidence score corresponding to each batch of data to the compute core in the PNM device.

[0047] Although not shown, the optimization method according to the embodiment of the present disclosure may further include: initiating a prefetch instruction to the PNM device at a preset time interval, the prefetch instruction being used to instruct the PNM memory to prefetch data from the PNM memory or a memory outside the PNM device.

[0048] The host device may send a pre-fetch instruction to the PNM device before starting training, so that the PNM memory pre-fetches data from the PNM memory of the PNM device or a memory outside the PNM device, and the amount of data or the number of data pre-fetched may be greater than the amount of data in each batch, that is, the amount of data or the number of data pre-fetched may be greater than the batch size. As an example, multiple batches of data (e.g., at least two batches of data) may be pre-fetched. The amount of data pre-fetched may be determined according to the PNM memory capacity. The larger the PNM memory capacity, the larger the amount of data pre-fetched. For example, there are 100 pictures stored in the memory, and the PNM memory may pre-fetch 20 pictures therefrom, and each batch may include 5 pictures (i.e., 5 pictures are loaded each time).

[0049] In addition, after the nth batch of data training is completed, the host device may initiate a pre-fetch request to the PNM device, and the PNM memory of the PNM device may pre-fetch data from the PNM storage or other storage according to the request. The amount of data pre-fetched each time may be the same or different.

[0050] When n=1, the first batch of data is loaded from the PNM memory of the PNM device for deep learning training to obtain at least one of the loss value and the confidence score corresponding to each data in the first batch of data, and a difficult sample set can be generated or updated according to at least one of the loss value and the confidence score corresponding to each data in the first batch of data. Assuming that there are 5 data (for example, 5 pictures) in the first batch of data, the first data (for example, the first picture) and the second data (for example, the second picture) are determined to be difficult data (with a higher loss value) according to the loss value or confidence score corresponding to the 5 data, then these two difficult-to-learn data can be added to the difficult sample set in the PNM memory (this will be described in more detail later). The difficult sample set may not originally exist in the PNM memory, or the difficult sample set may be empty. The difficult sample set can be generated or updated as the deep learning training proceeds.

[0051] Figure 2 is a flow chart showing an optimization method according to a first embodiment of the present disclosure.

[0052] Figure 2 The optimization method according to the first embodiment of the present disclosure shown may also be executed by a host device, and the optimization method according to the first embodiment of the present disclosure may further include step S130, step S140 and step S150.

[0053] In step S130, before loading the n+1th batch of data for deep learning training, a request is initiated to the PNM device to generate a difficult sample set in the PNM memory or update the difficult sample set in the PNM memory, wherein the difficult sample set is generated or updated based on at least one of a loss value and a confidence score corresponding to each data in the nth batch of data.

[0054] For example, before loading the second batch of data for deep learning training, a request is initiated to the PNM device to generate a difficult sample set in the PNM memory or to update the difficult sample set in the PNM memory.

[0055] In step S140, the n+1th batch of data is loaded from the PNM memory for generating or updating the difficult sample set for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the n+1th batch of data.

[0056] By analogy, after the last batch of data is loaded, at least one of the loss value and the confidence score returned by the deep learning training may still be returned to the PNM device, but this is merely an example.

[0057] In step S150 , at least one of a loss value and a confidence score corresponding to each data in the (n+1)th batch of data is transmitted to the PNM device.

[0058] The (n+1)th batch of data includes the difficult-to-learn sample data in the difficult sample set and the easy-to-learn sample data pre-fetched by the PNM memory from the PNM memory of the PNM device or the memory outside the PNM device.

[0059] It should be noted that the total batch can be determined based on the ratio of the total sample data volume to the batch size. For example, if there are 100 pictures stored in the memory, and each batch includes 5 pictures (i.e., 5 pictures are loaded each time), then the total batch can be 20, and the last batch of data is the 20th batch of data. The above method of determining batches is also exemplary, and the total batch can be greater than 20. For example, in a certain epoch, the difficult sample set has a total of 20 pictures, the total data volume is 120, and the batch can be 24. The data in the difficult sample set may still be determined as a difficult sample after training, so the data in the difficult sample set may be trained multiple times in each epoch, and thus, there may be more batches.

[0060] That is to say, in the embodiment of the present disclosure, all data can be loaded into the host (e.g., the neural network model running in the GPU of the host) in batches in each cycle (epoch), but considering that the second batch and each batch of data after the second batch may include difficult-to-learn sample data, the total amount of data loaded in each epoch may be greater than the total amount of samples stored in the memory, but this is only an example, and the total amount of data loaded in each epoch may be equal to the total amount of samples stored in the memory (in each epoch, there is a part of the easy-to-learn sample data that is not loaded into the host). For example, the total amount of data stored in the memory is 100, each batch includes 5 photos, and the batch can be 20. After training, the data in the difficult sample set may still be determined as a difficult sample, but no matter how many times the difficult sample is learned (e.g., at least twice), the batch can be 20, that is, part of the data stored in the memory is not loaded into the host for learning. The amount of this part of the data can be limited, for example, not more than 10% of the total amount of data. Still taking the total amount of data as 100 as an example, the data that has not been loaded for learning may not exceed 10.

[0061] Figure 3 is a block diagram showing a data reading process of the prior art and a data loading process of the present disclosure.

[0062] Reference Figure 3 In the prior art, the GPU memory in the host device reads the sample data to be trained in the storage device (for example, SSD) at a reading speed of only about 7 Gb / s. However, the embodiments of the present disclosure pre-fetch multiple batches of data through the memory of the PNM device, and the host device can load the pre-fetched data in the PNM device memory in batches, and the loading speed can reach 35 Gb / s.

[0063] That is, according to the optimization method of the embodiment of the present disclosure, data is pre-fetched from the PNM memory or the external memory to the PNM memory, so that access to the data can be achieved at a higher speed, eliminating IO access to the slower storage layer. In addition, when loading data in batches for training in each cycle, what is loaded is the pre-fetched data in the memory of the PNM device and the difficult sample data selected from the dynamically adjusted difficult sample set. In this way, the training time can be improved and reduced, and the efficiency and accuracy of deep learning can be improved. In addition, the sample quality in the later stage of deep learning training is improved, and the training performance is optimized.

[0064] Figure 4 is a block diagram illustrating a computer system of the present disclosure.

[0065] The computer system according to an embodiment of the present disclosure may include a host (or host device) and / or a PNM device. The PNM device here may include a PNM memory, a PNM storage, and a control module (including a computing core). Figure 4 The memory shown is integrated with the PNM memory, that is, Figure 4 The memory shown is part of the PNM device. However, this is only an example, and the memory may also be a memory external to the PNM device.

[0066] Reference Figure 4 , the host's update data agent (a software module of the host CPU) communicates with the PNM device, specifically, communicates with the computing core of the PNM device, passes the loss value and / or confidence score returned by the deep learning training, and sends an instruction to pre-fetch the next batch of samples to the PNM device. After performing forward propagation calculations in the GPU, the update data agent can pass the calculated loss value, etc. to the computing core of the PNM device, and initiate a request to update the difficult sample set. After that, the PNM device will pre-fetch randomly arranged samples according to the request and store them in the PNM memory, and update the difficult sample set.

[0067] The computing core of the PNM device can check whether the sample is difficult or easy and update the difficult sample set. In addition, the computing core can also determine the ratio of difficult sample data and easy sample data in the next batch. That is, in the PNM memory of the PNM device, multiple batches of samples (ordinary samples or easy samples) and difficult samples are stored. When the host requests the next batch of samples, the PNM memory of the PNM device can prepare the next batch of samples. The specific execution steps of the PNM device will be described in detail later.

[0068] As the number of iterations increases, the deep learning model becomes more powerful, and many samples become easy-to-learn samples, which are easier to be classified by the deep learning model, resulting in the gradient in the back propagation being dominated by the loss of simple samples. Therefore, the computing core of the PNM device can check whether the sample is difficult or simple, and update the current set of difficult samples.

[0069] In the initial stage of training, multiple batches of data will be pre-fetched and cached in the PNM memory. Later, when the host device requests the next batch of data, the computing core can determine whether the sample is a difficult sample. If it is a difficult sample, it will be saved in the PNM memory; if not, it will be marked as cleared. When the PNM memory space is insufficient, the newly pre-fetched samples will replace the samples marked as cleared.

[0070] Specifically, the ratio of difficult-to-learn sample data to easy-to-learn sample data in the n+1th batch of data can be determined based on at least one of the loss value and the confidence score corresponding to each data in the nth batch of data, the batch size of deep learning, and the total number of cycles of deep learning. That is, the "determination" here can refer to determining a specific ratio value, but this is only an example, and the "determination" here can also refer to adjusting the ratio of difficult-to-learn sample data to easy-to-learn sample data.

[0071] It should be noted that all pre-fetched data can be simple data (i.e., easy-to-learn data or ordinary data), but this is only an example, and the pre-fetched data can be classified. For example, the pre-fetched data is classified into difficult-to-learn data and easy-to-learn data based on past experience before learning, and the pre-fetched difficult-to-learn data can be stored in a difficult sample set.

[0072] Figures 5 to 7 is a flowchart showing an optimization method according to a second embodiment of the present disclosure, Figure 8 It is a schematic diagram showing the method of selecting the n+1th batch of data.

[0073] The optimization method according to the second embodiment of the present disclosure may be executed by a PNM device.

[0074] Reference Figure 5 , the optimization method according to the second embodiment of the present disclosure may include step S210.

[0075] In step S210, data is pre-fetched from a PNM memory of the PNM device or a memory outside the PNM device in response to a pre-fetch instruction request of the host device. The PNM device of the present disclosure may be a CXL-PNM device.

[0076] The pre-fetch instruction request as described above may be initiated to the PNM device at a preset time interval, but this is only an example. When the PNM memory capacity is sufficient, all data may be pre-fetched at one time. As an example, when data needs to be pre-fetched multiple times, the amount of data pre-fetched each time may be different or the same. When pre-fetching data, the total data in the PNM memory may always remain unchanged, for example, eighty percent of the PNM memory capacity may always be used.

[0077] Reference Figure 6 In addition to step S210, the optimization method according to the second embodiment of the present disclosure may further include step S220.

[0078] In step S220, in response to a sample request from the host device, a difficult sample set in the PNM memory is generated or updated according to at least one of a loss value and a confidence score corresponding to each data in the nth batch of data obtained by performing deep learning training on the nth batch of data, wherein n is a positive integer greater than or equal to 1, and n may be less than or equal to the ratio of the total data amount to the batch size. Due to the existence of difficult sample data, n may also be greater than the ratio of the total data amount to the batch size.

[0079] Reference Figure 7 In addition to step S210 and step S220, the optimization method according to the second embodiment of the present disclosure may further include step S230.

[0080] In step S230, the ratio of difficult-to-learn sample data to easy-to-learn sample data in the n+1th batch of data to be trained is determined, wherein the easy-to-learn sample data in the n+1th batch of data to be trained comes from data in the pre-fetched data of the PNM memory except for data corresponding to the difficult-to-learn sample data, and the difficult-to-learn sample data in the n+1th batch of data to be trained comes from the difficult sample set.

[0081] Reference Figure 8 , assuming that the batch size is 5, the number of difficult-to-learn sample data in the difficult sample set is 5, and the number of pre-taken easy-to-learn sample data is 15 (excluding data marked as cleared or data marked as cleared has been covered). After learning the nth batch of data, it is determined that the ratio of difficult-to-learn sample data to easy-to-learn sample data is two-fifths and three-fifths respectively, that is, it is determined that the n+1th batch of data includes 2 difficult-to-learn sample data and 3 easy-to-learn sample data. These 2 difficult-to-learn sample data can be randomly selected from the difficult sample set. Similarly, the 3 easy-to-learn sample data can also be randomly selected from the pre-taken easy-to-learn data. In addition, as an example, each data can be numbered, and when selecting, it can be selected in the order of numbers.

[0082] As described above, the ratio of the difficult-to-learn sample data and the easy-to-learn sample data in the n+1th batch of data to be trained can be determined according to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data, the batch size of deep learning, and the total number of cycles of deep learning. The size of the ratio of the difficult-to-learn sample data and the easy-to-learn sample data in the next batch of data to be trained can be specifically determined, that is, the number of the difficult-to-learn sample data and the easy-to-learn sample data in the next batch of data to be trained can be determined.

[0083] In addition, the step of determining the proportion of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained may include: in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a first condition, reducing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and increasing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained; in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a second condition, increasing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and reducing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained, wherein the first condition includes that the loss value of each data in the nth batch of data is greater than a first threshold value or the confidence score of each data in the nth batch of data is less than a second threshold value, and the second condition includes that the loss value of each data in the nth batch of data is less than or equal to the first threshold value or the confidence score of each data in the nth batch of data is greater than or equal to the second threshold value.

[0084] That is, if the loss value of the previous batch of data is high (for example, misclassification), then the number of easy-to-learn examples in the next batch will increase and the number of hard-to-learn examples will decrease; conversely, the number of easy-to-learn examples in the next batch will decrease and the number of hard-to-learn examples will increase. As the number of training iterations increases, the number of misclassification examples becomes smaller and smaller. Therefore, in the later stages of training, the number of hard-to-learn examples and easy-to-learn examples in a batch of data will tend to be similar.

[0085] After the data ratio is determined, the PNM device will allocate difficult and easy samples in proportion. When the host wants to load the next batch of data, the assembled data batch will be returned to the host from the PNM memory, which can effectively reduce the reading time compared to randomly taking samples from the storage device.

[0086] Although not shown, the optimization method according to the second embodiment of the present disclosure may further include: before determining the n+1th batch of data to be trained, marking the pre-fetched data corresponding to the sample data determined to be difficult to learn in the PNM memory as cleared. Compared with clearing the data directly in real time, the subsequent use of the new pre-fetched data to overwrite the data marked as cleared can effectively reduce energy consumption.

[0087] Fig. 9is a flowchart illustrating an optimization method according to a third embodiment of the present disclosure.

[0088] Reference Fig. 9 , the optimization method according to the third embodiment of the present disclosure may include step S310, step S320 and step S330.

[0089] In step S310, data is pre-fetched from a PNM memory of the PNM device or a memory outside the PNM device. As an example, data can be pre-fetched from the memory according to a pre-fetch data request of the host device. The amount of data pre-fetched each time and the number of pre-fetches can be determined according to the PNM memory.

[0090] In step S320, in each cycle of deep learning, the nth batch of data is loaded from the PNM memory for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the nth batch of data. The loss value and the confidence score here can be obtained in an existing manner.

[0091] In step S330, a difficult sample set is generated in the PNM memory or the difficult sample set in the PNM memory is updated according to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data obtained by deep learning training for the nth batch of data, where n is a positive integer greater than or equal to 1.

[0092] In addition, the optimization method according to the third embodiment of the present disclosure may also include: determining the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained, wherein the easy-to-learn sample data in the n+1th batch of data to be trained comes from data in the pre-fetched data of the PNM memory except for data corresponding to the sample data determined to be difficult-to-learn, and the difficult-to-learn sample data in the n+1th batch of data to be trained comes from a difficult sample set.

[0093] As described above, the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained can be determined based on at least one of the loss value and the confidence score corresponding to each data in the nth batch of data, the batch size of deep learning, and the total number of cycles of deep learning.

[0094] The step of determining the proportion of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained includes: in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a first condition, reducing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and increasing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained; in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a second condition, increasing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and reducing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained, wherein the first condition includes that the loss value of each data in the nth batch of data is greater than a first threshold value or the confidence score of each data in the nth batch of data is less than a second threshold value, and the second condition includes that the loss value of each data in the nth batch of data is less than or equal to the first threshold value or the confidence score of each data in the nth batch of data is greater than or equal to the second threshold value. The first threshold value and the second threshold value here are not specifically limited and can be adjusted as needed.

[0095] The optimization method according to the third embodiment of the present disclosure may also include: loading the n+1th batch of data from the PNM memory in which the difficult sample set is generated or updated for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the n+1th batch of data.

[0096] As described above, before determining the n+1th batch of data to be trained, the pre-fetched data corresponding to the sample data determined to be difficult to learn in the PNM memory may be marked as cleared.

[0097] Fig.10 is a block diagram illustrating a host device and a PNM device according to an embodiment of the present disclosure.

[0098] According to an embodiment of the present disclosure, the host device may include a CPU and a GPU, etc., the PNM memory of the PNM device may pre-fetch data from the data set of the memory, and the update data agent may run in the CPU. The loaded batch of data may be input to the deep learning model for training, and the update data agent may transmit the value returned by the training to the PNM device, for example, to the computing core of the PNM device.

[0099] The host device according to an embodiment of the present disclosure may include a processing unit and a storage unit, wherein the storage unit stores a program or an instruction, and when the program or the instruction is executed by the processing unit, the processing unit is prompted to execute the above-mentioned optimization method.

[0100] The PNM device according to an embodiment of the present disclosure may include a PNM memory, an update hard sample module (UHS), and an adaptive sample allocation module (ASA).

[0101] The PNM memory may pre-fetch data from the PNM memory or a memory outside the PNM device. For example, the PNM memory may pre-fetch data at predetermined time intervals according to a request from the host device.

[0102] The update difficult sample module (UHS) can generate or update the difficult sample set in the PNM memory according to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data obtained by performing deep learning training on the nth batch of data.

[0103] The adaptive sample allocation module (ASA) can determine the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained, wherein the easy-to-learn sample data in the n+1th batch of data to be trained comes from the data in the pre-fetched data of the PNM memory except the data corresponding to the difficult-to-learn sample data, and the difficult-to-learn sample data in the n+1th batch of data to be trained comes from the difficult sample set.

[0104] The adaptive sample allocation module may reduce the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and increase the proportion of easy-to-learn sample data in the n+1th batch of data to be trained in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a first condition; and increase the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and reduce the proportion of easy-to-learn sample data in the n+1th batch of data to be trained in response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying a second condition, wherein the first condition includes that the loss value of each data in the nth batch of data is greater than a first threshold or the confidence score of each data in the nth batch of data is less than a second threshold, and the second condition includes that the loss value of each data in the nth batch of data is less than or equal to the first threshold or the confidence score of each data in the nth batch of data is greater than or equal to the second threshold.

[0105] That is to say, the adaptive sample allocation module can increase the easy-to-learn sample data in the next batch and reduce the difficult-to-learn sample data when the loss value of the previous batch of data is high (for example, classification error); conversely, it can reduce the easy-to-learn sample data in the next batch and increase the difficult-to-learn sample data.

[0106] After the ratio of difficult to learn sample data to easy to learn sample data is determined, the adaptive sample allocation module will allocate difficult to learn samples and easy to learn samples in proportion. When the host wants to load the next batch of data, the assembled data batch will be returned to the host from the PNM memory, which can effectively reduce the reading time compared to randomly taking samples from the storage device.

[0107] Although not shown, before determining the n+1th batch of data to be trained, the update difficult sample module can mark the pre-fetched data corresponding to the sample data determined to be difficult to learn in the PNM memory as cleared. Compared with clearing the data directly in real time, the subsequent use of new pre-fetched data to overwrite the data marked as cleared can effectively reduce energy consumption.

[0108] The above has been referred to Figures 1 to 10 The optimization method and the like according to the exemplary embodiments of the present disclosure are described. However, it should be understood that the devices, units, equipment, etc. shown in the drawings can be configured as software, hardware, firmware, or any combination of the above items to perform specific functions. For example, these units and devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these systems or devices may also be uniformly performed by components in physical entity devices (e.g., processors, clients, or servers, etc.).

[0109] Instructions or programs corresponding to the optimization method according to the above embodiments of the present disclosure may be stored in a computer-readable storage medium. A computer system according to an embodiment of the present disclosure may include the above host device and / or the above PNM device (CXL-PNM device).

[0110] The instructions stored in the computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. It should be noted that the instructions can also be used to perform additional steps in addition to the above steps or perform more specific processing when performing the above steps. The contents of these additional steps and further processing have been described in detail in reference to Figures 1 to 10 It is mentioned in the description of related devices and methods, so it will not be repeated here to avoid repetition.

[0111] It should be noted that the optimization method according to the exemplary embodiment of the present disclosure can completely rely on the operation of computer programs or instructions to realize the corresponding functions, that is, each device corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding function.

[0112] The present disclosure utilizes the near memory processing characteristics of the PNM device to pre-fetch the data required for deep learning training into the PNM memory in advance, and deploys UHS and ASA modules in the PNM computing unit to effectively allocate difficult-to-learn samples, thereby improving the training efficiency and performance of deep learning. In addition, the present disclosure improves the overall I / O throughput based on the CXL-PNM device by pre-fetching samples. The present disclosure improves the performance of deep learning workloads by effectively utilizing PNM memory and computing cores to actively pre-fetch and cache training data. Furthermore, the optimization method of the present disclosure not only reduces the training time, but also effectively improves the sample quality in the later stage of training and improves the training performance.

[0113] The optimization method disclosed herein can be run in a device having a computational storage driver and an extended CXL device. The optimization method disclosed herein can be applied to system optimization of a CXL-PNM memory expander.

[0114] The optimization method disclosed in the present invention is applicable to the CXL-PNM memory expansion device.

[0115] Although the various embodiments of the present disclosure have been described using specific terms, the description and drawings are considered to be illustrative rather than restrictive in order to help understand the present disclosure. Without departing from the broader spirit and scope of the present disclosure, it is obvious that those skilled in the art may make various modifications and changes (for example, different features in different embodiments may be combined). Therefore, the scope of the present disclosure is not limited by specific implementation methods and embodiments, but by the claims and their equivalents.

Claims

1. An optimization method for deep learning, characterized in that: The optimization method comprises: In each cycle of deep learning, load the nth batch of data from the PNM memory of the PNM device for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the nth batch of data, wherein n is a positive integer greater than or equal to 1; delivering at least one of a loss value and a confidence score corresponding to each data in the nth batch of data to the PNM device, The nth batch of data at least includes data pre-fetched by the PNM memory from the PNM memory of the PNM device or a memory outside the PNM device.

2. The optimization method for deep learning according to claim 1, characterized in that: The optimization method also includes: before loading the n+1th batch of data for deep learning training, initiating a request to the PNM device to generate a difficult sample set in the PNM memory or update the difficult sample set in the PNM memory, wherein the difficult sample set is generated or updated based on at least one of a loss value and a confidence score corresponding to each data in the nth batch of data.

3. The optimization method for deep learning according to claim 2, characterized in that: The optimization method further comprises: Loading the n+1th batch of data from the PNM memory for generating or updating the difficult sample set for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the n+1th batch of data; transmitting at least one of a loss value and a confidence score corresponding to each data in the n+1th batch of data to the PNM device, The (n+1)th batch of data includes the difficult-to-learn sample data in the difficult sample set and the easy-to-learn sample data pre-fetched by the PNM memory from the PNM memory of the PNM device or the memory outside the PNM device.

4. The optimization method for deep learning according to claim 2, characterized in that: The ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data is determined based on at least one of the loss value and the confidence score corresponding to each data in the nth batch of data, the batch size of the deep learning, and the total number of cycles of the deep learning.

5. The optimization method for deep learning according to claim 1, characterized in that: The optimization method further includes: initiating a pre-fetch instruction to the PNM device at a preset time interval, wherein the pre-fetch instruction is used to instruct the PNM memory to pre-fetch data from the PNM memory or a memory outside the PNM device.

6. An optimization method for deep learning, characterized in that: The optimization method comprises: Data is pre-fetched from a PNM memory of the PNM device or a memory external to the PNM device in response to a pre-fetch instruction request of the host device.

7. The optimization method for deep learning according to claim 6, characterized in that: The optimization method further comprises: In response to a sample request from a host device, a difficult sample set in a PNM memory is generated or updated according to at least one of a loss value and a confidence score corresponding to each data in the nth batch of data obtained by performing deep learning training on the nth batch of data, where n is a positive integer greater than or equal to 1.

8. The optimization method for deep learning according to claim 7, characterized in that: The optimization method further comprises: Determine the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained, wherein the easy-to-learn sample data in the n+1th batch of data to be trained are derived from data in the pre-fetched data of the PNM memory except for data corresponding to the difficult-to-learn sample data, and the difficult-to-learn sample data in the n+1th batch of data to be trained are derived from the difficult sample set.

9. The optimization method for deep learning according to claim 8, characterized in that: The ratio of difficult-to-learn sample data to easy-to-learn sample data in the n+1th batch of data to be trained is determined based on at least one of a loss value and a confidence score corresponding to each data in the nth batch of data, a batch size of the deep learning, and a total number of cycles of the deep learning.

10. The optimization method for deep learning according to claim 8, characterized in that: The steps of determining the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained include: In response to at least one of a loss value and a confidence score corresponding to each data in the nth batch of data satisfying a first condition, reducing a proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and increasing a proportion of easy-to-learn sample data in the n+1th batch of data to be trained; In response to at least one of a loss value and a confidence score corresponding to each data in the nth batch of data satisfying a second condition, increasing a proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and decreasing a proportion of easy-to-learn sample data in the n+1th batch of data to be trained, The first condition includes that the loss value of each data in the nth batch of data is greater than a first threshold or the confidence score of each data in the nth batch of data is less than a second threshold, The second condition includes that the loss value of each data in the nth batch of data is less than or equal to a first threshold or the confidence score of each data in the nth batch of data is greater than or equal to a second threshold.

11. The optimization method for deep learning according to claim 8, characterized in that: The optimization method further includes: before determining the n+1th batch of data to be trained, marking the pre-fetched data in the PNM memory corresponding to the sample data determined to be difficult to learn as cleared.

12. An optimization method for deep learning, characterized in that: The optimization method comprises: pre-fetching data from a PNM memory of a PNM device or a memory external to the PNM device; In each cycle of deep learning, load the nth batch of data from the PNM memory to perform deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the nth batch of data; Generate a difficult sample set in the PNM memory or update the difficult sample set in the PNM memory based on at least one of the loss value and the confidence score corresponding to each data in the nth batch of data obtained by deep learning training on the nth batch of data, where n is a positive integer greater than or equal to 1.

13. The optimization method for deep learning according to claim 12, characterized in that: The optimization method further comprises: Determine the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained, wherein the easy-to-learn sample data in the n+1th batch of data to be trained are derived from data in the pre-fetched data of the PNM memory except for data corresponding to the difficult-to-learn sample data, and the difficult-to-learn sample data in the n+1th batch of data to be trained are derived from the difficult sample set.

14. The optimization method for deep learning according to claim 13, characterized in that: The ratio of difficult-to-learn sample data to easy-to-learn sample data in the n+1th batch of data to be trained is determined based on at least one of a loss value and a confidence score corresponding to each data in the nth batch of data, a batch size of the deep learning, and a total number of cycles of the deep learning.

15. The optimization method for deep learning according to claim 13, characterized in that: The steps of determining the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained include: In response to at least one of a loss value and a confidence score corresponding to each data in the nth batch of data satisfying a first condition, reducing a proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and increasing a proportion of easy-to-learn sample data in the n+1th batch of data to be trained; In response to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data satisfying the second condition, increasing the proportion of difficult-to-learn sample data in the n+1th batch of data to be trained and decreasing the proportion of easy-to-learn sample data in the n+1th batch of data to be trained, The first condition includes that the loss value of each data in the nth batch of data is greater than a first threshold or the confidence score of each data in the nth batch of data is less than a second threshold, The second condition includes that the loss value of each data in the nth batch of data is less than or equal to a first threshold or the confidence score of each data in the nth batch of data is greater than or equal to a second threshold.

16. The optimization method for deep learning according to claim 13, characterized in that: The optimization method further comprises: The n+1th batch of data is loaded from the PNM memory in which the difficult sample set is generated or updated for deep learning training to obtain at least one of a loss value and a confidence score corresponding to each data in the n+1th batch of data.

17. The optimization method for deep learning according to claim 13, characterized in that: Before determining the n+1th batch of data to be trained, the pre-fetched data corresponding to the sample data determined to be difficult to learn in the PNM memory is marked as cleared.

18. A host device for deep learning, characterized in that: include: A processing unit and a storage unit, wherein the storage unit stores a program or an instruction, and when the program or the instruction is executed by the processing unit, the processing unit is prompted to execute the optimization method according to any one of claims 1-5.

19. A PNM device for deep learning, characterized in that: include: PNM memory, pre-fetching data from the PNM memory or a memory external to the PNM device; The difficult sample update module is configured to generate or update the difficult sample set in the PNM memory according to at least one of the loss value and the confidence score corresponding to each data in the nth batch of data obtained by performing deep learning training on the nth batch of data, wherein n is a positive integer greater than or equal to 1, An adaptive sample allocation module determines the ratio of difficult-to-learn sample data and easy-to-learn sample data in the n+1th batch of data to be trained, wherein the easy-to-learn sample data in the n+1th batch of data to be trained comes from the data in the pre-fetched data of the PNM memory except the data corresponding to the difficult-to-learn sample data, and the difficult-to-learn sample data in the n+1th batch of data to be trained comes from the difficult sample set.

20. A computer system, characterized in that: The computer system comprises the host device according to claim 18 and / or the PNM device according to claim 19.

Images