Image processing model training method and apparatus

Abstract

This specification provides an image processing model training method and apparatus. The image processing model training method includes: writing a batch sample dataset to be processed into video memory, splitting the batch sample dataset to generate at least two corresponding batch sample data subsets, inputting the image data contained in each batch sample data subset into an image processing model for processing, obtaining the processing result output by the image processing model, writing the processing result into memory, writing the processing result into video memory after the at least two batch sample data subsets have been processed, performing gradient calculation based on the processing result to generate a calculation result corresponding to each batch sample data subset, performing gradient fusion on the calculation result, and adjusting the model parameters of the image processing model based on the fusion result.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an image processing model training method. One or more embodiments of this specification also relate to an image processing model training apparatus, an object recognition system, a computing device, and a computer-readable storage medium. Background Technology

[0002] Over the past few years, the scale of neural networks has continued to expand, and training may require a large amount of data and computational resources. For example, one of the classic CNN models, Inception-v3, has 23 million parameters. The larger the model, the larger the training data required. Training neural networks typically uses batch stochastic gradient descent, which divides the samples into m samples (mini-batches), each mini-batch containing n samples (mini-batches); the gradient of each sample (mini-batch) is calculated, and then the gradients of these mini-batches are summed and averaged to obtain the final gradient used to update the parameters.

[0003] Generally speaking, the more consistent the distribution of samples in each mini-batch is with the overall population, the better the training effect. A larger batch size is more likely to match the overall distribution. However, the memory (if using CPU) or GPU memory (if using GPU) required for training increases proportionally with the batch size. Since memory and GPU memory are always limited, overcoming the bottleneck of GPU memory capacity becomes a pressing issue. Summary of the Invention

[0004] In view of this, embodiments of this specification provide an image processing model training method. One or more embodiments of this specification also relate to an image processing model training apparatus, an object recognition system, a computing device, and a computer-readable storage medium, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an image processing model training method is provided, comprising:

[0006] Write the sample dataset to be processed into video memory, and split the sample dataset to generate at least two sample data subsets;

[0007] The image data contained in each sample data subset is input into the image processing model for processing, the processing result output by the image processing model is obtained, and the processing result is written into memory;

[0008] After the processing of at least two subsets of sample data is completed, the processing results are written to the video memory, and gradient calculation is performed based on the processing results to generate the calculation results corresponding to each subset of sample data.

[0009] The calculation results are gradient fused, and the model parameters of the image processing model are adjusted based on the fusion results.

[0010] According to a second aspect of the embodiments of this specification, an image processing model training apparatus is provided, comprising:

[0011] The splitting module is configured to write the sample dataset to be processed into video memory and split the sample dataset to generate at least two sample data subsets.

[0012] The processing module is configured to input the image data contained in each sample data subset into the image processing model for processing, obtain the processing result output by the image processing model, and write the processing result into memory;

[0013] The calculation module is configured to write the processing result to the video memory after the processing of at least two subsets of sample data is completed, and to perform gradient calculation based on the processing result to generate the calculation result corresponding to each subset of sample data.

[0014] The adjustment module is configured to perform gradient fusion on the calculation results and adjust the model parameters of the image processing model based on the fusion results.

[0015] According to a third aspect of the embodiments of this specification, an object recognition system is provided, comprising:

[0016] First processor and second processor;

[0017] The first processor is configured to read image data of the image to be identified into memory and perform matrix stitching of the image data to generate batch image data;

[0018] The second processor is configured to read the batch image data into video memory, split the batch image data into at least two batch image sub-data, input the batch image sub-data into an image processing model for processing, generate corresponding batch image sub-features, write the batch image sub-features into the memory, and after the at least two batch image sub-data are processed, read the batch image sub-features in the memory into the video memory, fuse the batch image sub-features to generate batch image features, and write the batch image features into the memory.

[0019] The first processor is configured to decompose the features of the batch images, generate image feature vectors corresponding to the images to be identified, and identify the images to be identified based on the image feature vectors to generate corresponding recognition results.

[0020] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0021] Memory and processor;

[0022] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the image processing model training method.

[0023] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the image processing model training method.

[0024] One embodiment of this specification involves writing the sample dataset to be processed into video memory, splitting the sample dataset into at least two sample data subsets, inputting the image data contained in each sample data subset into an image processing model for processing, obtaining the processing result output by the image processing model, and writing the processing result into memory. After the processing of the at least two sample data subsets is completed, the processing result is written into the video memory, and gradient calculation is performed based on the processing result to generate a calculation result corresponding to each sample data subset. Gradient fusion is performed on the calculation results, and the model parameters of the image processing model are adjusted based on the fusion result.

[0025] In this embodiment, after writing the sample data from the sample dataset into the video memory, the sample dataset (batch) is split into several sample data subsets (small batches). When the data from the sample data subsets is input into the model for processing, for the part with a large number of model parameters, the variables generated in that part can be transferred to memory in batches for caching, thereby overcoming the limitation of video memory capacity and improving the utilization rate of memory storage space. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating an image processing model training method provided in one embodiment of this specification.

[0027] Figure 2 This is a schematic diagram of an image processing model training process provided in one embodiment of this specification;

[0028] Figure 3This is a flowchart illustrating the processing procedure of an image processing model training method provided in one embodiment of this specification.

[0029] Figure 4 This is an architecture diagram of an object recognition system provided in one embodiment of this specification;

[0030] Figure 5 This is a schematic diagram of an image processing model training device provided in one embodiment of this specification;

[0031] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0032] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0033] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0035] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0036] Deep model training: The process of using a large amount of sample data and combining different deep model structures to make the model parameters fit the specific sample target as closely as possible.

[0037] GPU memory limitations: Since GPU platforms typically have very limited video memory, and during deep model training based on GPUs, forward and backward propagation will generate a large number of variable values ​​that need to be temporarily stored, video memory capacity becomes an important constraint for model training.

[0038] As business problems become increasingly complex, algorithmic model structures become more intricate, and the amount of variable data grows significantly. To cope with this increased computational difficulty, GPUs (Graphics Processing Units), a hardware platform suitable for large-scale parallel numerical computation, are widely used for deep model training. However, due to cost and technological limitations, the video memory used for GPU computation to cache data is often relatively small. Therefore, for training models with complex structures and a large number of parameters, computation time becomes a bottleneck when using a traditional CPU platform, while video memory capacity becomes a bottleneck when using a GPU platform. This specification's embodiments aim to resolve this contradiction by fully utilizing the idle memory on the GPU platform, thus overcoming the video memory bottleneck.

[0039] To address the training resource problem of large-scale models, the industry typically uses distributed methods, reduces batch size, and lowers computational complexity.

[0040] Distributed approach: Using multiple GPUs to compute parameter updates separately. This approach distributes computational tasks to various computing units, and then updates the parameters uniformly using various methods after the updates are complete. The drawback of this approach is that network transmission bottlenecks during parameter updates lead to inefficiencies as computing units wait for data, and for some complex models, the distributed acceleration effect can even be negative.

[0041] Reducing the batch size: Reducing the batch size linearly reduces the amount of data that needs to be cached during model training, effectively avoiding the problem of insufficient GPU memory. The drawback of this approach is that the batch size, as one of the hyperparameters for deep model training, often needs to be adjusted to obtain a better solution and avoid getting trapped in local optima. However, setting an upper limit on the batch size will narrow the range of hyperparameter adjustments, resulting in the model not being able to be effectively tuned.

[0042] Reducing computational complexity: By reducing the variables in the network from 32-bit floating-point numbers to 16-bit or even 8-bit floating-point numbers, the utilization of GPU memory will be reduced significantly. Adjusting the number of parameter layers and the dimension of hidden layers in the model can also reduce GPU memory utilization. The drawback of this approach is that some older platforms do not support adjusting the number of floating-point bits, and reducing the number of floating-point bits and simplifying the model structure will undoubtedly reduce the model's fitting effect.

[0043] Based on this, the embodiments in this specification use the API tf.while_loop(), which is customized for training RNN networks in TensorFlow, to split the batch into several smaller batches. The variables generated by the part with a large number of parameters in the model are calculated in batches and then transferred to memory for caching, thereby achieving the purpose of overcoming the GPU memory limit.

[0044] This specification provides an image processing model training method, and also relates to an image processing model training device, an object recognition system, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0045] Figure 1 A flowchart of an image processing model training method according to an embodiment of this specification is shown, including steps 102 to 108.

[0046] Step 102: Write the sample dataset to be processed into video memory, and split the sample dataset to generate at least two sample data subsets.

[0047] Specifically, the image processing model training method provided in the embodiments of this specification is applied to a terminal, which includes a CPU and a graphics processing unit (GPU).

[0048] A CPU is a very large-scale integrated circuit, which is the core of an electronic device for computation and control, and is mainly used for data processing; a graphics processing unit (GPU) is a microprocessor used for image processing.

[0049] In the embodiments described in this specification, when training an image processing model via a terminal, the image data used for model training can typically be written to the video memory in batches, so that the GPU can process the image data stored in the video memory to achieve model training. Specifically, the image data to be processed can be divided according to a preset batch size to generate sample datasets (batches), wherein each sample dataset contains a number of sample data consistent with the batch size; then, each sample dataset can be written to the video memory sequentially, and the graphics processing unit (GPU) can process the data in the sample datasets sequentially.

[0050] In practical applications, due to cost and technology constraints, the video memory used for GPU computing to cache data is often small. Therefore, for training models with complex structures and a large number of parameters, training time becomes a bottleneck when using a traditional CPU platform, while video memory capacity becomes a bottleneck when using a GPU platform.

[0051] Based on this, the embodiments in this specification fully utilize the idle memory on the GPU platform during training to overcome the GPU memory bottleneck. Specifically, the embodiments in this specification use the API tf.while_loop(), which is customized for image processing model training, to split each sample dataset (batch) in GPU memory into several sample data subsets (small batches). Then, during model training, the GPU sequentially inputs each sample data subset into the model for processing. This allows for the transfer of variables produced in batches to memory for temporary storage, especially for parts of the model with a large number of parameters, thereby overcoming the GPU memory limitation.

[0052] A schematic diagram of an image processing model training process provided in the embodiments of this specification is shown below. Figure 2 As shown, the sample dataset written to the video memory can be split into n sample data subsets. In the actual model training process, the n sample data subsets are input into the image processing model in sequence for processing. This allows the output variables of the part with a large number of model parameters to be stored in memory in batches, thereby overcoming the limitations of video memory.

[0053] In practice, the sample dataset to be processed is written to the video memory, including:

[0054] The acquired image data of the image to be processed is written into memory, and the image data is divided according to the preset batch sample number to generate the corresponding sample dataset;

[0055] The image data contained in each sample dataset are concatenated into a matrix, and the concatenation result is written to the video memory.

[0056] Specifically, the preset batch size can be calculated based on the image size and video memory capacity of the image to be processed.

[0057] In this embodiment of the specification, when training an image processing model using image data of an image to be processed, the CPU first writes the acquired image data of the image to be processed into memory. Then, the image data to be processed is divided according to a preset batch size to generate a sample dataset (batch). Next, the image data contained in each sample dataset can be sequentially matrix-stitched to generate a corresponding stitching result, which is first stored in the memory. When model training is required, the GPU writes each stitching result into video memory, and the GPU processes the data in each stitching result sequentially.

[0058] The GPU writes the stitching results into video memory, specifically storing the matrix stitching results of the image data in the sample dataset into video memory. If the image to be processed is divided into multiple sets, it needs to be written multiple times.

[0059] In practical applications, before matrix stitching, it is necessary to check whether the image sizes of the images to be processed are the same. If the image sizes are the same, the stitching can be carried out directly. If the image sizes are different, the images to be processed are corrected to the same size before stitching.

[0060] Then, based on the image size and video memory capacity of the image to be processed, the number of images that can be written to the video memory at the same time can be calculated. Based on the number of images to be processed and the number of images, the images to be processed can be divided into one or more groups. Then, the images to be processed in each group are matrix-stitched to generate a stitched result.

[0061] Matrix concatenation can be achieved using matrix concatenation functions in programming languages. If the image data of a single image to be processed is a c*h*w matrix, then matrix concatenation is to merge n images to be processed belonging to the same group into the same tensor matrix.

[0062] Furthermore, the sample dataset is split to generate at least two subsets of sample data, including:

[0063] The number of samples to be split is determined based on the video memory size;

[0064] Based on the amount of data to be split, the sample dataset is split to generate at least two corresponding sample data subsets.

[0065] Specifically, the sample dataset is split, with the criterion being to ensure that the video memory can fully handle the data caching requirements of a small batch (a subset of sample data).

[0066] In practical applications, the sample dataset (batch) is split into several sample data subsets (small batches), and the splitting results are stored using TensorArray.

[0067] Step 104: Input the image data contained in each sample data subset into the image processing model for processing, obtain the processing result output by the image processing model, and write the processing result into memory.

[0068] Specifically, after writing the sample dataset into the video memory and splitting it into several sample data subsets, the GPU can input the image data contained in each sample data subset into the image processing model for processing.

[0069] Since there are some parts of the model with a large number of parameters, such as the embedding layer, the embedding layer can encode the input image data into a 128-dimensional vector. When there are multiple input image data, the number of parameters corresponding to the output result of the embedding layer may be greater than the preset parameter threshold. Therefore, if it is determined that the number of parameters corresponding to its output result is greater than the preset parameter threshold, the output result needs to be written into memory.

[0070] In specific implementation, the image data contained in each sample data subset is input into the image processing model for processing, the processing result output by the image processing model is obtained, and the processing result is written into memory, including:

[0071] The image data contained in each sample data subset is input into the target processing layer of the image processing model for processing, and the corresponding processing results are generated.

[0072] Determine whether the number of parameters corresponding to the processing result is greater than a preset parameter threshold;

[0073] If so, the processing result is written into memory.

[0074] If not, the processing result is written to the video memory.

[0075] Specifically, as mentioned earlier, since there are some parts of the model with a large number of parameters, such as the embedding layer, the embedding layer can encode the input image data into a 128-dimensional vector. When there are multiple input image data, the number of parameters corresponding to the output result of the embedding layer may be greater than the preset parameter threshold. Therefore, if it is determined that the number of parameters corresponding to its output result (the product between the vector dimension and the number of image data contained in the sample data subset) is greater than the preset parameter threshold, the output result needs to be written into memory.

[0076] For some layers in the model, such as the MLP layer, the data input to the layer can be processed and output as a 2D vector. Therefore, when a certain amount of image data is output, the number of parameters corresponding to the output of the MLP layer may not be greater than the preset parameter threshold. In this case, the output of the layer does not need to be written to memory and can be directly stored in the video memory.

[0077] Based on this, in the embodiments of this specification, after inputting the image data contained in each sample data subset into the target processing layer (e.g., embedding layer or MLP layer) of the image processing model for processing and generating the corresponding processing result, it can first determine whether the parameter quantity (the product between the vector dimension and the number of image data contained in the sample data subset) corresponding to the processing result is greater than a preset parameter quantity threshold. If it is greater, the processing result is written to memory; if it is not greater, the processing result is written to video memory.

[0078] In the embodiments of this specification, when inputting data from a subset of sample data into the model for processing, for the part with a large number of model parameters, the variables generated in that part can be transferred to memory in batches as a subset of sample data for caching, thereby overcoming the limitation of video memory capacity and improving the utilization rate of memory storage space.

[0079] In specific implementation, when it is determined that the processing results need to be written to memory, the image data contained in each sample data subset is input into the image processing model for processing, the processing results output by the image processing model are obtained, and the processing results are written to memory, including:

[0080] The image data contained in the i-th sample data subset is input into the image processing model for processing to obtain the processing result output by the image processing model, wherein i is a positive integer less than or equal to n, i starts from 1, and n is the total number of sample data subsets;

[0081] Write the processing result into memory and determine whether i equals n;

[0082] If not, then i is incremented by 1, and the process returns to the step of inputting the image data contained in the i-th sample data subset into the image processing model for processing, and obtaining the processing result output by the image processing model.

[0083] If so, then it is determined that the processing of at least two subsets of sample data has been completed.

[0084] Specifically, the sample dataset is written into the video memory, and the sample dataset is split into several (n) sample data subsets. For these n sample data subsets, the image data contained in the first sample data subset can be input into the image processing model for processing to obtain the processing result output by the image processing model, and the processing result is written into memory.

[0085] Then, it can be determined whether the n sample data subsets have been processed, that is, whether i is equal to n. If they are equal, the processing is considered complete; if they are not equal, the image data contained in the second sample data subset is input into the image processing model for processing to obtain the processing result output by the image processing model and write the processing result into memory. The process continues to determine whether the n sample data subsets have been processed, so that the image data contained in the first to nth sample data subsets are input into the image processing model for processing in sequence, and the processing result output by the model is written into memory in sequence.

[0086] A schematic diagram of an image processing model training process provided in the embodiments of this specification is shown below. Figure 2 As shown, the tf.while_loop() API splits the sample data into subsets (sample data subset 1 to sample data subset n) of a TensorArray, runs the code contained therein in batches, that is, it inputs the image data contained in each sample data subset into the image processing model for processing in batches, and transfers the temporary data cache (the processing result output by the image processing model) generated in the code from the video memory to the memory. When calculating the gradient later, it is retrieved from the memory, and finally produces a TensorArray that stores the results of the corresponding small batch calculation.

[0087] The implementation process of tf.while_loop() is as follows: First, the image data contained in sample data subset 1 is input into the image processing model for processing, and the processing result output by the image processing model is obtained and written into memory. Then, it is determined whether 1 is equal to n. If they are equal, the processing is considered complete. If they are not equal, the image data contained in sample data subset 2 is input into the image processing model for processing, and the processing result output by the image processing model is obtained and written into memory. The process continues to determine whether n sample data subsets have been processed, so that the image data contained in sample data subsets 1 to n are input into the image processing model for processing in sequence, and the processing result output by the model is written into memory in sequence.

[0088] In addition, before writing the processing result to memory, it can be determined whether the amount of data (parameters) corresponding to the processing result is greater than the preset parameter threshold. If it is not greater, the processing result can also be written to the video memory.

[0089] In the embodiments of this specification, when inputting data from a subset of sample data into the model for processing, the processing results can be written to memory. Alternatively, for a portion of the model with a large number of parameters, the variables generated in that portion can be transferred to memory in batches, in units of a subset of sample data, for caching. For a portion of the model with a relatively small number of parameters, the variables generated in that portion can be written to video memory, thereby overcoming the limitations of video memory capacity and improving the utilization rate of memory storage space.

[0090] Step 106: After the processing of at least two subsets of sample data is completed, the processing result is written to the video memory, and gradient calculation is performed based on the processing result to generate the calculation result corresponding to each subset of sample data.

[0091] Specifically, as mentioned earlier, after splitting the sample dataset into sample data subsets and sequentially inputting the image data contained in each sample data subset into the image processing model for processing, generating corresponding processing results, and writing the processing results into memory, once it is determined that each sample data subset has been processed, the processing results corresponding to each sample data subset stored in memory can be rewritten into the display memory to calculate the gradient corresponding to each sample data subset based on each processing result and generate the corresponding calculation result.

[0092] Step 108: Perform gradient fusion on the calculation results, and adjust the model parameters of the image processing model based on the fusion results.

[0093] Specifically, after processing each subset of sample data to generate corresponding processing results, and calculating the gradient calculation results corresponding to each subset of sample data based on each processing result, gradient fusion can be performed on each gradient calculation result to adjust the model parameters of the image processing model based on the fusion result, thereby optimizing the image processing model.

[0094] In specific implementation, the step of performing gradient fusion on the calculation results and adjusting the model parameters of the image processing model based on the fusion results includes:

[0095] The calculation results are then concatenated into a matrix to generate the corresponding concatenation result;

[0096] The model parameters of the image processing model are adjusted based on the stitching result.

[0097] Specifically, the image data contained in each sample data subset is input into the image processing model in batches for processing, and the processing results output by the image processing model are transferred from the video memory to the memory. The results are then retrieved from the memory when calculating the gradient, and finally a TensorArray containing the results of the corresponding small batch is produced.

[0098] The embodiments in this specification aggregate the generated TensorArray according to the batch dimension. The resulting Tensor is completely consistent with the Tensor obtained by directly performing corresponding calculations without splitting. The remaining part of the subsequent calculation model and the model Loss are consistent with those in the normal model.

[0099] The gradient calculation based on the processing result includes:

[0100] Based on a preset loss function, calculate the loss value between the processing result corresponding to the target sample data subset and the sample label corresponding to the target sample data subset;

[0101] Accordingly, the step of performing gradient fusion on the calculation results and adjusting the model parameters of the image processing model based on the fusion results includes:

[0102] Based on the loss value, the feedback tuning parameters are calculated, and the model parameters of the image processing model are adjusted based on the feedback tuning parameters.

[0103] Specifically, after processing each subset of sample data and generating the corresponding processing result, the difference (loss value) between the processing result corresponding to each subset of sample data and the sample label corresponding to that subset of sample data can be calculated. The sum of the differences corresponding to each subset of sample data is used as the target loss value of the image processing model. The target loss value is used to determine the tuning parameters of the image processing model, and the tuning parameters are used to adjust the model parameters of the image processing model in reverse, thereby optimizing the image processing model.

[0104] The image processing model processing method provided in the embodiments of this specification, compared with the idea of ​​distributed computing, still remains within the computing scope of a single machine / single card. There is no parameter transmission between various computing units, only data exchange between the video memory and the main memory within a single machine. However, the data exchange in distributed computing between different hosts cannot avoid the data exchange between the video memory and the main memory. Therefore, this method avoids the bottleneck of network transmission latency of parameters in general.

[0105] The image processing model method provided in the embodiments of this specification, compared with the idea of ​​reducing batch size, greatly relaxes the restrictions on batch size because the cost of main memory is much lower than that of video memory, and its capacity can usually be tens or hundreds of times that of video memory. During the parameter adjustment process, the influence of the upper limit of batch size can be ignored.

[0106] The image processing model method provided in the embodiments of this specification, compared with the approach of reducing computational complexity, does not affect the accuracy of variables or the structure of the model, and therefore there is no difference in computational accuracy or model fitting ability.

[0107] This specification's embodiments fully utilize the main memory of a single machine that was previously unused when training deep learning models using a GPU, breaking the GPU memory bottleneck. This eliminates the need to consider GPU memory limits when tuning batch size, thereby enabling the acquisition of a superior image processing model.

[0108] One embodiment of this specification involves writing the sample dataset to be processed into video memory, splitting the sample dataset into at least two sample data subsets, inputting the image data contained in each sample data subset into an image processing model for processing, obtaining the processing result output by the image processing model, and writing the processing result into memory. After the processing of the at least two sample data subsets is completed, the processing result is written into the video memory, and gradient calculation is performed based on the processing result to generate a calculation result corresponding to each sample data subset. Gradient fusion is performed on the calculation results, and the model parameters of the image processing model are adjusted based on the fusion result.

[0109] In this embodiment, after writing the sample data from the sample dataset into the video memory, the sample dataset (batch) is split into several sample data subsets (small batches). When the data from the sample data subsets is input into the model for processing, for the part with a large number of model parameters, the variables generated in that part can be transferred to memory in batches for caching, thereby overcoming the limitation of video memory capacity and improving the utilization rate of memory storage space.

[0110] The following is in conjunction with the appendix Figure 3 Taking the image processing model training method provided in this specification as an example in an image recognition scenario, the image processing model training method will be further explained. Among other things, Figure 3 The flowchart of an image processing model training method according to an embodiment of this specification is shown, with specific steps including steps 302 to 318.

[0111] Step 302: The CPU writes the acquired image data of the image to be identified into memory, and divides the image data according to the preset batch sample number to generate the corresponding sample dataset.

[0112] Step 304: Perform matrix concatenation on the image data contained in each sample dataset and write the concatenation result to the video memory.

[0113] Step 306: The GPU splits the sample dataset to generate at least two subsets of sample data.

[0114] Step 308: Input the image data contained in the i-th sample data subset into the image recognition model for processing, and obtain the processing result output by the image recognition model, where i is a positive integer less than or equal to n, i starts from 1, and n is the total number of sample data subsets.

[0115] Step 310: If the number of parameters corresponding to the processing result is greater than the preset parameter threshold, write the processing result into memory.

[0116] Step 312: Determine if i is equal to n.

[0117] If the result of step 312 is negative, then proceed to step 314.

[0118] Step 314: Increment i by 1, and return to execute step 308.

[0119] Step 316: After processing at least two subsets of sample data, write the processing results to the video memory, and perform gradient calculation based on the processing results to generate the calculation results corresponding to each subset of sample data.

[0120] Step 318: Perform gradient fusion on the calculation results, and adjust the model parameters of the image recognition model based on the fusion results.

[0121] In this embodiment, after writing the sample data from the sample dataset into the video memory, the sample dataset (batch) is split into several sample data subsets (small batches). When the data from the sample data subsets is input into the model for processing, for the part with a large number of model parameters, the variables generated in that part can be transferred to memory in batches for caching, thereby overcoming the limitation of video memory capacity and improving the utilization rate of memory storage space.

[0122] Figure 4 An architecture diagram of an object recognition system according to an embodiment of this specification is shown, including:

[0123] First processor 402 and second processor 404;

[0124] The first processor 402 is used to read the image data of the image to be identified into memory, and to perform matrix stitching of the image data to generate batch image data;

[0125] The second processor 404 is configured to read the batch image data into video memory, split the batch image data into at least two batch image sub-data, input the batch image sub-data into an image processing model for processing, generate corresponding batch image sub-features, write the batch image sub-features into the memory, and after the at least two batch image sub-data are processed, read the batch image sub-features in the memory into the video memory, fuse the batch image sub-features to generate batch image features, and write the batch image features into the memory.

[0126] The first processor 402 is configured to decompose the features of the batch images, generate image feature vectors corresponding to the images to be identified, and identify the images to be identified based on the image feature vectors to generate corresponding recognition results.

[0127] Specifically, the first processor is the CPU, and the second processor is the GPU.

[0128] The embodiments in this specification achieve batch processing of image recognition by concatenating the image data of multiple images to be recognized into a matrix and reading them into the display memory at one time, thereby saving transmission time and improving the recognition speed of image recognition.

[0129] In addition, in the embodiments of this specification, after writing the batch image data into the video memory, the batch image data is then split into several batch image sub-data (small batches). This allows the variables generated in the part with a large number of model parameters to be transferred to memory in small batches for caching when the image data in the batch image sub-data is input into the model for processing. This overcomes the limitation of video memory capacity and helps to improve the utilization rate of memory storage space.

[0130] The above is an illustrative scheme of an object recognition system according to this embodiment. It should be noted that the technical solution of this object recognition system and the technical solution of the image processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the object recognition system, please refer to the description of the technical solution of the image processing model training method described above.

[0131] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing model training device. Figure 5 A schematic diagram of an image processing model training apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0132] The splitting module 502 is configured to write the sample dataset to be processed into the video memory and split the sample dataset to generate at least two sample data subsets.

[0133] The processing module 504 is configured to input the image data contained in each sample data subset into the image processing model for processing, obtain the processing result output by the image processing model, and write the processing result into memory;

[0134] The calculation module 506 is configured to write the processing result to the video memory after the processing of the at least two sample data subsets is completed, and to perform gradient calculation based on the processing result to generate the calculation result corresponding to each sample data subset.

[0135] The adjustment module 508 is configured to perform gradient fusion on the calculation results and adjust the model parameters of the image processing model based on the fusion results.

[0136] Optionally, the splitting module 502 is further configured to:

[0137] The acquired image data of the image to be processed is written into memory, and the image data is divided according to the preset batch sample number to generate the corresponding sample dataset;

[0138] The image data contained in each sample dataset are concatenated into a matrix, and the concatenation result is written to the video memory.

[0139] Optionally, the splitting module 502 is further configured to:

[0140] The number of samples to be split is determined based on the video memory size;

[0141] Based on the amount of data to be split, the sample dataset is split to generate at least two corresponding sample data subsets.

[0142] Optionally, the processing module 504 is further configured to:

[0143] The image data contained in the i-th sample data subset is input into the image processing model for processing to obtain the processing result output by the image processing model, wherein i is a positive integer less than or equal to n, i starts from 1, and n is the total number of sample data subsets;

[0144] Write the processing result into memory and determine whether i equals n;

[0145] If not, then i is incremented by 1, and the process returns to the step of inputting the image data contained in the i-th sample data subset into the image processing model for processing, and obtaining the processing result output by the image processing model.

[0146] Optionally, if the result of determining whether i is equal to n is yes, then it is determined that the processing of the at least two sample data subsets is complete.

[0147] Optionally, the processing module 504 is further configured to:

[0148] The image data contained in each sample data subset is input into the target processing layer of the image processing model for processing, and the corresponding processing results are generated.

[0149] Determine whether the number of parameters corresponding to the processing result is greater than a preset parameter threshold;

[0150] If so, the processing result is written into memory.

[0151] Optionally, if the execution result of determining whether the number of parameters corresponding to the processing result is greater than a preset parameter threshold is negative, the processing result is written to the video memory.

[0152] Optionally, the adjustment module 508 is further configured to:

[0153] The calculation results are then concatenated into a matrix to generate the corresponding concatenation result;

[0154] The model parameters of the image processing model are adjusted based on the stitching result.

[0155] Optionally, the computing module 506 is further configured to:

[0156] Based on a preset loss function, calculate the loss value between the processing result corresponding to the target sample data subset and the sample label corresponding to the target sample data subset;

[0157] Accordingly, the adjustment module 508 is further configured to:

[0158] Based on the loss value, the feedback tuning parameters are calculated, and the model parameters of the image processing model are adjusted based on the feedback tuning parameters.

[0159] In one embodiment of this specification, a splitting module writes the sample dataset to be processed into video memory and splits the sample dataset to generate at least two sample data subsets. A processing module inputs the image data contained in each sample data subset into an image processing model for processing, obtains the processing result output by the image processing model, and writes the processing result into memory. After the at least two sample data subsets have been processed, a calculation module writes the processing result into the video memory and performs gradient calculation based on the processing result to generate a calculation result corresponding to each sample data subset. An adjustment module performs gradient fusion on the calculation result and adjusts the model parameters of the image processing model based on the fusion result.

[0160] In this embodiment, after writing the sample data from the sample dataset into the video memory, the sample dataset (batch) is split into several sample data subsets (small batches). When the data from the sample data subsets is input into the model for processing, for the part with a large number of model parameters, the variables generated in that part can be transferred to memory in batches for caching, thereby overcoming the limitation of video memory capacity and improving the utilization rate of memory storage space.

[0161] The above is a schematic scheme of an image processing model training device according to this embodiment. It should be noted that the technical solution of this image processing model training device and the technical solution of the image processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the image processing model training device, please refer to the description of the technical solution of the image processing model training method described above.

[0162] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0163] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0164] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0165] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 600 can also be a mobile or stationary server.

[0166] The memory 610 is used to store computer-executable instructions, and the processor 620 is used to execute the following computer-executable instructions to implement the steps of the image processing model training method.

[0167] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image processing model training method described above.

[0168] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, are used to implement the steps of the image processing model training method.

[0169] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the image processing model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image processing model training method described above.

[0170] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0171] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0172] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0173] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training an image processing model, comprising: Write the sample dataset to be processed into video memory, and split the sample dataset in video memory to generate at least two sample data subsets; The image data contained in each sample data subset is input into the image processing model for processing to obtain the processing result output by the image processing model. If the number of parameters corresponding to the processing result is greater than a preset parameter threshold, the processing result is written into memory. After the processing of at least two subsets of sample data is completed, the processing results are written to the video memory, and gradient calculation is performed based on the processing results to generate the calculation results corresponding to each subset of sample data. The calculation results are gradient fused, and the model parameters of the image processing model are adjusted based on the fusion results.

2. The image processing model training method according to claim 1, wherein writing the sample dataset to be processed into the video memory comprises: The acquired image data of the image to be processed is written into memory, and the image data is divided according to the preset batch sample number to generate the corresponding sample dataset; The image data contained in each sample dataset are concatenated into a matrix, and the concatenation result is written to the video memory.

3. The image processing model training method according to claim 1 or 2, wherein splitting the sample dataset to generate at least two sample data subsets includes: The number of samples to be split is determined based on the video memory size; Based on the amount of data to be split, the sample dataset is split to generate at least two corresponding sample data subsets.

4. The image processing model training method according to claim 1, wherein the step of inputting the image data contained in each sample data subset into the image processing model for processing, obtaining the processing result output by the image processing model, and writing the processing result into memory includes: The image data contained in the i-th sample data subset is input into the image processing model for processing to obtain the processing result output by the image processing model, wherein i is a positive integer less than or equal to n, i starts from 1, and n is the total number of sample data subsets; Write the processing result into memory and determine whether i equals n; If not, then i is incremented by 1, and the process returns to the step of inputting the image data contained in the i-th sample data subset into the image processing model for processing, and obtaining the processing result output by the image processing model.

5. In the image processing model training method according to claim 4, if the result of determining whether i is equal to n is yes, then it is determined that the processing of the at least two sample data subsets is complete.

6. The image processing model training method according to claim 1, wherein the step of inputting the image data contained in each sample data subset into the image processing model for processing, obtaining the processing result output by the image processing model, and writing the processing result into memory includes: The image data contained in each sample data subset is input into the target processing layer of the image processing model for processing, and the corresponding processing results are generated. Determine whether the number of parameters corresponding to the processing result is greater than a preset parameter threshold; If so, the processing result is written into memory.

7. The image processing model training method according to claim 6, wherein if the execution result of determining whether the number of parameters corresponding to the processing result is greater than a preset parameter threshold is negative, the processing result is written into the video memory.

8. The image processing model training method according to claim 1, wherein performing gradient fusion on the calculation results and adjusting the model parameters of the image processing model based on the fusion results includes: The calculation results are then concatenated into a matrix to generate the corresponding concatenation result. The model parameters of the image processing model are adjusted based on the stitching result.

9. The image processing model training method according to claim 8, wherein the gradient calculation based on the processing result includes: Based on a preset loss function, calculate the loss value between the processing result corresponding to the target sample data subset and the sample label corresponding to the target sample data subset; Accordingly, the step of performing gradient fusion on the calculation results and adjusting the model parameters of the image processing model based on the fusion results includes: Based on the loss value, the feedback tuning parameters are calculated, and the model parameters of the image processing model are adjusted based on the feedback tuning parameters.

10. An image processing model training device, comprising: The splitting module is configured to write the sample dataset to be processed into video memory and split the sample dataset in the video memory to generate at least two sample data subsets. The processing module is configured to input the image data contained in each sample data subset into the image processing model for processing, obtain the processing result output by the image processing model, and if the number of parameters corresponding to the processing result is greater than a preset parameter threshold, write the processing result into memory. The calculation module is configured to write the processing result to the video memory after the processing of at least two sample data subsets is completed, and to perform gradient calculation based on the processing result to generate the calculation result corresponding to each sample data subset. The adjustment module is configured to perform gradient fusion on the calculation results and adjust the model parameters of the image processing model based on the fusion results.

11. An object recognition system, comprising: First processor and second processor; The first processor is configured to read image data of the image to be identified into memory and perform matrix stitching of the image data to generate batch image data; The second processor is configured to read the batch image data into video memory, split the batch image data into at least two batch image sub-data in the video memory, input the batch image sub-data into an image processing model for processing, generate corresponding batch image sub-features, and write the batch image sub-features into the memory; after the at least two batch image sub-data are processed, read the batch image sub-features in the memory into the video memory, fuse the batch image sub-features to generate batch image features, and write the batch image features into the memory. The first processor is configured to decompose the features of the batch images, generate image feature vectors corresponding to the images to be identified, and identify the images to be identified based on the image feature vectors to generate corresponding recognition results.

12. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the image processing model training method according to any one of claims 1 to 9.

13. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the image processing model training method according to any one of claims 1 to 9.

Images