An image processing method and apparatus

Abstract

Embodiments of the present application provide an image processing method and device, and relate to the technical field of deep learning. The method comprises: when an image needs to be processed, a preset model library file is called; the preset model library file comprises: a target model file of a target deep learning model corresponding to each of a plurality of preset image dimension information; the target model file contains weight parameters of the target deep learning model; a target memory space used for processing each to-be-processed image is allocated according to a memory size required for processing an image with the largest preset image dimension information by the target deep learning model; for each to-be-processed image, a target model file corresponding to preset image dimension information matched with the to-be-processed image is determined from the preset model library file; and the to-be-processed image is processed based on the determined target model file and the target memory space. In this way, the efficiency of image processing can be improved.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to an image processing method and apparatus. Background Technology

[0002] In related technologies, image processing based on deep learning models can include the following steps: Step 1: memory allocation; Step 2: model loading; Step 3: forward inference; and Step 4: memory release. The allocated memory is used to store the input data, output data, and the deep learning model itself.

[0003] When processing images using deep learning models, the required memory space is related to the dimensional information of the input image. Therefore, when the input image to be processed corresponds to different dimensional information, the image processing device needs to repeatedly execute steps one through four above. That is, the image processing device needs to frequently allocate and release memory to adapt to images with different dimensional information. The memory allocation and release process consumes a lot of time, thereby reducing the efficiency of image processing. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method and apparatus that can improve the efficiency of image processing. The specific technical solution is as follows:

[0005] Firstly, in order to achieve the above objectives, embodiments of this application disclose an image processing method, the method comprising:

[0006] When image processing is required, a preset model library file is invoked; wherein, the preset model library file includes: target model files corresponding to target deep learning models for each of the multiple preset image dimensional information; the target model files contain the weight parameters of the target deep learning models;

[0007] Based on the memory size required for the target deep learning model to process the image with the largest preset image dimensional information, allocate target memory space for processing each image to be processed;

[0008] For each image to be processed, a target model file corresponding to the preset image dimension information that matches the image to be processed is determined from the preset model library file;

[0009] Based on the determined target model file and the target memory space, the image to be processed is processed.

[0010] Optionally, before determining the target model file corresponding to the preset image dimension information that matches the image to be processed from the preset model library file, the method further includes:

[0011] For each image to be processed, determine whether there is a first image dimension information that is consistent with the image dimension information of the image to be processed in each preset image dimension information;

[0012] If the first image dimension information exists, then it is determined that the first image dimension information matches the image to be processed;

[0013] If the first image dimension information does not exist, then the second image dimension information among the multiple preset image dimension information is determined to match the image to be processed; wherein, among the multiple preset image dimension information, the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest.

[0014] Optionally, image dimension information includes batch number, number of channels, and image height and image width;

[0015] The step of determining whether the second image dimension information among multiple preset image dimension information matches the image to be processed includes:

[0016] From multiple preset image dimension information, determine the image dimension information that is not less than the image to be processed in terms of batch number, number of channels, and image height and image width, and use it as the third image dimension information;

[0017] Based on the differences between the image to be processed and the image to be processed in the dimensions of batch number, number of channels, and image height and image width, the second image dimension information that matches the image to be processed is determined from the third image dimension information.

[0018] Optionally, determining the second image dimension information matching the image to be processed from the third image dimension information based on the differences between the image to be processed and the image to be processed in the dimensions of batch number, number of channels, and image height and image width includes:

[0019] From the third image dimension information, determine the image dimension information with the smallest difference from the image to be processed in the dimension of batch number, and use it as the fourth image dimension information;

[0020] From the fourth image dimension information, determine the image dimension information with the smallest difference from the image to be processed in the dimension of the number of channels, and use it as the fifth image dimension information;

[0021] The image dimension information with the smallest difference in the dimensions of image height and image width compared to the image to be processed in the fifth image dimension information is determined as the second image dimension information that matches the image to be processed.

[0022] Optionally, the process of building the preset model library file includes:

[0023] Obtain information on each preset image dimension, as well as the initial model file representing the weight parameters of the initial deep learning model;

[0024] For each preset image dimension information, a model file of a target deep learning model that matches the preset image dimension information is generated based on the initial model file, and this model file is used as the target model file; wherein, the target deep learning model that matches the preset image dimension information means: a deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information.

[0025] The preset model library file is obtained by encapsulating each target model file.

[0026] Optionally, the step of encapsulating each target model file to obtain the preset model library file includes:

[0027] Each target model file and the model loading instruction are encapsulated to obtain the preset model library file; wherein, the model loading instruction is used to call the target model file to be run from the preset model library file; the preset model library file contains the common parts of each target model file and the private parts of each target model file.

[0028] Optionally, the preset model library file can be a dynamic library file or a static library file.

[0029] Secondly, in order to achieve the above objectives, embodiments of this application disclose an image processing apparatus, the apparatus comprising:

[0030] The preset model library file calling module is used to call preset model library files when image processing is required; wherein, the preset model library files include: target model files of target deep learning models corresponding to multiple preset image dimensional information; the target model files contain the weight parameters of the target deep learning models;

[0031] The target memory space allocation module is used to allocate target memory space for processing each image to be processed, according to the memory size required for the image with the largest preset image dimension information to be processed by the target deep learning model.

[0032] The target model file determination module is used to determine, for each image to be processed, the target model file corresponding to the preset image dimension information that matches the image to be processed from the preset model library file;

[0033] The image processing module is used to process the image to be processed based on the determined target model file and the target memory space.

[0034] Optionally, the device further includes:

[0035] The judgment module is used to determine, for each image to be processed, whether there is first image dimension information that is consistent with the image dimension information of the image to be processed in each preset image dimension information before determining the target model file corresponding to the preset image dimension information that matches the image to be processed from the preset model library file.

[0036] The first processing module is configured to determine that the first image dimension information matches the image to be processed if the first image dimension information exists.

[0037] The second processing module is configured to determine, if the first image dimension information does not exist, that the second image dimension information among a plurality of preset image dimension information matches the image to be processed; wherein, among the plurality of preset image dimension information, the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest.

[0038] Optionally, image dimension information includes batch number, number of channels, and image height and image width;

[0039] The second processing module includes:

[0040] The third image dimension information determination submodule is used to determine the image dimension information that is not less than the image to be processed in terms of batch number, number of channels, and image height and image width from multiple preset image dimension information, and use it as the third image dimension information.

[0041] The second image dimension information determination submodule is used to determine the second image dimension information that matches the image to be processed from the third image dimension information based on the difference between the image to be processed and the image to be processed in the dimensions of batch number, number of channels, and image height and image width.

[0042] Optionally, the second image dimension information determination submodule is specifically used to determine the image dimension information with the smallest difference in the dimension of the batch number from the third image dimension information, and use it as the fourth image dimension information;

[0043] From the fourth image dimension information, determine the image dimension information with the smallest difference from the image to be processed in the dimension of the number of channels, and use it as the fifth image dimension information;

[0044] The image dimension information with the smallest difference in the dimensions of image height and image width compared to the image to be processed in the fifth image dimension information is determined as the second image dimension information that matches the image to be processed.

[0045] Optionally, the device further includes:

[0046] The preset model library file building module is used to obtain the dimensional information of each preset image, as well as the initial model file representing the weight parameters of the initial deep learning model;

[0047] For each preset image dimension information, a model file of a target deep learning model that matches the preset image dimension information is generated based on the initial model file, and this model file is used as the target model file; wherein, the target deep learning model that matches the preset image dimension information means: a deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information.

[0048] The preset model library file is obtained by encapsulating each target model file.

[0049] Optionally, the preset model library file construction module is specifically used to encapsulate each target model file and the model loading instruction to obtain the preset model library file; wherein, the model loading instruction is used to call the target model file to be run from the preset model library file; the preset model library file contains the common parts of each target model file and the private parts of each target model file.

[0050] Optionally, the preset model library file can be a dynamic library file or a static library file.

[0051] In another aspect of this application, in order to achieve the above objectives, an embodiment of this application also discloses an electronic device, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0052] The memory is used to store computer programs;

[0053] When the processor executes the program stored in the memory, it implements any of the image processing methods described above.

[0054] In another aspect of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the image processing method as described above.

[0055] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the image processing methods described above.

[0056] Beneficial effects of the embodiments in this application:

[0057] The image processing method provided in this application embodiment calls a preset model library file when image processing is required. The preset model library file includes: target model files corresponding to multiple preset image dimensional information; the target model files contain weight parameters of the target deep learning models; target memory space is allocated for processing each image to be processed according to the memory size required for processing the image with the largest preset image dimensional information by the target deep learning model; for each image to be processed, a target model file corresponding to the preset image dimensional information matching the image to be processed is determined from the preset model library file; and the image to be processed is processed based on the determined target model file and the target memory space.

[0058] Based on the above processing, since the allocated target memory space is determined according to the memory size required to process the image with the largest preset image dimension information, the target memory space can meet the memory size required to process any input image with a dimension smaller than the largest preset image dimension information. That is, only one memory allocation is needed to process multiple images of different sizes with input dimension smaller than the largest preset image dimension information, eliminating the need for frequent memory allocation and deallocation, thus improving image processing efficiency.

[0059] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0061] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0062] Figure 2 A flowchart illustrating another image processing method provided in this application embodiment;

[0063] Figure 3A A flowchart illustrating another image processing method provided in this application embodiment;

[0064] Figure 3B A flowchart illustrating another image processing method provided in this application embodiment;

[0065] Figure 4 This is a schematic diagram illustrating the principle of image processing provided in an embodiment of this application;

[0066] Figure 5 A flowchart of an image processing method provided in this application embodiment;

[0067] Figure 6 A flowchart for determining the preset image dimension information that is closest to the input image, provided in an embodiment of this application;

[0068] Figure 7 A structural diagram of an image processing apparatus provided in an embodiment of this application;

[0069] Figure 8 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0071] This application provides an image processing method applicable to electronic devices that can process images based on a deep learning model. The deep learning model used in this application can be a CNN (Convolutional Neural Network) model or an RNN (Recurrent Neural Network) model, but is not limited to these. Correspondingly, image processing can involve identifying the category of objects (e.g., people, vehicles, etc.) in the image, identifying the image region to which an object belongs, or identifying whether a specified object exists in the image, but is not limited to these.

[0072] See Figure 1 , Figure 1 A flowchart of an image processing method provided in this application embodiment, the method may include the following steps:

[0073] S101: When image processing is required, call the preset model library file.

[0074] The preset model library files include target model files for multiple preset image dimensional information corresponding to the target deep learning models. Each target model file contains the weight parameters of the target deep learning model.

[0075] S102: Allocate target memory space for processing each image to be processed, based on the memory size required for the target deep learning model to process the image with the largest preset image dimensional information.

[0076] S103: For each image to be processed, determine the target model file corresponding to the preset image dimension information that matches the image to be processed from the preset model library file.

[0077] S104: Based on the determined target model file and target memory space, process the image to be processed.

[0078] Based on the image processing method provided in this application, since the allocated target memory space is determined according to the memory size required to process the image with the largest preset image dimension information, the target memory space can meet the memory size required to process any input image with a dimension smaller than the largest preset image dimension information. That is, only one memory allocation is needed to process multiple images of different sizes with input dimension smaller than the largest preset image dimension information, eliminating the need for frequent memory allocation and release, thus improving image processing efficiency.

[0079] For step S101, each target model file may also include the relationship between each network layer in the corresponding target deep learning model, as well as the network parameters of each network layer.

[0080] Multiple preset image dimension information can be set according to needs. For example, it can be set to the image dimension information of images commonly used in the current business.

[0081] In one embodiment, the image dimension information includes the number of batches, the number of channels, and the image height and width. That is, the image dimension information may include N (Num, number of batches), C (Channels), H (Height), and W (Width). N represents the number of batches, i.e., the number of image frames / feature maps processed simultaneously per unit time based on the deep learning model; C represents the number of channels; H represents the height of the image frame; and W represents the width of the image frame.

[0082] Regarding step S102, when processing images based on a deep learning model, it is necessary to acquire input data (i.e., the input image), output data (i.e., the processing result), and intermediate cached data (e.g., feature maps generated by the network layers). Therefore, before processing images based on a deep learning model, memory space needs to be pre-allocated for the deep learning model to store the aforementioned data. The memory space required for the deep learning model to process images is related to the image dimensionality information of the input image; that is, the larger the image dimensionality information of the input image, the larger the required memory space; the smaller the image dimensionality information of the input image, the smaller the required memory space.

[0083] In this embodiment, the memory size required to process an image with the largest preset image dimensionality based on a deep learning model can be calculated, and memory space (i.e., target memory space) can be allocated according to this memory size. Therefore, for any input image with less than the largest preset image dimensionality, the target memory space can satisfy the memory space required to process that image.

[0084] In one embodiment, prior to step S102 described above, the electronic device may also perform other initialization processes, such as image loading.

[0085] For steps S103 and S104, the image to be processed represents the input image that needs to be processed. It can be an image acquired by other devices, or it can be a feature map.

[0086] After allocating the target memory space, the electronic device can process multiple images with different image dimensional information based on the target memory space.

[0087] In one implementation, the electronic device can process each acquired image to be processed sequentially. For example, for each image to be processed, the electronic device can determine the image dimension information that matches the image to be processed from preset image dimension information, that is, the image dimension information that matches the image dimension information of the image to be processed. Then, the electronic device can obtain the target model file corresponding to the determined image dimension information from a preset model library file. Then, the electronic device can load the target model file to run the target deep learning model corresponding to the image dimension information, and process the image to be processed based on the allocated target memory space. Based on the above processing, memory space reuse can be achieved when processing images multiple times.

[0088] In one embodiment, after processing each image to be processed is completed, the electronic device can release the target memory space to avoid wasting memory space.

[0089] In one embodiment, see Figure 2 ,exist Figure 1 Based on the above step S103, the method may further include the following steps:

[0090] S105: For each image to be processed, determine whether there is a first image dimension information in each preset image dimension information that is consistent with the image dimension information of the image to be processed; if yes, then execute step S106; if no, then execute step S107.

[0091] S106: Determine that the dimension information of the first image matches the image to be processed.

[0092] S107: Determine that the second image dimension information among multiple preset image dimension information matches the image to be processed.

[0093] Among the multiple preset image dimension information, the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest.

[0094] In this embodiment, if among multiple preset image dimension information, there is image dimension information that matches the image to be processed (i.e., the first image dimension information), the electronic device can determine that the first image dimension information matches the image to be processed. That is, the image to be processed matches the target deep learning model corresponding to the first image dimension information. Furthermore, when processing the image, the electronic device can load the target model file corresponding to the first image dimension information to run the target deep learning model and process the image to be processed. This improves the efficiency of the target deep learning model in processing the image to be processed.

[0095] If the first image dimension information does not exist, the electronic device determines that the second image dimension information among multiple preset image dimension information matches the image to be processed. Since the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between them is minimal (i.e., the second image dimension information is closest to the image dimension information of the image to be processed), when processing the image, the electronic device can load the target model file corresponding to the second image dimension information to run the target deep learning model corresponding to the second image dimension information to process the image. Thus, when processing the image based on the target deep learning model corresponding to the second image dimension information, the electronic device only needs to perform minor optimization, improving the efficiency of the target deep learning model in processing the image.

[0096] In one embodiment, the image dimension information includes the batch size, the number of channels, and the image height and width. See also... Figure 3A ,exist Figure 2 Based on this, step S107 above may include the following steps:

[0097] S1071: From multiple preset image dimension information, determine the image dimension information that is not less than the image to be processed in terms of batch number, number of channels, and image height and image width, and use it as the third image dimension information.

[0098] S1072: Based on the differences between the image to be processed and the image to be processed in the dimensions of batch number, number of channels, and image height and image width, determine the second image dimension information that matches the image to be processed from the third image dimension information.

[0099] The third image dimension information can be one or more. If the third image dimension information is one, the electronic device directly determines that the third image dimension information matches the image to be processed.

[0100] If there are multiple third-dimensional image information pieces, since each third-dimensional image information piece is no less than the image dimension information of the image to be processed in all four dimensions—batch size, number of channels, image height, and image width—the electronic device can select one image dimension information piece from the various third-dimensional image information pieces to process the image to be processed based on the corresponding target deep learning model.

[0101] For example, the electronic device can compare each preset image dimension information with the image dimension information of the image to be processed, and filter out the preset image dimension information that is not less than the image to be processed in terms of batch number, number of channels, and dimensions of image height and image width, to obtain the third image dimension information.

[0102] In one implementation, for each third image dimension, the electronic device can calculate the difference between it and the image to be processed in four dimensions: batch size, number of channels, image height, and image width. Then, based on these four differences, the electronic device calculates the total difference between the third image dimension and the image dimension of the image to be processed. For example, the electronic device can calculate the weighted sum of the four differences as the total difference. Furthermore, the electronic device can determine the third image dimension with the smallest total difference as the second image dimension that matches the image to be processed.

[0103] In another implementation, after determining the third image dimension information, the electronic device can also randomly select one image dimension information from the third image dimension information and process the image to be processed based on the target deep learning model corresponding to the selected image dimension information.

[0104] In one implementation, after determining the third image dimension information, the electronic device can also determine the second image dimension information that matches the image to be processed from the third image dimension information according to a specified dimension order.

[0105] In one embodiment, specifying the dimensional order can represent the batch size, number of channels, image height, and image width; see corresponding examples. Figure 3B ,exist Figure 3A Based on this, step S1072 above may include the following steps:

[0106] S10721: From the third image dimension information, determine the image dimension information with the smallest difference in the dimension of batch number with the image to be processed, and use it as the fourth image dimension information.

[0107] S10722: From the fourth image dimension information, determine the image dimension information with the smallest difference from the image to be processed in the dimension of the number of channels, and use it as the fifth image dimension information.

[0108] S10723: Determine the image dimension information in the fifth image dimension information that has the smallest difference with the image to be processed in the dimensions of image height and image width, and use it as the second image dimension information that matches the image to be processed.

[0109] In one implementation, since the number of batches has the greatest impact on the memory occupied, the electronic device can determine the second image dimension information in the order of batch number, number of channels, image height, and image width.

[0110] For example, the electronic device determines the image dimension information (i.e., the fourth image dimension information) that has the smallest difference with the image to be processed in the dimension of batch processing number from the third image dimension information. If there is only one fourth image dimension information, the electronic device directly determines that the fourth image dimension information matches the image to be processed.

[0111] If there are multiple fourth image dimension information, the electronic device determines the image dimension information (i.e., the fifth image dimension information) that has the smallest difference with the image to be processed in the dimension of the number of channels. If there is only one fifth image dimension information, the electronic device directly determines that the fifth image dimension information matches the image to be processed.

[0112] If there are multiple fifth image dimension information, the electronic device determines that the image dimension information with the smallest difference in the dimensions of image height and image width from the image to be processed matches the image to be processed.

[0113] Based on the above processing, it is not necessary to list all the image dimensional information. That is, the preset model library file does not need to contain the target model file corresponding to all image dimensional information. By dynamically matching, the nearest neighbor image dimensional information can be selected for forward inference. While realizing memory space reuse, it can improve the efficiency of image processing.

[0114] Alternatively, the specified dimension order can be batch number, image height, image width, number of channels, or image height, image width, number of channels, batch number, but is not limited to these. Correspondingly, the method for determining the second image dimension information can refer to steps S10721-S10723 above.

[0115] Since the determined second image dimension information is not consistent with the image dimension information of the image to be processed (i.e., the second image dimension information is greater than the image dimension information of the image to be processed), optimization can be performed when the target deep learning model based on the second image dimension information processes the image to be processed.

[0116] For example, if the dimension information of the second image is greater than the dimension information of the image to be processed in the batch, the input data can be supplemented (for example, by supplementing image frames containing 0 data). Accordingly, after obtaining the output data, the output data corresponding to the supplemented data can be cropped to obtain the final output result.

[0117] For example, if the second image dimension information is greater than the image dimension information of the image to be processed in terms of image height and image width, then each image frame in the image to be processed can be padded (for example, the data of the padded part is 0). Accordingly, after obtaining the output data, the output data corresponding to the padded part can be cropped to obtain the final output result.

[0118] Based on the above processing, since the difference between the second image dimension information and the image dimension information of the image to be processed is minimal, the electronic device only needs to perform less optimization processing when the target deep learning model based on the second image dimension information processes the image to be processed, which can improve the efficiency of the target deep learning model in processing the image to be processed.

[0119] In one embodiment, the process of building the preset model library file includes:

[0120] Step 1: Obtain the dimensional information of each preset image, as well as the initial model file representing the weight parameters of the initial deep learning model.

[0121] Step 2: For each preset image dimension information, generate a model file of the target deep learning model that matches the preset image dimension information based on the initial model file, and use it as the target model file.

[0122] Step 3: Based on the encapsulation of each target model file, a preset model library file is obtained.

[0123] The target deep learning model that matches the preset image dimension information is represented as: the deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information.

[0124] In this embodiment of the application, the preset model library file can be generated by the electronic device or it can be generated in advance by other devices.

[0125] In one implementation, the electronic device can pre-train a deep learning model of the initial structure based on a training sample set (i.e., sample images) to obtain an initial deep learning model. Then, the weight parameters of the initial deep learning model can be obtained.

[0126] To improve the efficiency of image processing using a target-based deep learning model, preset image dimensions can be set according to preset step sizes. For example, the batch size can be set to 1, 2, 4, 8, 16, 24, or 32. Image height and width can be set to commonly used resolutions (e.g., 540P, 720P, 1080P). For instance, the maximum preset image dimension can be 32*3*1080*1920, resulting in a maximum supported image processing size of 32*3*1080*1920.

[0127] In one implementation, the electronic device can generate a preset model library file through a model generation engine. Correspondingly, the dimensional information of each preset image can be recorded in a preset JSON file for loading by the model generation engine. Additionally, this JSON file can also record the data format of the input image, for example, it can be Int8 format, FP16 format, or Float format.

[0128] Furthermore, for each preset image dimension, the model generation engine can optimize the initial deep learning model based on the initial model file. In one implementation, the electronic device can fuse the network layers in the initial deep learning model based on the aforementioned data. For example, it can fuse Conv layers, BN (Batch Normalization), and ReLU (Rectified Linear Unit) layers; and / or fuse Concat layers and convolutional layers.

[0129] Based on the above processing, weight sharing among network layers in a deep learning model can be achieved, reducing the space occupied by each target model file, thereby reducing the space occupied by the pre-defined model library file. Correspondingly, if image processing is performed based on an NPU (Nneural-network Processing Unit), the on-chip resource utilization of the NPU can also be improved.

[0130] In addition, for each preset image dimension information, the electronic device can also determine the dimension of the input and output data of each network layer in the corresponding target deep learning model, and determine the memory reuse method between each network layer in combination with the relationship between each network layer, and record the memory reuse method in the corresponding target model file.

[0131] In one embodiment, for each preset image dimension, the electronic device can determine the dimension of the input and output data of each network layer in the corresponding target deep learning model, and, in conjunction with the current hardware characteristics (such as on-chip memory size), determine the data loading method for each network layer. For example, for each network layer, if the dimension of the input data of that network layer is greater than a threshold, the data recorded in its corresponding target model file can be loaded in chunks. This threshold can be determined based on the on-chip memory size mentioned above. For example, loading 1 / 4 of the input data each time, or loading 1 / 8 of the input data each time. If the dimension of the input data of that network layer is less than the threshold, the data recorded in its corresponding target model file can be loaded entirely, that is, the entire input data is loaded directly.

[0132] In one embodiment, the model generation engine can also support user-defined plugins and generate a preset model library file by including user-defined network layer configuration information (e.g., weight information, implementation methods, etc.). This improves the flexibility of the generated preset model library file, thereby enhancing the flexibility of image processing. The implementation methods may include memory reuse between network layers and data loading methods for network layers.

[0133] In one embodiment, to further reduce the space occupied by the preset model library file, step three above may include the following steps: encapsulating each target model file and the model loading instructions to obtain the preset model library file.

[0134] The model loading instruction is used to call the target model file to be run from the preset model library file. The preset model library file contains the common parts of all target model files, as well as the private parts of each target model file.

[0135] In this embodiment, the target deep learning models corresponding to the preset image dimensional information have common parts. Therefore, in order to further reduce the space occupied by the preset model library file, the common parts (i.e., common parts) of each target model file can be recorded only once. Correspondingly, the parts of each target model file that are different from other target model files (i.e., private parts) can be recorded separately.

[0136] For example, if a common part represents the same network layer in multiple target model files, then in the preset model library file, only one copy of this common network layer is recorded for each of the multiple target model files, and this network layer is marked as a common part of the multiple target model files. Correspondingly, the preset model library file can also record other network layers in the multiple target model files besides the common network layers, and mark the target model file to which each of these network layers belongs.

[0137] In one implementation, the electronic device can read the aforementioned common and private portions in binary format and write them to a C file. Then, the model loading instructions (e.g., vload instructions) can be encapsulated together with the data to obtain a preset model library file. For example, the electronic device can encapsulate the data in Fatbin format. Alternatively, the electronic device can also read the aforementioned common and private portions in binary format and write them to a C++ file to generate a preset model library file. In this application, the file type of the written file is not limited to C files and C++ files, but can also be other types.

[0138] In one embodiment, the preset model library file can be a dynamic library file, that is, the electronic device can compile the above data into a dynamic library file to obtain the preset model library file.

[0139] For example, a dynamic library file can be generated using the following instructions:

[0140] gcc-shared–fpic XXX.c lib XXX.so

[0141] In the above instructions, `shared` indicates that the output result is a shared library type; `fpic` indicates that the output file is generated using address-independent code; and `XXX` indicates the name of the generated dynamic library file.

[0142] In one implementation, the code corresponding to the forward inference process (i.e., processing the input image) can be encapsulated into a library file to obtain an inference library. The electronic device can perform forward inference by calling the inference library. If the preset model library file is a dynamic library file, the electronic device can link to the preset model library file via dynamic linking during the inference library call process to process the image based on the target deep learning model. Because of the dynamic linking method, the inference library only needs to record the entry address of the preset model library file, not the entire preset model library file, thus saving memory space occupied by the inference library.

[0143] In one embodiment, the preset model library file can be a static library file, that is, the electronic device can compile the above data into a static library file to obtain the preset model library file.

[0144] For example, a static library file can be generated using the following instructions:

[0145] gcc –c XXX.c

[0146] ar–cr XXX.a XXX.o

[0147] In the above command, ar -cr means to package multiple compiled files into a single static library file, and XXX represents the name of the generated static library file.

[0148] See Figure 4 , Figure 4 This is a schematic diagram illustrating the principle of image processing provided in an embodiment of this application.

[0149] The offline implementation includes the process of generating preset model library files before deploying deep learning models online. For example, a model generation engine can generate model files corresponding to each preset image dimension based on a JSON file package (containing information about each preset image dimension) and a weight file. These model files, along with the model loading instructions, are packaged to generate a .so library (i.e., a dynamic library), resulting in the preset model library files. Furthermore, the model generation engine can support user-defined plugins during model file generation, incorporating user-defined network layer configuration information (e.g., weight information, implementation methods) into the preset model library files, thereby improving the flexibility of the generated preset model library files.

[0150] The forward inference part includes the image processing after the deep learning model is deployed. For example, after acquiring input data (e.g., an image), the inference library interface can be called to invoke the inference library (i.e., a library obtained by encapsulating the code corresponding to the forward inference process) to process the input data. Specifically, during processing, the generated .so library mentioned above can be called to process the input data based on the model file that matches the input data, and obtain the output result.

[0151] See Figure 5 , Figure 5 This is a flowchart of an image processing method provided in an embodiment of this application.

[0152] The process involves loading the model, specifically calling a preset model library file, which includes model files corresponding to multiple preset image dimensional information. Next, the maximum memory space is allocated, specifically, memory space is allocated according to the memory size required to process the image with the largest preset image dimensional information. Then, it can be determined whether any of the preset image dimensional information contains preset image dimensional information that matches the input image.

[0153] If it exists, the input image is processed using the model file corresponding to the consistent preset image dimension information, based on the allocated memory space.

[0154] If the model file does not exist, it is selected based on the allocated memory space and corresponds to the model file with the preset image dimension information that is closest to the input image. The preset image dimension information that is closest to the input image is defined as the preset image dimension information that is not smaller than the input image and has the smallest difference between its dimension information and that of the input image. Additionally, the output data can be cropped. That is, when processing the input image, padding and / or supplementing the input data can be performed. Correspondingly, after obtaining the output data, the output data corresponding to the padded and / or supplemented parts can be cropped.

[0155] After outputting the processing results, the allocated memory space can be released.

[0156] See Figure 6 , Figure 6 This is a flowchart illustrating how to determine the preset image dimension information that most closely matches the input image, as provided in an embodiment of this application.

[0157] When obtaining the image dimension information (i.e., N, C, H, W) of the input image, N, C, H, and W can be compared one by one to determine whether there is a third image dimension information among the preset image dimension information whose four dimensions are not less than the image dimension information of the input image.

[0158] If it does not exist, then it is determined that processing the input image is not supported.

[0159] If it exists, determine the fourth image dimension information that has the smallest difference with the input image in the N dimension from the third image dimension information. Then, determine the fifth image dimension information that has the smallest difference with the input image in the C dimension from the fourth image dimension information. Then, determine the image dimension information that has the smallest difference with the input image in the H and W dimensions from the fifth image dimension information, and use it as the preset image dimension information that is closest to the input image.

[0160] Based on the same inventive concept, this application also provides an image processing apparatus, see [link to relevant documentation]. Figure 7 , Figure 7 This application provides a structural diagram of an image processing apparatus, which may include:

[0161] The preset model library file calling module 701 is used to call the preset model library file when image processing is required; wherein, the preset model library file includes: target model files of target deep learning models corresponding to multiple preset image dimensional information; the target model files contain the weight parameters of the target deep learning models;

[0162] The target memory space allocation module 702 is used to allocate target memory space for processing each image to be processed according to the memory size required for the image with the largest preset image dimension information processed by the target deep learning model.

[0163] The target model file determination module 703 is used to determine, for each image to be processed, the target model file corresponding to the preset image dimension information that matches the image to be processed from the preset model library file;

[0164] The image processing module 704 is used to process the image to be processed based on the determined target model file and the target memory space.

[0165] Optionally, the device further includes:

[0166] The judgment module is used to determine, for each image to be processed, whether there is first image dimension information that is consistent with the image dimension information of the image to be processed in each preset image dimension information before determining the target model file corresponding to the preset image dimension information that matches the image to be processed from the preset model library file.

[0167] The first processing module is configured to determine that the first image dimension information matches the image to be processed if the first image dimension information exists.

[0168] The second processing module is configured to determine, if the first image dimension information does not exist, that the second image dimension information among a plurality of preset image dimension information matches the image to be processed; wherein, among the plurality of preset image dimension information, the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest.

[0169] Optionally, image dimension information includes batch number, number of channels, and image height and image width;

[0170] The second processing module includes:

[0171] The third image dimension information determination submodule is used to determine the image dimension information that is not less than the image to be processed in terms of batch number, number of channels, and image height and image width from multiple preset image dimension information, and use it as the third image dimension information.

[0172] The second image dimension information determination submodule is used to determine the second image dimension information that matches the image to be processed from the third image dimension information based on the difference between the image to be processed and the image to be processed in the dimensions of batch number, number of channels, and image height and image width.

[0173] Optionally, the second image dimension information determination submodule is specifically used to determine the image dimension information with the smallest difference in the dimension of the batch number from the third image dimension information, and use it as the fourth image dimension information;

[0174] From the fourth image dimension information, determine the image dimension information with the smallest difference from the image to be processed in the dimension of the number of channels, and use it as the fifth image dimension information;

[0175] The image dimension information with the smallest difference in the dimensions of image height and image width compared to the image to be processed in the fifth image dimension information is determined as the second image dimension information that matches the image to be processed.

[0176] Optionally, the device further includes:

[0177] The preset model library file building module is used to obtain the dimensional information of each preset image, as well as the initial model file representing the weight parameters of the initial deep learning model;

[0178] For each preset image dimension information, a model file of a target deep learning model that matches the preset image dimension information is generated based on the initial model file, and this model file is used as the target model file; wherein, the target deep learning model that matches the preset image dimension information means: a deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information.

[0179] The preset model library file is obtained by encapsulating each target model file.

[0180] Optionally, the preset model library file construction module is specifically used to encapsulate each target model file and the model loading instruction to obtain the preset model library file; wherein, the model loading instruction is used to call the target model file to be run from the preset model library file; the preset model library file contains the common parts of each target model file and the private parts of each target model file.

[0181] Optionally, the preset model library file can be a dynamic library file or a static library file.

[0182] This application also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804.

[0183] Memory 803 is used to store computer programs;

[0184] When processor 801 executes a program stored in memory 803, it performs the following steps:

[0185] When image processing is required, a preset model library file is invoked; wherein, the preset model library file includes: target model files corresponding to target deep learning models for each of the multiple preset image dimensional information; the target model files contain the weight parameters of the target deep learning models;

[0186] Based on the memory size required for the target deep learning model to process the image with the largest preset image dimensional information, allocate target memory space for processing each image to be processed;

[0187] For each image to be processed, a target model file corresponding to the preset image dimension information that matches the image to be processed is determined from the preset model library file;

[0188] Based on the determined target model file and the target memory space, the image to be processed is processed.

[0189] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0190] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0191] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0192] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0193] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image processing methods.

[0194] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the image processing methods described above.

[0195] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0196] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0197] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0198] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image processing method, characterized by, The method comprises: When an image needs to be processed, a preset model library file is called; wherein the preset model library file comprises: target model files of target deep learning models corresponding to a plurality of preset image dimension information respectively; the target model file contains weight parameters of the target deep learning model; the image dimension information comprises batch number, channel number, image height and image width; According to the memory size required by the target deep learning model to process the image with the largest preset image dimension information, a target memory space for processing each to-be-processed image is allocated; For each to-be-processed image, it is judged whether there is first image dimension information consistent with the image dimension information of the to-be-processed image in each preset image dimension information; If the first image dimension information exists, it is determined that the first image dimension information matches the to-be-processed image; If the first image dimension information does not exist, from the plurality of preset image dimension information, determine the third image dimension information which is not less than the image dimension information of the to-be-processed image in the dimensions of batch number, channel number, image height and image width; Based on the difference in the dimensions of batch number, channel number, image height and image width between the to-be-processed image and the third image dimension information, determine the second image dimension information matching the to-be-processed image from the third image dimension information; wherein the second image dimension information is not less than the image dimension information of the to-be-processed image in the plurality of preset image dimension information, and the difference between the image dimension information of the to-be-processed image and the second image dimension information is the smallest; For each to-be-processed image, determine the target model file corresponding to the preset image dimension information matching the to-be-processed image from the preset model library file; Based on the determined target model file and the target memory space, process the to-be-processed image.

2. The method of claim 1, wherein, The method comprises: From the third image dimension information, determine the image dimension information with the smallest difference in the dimension of batch number with the to-be-processed image as the fourth image dimension information; From the fourth image dimension information, determine the image dimension information with the smallest difference in the dimension of channel number with the to-be-processed image as the fifth image dimension information; Determine the image dimension information with the smallest difference in the dimensions of image height and image width with the to-be-processed image in the fifth image dimension information as the second image dimension information matching the to-be-processed image.

3. The method of claim 1, wherein, The construction process of the preset model library file comprises: Obtain each preset image dimension information and an initial model file representing weight parameters of an initial deep learning model; For each preset image dimension information, a model file of a target deep learning model matching the preset image dimension information is generated as a target model file according to the initial model file, wherein the target deep learning model matching the preset image dimension information represents a deep learning model obtained by model optimization of the initial deep learning model according to the preset image dimension information. The preset model library file is obtained by packaging the target model files.

4. The method of claim 3, wherein, The preset model library file is obtained by packaging the target model files. The preset model library file is obtained by packaging the target model files.

5. The method according to any one of claims 1 to 4, characterized in that, The preset model library file is obtained by packaging the target model files.

6. An image processing apparatus characterized by comprising: The preset model library file is a dynamic library file or a static library file. The apparatus comprises: A preset model library file calling module is configured to call a preset model library file when an image needs to be processed, wherein the preset model library file comprises target model files of target deep learning models corresponding to a plurality of preset image dimension information respectively, and the target model files contain weight parameters of the target deep learning models; the image dimension information comprises batch number, channel number, image height and image width; A target memory space allocation module is configured to allocate a target memory space for processing each to-be-processed image according to a memory size required for processing an image with the largest preset image dimension information by the target deep learning model; A judgment module is configured to judge, for each to-be-processed image, whether there is first image dimension information consistent with image dimension information of the to-be-processed image in the preset image dimension information; A first processing module is configured to determine that the first image dimension information matches the to-be-processed image if the first image dimension information exists; A second processing module is configured to determine, if the first image dimension information does not exist, third image dimension information that is not less than the image dimension information of the to-be-processed image in the dimensions of batch number, channel number, image height and image width from the plurality of preset image dimension information; determine second image dimension information matching the to-be-processed image from the third image dimension information based on the difference between the to-be-processed image in the dimensions of batch number, channel number, image height and image width; wherein the second image dimension information is not less than the image dimension information of the to-be-processed image and has the smallest difference with the image dimension information of the to-be-processed image in the plurality of preset image dimension information; A target model file determination module is configured to determine, for each to-be-processed image, a target model file corresponding to preset image dimension information matching the to-be-processed image from the preset model library file; An image processing module is configured to process the to-be-processed image based on the determined target model file and the target memory space.

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps in any one of claims 1-5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps in any one of claims 1-5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, ​

Images