Image processing method and image processing apparatus

Abstract

The application provides an image processing method and an image processing device, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring resource parameters of a neural network processing system and model parameters of a first neural network; determining a first slice size and a first padding size of an input image and a number of layers of a splicing layer in the first neural network according to the resource parameters and the model parameters, wherein the quality of an output image obtained by processing the input image according to the first slice size, the first padding size and the splicing layer in the first neural network is within a preset threshold range. According to the scheme, the appropriate slice size, padding size and splicing layer are determined according to the system resource parameters and the model parameters on the premise that the quality of the output image meets the preset threshold range, so that the operation cost and the memory consumption can be effectively reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to an image processing method and an image processing apparatus. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, neural networks (such as deep neural networks) have achieved great success in the processing and analysis of various media signals, including images, videos, and audio, in recent years. In image processing, given the high demands of processing high-resolution images using deep neural network models, reducing computational load and memory consumption has become crucial.

[0003] To address the issues of high computational and memory consumption in processing high-resolution input images, the concept of slicing emerged. This involves dividing the input image into multiple slices, processing each slice separately, and then stitching the processed results together. To ensure the stitched image is as similar as possible to the output image obtained without slicing, adjacent pixels are padded into the slices. However, existing slicing techniques are prone to producing blocky artifacts at the junctions, leading to a decline in visual quality and detection success rate. Since padding increases computational load, existing image segmentation methods (including slicing and padding) either reduce computational and memory consumption but result in excessive distortion of the output image, or they maintain relatively low distortion but fail to effectively reduce computational and memory consumption.

[0004] Therefore, how to reduce computational load and memory consumption while ensuring the quality of output images is an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides an image processing method and an image processing apparatus that can reduce computational load and memory consumption while ensuring the quality of the output image.

[0006] In a first aspect, an image processing method is provided, the method comprising: acquiring resource parameters of a neural network processing system and model parameters of a first neural network; determining, based on the resource parameters and model parameters, a first slice size and a first padding size of the input image of the first neural network and the number of stitching layers in the first neural network, wherein the quality of a first output image is within a preset threshold range, and the first output image is the output image obtained when the first neural network processes the input image based on the first slice size, the first padding size and the stitching layers.

[0007] In the technical solution of this application, under the premise that the quality of the output image meets the preset threshold range, the appropriate slice size and padding size, as well as the number of layers of the splicing layer in the first neural network, are determined according to the system resource parameters and model parameters. This can effectively reduce the computational overhead and memory consumption, and avoid excessive computation or severe distortion. This resource parameter represents the computing and storage capabilities of the neural network processing system. The first neural network refers to any neural network (i.e., a neural network model) that can be called by the neural network processing system; it can also be understood as any neural network that can run within the neural network processing system.

[0008] Neural network processing systems can be processors, chips, hardware modules, etc.

[0009] Optionally, network architecture analysis can be performed on the first neural network to obtain its model parameters. The model parameters of the first neural network represent the computational and storage requirements of the first neural network, and may include parameters such as its receptive field, operators, and the computational cost of each layer of the neural network.

[0010] A concatenation layer can be understood as a neural network layer in the first neural network that processes individual slices and then concatenates the results into a unified feature map. "The number of the concatenation layer in the first neural network" means which layer (or which number) in the first neural network is the concatenation layer. Therefore, it can be seen that if the first neural network consists of layers 0 to L-1, for a total of L layers, then the number of the concatenation layer in the first neural network must be a value between 0 and L-1, where L is a positive integer greater than 1.

[0011] In the embodiments of this application, quality can be represented by, for example, distortion or image precision. The higher the quality, the more accurate the results will be when the output image is used for subsequent image recognition, image classification and other processing.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the first slice size, the first padding size, and the number of stitching layers in the first neural network are values ​​that minimize the computational or storage requirements of the first neural network while ensuring that the quality of the first output image is within a preset threshold range. This further achieves the technical effect of better reducing computational overhead and memory consumption.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the size of the first slice is less than or equal to the size of the input image, the number of layers in the stitching layer is less than or equal to the total number of layers in the first neural network, the first padding size is determined based on a preset threshold range and the receptive field of the stitching layer, and the amount of computation or storage required for each layer of the first neural network does not exceed the computational or storage capacity of the neural network system.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the method described above further includes: dividing the input image into multiple patches according to the first slice size and the first fill size, stitching the processing results corresponding to the multiple patches in the stitching layer, and obtaining a first output image based on the stitched image, wherein the processing results corresponding to the multiple patches are obtained by the first neural network processing the multiple patches.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the quality of the first output image being within a preset threshold range specifically means that the ratio of the quality of the first output image to the quality of the second output image is within a preset range, and the second output image is the output image obtained when the first neural network directly processes the input image.

[0016] In a second aspect, an image processing apparatus is provided, the apparatus including a unit for performing a method of any of the implementations of the first aspect described above.

[0017] Thirdly, an image processing apparatus is provided, comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute a method in any implementation of the first aspect.

[0018] Fourthly, a computer-readable medium is provided that stores program code for execution by a device, the program code including methods for performing any implementation of the first aspect.

[0019] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any of the implementations of the first aspect described above.

[0020] In a sixth aspect, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method in any of the implementations of the first aspect described above.

[0021] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in any of the implementations of the first aspect. Attached Figure Description

[0022] Figure 1 This is a schematic flowchart of an image processing method according to an embodiment of this application.

[0023] Figure 2 This is a schematic flowchart illustrating the execution process of the image processing method according to an embodiment of this application.

[0024] Figure 3 This is a schematic diagram illustrating the test results of the image processing method according to an embodiment of this application.

[0025] Figure 4 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application.

[0026] Figure 5 This is a schematic diagram of the hardware structure of the image processing apparatus according to an embodiment of this application. Detailed Implementation

[0027] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0028] This application relates to neural networks. In order to better understand the methods of this application, the relevant terms and concepts of neural networks will be introduced below.

[0029] In traditional methods, the input image is sliced ​​to reduce computation and memory consumption. However, current techniques do not consider better padding methods; they simply use zeros or similar values ​​for convenience, leading to blocky artifacts at the connection points. Furthermore, padding also increases computational load, and current techniques do not consider how to balance computational load and distortion. In short, current techniques do not account for the different computational load, memory consumption, and distortion levels resulting from different segmentation methods. This means that while the input image is segmented, these methods either reduce computational load and memory consumption but result in excessive distortion in the output image, or they ensure lower distortion in the output image but do not effectively reduce computational load and memory consumption. In other words, current techniques cannot effectively improve image processing performance.

[0030] To address the aforementioned problems, this application proposes an image processing scheme that determines the number of stitching layers, slice size, and fill size based on the resource parameters of the neural network processing system and the parameters of the neural network, while ensuring that the quality of the neural network's output is within a preset threshold range, thereby solving the problems caused by the aforementioned image segmentation.

[0031] In the embodiments of this application, image segmentation can be understood as slicing the input image and filling the slices, thereby dividing the input image into multiple patches.

[0032] Figure 1 This is a schematic flowchart of an image processing method according to an embodiment of this application. The following is a description of... Figure 1 The steps shown will be explained.

[0033] 101. Obtain the resource parameters of the neural network processing system and the model parameters of the first neural network.

[0034] This resource parameter represents the computing and storage capabilities of the neural network processing system. The first neural network refers to any neural network (i.e., a neural network model) that can be called by the neural network processing system; it can also be understood as any neural network that can run within the neural network processing system.

[0035] Neural network processing systems can be processors, chips, hardware modules, etc.

[0036] Optionally, network architecture analysis can be performed on the first neural network to obtain its model parameters. The model parameters of the first neural network represent the computational and storage requirements of the first neural network, and may include parameters such as its receptive field, operators, and the computational cost of each layer of the neural network.

[0037] 102. Based on the above resource parameters and model parameters, determine the first slice size and first padding size of the input image of the first neural network, as well as the number of stitching layers in the first neural network.

[0038] A concatenation layer can be understood as a neural network layer in the first neural network that processes individual slices and then concatenates the results into a unified feature map. "The number of the concatenation layer in the first neural network" means which layer (or which number) in the first neural network is the concatenation layer. Therefore, it can be seen that if the first neural network consists of layers 0 to L-1, for a total of L layers, then the number of the concatenation layer in the first neural network must be a value between 0 and L-1, where L is a positive integer greater than 1.

[0039] Optionally, based on the above resource parameters and model parameters, a candidate range of slice size, fill size and number of layers of splicing layer can be determined, and then any slice size and fill size can be selected from the candidate range as the above first slice size and first fill size.

[0040] To achieve better technical results, the optimal values ​​of slice size, fill size, and number of stitching layers can be determined based on the above resource parameters and model parameters, and then these optimal values ​​can be used as the first slice size, first fill size, and number of stitching layers.

[0041] In determining the first slice size, the first padding size, and the number of stitching layers in the first neural network, it is also necessary to ensure that the quality of the first output image is within a preset threshold range. This first output image is obtained by the first neural network processing the input image based on the aforementioned first slice size, first padding size, and number of stitching layers. This is because if only reducing computational load and memory consumption is considered without regard to the final output result, it would lead to a situation where, although the computational load is significantly reduced, the quality of the output image is very poor (e.g., severely distorted), rendering it unusable and negating the purpose of image processing.

[0042] In this embodiment, quality can be represented by distortion or image precision, for example. Higher quality results in more accurate images when used for subsequent image recognition, classification, or other processing. Specific quality parameters used to measure quality can include, for example, mAP and IoU (interchange of identity) as described below. This allows for the selection of slice and fill sizes, as well as the number of stitching layers, that result in lower computational load and memory consumption, while ensuring the output image quality remains within acceptable limits. In other words, it further reduces computational load and memory consumption while maintaining the quality of the output image.

[0043] Optionally, the slice size, padding size, and number of splicing layers can be determined by combining the performance metrics of the first neural network. These performance metrics can be computational load, memory consumption, latency, etc.—in other words, parameters used to evaluate the performance of the first neural network itself. For ease of understanding, this application's embodiments primarily use computational load as an example.

[0044] In some implementations, for example, a candidate range for slice size, fill size, and the number of stitching layers can be determined based on the aforementioned resource parameters and model parameters. Then, the optimal values ​​for slice size, fill size, and the number of stitching layers can be determined based on performance metrics and the aforementioned preset threshold range. Assuming the performance metric is computational complexity, this is equivalent to first determining, based on the aforementioned resource parameters and model parameters, that when the slice size, fill size, and the number of stitching layers are within a certain candidate range, the overall computational complexity of the first neural network processing the segmented image will not exceed the computational capacity of the neural network system, and the quality of the first output image is within a preset range. Then, from this candidate range, the slice size and fill size that optimize the aforementioned computational complexity can be found, and the number of stitching layers that optimizes the aforementioned computational complexity can also be found.

[0045] Assume the first neural network Π is an L-layer network, consisting of layers 0 to L-1, where layer 0 acquires the input image and layer L-1 outputs the processed image. The on-chip memory size of the neural network processing system is M words (1 word is 32 bits). For ease of calculation, the required storage precision for the input image, operator coefficients, output image, operator output, etc., is 1 word. The size of the input image is (H, W, C), where H represents the height of the input image, W represents the width of the input image, and C represents the number of channels of the input image. The hyperparameter that ensures the quality of the first output image is within a preset threshold range is λ. When λ=1, it represents the ratio of the quality of the first output image to the quality of the second output image, that is, the value range of λ is 0≤λ≤1. When λ=1, it means that the quality of the first output image and the quality of the second output image are the same. For ease of understanding, this can be regarded as the user's tolerance for the degradation of the quality of the output image after image segmentation.

[0046] Alternatively, the first slice size, the first fill size, and the number of splicing layers can be determined by minimizing the computational cost F of Π. This process can be represented by the following formula (1).

[0047] (1) in, This indicates the optimal number of layers for splicing. The optimal value for the height representing the slice size. F represents the optimal value of the slice size width. Π This represents the computational complexity in the first neural network processing step, (h) l w l c l Let represent the height, width, and number of channels of the input feature map of layer l, respectively, where l is an integer greater than or equal to 0 and less than or equal to L-1. Equation (1) is the formula that minimizes the computational complexity. The number of layers, the height of the first slice, and the width of the first slice are respectively determined. Equation (1) also needs to satisfy the following constraints.

[0048]

[0049] In other words, the first L constraints are to ensure that the total memory occupied by the inputs, outputs, and operator coefficients of each layer (from layer 0 to layer L-1) of the first neural network does not exceed the M-word limit; the last three constraints are to ensure that the concatenation layer... The layer number range is the same as the layer number range of the first neural network (from layer 0 to layer L-1), and the slice height is... The slice width is less than the height H of the input image (less than or equal to H-1). It is smaller than the width W of the input image (less than or equal to W-1).

[0050] Let NT represent the height of the input image as a slice of unfilled material. With width The number of slices obtained after segmentation, then In other words, the number of slices N T The integer part of the quotient obtained by dividing the input image size by the size of the unpadded slice. Also, the height and width (h) of the input feature map of layer l. l w l ) satisfies the following formula (2).

[0051] (2) Among them, the The input tiles of the l-th layer are stitched together; rl is the receptive field size of the input of the l-th layer; This is the size of the tile after filling, and the fill size is... Therefore, in equation (2), This refers to the size of the tile after filling at layer 0. Specifically, it is the size after filling by a specified amount. The size of the tile obtained after filling; Let h represent the height of the output feature map at the m-th layer when the height of the input feature map is h. Therefore, in equation (2) The height of the input feature map is At that time, the height of the output feature map in the (l-1)th layer; Let w represent the width of the output feature map at the m-th layer when the width of the input feature map is w. Therefore, in equation (2) The width of the input feature map is At that time, the width of the output feature map in the (l-1)th layer. Assume... Let represent the height and width of the l-th layer operator, respectively. If the operator is a convolution, then the height and width of the operator are the same as the height and width of the convolution kernel. If the operator is a pooling, then for max pooling, the height and width of the operator are both 0, while for average pooling, they are the same as the height and width of the pooling kernel. If the operator is an activation function, then assuming the activation function can be implemented as a lookup table that does not involve any computation, then the height and width of the operator are 0. In equation (2) This means that when l = lc, the value is... That is, N T When l ≠ lc, the value is [value to be filled in]. The value is 1, meaning that the value is NT in the splicing layer and 1 in the non-splicing layer.

[0052] When the splicing layer (the first) The input of the layer is N T There are 1 input tile, and the size of the already filled input tile at layer 0 is 1. The total computational cost of the entire first neural network at that time; Let represent the computational complexity from layer n to layer m, where the size of the input feature map of layer n is (x, y). Indicates from level 0 to level 1 The computational complexity of a layer is calculated by multiplying the computational complexity per tile by the number of tiles, since the number of tiles is NT. The size of the input feature map for layer 0 is... , Indicates from the first The computational complexity from layer L to layer L-1 is due to the The layer concatenates the processing results of multiple image patches to obtain a feature map, so the computational complexity is related to the size of the concatenated feature map. The size of the input feature map of the layer is .

[0053] The optimal value of the fill size in the embodiments of this application is .

[0054] By adjusting the hyperparameter λ, the preset threshold range for the output image can be adjusted. Assuming this preset threshold range represents the range of image quality degradation in the output image, the closer λ is to 1, the less the image quality degrades; conversely, the closer λ is to 0, the more the image quality degrades. The so-called image quality degradation in the output image can be understood as the quality of the first output image being worse than that of the second output image.

[0055] In determining the optimal values ​​for slice size and the number of stitching layers, an exhaustive search can be performed. , , The optimization is performed using the parameter space. For example, the search can begin in ascending order of parameter magnitude, thus iterating from the minimum allowed value of the slice size to the maximum allowed value, and Iterate from 0 (getting input) to L-1 (producing the final output). To speed up the optimization, the parameter search space can be reduced without sacrificing optimization performance if the following factors are considered. Assume that the search explores {...} while satisfying all L constraints. =0,…,m},{ }, { First, assume l=m is an activation function layer (e.g., ReLU, Sigmoid function, etc.) or a pooling layer, where (hm, wm) = (hm+1, wm+1). In both cases, for =m+1 and =m, the total computational complexity of the first neural network is the same as the conditions for satisfying L constraints, because M m+1 (m)≤M m (m) and M m+1 (i)=M m (i), i≠m, where, when When =l, M l (m) represents the storage requirement of the m-th level. Therefore, it can be skipped during the search. =m+1、{ }and{ Secondly, for a given tile size ( , ) and splicing layer Assume the l-th layer (l≤ The memory requirement exceeds M. Since r0≤r1≤…≤r L-1 Therefore, for splicing layers Tile size ( , ), the l-th layer (l≤l) c The memory requirements of the ') also exceed M. Therefore, the search splicing layer and tile size ( , () is unnecessary and can be terminated.

[0056] In other words, when determining the slice size from the candidate range, it is possible to skip the next layer in the activation function layer and pooling layer.

[0057] 103. Divide the input image into multiple patches according to the first slice size and the first fill size.

[0058] In other words, the input image is divided into multiple slices according to the slice size determined in step 102, and then multiple slices are filled according to the determined fill size to obtain multiple blocks.

[0059] 104. The processing results corresponding to the above multiple image patches are stitched together, and a first output image is obtained from the stitched image. The processing results corresponding to the above multiple image patches are obtained by the first neural network processing the multiple image patches.

[0060] In other words, by inputting the above-mentioned multiple image patches into the first neural network, the processing results corresponding to the multiple image patches can be obtained. After these processing results are stitched together, the stitched processed image can be obtained. The stitched processed image can be used as the input of the next layer of the first neural network, or as the target processing result of the first neural network on the input image, that is, the above-mentioned first output image.

[0061] In this embodiment of the application, it is assumed that the output image obtained by directly inputting the input image into the first neural network is called the second output image. The closer the quality of the first output image is to the quality of the second output image, the better.

[0062] The stitching process requires the processing results of all the aforementioned multiple tiles before it can be executed.

[0063] exist Figure 1 In the illustrated scheme, while ensuring the output image quality meets a preset threshold range, appropriate slice size, padding size, and the number of stitching layers are determined based on system resource parameters and model parameters. This effectively combines improving output image quality with reducing computational overhead and memory consumption, thus significantly improving image processing performance without excessive computation or severe distortion. Furthermore, determining the slice size, padding size, and the number of stitching layers minimizes the computational or storage requirements of the first neural network while still ensuring the output image quality meets the preset threshold range, further enhancing the reduction of computational overhead and memory consumption.

[0064] Figure 2 This is a schematic flowchart illustrating the execution process of the image processing method according to an embodiment of this application. Figure 2 It can be seen as Figure 1 An example.

[0065] 201. Analyze the neural network model to obtain the receptive field, operators, and computational cost of each layer of the neural network.

[0066] The neural network model can be analyzed using the network architecture analysis module to obtain the receptive field, operators, and computational complexity of each layer of the neural network.

[0067] This neural network model can be viewed as... Figure 1 An example of the first neural network in China.

[0068] 202. Obtain the memory size of the neural network processing system.

[0069] This memory size can be, for example, the storage space of fast on-chip memory, and can be used to evaluate the computational and storage capabilities of a neural network system. The aforementioned on-chip memory can include, for example, L1, L2, L3, HBM, DDR, etc. The memory size can be considered as... Figure 1 An example of the resource parameters.

[0070] Steps 201 and 202 can be seen as an example of step 101. Steps 201 and 202 can be executed simultaneously or at different times, and there is no restriction on the order of execution.

[0071] 203. Determine the first slice size and the first fill size of the input image, as well as the number of stitching layers.

[0072] Step 203 can be followed exactly as described in step 102.

[0073] 204(a) Divide the input image into multiple patches.

[0074] The content of the multiple tiles obtained in step 204 can be completely referred to the description in step 103.

[0075] 204(b) Set the current processing layer N of the neural network model to 0, and select the next patch to be processed from the multiple patches obtained in step 204(a).

[0076] 205. Input the patch to be processed into the Nth layer of the neural network model for processing, and obtain the processing result of the patch, which becomes the next patch to be processed.

[0077] 206. Determine whether the Nth layer is the last layer. If the result is "yes", proceed to step 207; if the result is "no", proceed to step 208.

[0078] 207. Output the first output image.

[0079] In other words, the first output image is the result of the last layer of the neural network model.

[0080] 208. Determine whether the Nth layer is a splicing layer. If the result is "yes", proceed to step 209; if the result is "no", proceed to step 210.

[0081] 209. Determine whether the parts can be spliced. If the result is "yes", proceed to step 211; if the result is "no", proceed to step 204(b).

[0082] Step 209 determines whether the tiles can be spliced ​​based on whether the Nth layer has processed all the multiple tiles.

[0083] 210. Increment the value of N by 1, and proceed to step 205.

[0084] In other words, the input image is processed using the next layer of the neural network. Step 210 involves traversing all neural network layers.

[0085] 211. Combine the processing results of multiple blocks, and the combined block becomes the next block to be processed. Proceed to step 210.

[0086] Steps 204-211 can be seen as an example of steps 103-104, mainly implementing the following process. First, the input image is divided into patches with padding of a selected size. Step 0: The current processing layer N of the neural network model is set to 0. At this time, the next patch from the input image can also be loaded into memory as data input. Step 1: After loading the kernel of layer N into memory, data processing of layer N begins. Assuming N is the last layer of the network, the output of layer N is the final output of the neural network model's processing of the input image. Assuming N is neither the last layer nor a concatenation layer, the output of layer N becomes the input of layer N+1, so it is kept in memory, N is incremented by 1, and step 1 is repeated. Assuming N is not the last layer but a concatenation layer, if layer N has processed multiple patches, all processing results of layer N are concatenated, the concatenated patch is kept in memory, N is incremented by 1, and step 1 is repeated; otherwise, step 0 is repeated.

[0087] To facilitate understanding of the technical effects of the solutions in this application's embodiments, a specific test example is provided below. The solutions in this application's embodiments can be used in various neural networks, such as convolutional neural networks. The following test uses a residual network (ResNet) as an example. Residual networks are a type of deep convolutional network proposed in 2015. Compared to traditional convolutional neural networks, residual networks are easier to optimize and can improve accuracy by significantly increasing their depth. The core of residual networks is solving the side effects (degradation problem) caused by increasing depth, thus improving network performance simply by increasing network depth. Residual networks typically contain many structurally identical sub-modules. A number is usually used to represent the number of times a sub-module is repeated, such as ResNet50, which indicates that the residual network has 50 sub-modules.

[0088] In this example, the input image resolution was adjusted to 1600x1600, and the 11th layer (Conv2_9) of the trained ResNet50 was set as the stitching layer. The receptive field size of the 11th layer of ResNet50 is 51, therefore it can be inferred that when the padding size is greater than or equal to 26 (calculated using 51 / 2), blocky artifacts will not appear in the stitched image. Three slice sizes were determined: 96x96, 192x192, and 384x384 (i.e., the test values ​​for slice size), and the test padding size values ​​were 0 and 64. The test results are as follows: Figure 3 As shown.

[0089] Figure 3 This is a schematic diagram illustrating the test results of the image processing method according to an embodiment of this application. Figure 3 In the diagram, curves A, B, and C represent the changes in quality parameters for slice sizes of 384x384, 192x192, and 96x96, respectively. From... Figure 3 It can be seen that when the fill size is equal to 64, which is equivalent to p=64 and λ=1, the quality parameters of the output image are the same as those without slicing. As the fill size decreases from 64 to 0, which is equivalent to p=0 and λ=0, the quality parameters of the output image decrease for all three slice sizes. However, with the same fill size, the larger the slice size, the higher the quality parameters of the output image. Compared to fill size = 64, when the fill size = 0, for a 384x384 slice, the quality decreases by 0.6% (i.e., (0.354-0.352) / 0.354), while the computational load is reduced by approximately 27%. For a slice of size 192 x 192, the quality decreased by 0.8% (i.e. (0.354 - 0.351) / 0.354), and the computational cost was reduced by approximately 44%; for a slice of size 96 x 96, the quality decreased by 2.3% (i.e. (0.354 - 0.346) / 0.354), and the computational cost was reduced by approximately 64%.

[0090] It should be noted that, in Figure 3In the example shown, the quality parameters mentioned above use a common evaluation metric for object detection: mAP@IoU. mAP represents mean average precision; IoU represents the intersection-over-union ratio. mAP@IoU represents the detection precision of the trained model across all classes at a specific IoU. IoU represents the overlap rate, or degree of overlap, between the generated candidate bounding boxes and the original ground truth bounding boxes; ideally, they should overlap completely, with a ratio of 1. Figure 3 In the example shown, the target detection performance index of the neural network model in processing the input image (resolution of 1600x1600) is mAP@IoU=0.5:0.95, that is, the mAP when the IoU is in the range of 0.5 and 0.95.

[0091] Therefore, all three types of slicing can achieve a significant reduction in computational load at the cost of very little quality degradation.

[0092] Let's combine the following... Figure 4 The image processing apparatus according to embodiments of this application will be described. Figure 4 The image processing apparatus shown can be used to execute the various steps of the image processing method of the embodiments of this application. The image processing apparatus can be a computer, server or other device with sufficient computing power to construct a neural network.

[0093] Figure 4 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application. Figure 4 The apparatus 2000 shown includes an acquisition unit 2001 and a processing unit 2002.

[0094] The apparatus 2000 can be used to perform the steps of the image processing method according to the embodiments of this application. For example, the acquisition unit 2001 can be used to perform... Figure 1 In step 101 of the method shown, the processing unit 2002 can be used to execute... Figure 1 Steps 102 to 104 in the method shown. For example, the acquisition unit 2001 can be used to perform... Figure 2 In the method shown, steps 201 and 202, the processing unit 2002 can be used to execute... Figure 2 Steps 203 to 211 in the method shown.

[0095] Figure 4 In the device 2000 shown, the acquisition unit 2001 can be equivalent to Figure 5The communication interface 3003 in the device 3000 shown can be used to obtain the aforementioned resource parameters and model parameters. Alternatively, the acquisition unit 2001 can also be equivalent to... Figure 5 The processor 3002 in the device 3000 shown can retrieve the aforementioned resource parameters and model parameters from the memory 3001 at this time.

[0096] In addition, the above Figure 4 The processing unit 2002 in the device 2000 shown can be equivalent to Figure 5 The processor 3002 in the device 3000 shown.

[0097] Figure 5 This is a schematic diagram of the hardware structure of the image processing apparatus according to an embodiment of this application. Figure 5 The illustrated device 3000 includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. The memory 3001, processor 3002, and communication interface 3003 are interconnected via the bus 3004.

[0098] The memory 3001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program, and when the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 and the communication interface 3003 are used to execute the various steps of the image processing method of the embodiments of this application.

[0099] The processor 3002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the image processing apparatus of this application embodiment, or to execute the various steps of the image processing method of this application embodiment.

[0100] The processor 3002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image processing method in this embodiment can be completed by the integrated logic circuitry in the processor 3002 or by software instructions.

[0101] The processor 3002 described above can also be a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the image processing method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 3001. The processor 3002 reads the information in memory 3001 and, in conjunction with its hardware, completes the functions required by the units included in the image processing apparatus of this application embodiment, or executes the various steps of the image processing method of this application embodiment.

[0102] The communication interface 3003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 3000 and other devices or communication networks. For example, control parameters corresponding to inference results can be sent through the communication interface 3003.

[0103] Bus 3004 may include a pathway for transmitting information between various components of device 3000 (e.g., memory 3001, processor 3002, communication interface 3003).

[0104] It should be noted that although the above-described device 3000 only shows a memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, device 3000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that device 3000 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that device 3000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 5 All the devices shown.

[0105] This application does not impose any particular limitation on the specific structure of the execution subject of the method provided in this application embodiment. As long as it is possible to communicate according to the method provided in this application embodiment by running a program that records the code of the method provided in this application embodiment. For example, the execution subject of the method provided in this application embodiment can be a terminal device or a network device, or a functional module in a terminal device or network device that can call and execute a program.

[0106] Various aspects or features of this application can be implemented as methods, apparatus, or articles of art using standard programming and / or engineering techniques. The term "article of art" as used herein can encompass a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media can include, but is not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.).

[0107] The various storage media described herein may represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable media" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0108] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) can be integrated into the processor.

[0109] It should also be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0110] Those skilled in the art will recognize that the units and steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of protection of this application.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of apparatus or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] In addition, the functional units in the various embodiments of this application can be integrated into one unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0115] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium and includes several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Obtain the resource parameters of the neural network processing system and the model parameters of the first neural network. The resource parameters are used to represent the computing power and storage capacity of the neural network processing system, and the model parameters are used to represent the amount of computing power and storage required by the first neural network. Based on the resource parameters and the model parameters, the first slice size, the first padding size, and the number of stitching layers in the first neural network are determined. The first slice size, the first padding size, and the number of stitching layers in the first neural network are values ​​that minimize the computational or storage requirements of the first neural network while ensuring that the quality of the first output image is within a preset threshold range. The first output image is the output image obtained when the first neural network processes the input image based on the first slice size, the first padding size, and the stitching layers. The method further includes: The input image is divided into multiple patches based on the first slice size and the first fill size; The processing results corresponding to the plurality of image patches are stitched together in the stitching layer, and the first output image is obtained from the stitched image. The processing results corresponding to the plurality of image patches are obtained by the first neural network processing the plurality of image patches.

2. The method as described in claim 1, characterized in that, The size of the first slice is less than or equal to the size of the input image, the number of layers in the stitching layer is less than or equal to the total number of layers in the first neural network, the first padding size is determined based on the preset threshold range and the receptive field of the stitching layer, and the amount of computation or storage required for each layer of the first neural network does not exceed the computational or storage capacity of the neural network processing system.

3. The method as described in claim 1 or 2, characterized in that, The quality of the first output image is within a preset threshold range, specifically: the ratio of the quality of the first output image to the quality of the second output image is within a preset range, and the second output image is the output image obtained when the first neural network directly processes the input image.

4. An image processing apparatus, characterized in that, include: An acquisition unit is used to acquire resource parameters of the neural network processing system and model parameters of the first neural network. The resource parameters are used to represent the computing power and storage capacity of the neural network processing system, and the model parameters are used to represent the amount of computing power and storage required by the first neural network. The processing unit is configured to determine, based on the resource parameters and the model parameters, the first slice size, the first padding size, and the number of stitching layers in the first neural network of the input image of the first neural network, wherein the first slice size, the first padding size, and the number of stitching layers in the first neural network are values ​​that minimize the computational or storage requirements of the first neural network while ensuring that the quality of the first output image is within a preset threshold range, and the first output image is the output image obtained when the first neural network processes the input image based on the first slice size, the first padding size, and the stitching layers; The processing unit is also used for: The input image is divided into multiple patches based on the first slice size and the first fill size; The processing results corresponding to the plurality of image patches are stitched together in the stitching layer. The processing results corresponding to the plurality of image patches are obtained by the first neural network processing the plurality of image patches.

5. The apparatus as described in claim 4, characterized in that, The size of the first slice is less than or equal to the size of the input image, the number of layers in the stitching layer is less than or equal to the total number of layers in the first neural network, the first padding size is determined based on the preset threshold range and the receptive field of the stitching layer, and the amount of computation or storage required for each layer of the first neural network does not exceed the computational or storage capacity of the neural network processing system.

6. The apparatus as described in claim 4 or 5, characterized in that, The quality of the first output image is within a preset threshold range, specifically: the ratio of the quality of the first output image to the quality of the second output image is within a preset range, and the second output image is the output image obtained when the first neural network directly processes the input image.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is for program code executed by the device, the program code including instructions for performing the method as described in any one of claims 1 to 3.

8. A chip, characterized in that, The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the method as described in any one of claims 1 to 3.

Images