Image processing method, device and storage medium

Abstract

The embodiment of the present application provides an image processing method, device and storage medium, the method comprising: inputting a first feature map corresponding to an image to be processed into a residual block, so as to sequentially perform convolution processing on the first feature map using at least one network layer in the residual block; in each network layer, sequentially performing convolution processing on the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size to obtain at least one second feature map, wherein the first size is larger than the second size; and obtaining a processing result of the image to be processed based on the at least one second feature map. Accordingly, in the process of performing convolution on the first feature map using the residual block, a larger receptive field can be obtained, thereby realizing the input of more information, thereby effectively improving the accuracy of image processing.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] At present, image processing technology based on deep learning has been widely used in many fields. Among them, the improved Fully Convolutional Network (FCN) is the most commonly used, such as the U-Net network.

[0003] However, the U-Net network has difficulty in handling complex image processing tasks. Therefore, a solution combining the U-Net network and the residual network ResNet was born. Although this solution enhances the network's recognition of high-level features, the accuracy of image processing is still not high. Summary of the invention

[0004] Multiple aspects of the present application provide an image processing method, device, and storage medium to improve the accuracy of image processing.

[0005] The present application provides an image processing method, including:

[0006] Inputting a first feature map corresponding to the image to be processed into a residual block, so as to sequentially perform convolution processing on the first feature map using at least one network layer in the residual block;

[0007] In each network layer, convolution processing is performed on the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence to obtain at least one second feature map, wherein the first size is larger than the second size;

[0008] Based on the at least one second feature map, a processing result of the image to be processed is obtained.

[0009] The embodiment of the present application also provides a computing device, including a memory and a processor;

[0010] The memory is used to store one or more computer instructions;

[0011] The processor is coupled to the memory and configured to execute the one or more computer instructions for:

[0012] Inputting a first feature map corresponding to the image to be processed into a residual block, so as to sequentially perform convolution processing on the first feature map using at least one network layer in the residual block;

[0013] In each network layer, convolution processing is performed on the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence to obtain at least one second feature map, wherein the first size is larger than the second size;

[0014] Based on the at least one second feature map, a processing result of the image to be processed is obtained.

[0015] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the image processing method according to any one of claims 1 to 10.

[0016] In an embodiment of the present application, based on an improved residual block, a convolution process is performed on the first feature map corresponding to the image to be processed. In each network layer of the residual block, convolution processes are performed using convolution kernels of a first size and a second size in sequence, and the first size is larger than the second size. Accordingly, in the process of convolving the first feature map using the residual block, a larger receptive field can be obtained, thereby realizing the input of more information, thereby effectively improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A flowchart of an image processing method provided in one embodiment of the present application;

[0019] Figure 2 A schematic diagram of the structure of a residual block provided in one embodiment of the present application;

[0020] Figure 3 A schematic diagram of the structure of an image processing system provided in one embodiment of the present application;

[0021] Figure 4 A schematic diagram of the structure of a computing device provided in accordance with another embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0023] At present, when a solution combining a U-Net network and a residual network ResNet is used for image processing, although the network's recognition of high-level features is enhanced, the accuracy of image processing is not high. In order to improve the problems existing in the prior art, in some embodiments of the present application: based on an improved residual block, a first feature map corresponding to the image to be processed is convolved. In each network layer of the residual block, convolution processing is performed using convolution kernels of a first size and a second size in turn, and the first size is larger than the second size. Accordingly, in the process of convolving the first feature map using the residual block, a larger receptive field can be obtained, thereby realizing the input of more information, thereby effectively improving the accuracy of image processing.

[0024] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0025] Figure 1 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0026] 100. Inputting a first feature map corresponding to the image to be processed into a residual block, so as to sequentially perform convolution processing on the first feature map using at least one network layer in the residual block;

[0027] 102. In each network layer, sequentially use a convolution kernel of a first size and a convolution kernel of a second size to perform convolution processing on the input of the network layer to obtain at least one second feature map, wherein the first size is larger than the second size;

[0028] 103. Obtain a processing result of the image to be processed based on at least one second feature map.

[0029] The image processing method provided in the embodiment of the present application can be applied to various image processing scenarios, for example, image matching, image classification, face feature extraction, face recognition, target tracking, automatic driving or image segmentation, etc., which is not limited in this embodiment. In this embodiment, the residual block is a component unit of the residual network. In this embodiment, the residual network can be used alone to process the image to be processed to obtain a processing result. Of course, the residual network can also be combined with other neural networks to form an image processing system, so that the image processing system is used to process the image to be processed to obtain a processing result. For example, the image processing system can be the U-Net network mentioned above and the residual network ResNet can be combined into a UR network. This embodiment does not limit the architecture of the image processing system to which the residual block belongs.

[0030] The technical solution of the present application will be described below from the dimension of the residual block.

[0031] In step 100, the first feature map corresponding to the image to be processed may be input into the residual block. In different application scenarios, the source of the first feature map may not be exactly the same. For example, the first feature map may be a feature map output by the previous residual block, or it may be the image to be processed itself, or it may be a feature map output after other processing is performed on the image to be processed, and this embodiment does not limit this. Accordingly, in this embodiment, the first feature map may be the image to be processed itself, or it may be a general term for a group of feature maps generated during the image processing process.

[0032] In addition, in this embodiment, the number of channels of the first feature map may be one or more. The number of channels is used to characterize the number of channels of a picture (such as R, G, B channels) or to characterize the number of feature maps. In this article, for the convenience of description, the number of feature maps and the number of channels of the picture are both described as the number of channels, but it should be understood that the meanings represented by the channels are different in different situations.

[0033] Figure 2 A schematic diagram of the structure of a residual block provided in an embodiment of the present application. Figure 2 As shown, in this embodiment, the residual block includes at least one network layer 10, each network layer 10 is connected in series in sequence, and the output of the previous network layer is used as the input of the subsequent network layer. Based on this, at least one network layer 10 in the residual block can be used to perform convolution processing on the first feature map 11 in sequence. That is, after the first network layer in the residual block performs convolution processing on the first feature map, the convolution processing result is used as the input of the second network layer, so as to continue the convolution processing using the second network layer, and so on, to achieve the convolution processing of the first feature map by at least one network layer in the residual block.

[0034] Accordingly, in this embodiment, the input of the first network layer in the residual block is the first feature map, and the input of other network layers is the output of the previous network layer.

[0035] It is worth mentioning that Figure 2 The residual block shown in FIG. 1 includes three network layers 10, but the number of network layers 10 in the residual block is not limited in this embodiment. According to actual needs, the number of network layers 10 in the residual block can be 2, 3 or other numbers. Moreover, when the aforementioned image processing system includes multiple residual blocks, the number of network layers in different residual blocks may not be exactly the same. Figure 2 In addition to the network layers shown in , the residual block may also include a pooling layer, an output layer, an activation function, and other components, which will not be described in detail here.

[0036] In step 101, the processing process inside the residual block is described from the dimension of the network layer. In each network layer, the input of the network layer is convolved using a convolution kernel of a first size and a convolution kernel of a second size in turn. The first size is larger than the second size.

[0037] In this embodiment, the size of the convolution kernel is used to determine the receptive field of the convolution processing. First, the input of the network layer is convolved using a convolution kernel of a first size to obtain a larger receptive field, so that more information can be obtained from the input of the network layer. After that, the convolution result of the convolution kernel of the first size is further convolved with a convolution kernel of a smaller size.

[0038] For example, if the first size is 3*3 and the second size is 1*1, the input information corresponding to the 3*3 receptive field can be obtained from the input of the network layer based on the 3*3 convolution kernel, and then the convolution is continued with the 1*1 receptive field. However, the residual block in the prior art needs to first obtain the input information from the input of the network layer with a 1*1 receptive field, which directly causes a large amount of input information loss.

[0039] Accordingly, the scheme of sequentially using convolution kernels of the first size and the second size to perform convolution processing on the input of the network layer has a larger receptive field during the convolution processing compared to the residual block processing scheme in the prior art. Therefore, more input information can be obtained, which can effectively improve the processing accuracy of the residual block.

[0040] Accordingly, each network layer in the residual block can output a second feature map. Among them, the parameters such as the size and number of channels of the second feature maps output by different network layers may not be completely the same, and this embodiment does not limit this.

[0041] In step 102, a processing result of the image to be processed may be obtained based on the second feature map output by each network layer in the residual block and the first feature map input to the residual block.

[0042] Specifically, the second feature map output by each network layer in the residual block can be subjected to residual processing with the first feature map of the input residual block to obtain the output of the residual block, and the output of the residual block can participate in other processing processes of the image to be processed, thereby finally obtaining the processing result of the image to be processed. The specific details will be described later in conjunction with the architecture of the image processing system.

[0043] In this embodiment, based on an improved residual block, the first feature map corresponding to the image to be processed is convolved. In each network layer of the residual block, convolution processing is performed using convolution kernels of a first size and a second size in turn, and the first size is larger than the second size. Accordingly, in the process of convolving the first feature map using the residual block, a larger receptive field can be obtained, thereby realizing the input of more information, thereby effectively improving the accuracy of image processing. In addition, based on the residual block in this embodiment, the quantization of the neural network can also be realized, the neural network can be compressed and accelerated, and the performance loss of various technical indicators caused by the quantization process can be reduced.

[0044] In the above or following embodiments, in each network layer of the residual block, a convolution kernel of a first size can be used to convolve the input of the network layer to obtain an intermediate feature map, and the number of channels of the intermediate feature map is consistent with the number of channels of the input of the network layer; the intermediate feature map is convolved using a convolution kernel of a second size to obtain a second feature map, wherein the number of channels of the convolution kernel of the second size is less than or equal to the number of channels of the input of the network layer.

[0045] The following will take one of the network layers as an example to provide a detailed description of the technical solution.

[0046] For the network layer, the number of convolution kernels of the first size can be configured to be consistent with the number of channels of the input of the network layer, so that the number of channels of the intermediate feature map obtained based on the network layer is consistent with the number of channels of the input of the network layer.

[0047] In one implementation, if the number of input channels of the network layer is N, the i-th convolution kernel of the first size can be convolved with each channel of the input of the network layer, and the convolution processing results obtained for each channel can be element-wise superimposed to obtain the data of the i-th channel of the intermediate feature map.

[0048] In this implementation, the number of channels of the convolution kernel of the first size may be consistent with the number of channels of the input of the network layer, or may be inconsistent, which is not limited here.

[0049] In another implementation, if the number of input channels of the network layer is N, the i-th convolution kernel of the first size can be convolved with the i-th input of the network layer to obtain the data of the i-th channel of the intermediate feature map.

[0050] In this implementation, preferably, the number of channels of the convolution kernel of the first size is 1. Of course, this is not limited to this.

[0051] Compared with the above two implementations, the latter implementation uses fewer parameters and has higher image processing efficiency. In the above two embodiments, N and i are positive integers.

[0052] Of course, the above two implementation methods are only exemplary. In this embodiment, the method of achieving the consistency between the number of channels of the intermediate feature map obtained based on the network layer and the number of channels of the input of the network layer is not limited to this.

[0053] Based on the intermediate feature map obtained by the convolution kernel of the first size, in this embodiment, the number of output channels of the network layer can continue to be adjusted by using the convolution kernel of the second size.

[0054] Optionally, the second size may be 1*1. Accordingly, the number of channels of the output of the network layer may be adjusted by using the 1*1 convolution kernel by configuring the number of 1*1 convolution kernels as needed. Of course, in this embodiment, the second size is not limited thereto.

[0055] The number of convolution kernels of the second size can be configured to be less than or equal to the number of input channels of the network layer, thereby reducing or maintaining the number of output channels of the network layer.

[0056] At this point, the second feature map corresponding to the network layer can be obtained.

[0057] In this embodiment, based on the different numbers of convolution kernels of the second size configured for different network layers, the number of channels output by different network layers may not be exactly the same. In some practical applications, the number of channels output by the network layer may be reduced in sequence to reduce the amount of calculation of the residual block.

[0058] For example, Figure 2 As shown, if the number of channels of the first feature map 11 is 192, the number of channels of the output of the first network layer of the residual block can be 96, the number of channels of the output of the second network layer can be 48, and the number of channels of the output of the third network layer can remain at 48. Of course, this is only exemplary, and the number of channels of the output of each network layer can be adjusted according to actual needs.

[0059] In this embodiment, by reducing the number of channels of the output of each network layer in the residual block, the number of parameters can be reduced in the process of the residual block processing the first feature map, thereby reducing the amount of calculation and improving processing efficiency. Therefore, although compared with the residual block in the prior art, the technical means of obtaining more input information by expanding the receptive field in the aforementioned embodiment increases some calculation amount, the technical means of reducing the number of channels within the residual block proposed in this embodiment can compensate for the increase in calculation amount caused by expanding the receptive field. That is, in this embodiment, a larger receptive field can be obtained in the convolution process while maintaining the amount of calculation equivalent to that of the existing residual block, thereby obtaining more input information to improve the accuracy of image processing.

[0060] Furthermore, in this embodiment, the second feature maps output by each network layer can be combined to obtain a post-convolution feature map; a third feature map is generated based on the post-convolution feature map and the first feature map, so as to obtain a processing result of the image to be processed based on the third feature map.

[0061] For example, Figure 2 As shown, in this embodiment, the outputs of the first network layer, the second network layer, and the third network layer of the residual block can be channel-superimposed to obtain a post-convolution feature map. Figure 2 In the figure, the number of channels of the output of the first network layer is 96, the number of channels of the output of the second network layer is 48, and the number of channels of the output of the third network layer is 48. The number of channels of the convolutional feature map obtained after channel superposition is: 96+48+48=192.

[0062] In the above example, the number of channels of the feature map after convolution is consistent with the number of channels of the first feature map. However, it should be understood that this embodiment is not limited to this, and the number of channels of the feature map after convolution may also be inconsistent with the number of channels of the first feature map. Of course, for the convenience of calculation, when the number of channels of the feature map after convolution is inconsistent with the number of channels of the first feature map, the channel filling of the feature map after convolution may be performed to make the number of channels of the feature map after convolution consistent with the number of channels of the first feature map.

[0063] In addition, as mentioned above, the sizes of the second feature maps output by different network layers may not be exactly the same. Based on this, the sizes of feature maps under different channels in the convolutional feature map may not be exactly the same. In this embodiment, the sizes of feature maps under different channels in the convolutional feature map may also be padded, such as zero padding, so that the sizes of feature maps under each channel in the convolutional feature map are consistent with the size of the first feature map.

[0064] The above-mentioned processing of channel filling and size filling is to facilitate the subsequent calculation process between the feature map after convolution and the first feature map, but it should be understood that when different calculation methods are adopted, the above-mentioned single or multiple processing may not be necessary. Therefore, the above-mentioned two processings are optional and can be selected according to actual needs.

[0065] In some practical applications, such as Figure 2 As shown, the convolution feature map and the first feature map can be element-wise superimposed to obtain a third feature map 12. Of course, in this implementation, the combination of the convolution feature map and the first feature map is not limited to this implementation. Other implementations can also be used to combine the convolution feature map and the first feature map to correct the output of the residual block through the first feature map.

[0066] At this point, the output of the residual block can be obtained: the third feature map. The third feature map can be used in the subsequent processing of the image to be processed.

[0067] Based on the different architectures of the image processing system to which the residual block belongs, the destination of the third feature map may not be exactly the same. For example, if the residual block is used in the UR network mentioned above, the third feature map may be used as the input of the next residual block.

[0068] In the above or below embodiments, on one hand, the image to be processed may be binarized to obtain a binarized image.

[0069] As mentioned above, the first feature map may be the image to be processed itself, or may be a feature map generated after the image to be processed is convolved, etc. Based on the binarization process performed on the image to be processed, the first feature map will also be binarized.

[0070] The specific implementation method of the binarization processing is not limited in this embodiment. For example, a standard value may be preset, and the elements in each channel of the image to be processed may be compared with the standard value, and the elements greater than the standard value may be converted to 1, and the elements less than or equal to the standard value may be converted to -1, thereby realizing the binarization of the image to be processed.

[0071] Accordingly, the first feature map can be binarized based on the binarization of the image to be processed. Of course, in this embodiment, when the first feature map is obtained by performing convolution processing on the image to be processed, the first feature map can also be directly binarized, and the binarized first feature map is used as the input of the residual block. This embodiment does not limit this.

[0072] Compared with the floating point number method used for the input of the residual block in the prior art, the binarized first feature map, as the input of the residual block, can greatly reduce the calculation amount of the residual block and improve the inference speed.

[0073] On the other hand, the convolution kernel of the first size and the convolution kernel of the second size may also be binarized. In each network layer of the residual block, convolution processing may be performed based on the binarized convolution kernel. The specific binarization process may also refer to the implementation method of the binarization processing provided above, which will not be repeated here.

[0074] Compared with the floating point method used in the convolution kernel in the residual block in the prior art, the binarized convolution kernel can save a lot of storage resources. Taking the binarization in the form of 1 and -1 as an example, the existing 32-bit floating point number itself has a sign bit, and the binarization result in this embodiment also has a sign bit. Therefore, even considering the existence of the sign bit, in this embodiment, the binarized convolution kernel can save 31 / 32 of the storage resources compared with the existing 32-bit floating point convolution kernel. And because the convolution kernel usually needs to be stored in memory, the binarization process can greatly reduce the memory usage.

[0075] In addition, binarization processing can reduce the computational complexity of the convolution process. Based on this, when the amount of data becomes smaller, the memory can read more data at a time. Therefore, binarization processing can significantly improve the efficiency of image processing.

[0076] It is worth noting that the above binarization process of the first feature map and the binarization process of the convolution kernel can be performed separately or in combination. When the two are performed in combination, a better effect of saving computing resources can be obtained.

[0077] When both the first feature map and the convolution kernel are binarized, the convolution operation can be performed by bit operation, which can greatly improve the inference speed of the residual block.

[0078] In this embodiment, the binarization processing method of the convolution kernel can also be extended to the training process of the residual block, that is, when the residual block is trained, the convolution kernel can be binarized during the forward propagation process, and during the reverse propagation process, the floating point method can be maintained without binarization of the convolution kernel, so as to ensure the accuracy of the derivative calculation during the reverse propagation process. Of course, this embodiment is not limited to this.

[0079] In this embodiment, by binarizing the convolution kernel involved in the first feature map and the residual block, the amount of calculation of the residual block can be greatly reduced, thereby effectively saving computing resources. The loss of precision caused by the binarization process can be compensated by the technical means of expanding the receptive field described in the above embodiments, and adjusting the number of channels output by each network layer can further reduce the amount of calculation. Therefore, in general, the image processing method provided in this embodiment can significantly improve the processing speed and save computing resources while maintaining good neural network performance.

[0080] In the above or following embodiments, if the image processing system to which the residual block belongs adopts a U-Net network, the image processing system will include a downsampling process and an upsampling process.

[0081] Figure 3 FIG. 1 is a schematic diagram of the structure of an image processing system provided in one embodiment of the present application. Figure 3 As shown, the image processing system adopts the U-Net network architecture, and uses residual blocks to implement upsampling (down_con) and downsampling (up_conv) under the U-net network architecture.

[0082] In this embodiment, if the residual block is used to implement upsampling in an image processing system, in addition to inputting the aforementioned first feature map into the residual block, it is also possible to extract robust features corresponding to the size of the first feature map from the image to be processed based on the variational autoencoder, and use the extracted robust features as input to the residual block to perform process sampling correction in the process of sequentially performing convolution processing on the first feature map using at least one network layer in the residual block.

[0083] Of course, in this embodiment, the residual block can also be used to implement downsampling in the image processing system, which is not limited in this embodiment.

[0084] like Figure 3 As shown in the figure, a variational autoencoder is added to the image processing system. The variational autoencoder can extract robust features of four sizes from the image to be processed, and provide them to four upsampling operations respectively. Among them, the variational autoencoder can extract robust features of different sizes from the image to be processed based on different sizes. Robust features refer to those universal, stable or core features at different sizes. Each upsampling operation can perform sampling correction based on the robust features of its corresponding size, thereby strengthening the robust features in the image to be processed during the upsampling process and realizing the transmission of more effective information.

[0085] Accordingly, robust features of different sizes extracted from the image to be processed based on the variational autoencoder are directly used in the upsampling process, and there is no need to calculate the conduction features based on the upsampling process, thereby saving computing resources and time. At the same time, during the upsampling process, the effective features in the image to be processed can be enhanced, and the transmission of more effective information can be achieved, which effectively alleviates the sensitivity of the upsampling process to objects of different sizes and improves the robustness of image processing. Among them, the conduction features in the prior art contain many redundant features, and the effective features in the image to be processed are not transmitted, which will greatly reduce the accuracy of image processing. The variational autoencoder introduced in this embodiment can effectively improve this problem.

[0086] In this embodiment, before using the variational autoencoder, the variational autoencoder may be trained. In practical applications, different image databases may be used as training samples of the variational autoencoder, and the parameters of the variational autoencoder may be reversely corrected according to the output results of the variational autoencoder. The correction process may be aimed at maximizing the mean of the accuracy of the output results of different image databases. Of course, this embodiment is not limited to this. Among them, the properties such as content and size of the images in different image databases may not be exactly the same.

[0087] Based on this, a variational autoencoder that can generate highly robust features can be trained. Therefore, when the variational autoencoder is applied to the image processing process, robust features suitable for different sizes can be generated, avoiding the problem of insufficient algorithm robustness due to the diversity of sampling sizes. For example, when the image processing system can be used for image segmentation and image recognition, based on the variational autoencoder, it can effectively deal with the situation where the objects to be segmented have more deformations, reduce the calculation pressure of subsequent image recognition, and improve robustness.

[0088] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 100, 101, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different sizes, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0089] Figure 4 A schematic diagram of a computing device provided in another embodiment of the present application is shown in FIG. Figure 4 As shown, the computing device includes: a memory 40 and a processor 41 .

[0090] The memory 40 is used to store computer programs and can be configured to store various other data to support operations on the computing device. Examples of such data include instructions, messages, pictures, videos, etc. for any application or method operating on the computing device.

[0091] The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0092] The processor 41 is coupled to the memory 40 and is used to execute the computer program in the memory 40 to:

[0093] Inputting a first feature map corresponding to the image to be processed into a residual block, so as to sequentially perform convolution processing on the first feature map using at least one network layer in the residual block;

[0094] In each network layer, convolution processing is performed on the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence to obtain at least one second feature map, wherein the first size is larger than the second size;

[0095] Based on the at least one second feature map, a processing result of the image to be processed is obtained.

[0096] In an optional embodiment, when the processor 41 sequentially uses the convolution kernel of the first size and the convolution kernel of the second size to perform convolution processing on the input of the network layer to obtain at least one second feature map, it is used to:

[0097] The input of the network layer is convolved with a convolution kernel of the first size to obtain an intermediate feature map, where the number of channels of the intermediate feature map is consistent with the number of channels of the input of the network layer;

[0098] The intermediate feature map is convolved with a convolution kernel of a second size to obtain a second feature map, wherein the number of convolution kernels of the second size is less than or equal to the number of channels of the input of the network layer.

[0099] In an optional embodiment, if the number of channels of the input of the network layer is N, the processor 41, when performing convolution on the input of the network layer using the convolution kernel of the first size, is used to:

[0100] Convolve the i-th convolution kernel of the first size with the i-th input of the network layer to obtain the data of the i-th channel of the intermediate feature map, where i = 1, 2…N.

[0101] In an optional embodiment, the second size is 1*1.

[0102] In an optional embodiment, when the processor 41 obtains the processing result of the image to be processed based on at least one second feature map, it is configured to:

[0103] Channel-superimpose at least one second feature map to obtain a post-convolution feature map;

[0104] Superimposing the element-wise product of the convolution feature map and the first feature map to obtain a third feature map;

[0105] Based on the third feature map, a processing result of the image to be processed is obtained.

[0106] In an optional embodiment, the number of channels of the feature map after convolution is consistent with the number of channels of the first feature map.

[0107] In an optional embodiment, if the image processing system to which the residual block belongs adopts a U-Net network, and the residual block is used in an upsampling process of the U-Net network, the processor 41 is further used to:

[0108] Based on the variational autoencoder, a robust feature corresponding to the size of the first feature map is extracted from the image to be processed;

[0109] According to the robust features, sampling correction is performed on the operation of sequentially performing convolution processing on the first feature map using at least one network layer in the residual block.

[0110] In an optional embodiment, before inputting the first feature map corresponding to the image to be processed into the residual block, the processor 41 is further configured to:

[0111] Perform binarization processing on the image to be processed to obtain a binarized image;

[0112] The binary image is convolved to obtain the first feature map.

[0113] In an optional embodiment, before sequentially using the convolution kernel of the first size and the convolution kernel of the second size to perform convolution processing on the input of the network layer, the processor 41 is further configured to:

[0114] The convolution kernel of the first size and the convolution kernel of the second size are binarized.

[0115] In an optional embodiment, when the processor 41 sequentially uses the convolution kernel of the first size and the convolution kernel of the second size to perform convolution processing on the input of the network layer, it is used to:

[0116] Based on bit operations, the input of the network layer is convolved using the convolution kernel of the first size and the convolution kernel of the second size in sequence.

[0117] It is worth noting that for the technical details in the above-mentioned embodiments of the computing device, reference can be made to the relevant descriptions in the above-mentioned embodiments of the image processing method. In order to save space, they will not be repeated here, but this should not cause loss of the scope of protection of the present application.

[0118] Further, if Figure 4 As shown, the computing device also includes: a communication component 42, a power supply component 43 and other components. Figure 4 Only some components are shown schematically, and it does not mean that the computing device only includes Figure 4 Components shown.

[0119] Among them, the communication component 42 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology or other technologies to facilitate short-range communication.

[0120] The power supply component 43 provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0121] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a computing device in the above method embodiment.

[0122] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0128] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0130] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: include: Inputting a first feature map corresponding to the image to be processed into a residual block, so as to sequentially perform convolution processing on the first feature map using at least one network layer in the residual block, so as to implement a downsampling process of an image processing system to which the residual block belongs, wherein the image processing system includes an upsampling process and a downsampling process; In each network layer, convolution processing is performed on the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence, wherein the first size is larger than the second size; Extracting robust features corresponding to the size of the first feature map from the image to be processed based on a variational autoencoder; The robust feature is applied to the upsampling process of the image processing system to perform sampling correction on the operation of sequentially performing convolution processing on the first feature map using at least one network layer in the residual block, so as to obtain a processing result of the image to be processed.

2. The method according to claim 1, characterized in that The step of sequentially using a convolution kernel of a first size and a convolution kernel of a second size to perform convolution processing on the input of the network layer includes: Convolving the input of the network layer using the convolution kernel of the first size to obtain an intermediate feature map, wherein the number of channels of the intermediate feature map is consistent with the number of channels of the input of the network layer; The intermediate feature map is convolved using a convolution kernel of the second size, wherein the number of convolution kernels of the second size is less than or equal to the number of input channels of the network layer.

3. The method according to claim 2, characterized in that If the number of channels of the input of the network layer is N, performing convolution on the input of the network layer using the convolution kernel of the first size includes: Convolve the i-th convolution kernel of the first size with the i-th input of the network layer to obtain data of the i-th channel of the intermediate feature map, where i=1,2…N.

4. The method according to claim 2, characterized in that: The second size is 1*1.

5. The method according to claim 2, characterized in that: The downsampling process further includes: Convolutionally processing the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence to obtain at least one second feature map; Perform channel superposition on the at least one second feature map to obtain a post-convolution feature map; The convolution feature map and the first feature map are element-wise superimposed to obtain a convolution result of the downsampling process.

6. The method according to claim 5, characterized in that The number of channels of the convolved feature map is consistent with the number of channels of the first feature map.

7. The method according to claim 1, characterized in that Before the first feature map corresponding to the image to be processed is input into the residual block, the method further includes: Performing binarization processing on the image to be processed to obtain a binarized image; Perform convolution processing on the binary image to obtain the first feature map.

8. The method according to claim 1 or 7, characterized in that: Before sequentially using the convolution kernel of the first size and the convolution kernel of the second size to perform convolution processing on the input of the network layer, the method further includes: Binarization is performed on the convolution kernel of the first size and the convolution kernel of the second size.

9. The method according to claim 8, characterized in that The step of sequentially using a convolution kernel of a first size and a convolution kernel of a second size to perform convolution processing on the input of the network layer includes: Based on bit operations, convolution processing is performed on the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence.

10. A computing device, characterized in that: including memory and processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and configured to execute the one or more computer instructions for: Inputting a first feature map corresponding to the image to be processed into a residual block, so as to sequentially perform convolution processing on the first feature map using at least one network layer in the residual block, so as to implement a downsampling process of an image processing system to which the residual block belongs, wherein the image processing system includes an upsampling process and a downsampling process; In each network layer, convolution processing is performed on the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence, wherein the first size is larger than the second size; Extracting robust features corresponding to the size of the first feature map from the image to be processed based on a variational autoencoder; The robust feature is applied to the upsampling process of the image processing system to perform sampling correction on the operation of sequentially performing convolution processing on the first feature map using at least one network layer in the residual block, so as to obtain a processing result of the image to be processed.

11. The computing device according to claim 10, characterized in that When the processor sequentially uses a convolution kernel of a first size and a convolution kernel of a second size to perform convolution processing on the input of the network layer, it is used to: Convolving the input of the network layer using the convolution kernel of the first size to obtain an intermediate feature map, wherein the number of channels of the intermediate feature map is consistent with the number of channels of the input of the network layer; The intermediate feature map is convolved using a convolution kernel of the second size, wherein the number of convolution kernels of the second size is less than or equal to the number of input channels of the network layer.

12. The computing device according to claim 11, characterized in that If the number of channels of the input of the network layer is N, the processor, when performing convolution on the input of the network layer using the convolution kernel of the first size, is used to: Convolve the i-th convolution kernel of the first size with the i-th input of the network layer to obtain data of the i-th channel of the intermediate feature map, where i=1,2…N.

13. The computing device according to claim 11, characterized in that The second size is 1*1.

14. The computing device according to claim 11, characterized in that During the downsampling process, the processor is further used for: Convolutionally processing the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence to obtain at least one second feature map; Perform channel superposition on the at least one second feature map to obtain a post-convolution feature map; The convolution feature map and the first feature map are element-wise superimposed to obtain a convolution result of the downsampling process.

15. The computing device according to claim 14, characterized in that The number of channels of the convolved feature map is consistent with the number of channels of the first feature map.

16. The computing device according to claim 10, characterized in that Before inputting the first feature map corresponding to the image to be processed into the residual block, the processor is further used to: Performing binarization processing on the image to be processed to obtain a binarized image; Perform convolution processing on the binary image to obtain the first feature map.

17. The computing device according to claim 10 or 16, characterized in that Before sequentially using a convolution kernel of a first size and a convolution kernel of a second size to perform convolution processing on the input of the network layer, the processor is further used to: Binarization is performed on the convolution kernel of the first size and the convolution kernel of the second size.

18. The computing device according to claim 17, characterized in that When the processor sequentially uses a convolution kernel of a first size and a convolution kernel of a second size to perform convolution processing on the input of the network layer, it is used to: Based on bit operations, convolution processing is performed on the input of the network layer using a convolution kernel of a first size and a convolution kernel of a second size in sequence.

19. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the image processing method according to any one of claims 1 to 9.

Images