Image processing method and device, chip and storage medium

Abstract

The invention discloses an image processing method and device, a chip and a storage medium, the method is applied to a mobile terminal device, and the method comprises the steps: obtaining a to-be-processed image and at least two image blocks of the to-be-processed image; performing convolution processing on the to-be-processed image to obtain an initial feature map; calculating target correlation of the at least two image blocks, and performing weighting processing on the initial feature map according to the target correlation to obtain a weighted feature map; and performing noise reduction processing on the initial feature map and the weighted feature map to obtain a target image. Therefore, the real-time performance of the target image obtained after noise reduction processing can be improved; and the target image obtained after noise reduction processing can keep more image detail textures, so that the noise reduction effect is enhanced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, chip and storage medium. Background Art

[0002] In the process of image acquisition and transmission, image denoising is an essential part of image processing. Image denoising models in related technologies can be divided into two types: one is a large convolutional neural network (CNN) model deployed on cloud devices, which has low real-time image denoising performance; the other is a lightweight CNN model deployed on mobile devices, which loses more image detail textures during the image denoising process. Therefore, the denoising effects of these two CNN models are still poor. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, chip and storage medium, which can not only improve the real-time performance of the target image obtained after noise reduction processing, but also enable the target image obtained after noise reduction processing to retain more image detail texture, thereby enhancing the noise reduction effect.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to a mobile terminal device, and the method includes:

[0005] Acquire an image to be processed and at least two image blocks of the image to be processed;

[0006] Perform convolution processing on the image to be processed to obtain the initial feature map;

[0007] Calculating target correlation of at least two image blocks, and performing weighted processing on the initial feature map according to the target correlation to obtain a weighted feature map;

[0008] The initial feature map and the weighted feature map are denoised to obtain the target image.

[0009] In a second aspect, an embodiment of the present application provides a mobile terminal device, the mobile terminal device includes an acquisition unit, a processing unit and a calculation unit; wherein:

[0010] An acquisition unit, configured to acquire an image to be processed and at least two image blocks of the image to be processed;

[0011] A processing unit configured to perform convolution processing on the image to be processed to obtain an initial feature map;

[0012] A calculation unit configured to calculate a target correlation between at least two image blocks;

[0013] The processing unit is further configured to perform weighted processing on the initial feature map according to the target correlation to obtain a weighted feature map; and perform noise reduction processing on the initial feature map and the weighted feature map to obtain a target image.

[0014] In a third aspect, an embodiment of the present application provides a mobile terminal device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the image processing method described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a chip for implementing the image processing method described in the first aspect above.

[0016] Specifically, the chip includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the image processing method described in the first aspect above.

[0017] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the image processing method described in the first aspect above.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which enable a computer to execute the image processing method described in the first aspect above.

[0019] In a seventh aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the image processing method described in the first aspect above.

[0020] The embodiments of the present application provide an image processing method, device, chip and storage medium, which can be applied to mobile devices. First, an image to be processed and at least two image blocks of the image to be processed can be obtained; secondly, a convolution process is performed on the image to be processed to obtain an initial feature map; then, the target correlation of at least two image blocks is calculated, and the initial feature map is weighted according to the target correlation to obtain a weighted feature map; finally, the initial feature map and the weighted feature map are subjected to denoising to obtain a target image. In this way, on the one hand, when the image processing method is applied to a mobile device, the real-time performance of the target image obtained after the denoising process can be improved; on the other hand, according to the initial feature map obtained by convolution processing of the image to be processed, and the weighted feature map obtained by weighting the initial feature map according to the target correlation, the denoising process can be performed, so that the obtained target image retains more image detail textures and enhances the denoising effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1It is a flowchart of an image processing method provided in an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a scene of image segmentation to be processed provided by an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of a framework for noise reduction using a noise reduction model provided in an embodiment of the present application;

[0024] Figure 4 is a flowchart of another image processing method provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of a framework of a lightweight Transformer algorithm provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of a framework for determining an initial feature map and a weighted feature map provided in an embodiment of the present application;

[0027] Figure 7 It is a schematic diagram of the composition structure of a mobile terminal device provided in an embodiment of the present application;

[0028] Figure 8 is a schematic structural diagram of a mobile terminal device provided in an embodiment of the present application;

[0029] Fig. 9 It is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0034] In the embodiments of the present application, the term "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, object A and / or object B may represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0035] In the embodiment of the present application, the term "at least two" may refer to two or more than two. For example, at least two image blocks may refer to two or more than two image blocks.

[0036] In the process of image acquisition and transmission, different types of noise will be introduced due to various factors. Noise not only affects the visual effect of the image, but also brings difficulties to the subsequent image analysis and processing. Therefore, reducing the noise in the image is an essential part of the image processing process.

[0037] In the related art, most artificial intelligence (AI) denoising algorithms are traditional convolution algorithms, and usually an image denoising model based on a convolution algorithm can be used to denoise an image. Image denoising models based on a convolution algorithm can be divided into the following two types:

[0038] One is a large CNN model deployed on cloud devices, which obtains data and returns calculation results through mobile communications, which results in low real-time performance of image denoising.

[0039] The other is a lightweight CNN model deployed on mobile devices. The lightweight CNN model on mobile devices often has a simple structure and will lose more image detail texture during the image denoising process. The reconstruction of image details is worse than that of large CNN models.

[0040] Therefore, the denoising effects of the large CNN model deployed on cloud devices and the lightweight CNN model deployed on mobile devices are still poor.

[0041] In addition, AI denoising algorithms can also be Transformer algorithms. The concept of the Transformer algorithm and related applications have brought the capabilities of traditional CNN models to a new level. However, most Transformer algorithms in related technologies are not lightweight. They are usually deployed on cloud devices due to their large size and are rarely seen on mobile devices, resulting in low real-time performance of image denoising.

[0042] Based on this, an embodiment of the present application provides an image processing method that can be applied to a mobile terminal device. First, an image to be processed and at least two image blocks of the image to be processed can be obtained; secondly, a convolution process is performed on the image to be processed to obtain an initial feature map; then, the target correlation of at least two image blocks is calculated, and the initial feature map is weighted according to the target correlation to obtain a weighted feature map; finally, the initial feature map and the weighted feature map are subjected to denoising to obtain a target image. In this way, on the one hand, when the image processing method is applied to a mobile terminal device, the real-time performance of the target image obtained after the denoising process can be improved; on the other hand, according to the initial feature map obtained by convolution processing of the image to be processed, and the weighted feature map obtained by weighting the initial feature map according to the target correlation, the denoising process can be performed, so that the obtained target image retains more image detail textures and enhances the denoising effect.

[0043] It should be noted that the image processing method provided in the embodiment of the present application can be used to improve the Transformer algorithm in the related technology, so that the improved Transformer algorithm can be deployed on mobile devices, and the image denoising by the improved Transformer algorithm can retain more image detail texture.

[0044] Figure 1 1 is a flow chart of an image processing method provided in an embodiment of the present application, which is applied to a mobile terminal device. Figure 1 As shown, the method may include the following steps.

[0045] S110: Acquire an image to be processed and at least two image blocks of the image to be processed.

[0046] It should be noted that in the embodiments of the present application, the image processing method can be applied to mobile devices. The mobile devices can be implemented in various forms, for example, the mobile devices can include such as smart phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, digital cameras, etc., which are not limited here.

[0047] It should also be noted that the image to be processed can be obtained by splicing at least two image blocks.

[0048] Considering that it takes a long time to directly process the image to be processed in blocks, in some embodiments, obtaining at least two image blocks may include: processing the image to be processed in blocks according to a graphics processing unit (GPU) core to obtain at least two image blocks.

[0049] It should be noted that the process of performing block processing on the image to be processed according to the GPU core can be implemented using an open computing language (Open Computing Language, OpenCL).

[0050] Among them, OpenCL is an open cross-platform parallel computing standard, which can be composed of accelerators such as central processing unit (CPU), GPU and digital signal processing (DSP).

[0051] It should also be noted that a GPU is generally composed of a large number of small processing cores (i.e., GPU cores) and has powerful computing capabilities.

[0052] Through this method, the GPU core is used to process the image in blocks, which can, on the one hand, achieve miniaturization of the network scale, and on the other hand, reduce the time consumption of image block processing.

[0053] Further, in some embodiments, the GPU core may include at least two sub-GPU cores, and performing block processing on the image to be processed according to the GPU core may include: determining a starting position when at least two sub-GPU cores each read the image to be processed; and using at least two sub-GPU cores concurrently to perform block processing on the image to be processed according to the starting position when at least two sub-GPU cores each read the image to be processed.

[0054] It should be noted that the number of at least two sub-GPU cores can be equal to the number of at least two image blocks. One sub-GPU core can read a starting position in the image to be processed as the starting point of an image block, and based on the starting point of the image block, divide the image to be processed into one image block; therefore, at least two sub-GPU cores can read at least two starting positions in the image to be processed as the starting points of at least two image blocks, and based on the starting points of the at least two image blocks, divide the image to be processed into at least two image blocks.

[0055] For example, Figure 2is a schematic diagram of a scene of image segmentation to be processed provided by an embodiment of the present application. Figure 2 As shown in the figure, assuming that the image to be processed needs to be divided into 16 image blocks, the number of sub-GPU cores required is also 16 ( Figure 2 The 16 sub-GPU cores can read 16 starting positions in the image to be processed ( Figure 2 In the example, 3 starting positions in the image to be processed are read, and the 16 starting positions are used as the starting points of the 16 image blocks. Based on the starting points of the 16 image blocks, one pixel is read at a time and stored in the memory, so that 16 image blocks can be obtained from the image to be processed.

[0056] It should also be noted that using at least two sub-GPU cores to concurrently process the image to be processed in blocks can be understood as the interval between the start times when the at least two sub-GPU cores each process the image to be processed in blocks is less than the second parameter.

[0057] Exemplarily, the second parameter may be predefined or manually set, which is not limited in the embodiments of the present application.

[0058] It should also be noted that, due to the high concurrency of the GPU, the amount of computation required to use at least two sub-GPU cores to concurrently process the image in blocks may be determined by the size of the image blocks.

[0059] For example, the image to be processed is divided into n×n blocks, and the time consumption is n 2 The time of iteration.

[0060] In addition, since the size of the image to be processed is often small, generally 8×8 or 16×16, the computational complexity of using at least two sub-GPU cores to concurrently process the image in blocks is small and does not take long.

[0061] It should also be noted that the core technology for at least two GPU sub-cores to process the image in blocks concurrently is the Single Instruction Multiple Data (SMID) instruction. The SMID instruction can apply one instruction to multiple image blocks that need to be segmented at the same time without having to execute them one by one, thus enabling large-scale parallel computing.

[0062] Through this method, at least two sub-GPU cores are used to concurrently process the image to be processed in blocks according to the starting positions of the at least two sub-GPU cores when they each read the image to be processed, which can not only achieve miniaturization of the network scale, but also reduce the time consumption of image segmentation.

[0063] S120, performing convolution processing on the image to be processed to obtain an initial feature map.

[0064] It should be noted that performing convolution processing on the image to be processed to obtain the initial feature map may include: passing the image to be processed through a convolution layer, so as to expand the image to be processed into the initial feature map.

[0065] S130, calculating the target correlation of at least two image blocks, and performing weighted processing on the initial feature map according to the target correlation to obtain a weighted feature map.

[0066] Exemplarily, at least two image blocks may be straightened image blocks.

[0067] It should be noted that the target correlation of at least two image blocks may include: the autocorrelation of at least two image blocks, and / or the mutual correlation of at least two image blocks.

[0068] Exemplarily, the autocorrelation of at least two image blocks can be understood as the correlation between at least two image blocks themselves. For example, the correlation between image block 1 and image block 1 is the autocorrelation of image block 1.

[0069] Exemplarily, the mutual correlation of at least two image blocks can be understood as the correlation between every two image blocks in the at least two image blocks. For example, the correlation between image block 1 and image block 2 is the mutual correlation between image block 1 and image block 2.

[0070] In some embodiments, the number of channels of the initial feature map is equal to the number of at least two image patches.

[0071] It should be noted that when the initial feature map is weighted according to the target correlation, the number of channels of the initial feature map may be the same as the number of at least two image blocks.

[0072] Furthermore, weighting the initial feature map according to the target correlation to obtain the weighted feature map may include: performing an element-wise (Element-Wise) product of the target correlation and the initial feature map, thereby obtaining the weighted feature map.

[0073] In some embodiments, before calculating the target correlation of at least two image blocks, the method may further include: performing dimensionality reduction processing on the at least two image blocks.

[0074] It should be noted that when performing dimensionality reduction processing on at least two image blocks, the at least two image blocks may be input into a multilayer perceptron (MLP), so that the at least two image blocks after dimensionality reduction may be obtained using the following formula (1):

[0075] Q=XP (1)

[0076] Among them, Q is at least two image blocks after dimensionality reduction; X is at least two image blocks before dimensionality reduction; P is the weight of MLP.

[0077] Furthermore, the loss function (Loss) of network training can be used as the weight of MLP. The loss is Charbonnier Loss, and the prior is added. The calculation formula of the loss is:

[0078] Loss=Charbonnier(N(x)-y)+λΦ(N(x),y) (2)

[0079] Wherein, x is the input image, which can be a low quality (LQ) image, N(x) is the output image, y is the credible ground truth (GT) image, λ is a constant, Φ(N(x), y) is a regularization term, and Φ(N(x), y) can be used to constrain the calculation results of Charbonnier(N(x)-y).

[0080] Through this method, after the dimensionality reduction processing is performed on at least two image blocks, the computational complexity of the subsequent calculation of the target correlation of the at least two image blocks can be reduced, which is more conducive to the application of the image processing method provided in the embodiment of the present application on a mobile terminal device.

[0081] S140, performing noise reduction processing on the initial feature map and the weighted feature map to obtain a target image.

[0082] In some embodiments, performing denoising on the initial feature map and the weighted feature map to obtain a target image may include: performing splicing on the initial feature map and the weighted feature map to obtain an initial target image; and inputting the initial target image into a denoising model to obtain a target image.

[0083] Exemplarily, the denoising model may be a CNN model. For example, the denoising model may be a UNet model.

[0084] In addition, the noise reduction model may also be other models, which is not limited in the embodiments of the present application.

[0085] Take the UNet model as an example. Figure 3 is a schematic diagram of a framework for noise reduction using a noise reduction model provided in an embodiment of the present application. Figure 3 As shown, the initial feature map 310 and the weighted feature map 320 can be spliced ​​to obtain an initial target image 330, and the initial target image is input into the UNet model 340 to obtain a target image 350.

[0086] It should be noted that the denoising model may also have weights. In this case, the loss of network training can be used as the weight in the denoising model, and the above formula (2) can be used to calculate the loss of network training.

[0087] Through this method, the initial feature map and the weighted feature map are spliced ​​to obtain the initial target image, which is input into the denoising model. This can make the obtained target image retain more image detail texture and enhance the denoising effect.

[0088] The embodiment of the present application provides an image processing method that can be applied to a mobile terminal device. First, an image to be processed and at least two image blocks of the image to be processed can be obtained; secondly, a convolution process is performed on the image to be processed to obtain an initial feature map; then, the target correlation of at least two image blocks is calculated, and the initial feature map is weighted according to the target correlation to obtain a weighted feature map; finally, the initial feature map and the weighted feature map are subjected to denoising to obtain a target image. In this way, on the one hand, when the image processing method is applied to a mobile terminal device, the real-time performance of the target image obtained after the denoising process can be improved; on the other hand, according to the initial feature map obtained by convolution processing of the image to be processed, and the weighted feature map obtained by weighting the initial feature map according to the target correlation, the denoising process can be performed, so that the obtained target image retains more image detail textures and enhances the denoising effect.

[0089] Based on S130, the target correlation of at least two image blocks can be calculated. Figure 4 A method for calculating an object correlation of at least two image patches is described.

[0090] Figure 4 FIG. 1 is a flow chart of another image processing method provided in an embodiment of the present application, which is applied to a mobile terminal device. Figure 4 As shown, the method may include the following steps:

[0091] S410, calculating a first initial correlation between at least two image blocks;

[0092] S420, normalizing the first initial correlation to obtain a second initial correlation of at least two image blocks;

[0093] S430 , mapping the second initial correlation according to a normalized exponential function to obtain target correlations of at least two image blocks.

[0094] For example, assuming there are two image blocks, denoted as G and H, the target correlation between G and H is calculated as follows:

[0095]

[0096] Among them, through G.H. T The first initial correlation between G and H can be obtained; It can be the first initial correlation G·H T Normalization is performed to obtain the second initial correlation between G and H; SoftMax is a normalized exponential function, through which the second initial correlation can be obtained. A mapping process is performed to obtain the target correlation between G and H.

[0097] In some embodiments, the method may further include: determining a first parameter, suppressing the autocorrelation of at least two image blocks in the target correlation according to the first parameter, and / or enhancing the mutual correlation of at least two image blocks in the target correlation according to the first parameter.

[0098] Exemplarily, the first parameter may be predefined or manually set, which is not limited in the embodiments of the present application.

[0099] It should be noted that the first parameter can be a fixed constant matrix, the first parameter can suppress the diagonal values ​​in the target correlation, and / or, the first parameter can enhance the non-diagonal values ​​in the target correlation; wherein, the diagonal values ​​can represent the autocorrelation of at least two image blocks, and the non-diagonal values ​​can represent the mutual correlation of at least two image blocks.

[0100] It should also be noted that the first parameter and the target correlation may be Element-Wise multiplied.

[0101] For example, assuming that there are two image blocks, respectively denoted as G and H, based on the calculation formula of the target correlation between G and H in the above formula (3), a calculation formula for suppressing the autocorrelation between G and H and / or enhancing the mutual correlation between G and H by the first parameter can be obtained:

[0102]

[0103] Where A is the target correlation between G and H obtained by suppressing the autocorrelation between G and H and / or enhancing the mutual correlation between G and H through the first parameter; Mask is the first parameter; By performing an Element-Wise product with Mask, the diagonal value of the target correlation between G and H can be made less than 1, and the off-diagonal value can be made greater than 1, thereby achieving the purpose of suppressing the autocorrelation between G and H and / or enhancing the mutual correlation between G and H.

[0104] Through this method, the autocorrelation of at least two image blocks in the target correlation is suppressed based on the first parameter, and / or the mutual correlation of at least two image blocks in the target correlation is enhanced based on the first parameter, so that the weighted feature map obtained by subsequent weighted processing of the initial feature map based on the target correlation can retain more image detail texture.

[0105] The embodiment of the present application provides an image processing method that can be applied to a mobile device. First, a first initial correlation of at least two image blocks can be calculated; then, the first initial correlation can be normalized to obtain a second initial correlation of at least two image blocks; finally, the second initial correlation is mapped according to a normalized exponential function to obtain a target correlation of at least two image blocks. In this way, the weighted feature map obtained by subsequent weighted processing of the initial feature map based on the target correlation can retain more image detail textures.

[0106] The image processing method provided by the above embodiment is described in detail below in combination with specific application scenarios.

[0107] In the embodiment of the present application, the image processing method can be used to improve the Transformer algorithm in the related art to obtain a lightweight Transformer algorithm (i.e., the improved Transformer algorithm), which can be applied to mobile devices, and its computational complexity can meet the requirements of mobile device deployment, and based on the concept of the lightweight Transformer algorithm, the detailed texture of the target image can be well restored. The lightweight Transformer algorithm is described in detail below.

[0108] Figure 5 is a schematic diagram of a lightweight Transformer algorithm provided in an embodiment of the present application. Figure 5As shown, the framework of the lightweight Transformer algorithm may include a preprocessing module 510 , an attention module 520 , and an image denoising module 530 . First, the input data stream can be input into the preprocessing module 510, so that the input data stream can be processed into a to-be-processed image in the preprocessing module 510, and the to-be-processed image can be divided into blocks in the preprocessing module 510 to obtain at least two image blocks; then, the to-be-processed image and the at least two image blocks can be input into the attention module 520, so that the to-be-processed image can be convolved in the attention module 520 to obtain an initial feature map, and the target correlation of at least two image blocks can be calculated in the attention module 520, and the initial feature map can be weighted according to the target correlation to obtain a weighted feature map; finally, the initial feature map and the weighted feature map can be input into the image denoising module 530, so that the initial feature map and the weighted feature map can be spliced ​​in the image denoising module 530 to obtain an initial target image, and; the initial target image can be input into the denoising model in the image denoising module 530 to obtain a target image, and then the output data stream can be obtained based on the target image.

[0109] The functions of the preprocessing module 510, the attention module 520 and the image denoising module 530 are described in detail below.

[0110] (I) Preprocessing module 510

[0111] The preprocessing module 510 can process the input data stream into an image to be processed, and perform block processing on the image to be processed to obtain at least two image blocks, so that the preprocessing module 510 can input the image to be processed and the at least two image blocks into the attention module 520.

[0112] Considering that it takes a long time to directly process the image to be processed in blocks, the embodiment of the present application can use the GPU core to process the image to be processed in blocks, so as to obtain at least two image blocks.

[0113] The scene of dividing the image to be processed into blocks is as follows Figure 2 As shown, the process can be implemented using OpenCL.

[0114] For example, Figure 2 As shown, the reading point (i.e., the starting position) of the GPU core can be fixed at the starting point of each image block. For example, when divided into 16 image blocks, the number of GPU cores can be specified as 16, and the reading points of the 16 GPU cores can be fixed at the starting points of the 16 image blocks. Each GPU core can read one pixel at a time and store it in the memory.

[0115] Due to the high concurrency of GPU, the computational complexity of the block process of the image to be processed can be determined by the size of the image block. For example, if the image to be processed is divided into n×n blocks, the time consumption is n 2 Since the size of the image to be processed is usually small, usually 8×8 or 16×16, the computational complexity of the block processing of the image to be processed by the GPU core is small and the time consumption is not long.

[0116] (II) Attention Module 520

[0117] The attention module 520 can receive the image to be processed and at least two image blocks from the preprocessing module 510, and obtain an initial feature map and a weighted feature map by processing the image to be processed and at least two image blocks, so that the attention module 520 can input the initial feature map and the weighted feature map into the image denoising module 530.

[0118] Figure 6 Schematic diagram of a framework for determining an initial feature map and a weighted feature map provided in an embodiment of the present application. Figure 6 As shown, the framework for determining the initial feature map and the weighted feature map may include a convolution module 610, an MLP module 620, and a correlation module 630. First, the image to be processed may be input into the convolution module 610, so that the image to be processed can be expanded into an initial feature map through a convolution layer in the convolution module 610, and the number of channels of the initial feature map is the same as the number of at least two image blocks; secondly, at least two image blocks may be input into the MLP module 620, so that the at least two image blocks can be subjected to dimensionality reduction processing in the MLP module 620, and the calculation formula for the dimensionality reduction processing of the at least two image blocks is shown in the aforementioned formula (1); then, after the dimensionality reduction processing of the at least two image blocks is performed, the at least two image blocks may be input into the correlation module 630, so that the target correlation of the at least two image blocks can be calculated in the correlation model 630 using the aforementioned formula (3) and / or formula (4); finally, the initial feature map may be subjected to weighted processing according to the target correlation (such as performing an Element-Wise product of the target correlation and the initial feature map), so that the weighted feature map can be obtained.

[0119] It should be noted that at least two image blocks may be straightened image blocks.

[0120] (III) Image noise reduction module 530

[0121] The image denoising module 530 may receive the initial feature map and the weighted feature map from the attention module 520 , and obtain a target image by processing the initial feature map and the weighted feature map.

[0122] like Figure 3As shown, the initial feature map 310 and the weighted feature map 320 can be spliced ​​to obtain an initial target image 330, and then the initial target image 330 can be denoised through the UNet model 340 to obtain a target image 350.

[0123] It should be noted that, in addition to the UNet model 340, other denoising models may also be used in the denoising model in the embodiment of the present application. Using the denoising model in the lightweight Transformer algorithm may improve the performance of the denoising model.

[0124] It should also be noted that the embodiment of the present application can also add more multi-head attention modules 520 to the denoising model, and can recursively call the correlation module 630 to improve the expressiveness of the network.

[0125] Furthermore, the recursive calls of the correlation module 630 are adjustable.

[0126] In an embodiment of the present application, the lightweight Transformer algorithm may include two parts: preprocessing (such as performed by the preprocessing module 510) and network processing (such as performed by the attention module 520 and the image denoising module 530). In this way, on the one hand, the lightweight Transformer algorithm does not need to add hardware-unfriendly operators when it is integrated; on the other hand, the lightweight Transformer algorithm can also achieve time-sharing multiplexing. For example, when the network processes the current frame image, the GPU (located in the preprocessing module 510) can synchronously process the next frame image. In addition, integrating the network on the DSP can reduce the interference between network processing and GPU processing.

[0127] In an embodiment of the present application, after dimensionality reduction of at least two image blocks, the amount of computation required to calculate target correlations between at least two image blocks is reduced, thereby enabling the lightweight Transformer algorithm to be applied to mobile devices; the denoising effect of the image denoising module 530 is significantly improved compared to the denoising effect of the traditional CNN model.

[0128] The embodiment of the present application provides a lightweight Transformer algorithm for a mobile terminal, which can use the GPU to divide the image to be processed into blocks to achieve miniaturization of the network scale, and the dimensions of at least two image blocks can be reduced through the MIP module, thereby reducing the amount of calculation when calculating the target correlation of at least two image blocks. In addition, using the lightweight Transformer algorithm provided in the embodiment of the present application for denoising, the obtained target image can retain more image detail textures, thereby enhancing the denoising effect.

[0129] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the idea of ​​the present application, they should also be regarded as the contents disclosed in the present application. For another example, under the premise of no conflict, the various embodiments and / or the technical features in the various embodiments described in the present application can be arbitrarily combined with the prior art, and the technical solution obtained after the combination should also fall within the protection scope of the present application.

[0130] It should be understood that in the various method embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0131] Based on the same inventive concept as the above embodiments, Figure 7 A schematic diagram of the structure of a mobile terminal device provided in an embodiment of the present application is shown. Figure 7 As shown, the mobile terminal device 700 may include an acquisition unit 710, a processing unit 720 and a calculation unit 730, wherein:

[0132] An acquisition unit 710 is configured to acquire an image to be processed and at least two image blocks of the image to be processed;

[0133] The processing unit 720 is configured to perform convolution processing on the image to be processed to obtain an initial feature map;

[0134] A calculation unit 730, configured to calculate a target correlation of at least two image blocks;

[0135] The processing unit 720 is further configured to perform weighted processing on the initial feature map according to the target correlation to obtain a weighted feature map; and perform noise reduction processing on the initial feature map and the weighted feature map to obtain a target image.

[0136] In some embodiments, the calculation unit 730 is further configured to calculate a first initial correlation of at least two image blocks; the processing unit 720 is further configured to normalize the first initial correlation to obtain a second initial correlation of at least two image blocks, and map the second initial correlation according to a normalized exponential function to obtain a target correlation of at least two image blocks.

[0137] In some embodiments, Figure 7 As shown, the mobile terminal device 700 may further include a determining unit 740, wherein:

[0138] The determination unit 740 is configured to determine a first parameter, suppress the autocorrelation of at least two image blocks in the target correlation according to the first parameter, and / or enhance the mutual correlation of at least two image blocks in the target correlation according to the first parameter.

[0139] In some embodiments, the processing unit 720 is further configured to perform dimensionality reduction processing on at least two image blocks.

[0140] In some embodiments, the number of channels of the initial feature map is equal to the number of at least two image patches.

[0141] In some embodiments, the processing unit 720 is further configured to perform block processing on the image to be processed according to the GPU core of the graphics processing unit to obtain at least two image blocks.

[0142] In some embodiments, the determination unit 740 is further configured to determine the starting position of at least two sub-GPU cores when each reads the image to be processed; the processing unit 720 is further configured to use at least two sub-GPU cores to concurrently perform block processing on the image to be processed based on the starting position of at least two sub-GPU cores when each reads the image to be processed.

[0143] In some embodiments, Figure 7 As shown, the mobile terminal device 700 may further include a generating unit 750, wherein:

[0144] The processing unit 720 is further configured to perform splicing processing on the initial feature map and the weighted feature map to obtain an initial target image;

[0145] The generating unit 750 is configured to input the initial target image into the denoising model to obtain the target image.

[0146] The embodiment of the present application provides a mobile terminal device. First, an image to be processed and at least two image blocks of the image to be processed can be obtained; secondly, a convolution process is performed on the image to be processed to obtain an initial feature map; then, the target correlation of at least two image blocks is calculated, and the initial feature map is weighted according to the target correlation to obtain a weighted feature map; finally, the initial feature map and the weighted feature map are subjected to denoising to obtain a target image. In this way, on the one hand, when the image processing method is applied to a mobile terminal device, the real-time performance of the target image obtained after the denoising process can be improved; on the other hand, according to the initial feature map obtained by convolution processing of the image to be processed, and the weighted feature map obtained by weighting the initial feature map according to the target correlation, the denoising process can be performed, so that the obtained target image retains more image detail textures and enhances the denoising effect.

[0147] Those skilled in the art should understand that the relevant description of the above-mentioned mobile terminal device in the embodiment of the present application can be understood by referring to the relevant description of the image processing method in the embodiment of the present application.

[0148] Figure 8 is a schematic structural diagram of a mobile terminal device 700 provided in an embodiment of the present application. Figure 8 As shown, the mobile terminal device 700 may include a processor 810 and a memory 820. The memory 820 may store a computer program, and the processor 810 may call and run the computer program from the memory 820 to implement the method in the embodiment of the present application.

[0149] The memory 820 may be a separate device independent of the processor 810 , or may be integrated into the processor 810 .

[0150] In some embodiments, Figure 8 As shown, the mobile device 700 may further include a transceiver 830, and the processor 810 may control the transceiver 830 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0151] The transceiver 830 may include a transmitter and a receiver. The transceiver 830 may further include an antenna, and the number of the antennas may be one or more.

[0152] Fig. 9 is a schematic structural diagram of a chip provided in an embodiment of the present application. Fig. 9 As shown, the chip 900 includes a processor 910, and the processor 910 can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0153] In some embodiments, Fig. 9 As shown, the chip 900 may further include a memory 920. The processor 910 may call and run a computer program from the memory 920 to implement the method in the embodiment of the present application.

[0154] The memory 920 may be a separate device independent of the processor 910 , or may be integrated into the processor 910 .

[0155] In some embodiments, the chip 900 may further include an input interface 930. The processor 910 may control the input interface 930 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0156] In some embodiments, the chip 900 may further include an output interface 940. The processor 910 may control the output interface 940 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0157] In some embodiments, the chip can be applied to the mobile device in the embodiments of the present application. For the sake of brevity, it will not be described here.

[0158] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0159] It can be understood that the processor of the embodiment of the present application may be an integrated circuit chip with information processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0160] It can also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0161] It can also be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application can also be static random access memory (Static RAM, SRAM), dynamic random access memory (DynamicRAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Dynch Link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0162] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0163] In some embodiments, the computer-readable storage medium can be applied to the mobile terminal device in the embodiments of the present application, and when the computer program is executed by at least one processor, it implements the corresponding processes implemented by the mobile terminal device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0164] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0165] In some embodiments, the computer program product can be applied to the mobile terminal device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the mobile terminal device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0166] The embodiment of the present application also provides a computer program.

[0167] In some embodiments, the computer program can be applied to the mobile device in the embodiments of the present application. When the computer program runs on the computer, the computer executes the corresponding processes implemented by the mobile device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0168] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0173] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0174] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0175] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0176] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0177] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0178] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0179] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to a mobile device, the method comprises: Acquire an image to be processed and at least two image blocks of the image to be processed; Performing convolution processing on the image to be processed to obtain an initial feature map; Calculating the target correlation of the at least two image blocks, and performing weighted processing on the initial feature map according to the target correlation to obtain a weighted feature map; The initial feature map and the weighted feature map are subjected to denoising to obtain a target image.

2. The method according to claim 1, characterized in that The calculating the target correlation of the at least two image blocks comprises: Calculating a first initial correlation of the at least two image blocks; The first initial correlation is normalized to obtain a second initial correlation of the at least two image blocks, and the second initial correlation is mapped according to a normalized exponential function to obtain a target correlation of the at least two image blocks.

3. The method according to claim 2, characterized in that The method further comprises: determining a first parameter, suppressing the autocorrelation of the at least two image blocks in the target correlation according to the first parameter, and / or, The mutual correlation of the at least two image blocks in the target correlation is enhanced according to the first parameter.

4. The method according to any one of claims 1 to 3, characterized in that Before calculating the target correlation of the at least two image blocks, the method further includes: Perform dimensionality reduction processing on the at least two image blocks.

5. The method according to any one of claims 1 to 3, characterized in that The number of channels of the initial feature map is equal to the number of the at least two image blocks.

6. The method according to any one of claims 1 to 3, characterized in that Acquiring the at least two image blocks includes: The image to be processed is processed in blocks according to a graphics processing unit (GPU) core to obtain the at least two image blocks.

7. The method according to claim 6, characterized in that The GPU core includes at least two sub-GPU cores, and the processing of the image to be processed in blocks according to the GPU cores includes: Determine a starting position when each of the at least two sub-GPU cores reads the image to be processed; According to the starting positions when the at least two sub-GPU cores respectively read the image to be processed, the at least two sub-GPU cores are used to concurrently process the image to be processed in blocks.

8. The method according to any one of claims 1 to 3, characterized in that The performing denoising on the initial feature map and the weighted feature map to obtain a target image includes: Performing splicing processing on the initial feature map and the weighted feature map to obtain an initial target image; The initial target image is input into a denoising model to obtain the target image.

9. A mobile terminal device, characterized in that: The mobile terminal device includes an acquisition unit, a processing unit and a calculation unit; wherein: The acquisition unit is configured to acquire an image to be processed and at least two image blocks of the image to be processed; The processing unit is configured to perform convolution processing on the image to be processed to obtain an initial feature map; The calculation unit is configured to calculate the target correlation of the at least two image blocks; The processing unit is further configured to perform weighted processing on the initial feature map according to the target correlation to obtain a weighted feature map; and perform noise reduction processing on the initial feature map and the weighted feature map to obtain a target image.

10. A mobile terminal device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 8.

11. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes a method as claimed in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 8 is implemented.

Images