Image processing method, device, electronic device and readable storage medium

Abstract

The present disclosure provides an image processing method, device, electronic device and readable storage medium, the method comprising: acquiring an original image to be processed; performing downsampling processing on the original image according to N set multiples to obtain N thumbnail images corresponding to the set multiples one by one; wherein N is a positive integer; performing feature analysis on the N thumbnail images to obtain a first fusion feature characterizing the N thumbnail images; performing a first feature extraction processing on the original image to obtain the original feature of the original image; performing a fusion processing based on the first fusion feature and the original feature of the original image to obtain a denoised image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and more specifically, to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] With the development of multimedia technology, video and image data are growing exponentially. In order to save the bandwidth and bit rate required for video or image transmission, the sending device usually performs lossy encoding compression on the video or image before transmitting it. Due to lossy compression, the video or image received by the receiving device often has large compression noise, which makes the user's visual experience poor.

[0003] The current solution to image noise is usually for the receiving device to use a denoising algorithm to restore the noisy image to an image close to the original.

[0004] Commonly used denoising algorithms in the prior art, such as non-local mean (NLM) algorithm, BM3D algorithm, WNNM algorithm, MCWNNM algorithm, etc., are all based on the non-local similarity of images. These denoising algorithms find the rules from the original image with noise and then perform the corresponding denoising process. The final analytical solution or iterative solution is obtained through a series of conditional constraints. It is generally assumed that the noise signal and the speech signal are independent of each other, but this is usually an unreasonable assumption and affects the upper limit of the algorithm effect. This also determines that these denoising algorithms cannot solve complex denoising problems well. Summary of the invention

[0005] One object of the present disclosure is to provide a new technical solution for image denoising.

[0006] According to a first aspect of the present disclosure, there is provided an image processing method, comprising:

[0007] Obtain the original image to be processed;

[0008] According to N set multiples, downsampling the original image is performed to obtain N thumbnail images corresponding to the set multiples one by one; wherein N is a positive integer;

[0009] Performing feature analysis on the N thumbnail images to obtain first fusion features representing the N thumbnail images;

[0010] Performing a first feature extraction process on the original image to obtain original features of the original image;

[0011] A fusion process is performed based on the first fusion feature and the original feature of the original image to obtain a denoised image.

[0012] Optionally, the N thumbnail images are arranged in order of a set multiple;

[0013] The performing feature analysis on the N thumbnail images to obtain a first fusion feature characterizing the N thumbnail images includes:

[0014] Perform a second feature extraction process on the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image:

[0015] Performing a first feature extraction process on the nth thumbnail image to obtain an original feature of the nth thumbnail image; wherein n is a positive integer greater than 1 and less than or equal to N;

[0016] The original feature of the nth thumbnail image and the second fused feature corresponding to the n-1th thumbnail image are fused to obtain the second fused feature corresponding to the nth thumbnail image;

[0017] The second fusion feature corresponding to the Nth thumbnail image is used as the first fusion feature representing the N thumbnail images.

[0018] Optionally, performing a second feature extraction process on the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image includes:

[0019] Performing a first feature extraction process on the first thumbnail image to obtain original features of the first thumbnail image;

[0020] Based on the first residual dense network, performing local depth feature extraction processing on the original features of the first thumbnail image to obtain the local depth features of the first thumbnail image;

[0021] Based on the first residual efficient channel attention network, global deep feature extraction processing is performed on the local deep features of the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image.

[0022] Optionally, the first feature extraction process is performed in the following manner:

[0023] Based on the preset convolutional network, a first feature extraction process is performed on the input thumbnail image to obtain the original features of the corresponding thumbnail image.

[0024] Optionally, the fusing the original feature of the nth thumbnail image and the second fused feature corresponding to the (n-1)th thumbnail image to obtain the second fused feature corresponding to the nth thumbnail image includes:

[0025] Combining the original feature of the nth thumbnail image and the second fused feature corresponding to the n-1th thumbnail image to obtain a combined feature corresponding to the nth thumbnail image;

[0026] Based on the nth residual dense network, performing local depth feature extraction processing on the combined features corresponding to the nth thumbnail image to obtain the local depth features of the nth thumbnail image;

[0027] Based on the nth residual efficient channel attention network, a global deep feature extraction process is performed on the local deep features of the nth thumbnail image to obtain a second fusion feature corresponding to the nth thumbnail image.

[0028] Optionally, after obtaining the second fusion feature, the method further includes:

[0029] The second fused features are compressed.

[0030] Optionally, the performing fusion processing according to the first fusion feature and the original feature of the original image to obtain a denoised image includes:

[0031] Combining the first fused feature with the original feature of the original image to obtain a combined feature corresponding to the original image;

[0032] Based on the N+1th residual dense network, local depth feature extraction processing is performed on the combined features corresponding to the original image to obtain the local depth features of the original image;

[0033] Based on the N+1th residual efficient channel attention network, perform global deep feature extraction processing on the local deep features of the original image to obtain a third fusion feature corresponding to the original image;

[0034] The denoised image is generated according to the third fusion feature.

[0035] According to a second aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0036] An image acquisition module, used to acquire the original image to be processed;

[0037] A downsampling module, used to downsample the original image according to N set multiples to obtain N thumbnail images corresponding to the set multiples; wherein N is a positive integer;

[0038] A feature analysis module, used for performing feature analysis on the N thumbnail images to obtain a first fusion feature representing the N thumbnail images;

[0039] A feature extraction module, used for performing a first feature extraction process on the original image to obtain original features of the original image;

[0040] A fusion processing module is used to perform fusion processing according to the first fusion feature and the original feature of the original image to obtain a denoised image.

[0041] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0042] The device as described in the second aspect of the present disclosure; or,

[0043] A processor and a memory, wherein the memory is used to store instructions, and the instructions are used to control the processor to execute the method according to the first aspect of the present disclosure.

[0044] According to a fourth aspect of the present disclosure, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect of the present disclosure is implemented.

[0045] Through the embodiments of the present disclosure, the original image is denoised according to the thumbnail image corresponding to each set magnification, and the obtained denoised image can effectively restore the details in the original image and remove the noise in the original image to improve the visual effect of the denoised image. Based on the image processing method of this embodiment, the original image with a higher compression rate can be transmitted between devices to save the bandwidth and bit rate required for transmitting the original image.

[0046] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0048] Figure 1 is a block diagram showing an example of a hardware configuration of an electronic device that can be used to implement an embodiment of the present disclosure.

[0049] Figure 2 is a block diagram showing another example of a hardware configuration of an electronic device that can be used to implement an embodiment of the present disclosure.

[0050] Figure 3 A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0051] Figure 4 A flow chart showing steps of feature analysis according to an embodiment of the present disclosure.

[0052] Figure 5 A structural diagram of a machine learning model according to an embodiment of the present disclosure is shown.

[0053] Figure 6 A schematic diagram of the structure of a residual dense network according to an embodiment of the present disclosure is shown.

[0054] Figure 7 A schematic diagram of the structure of the residual efficient channel attention network of an embodiment of the present disclosure is shown.

[0055] Figure 8 A schematic diagram showing an example of an image processing method according to an embodiment of the present disclosure.

[0056] Fig. 9 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0057] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0059] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0060] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0061] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0062] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0063] <Hardware Configuration>

[0064] Figure 1 and Figure 2 It is a block diagram of a hardware configuration of an electronic device that can be used to implement the method of any embodiment of the present disclosure.

[0065] In one embodiment, Figure 1As shown, the electronic device may be a server 1100 .

[0066] Server 1100 provides a computer for processing, database, and communication facilities. Server 1100 can be a monolithic server or a distributed server across multiple computers or computer data centers. The server can be of various types, such as but not limited to, a network server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an interactive server, a database server, or a proxy server. In some embodiments, each server may include hardware, software, or an embedded logic component or a combination of two or more such components for executing the appropriate functions supported or implemented by the server. For example, a server such as a blade server, a cloud server, etc., or a server group consisting of multiple servers, which may include one or more of the above-mentioned types of servers, etc.

[0067] In this embodiment, the server 1100 can be as follows Figure 1 As shown, it includes a processor 1110 , a memory 1120 , an interface device 1130 , a communication device 1140 , a display device 1150 , and an input device 1160 .

[0068] In this embodiment, the server 1100 may also include a speaker, a microphone, etc., which are not limited here.

[0069] The processor 1110 may be a dedicated server processor, or a desktop processor or a mobile processor that meets the performance requirements, which is not limited here. The memory 1120 includes, for example, ROM (read-only memory), RAM (random access memory), non-volatile memory such as a hard disk, etc. The interface device 1130 includes, for example, various bus interfaces, such as a serial bus interface (including a USB interface), a parallel bus interface, etc. The communication device 1140 is capable of wired or wireless communication, for example. The display device 1150 is, for example, a liquid crystal display, an LED display, a touch display, etc. The input device 1160 may include, for example, a touch screen, a keyboard, etc.

[0070] In this embodiment, the memory 1120 of the server 1100 is used to store instructions, which are used to control the processor 1110 to operate to at least perform the method according to any embodiment of the present disclosure. A technician can design instructions according to the scheme disclosed in the present disclosure. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0071] Despite Figure 1 , multiple devices of the server 1100 are shown; however, the present disclosure may only involve some of the devices, for example, the server 1100 only involves the memory 1120 and the processor 1110 .

[0072] In one embodiment, the electronic device may be a terminal device 1200 such as a PC or a laptop computer used by an operator, which is not limited here.

[0073] In this embodiment, refer to Figure 2 As shown, the terminal device 1200 may include a processor 1210, a memory 1220, an interface device 1230, a communication device 1240, a display device 1250, an input device 1260, a speaker 1270, a microphone 1280, and the like.

[0074] The processor 1210 may be a mobile version processor. The memory 1220 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1230 may include, for example, a USB interface, a headphone interface, etc. The communication device 1240 may, for example, be capable of wired or wireless communication. The communication device 1240 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 1240 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 1250 may, for example, be a liquid crystal display, a touch display, etc. The input device 1260 may include, for example, a touch screen, a keyboard, etc. The user may input / output voice information through the speaker 1270 and the microphone 1280.

[0075] In this embodiment, the memory 1220 of the terminal device 1200 is used to store instructions, which are used to control the processor 1210 to operate to at least perform the method according to any embodiment of the present disclosure. A technician can design instructions according to the scheme disclosed in the present disclosure. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0076] Despite Figure 2 , multiple devices of the terminal device 1200 are shown, but the present disclosure may only involve some of the devices, for example, the terminal device 1200 only involves the memory 1220, the processor 1210, and the display device 1250.

[0077] <Method Example>

[0078] In this embodiment, an image processing method is provided. The method is implemented by an electronic device. The electronic device may be an electronic product having a processor and a memory. For example, it may be a desktop computer, a laptop computer, a mobile phone, a tablet computer, etc. In one example, the electronic device may be Figure 1 The server 1100 shown may also be Figure 2 The terminal device 1200 is shown.

[0079] according to Figure 3 As shown, the image processing method of this embodiment may include the following steps S3100 to S3300.

[0080] Step S3100, obtaining the original image to be processed.

[0081] In this embodiment, the original image may be a static image, any layer in a dynamic image, or any video frame in a video.

[0082] In this embodiment, the original image may be an image with noise after lossy coding compression, may be pre-stored in the electronic device executing this embodiment, or may be obtained by the electronic device from other electronic devices, where the other electronic devices may be terminal devices or servers.

[0083] In an example, the width of the original image may be W pixels, the height may be H pixels, and the number of channels may be 3, that is, the size of the original image may be expressed as W*H*3.

[0084] Step S3200, down-sampling the original image according to the N-multiple factor, to obtain N thumbnail images corresponding to the set multiples.

[0085] Wherein, N is a positive integer.

[0086] In this embodiment, the set multiple may be pre-set according to the application scenario or specific requirements, and the N set multiples may be consecutive positive integers starting from 2. For example, when the number of set multiples N is 3, the N set multiples may include 1, 2, and 3. For another example, when the number of set multiples N is 5, the N set multiples may include 1, 2, 3, 4, and 5.

[0087] The original image with a size of W*H*3 is downsampled according to the set magnification s, and the size of the thumbnail image corresponding to the set magnification s can be (W / 2 s )*(H / 2 s )*(3*4 s ).

[0088] For example, when the magnification is set to 1, the original image with a size of W*H*3 is downsampled according to the set magnification 1, and the size of the thumbnail image corresponding to the set magnification 1 can be (W / 2)*(H / 2)*(3*4).

[0089] For another example, when the magnification is set to 2, the original image with a size of W*H*3 is downsampled according to the magnification of 2, and the size of the thumbnail image corresponding to the magnification of 2 can be (W / 2 2 )*(H / 2 2 )*(3*4 2 )=(W / 4)*(H / 4)*48.

[0090] For another example, when the magnification is set to 3, the original image with a size of W*H*3 is downsampled according to the magnification of 3, and the size of the thumbnail image corresponding to the magnification of 3 can be (W / 2 3 )*(H / 2 3 )*(3*4 3 )=(W / 8)*(H / 8)*192.

[0091] Step S3300: Perform feature analysis on the N thumbnail images to obtain a first fusion feature representing the N thumbnail images.

[0092] In this embodiment, the N thumbnail images may be arranged in the order of the set magnifications. For example, the N thumbnail images may be arranged in the order of the set magnifications from large to small, or may be arranged in the order of the set magnifications from small to large, which is not limited here.

[0093] The first fusion feature in this embodiment can represent all thumbnail images.

[0094] When the thumbnail image is represented in the form of a matrix, the first fused feature is also in the form of a matrix.

[0095] In one embodiment of the present disclosure, feature analysis is performed on N thumbnail images to obtain first fusion features representing the N thumbnail images, which may include: Figure 4 Steps S3310 to S3340 shown:

[0096] Step S3310: Perform a second feature extraction process on the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image.

[0097] In one embodiment of the present disclosure, performing a second feature extraction process on the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image may include the following steps S3311 to S3313:

[0098] Step S3311, performing a first feature extraction process on the first thumbnail image to obtain the original features of the first thumbnail image.

[0099] In this embodiment, the first feature extraction process may be performed in the following manner:

[0100] The first thumbnail image is input into a preset convolutional network, and based on the preset convolutional network, a first feature extraction process is performed on the first thumbnail image to obtain the original features of the first thumbnail image.

[0101] The convolutional network preset in this embodiment may be pre-set according to the application scenario or specific requirements. For example, the convolutional network may be a 3*3 convolutional network.

[0102] In the case where the thumbnail images are represented in the form of a matrix, by performing the first feature extraction process on the first thumbnail image, the original features of the first thumbnail image obtained are also in the form of a matrix.

[0103] Step S3312: Based on the first residual dense network, local depth feature extraction processing is performed on the original features of the first thumbnail image to obtain the local depth features of the first thumbnail image.

[0104] The first residual dense network (Residual Dense Net) may correspond to the first thumbnail image, and may perform local deep feature extraction processing on the original features of the first thumbnail image to obtain the local deep features of the first thumbnail image.

[0105] Step S3313: Based on the first residual efficient channel attention network, global depth feature extraction processing is performed on the local depth features of the first thumbnail image to obtain the second fusion feature corresponding to the first thumbnail image.

[0106] The first residual efficient channel attention network (Residual Efficient Channel Attention Net) corresponds to the first residual dense network. The first residual efficient channel attention network can perform global deep feature extraction processing on the local deep features of the first thumbnail image output by the first residual dense network, and then obtain the second fusion feature corresponding to the first thumbnail image.

[0107] The residual efficient channel attention network used in this embodiment is a lightweight module. Therefore, based on the use of the residual efficient channel attention network, the application scope of the method of this embodiment can be expanded (including mobile terminals, CPU ends), the time for image processing can be saved, and the coding richness of the coding device used to encode the original image can be further improved.

[0108] After obtaining the second fusion feature corresponding to the first thumbnail image, the method of the present disclosure may further include: compressing the second fusion feature corresponding to the first thumbnail image.

[0109] In this embodiment, the amount of data processed by the electronic device can be reduced by compressing the second fusion feature corresponding to the first thumbnail image and applying the compressed second fusion feature to the determination process of the second fusion feature corresponding to the second thumbnail image.

[0110] Step S3320, performing a first feature extraction process on the nth thumbnail image to obtain the original features of the nth thumbnail image; wherein n is a positive integer greater than 1 and less than or equal to N.

[0111] In this embodiment, the first feature extraction process may be performed in the following manner:

[0112] The nth thumbnail image is input into a preset convolutional network, and based on the preset convolutional network, a first feature extraction process is performed on the nth thumbnail image to obtain the original feature of the nth thumbnail image.

[0113] The convolutional network preset in this embodiment may be pre-set according to the application scenario or specific requirements. For example, the convolutional network may be a 3*3 convolutional network.

[0114] In the case where the thumbnail image is represented in the form of a matrix, by performing the first feature extraction process on the nth thumbnail image, the original features of the nth thumbnail image obtained are also in the form of a matrix.

[0115] In one example, the convolutional network that performs the first feature extraction process on the first thumbnail image and the nth thumbnail image may be the same convolutional network.

[0116] Step S3330, fusing the original features of the nth thumbnail image and the second fused features corresponding to the (n-1)th thumbnail image to obtain the second fused features corresponding to the nth thumbnail image.

[0117] In one embodiment of the present disclosure, performing fusion processing on the original feature of the nth thumbnail image and the second fused feature corresponding to the n-1th thumbnail image to obtain the second fused feature corresponding to the nth thumbnail image may include the following steps S3331 to S3333:

[0118] Step S3331, combining the original features of the nth thumbnail image and the second fused features corresponding to the n-1th thumbnail image to obtain the combined features corresponding to the nth thumbnail image.

[0119] In this embodiment, the original features of the nth thumbnail image and the second fused features corresponding to the n-1th thumbnail image can both be represented in the form of a matrix, but the size (including width and height) of the original features of the nth thumbnail image and the size of the second fused features corresponding to the n-1th thumbnail image may be different.

[0120] Therefore, the second fusion feature corresponding to the n-1th thumbnail image can be first upsampled to obtain the upsampled second fusion feature, so that the upsampled second fusion feature matches the original feature of the nth thumbnail image, that is, the size of the upsampled second fusion feature is the same as the size of the original feature of the nth thumbnail image.

[0121] When the second fused feature after upsampling is obtained, the original feature of the n-th thumbnail image and the second fused feature after upsampling may be combined to obtain the combined feature corresponding to the n-th thumbnail image.

[0122] Step S3332: Based on the nth residual dense network, local depth feature extraction processing is performed on the combined features corresponding to the nth thumbnail image to obtain the local depth features of the nth thumbnail image.

[0123] The nth residual dense network in this embodiment corresponds to the combined features corresponding to the nth thumbnail image, and local depth feature extraction processing can be performed on the combined features corresponding to the nth thumbnail image to obtain the local depth features of the nth thumbnail image.

[0124] Step S3333: Based on the nth residual efficient channel attention network, a global deep feature extraction process is performed on the local deep features of the nth thumbnail image to obtain a second fusion feature corresponding to the nth thumbnail image.

[0125] The nth residual efficient channel attention network may correspond to the nth residual dense network, and may perform global deep feature extraction processing on the local deep features of the nth thumbnail image output by the nth residual dense network to obtain a second fusion feature corresponding to the nth thumbnail image.

[0126] After obtaining the second fusion feature corresponding to the nth thumbnail image, the method of the present disclosure may further include: compressing the second fusion feature corresponding to the nth thumbnail image.

[0127] In this embodiment, the amount of data processed by the electronic device can be reduced by compressing the second fusion feature corresponding to the nth thumbnail image and applying the compressed second fusion feature to the determination process of the second fusion feature corresponding to the n+1th thumbnail image.

[0128] Step S3340: Use the second fusion feature corresponding to the Nth thumbnail image as the first fusion feature representing the N thumbnail images.

[0129] Step S3400: performing a first feature extraction process on the original image to obtain the original features of the original image.

[0130] In this embodiment, the first feature extraction process may be performed in the following manner:

[0131] The original image is input into a preset convolutional network, and based on the preset convolutional network, a first feature extraction process is performed on the original image to obtain the original features of the original image.

[0132] The convolutional network preset in this embodiment may be pre-set according to the application scenario or specific requirements. For example, the convolutional network may be a 3*3 convolutional network.

[0133] When the original image is represented in the form of a matrix, the original features of the original image obtained by performing the first feature extraction process on the original image are also in the form of a matrix.

[0134] In one example, the convolutional network that performs the first feature extraction process on the first thumbnail image, the nth thumbnail image, and the original image may be the same convolutional network.

[0135] Step S3500, performing fusion processing based on the first fusion feature and the original feature of the original image to obtain a denoised image.

[0136] In this embodiment, the denoised image obtained by fusion processing based on the first fusion features of the N thumbnail images and the original features of the original image can effectively restore the details in the original image and remove the noise in the original image to improve the visual effect of the denoised image. Based on the image processing method of this embodiment, the original image with a higher compression rate can be transmitted between devices to save the bandwidth and bit rate required for transmitting the original image.

[0137] In one embodiment of the present disclosure, performing fusion processing according to the first fusion feature and the original feature of the original image to obtain a denoised image may include the following steps S3510 to S3540:

[0138] Step S3510, combining the first fused feature and the original feature of the original image to obtain a combined feature corresponding to the original image.

[0139] In this embodiment, the first fused feature and the original feature of the original image may both be represented in the form of a matrix, but the size (including width and height) of the first fused feature may be different from the size of the original feature of the original image.

[0140] Therefore, the first fused feature may be upsampled first to obtain the upsampled first fused feature, so that the upsampled first fused feature matches the original feature of the original image, that is, the size of the upsampled first fused feature is the same as the size of the original feature of the original image.

[0141] When the first fused feature after upsampling is obtained, the original feature of the original image and the first fused feature after upsampling may be combined to obtain the combined feature corresponding to the original image.

[0142] Step S3520: Based on N+1 residual dense networks, local depth feature extraction processing is performed on the combined features corresponding to the original image to obtain the local depth features of the original image.

[0143] The N+1th residual dense network in this embodiment corresponds to the combined features corresponding to the original image, and local depth feature extraction processing can be performed on the combined features corresponding to the N+1th thumbnail image to obtain local depth features of the original image.

[0144] Step S3530, based on the N+1th residual efficient channel attention network, perform global depth feature extraction processing on the local depth features of the original image to obtain the third fusion feature corresponding to the original image.

[0145] The N+1th residual efficient channel attention network may correspond to the N+1th residual dense network, and may perform global deep feature extraction processing on the local deep features of the original image output by the N+1th residual dense network to obtain the third fusion feature corresponding to the original image.

[0146] The third fusion feature in this embodiment can represent the original image and all thumbnail images.

[0147] Step S3540: Generate a denoised image based on the third fusion feature.

[0148] By using the method of this embodiment, a first fusion feature that can reflect the local features and global features of all compressed images can be obtained, and then a denoised image with higher definition can be obtained based on the first fusion feature.

[0149] In one embodiment of the present disclosure, the third fusion feature and the denoised image may both be in matrix form. The width and height corresponding to the third fusion feature may be the same as the width and height corresponding to the denoised image, the number of channels of the denoised image is the same as the number of channels of the original image, for example, it may be 3, and the number of channels of the third fusion feature may be much larger than the number of channels of the denoised image.

[0150] Then, according to the third fusion feature, the denoised image is generated by performing dimensionality reduction processing on the third fusion feature in the channel dimension to obtain the denoised image. For example, when the number of channels of the third fusion feature is m and the number of channels of the denoised image is 3, the number of channels m of the third fusion feature can be reduced in dimension so that the number of channels after dimensionality reduction is 3, that is, the denoised image is obtained.

[0151] In an example, the number of channels of the third fusion feature is much larger than the number of channels of the denoised image. In order to achieve a better denoising effect, the channel dimension corresponding to the third fusion feature can be firstly reduced in dimension to obtain the reduced-dimensional feature, so that the number of channels corresponding to the reduced-dimensional feature is k (where k is an integer less than m and greater than 3); then the channel dimension corresponding to the reduced-dimensional feature is reduced in dimension to obtain the denoised image.

[0152] On the basis of this embodiment, the channel dimension corresponding to the reduced-dimensional feature may be subjected to at least one dimensionality reduction process to obtain a denoised image.

[0153] In one example, steps S3300 to S3500 in the image processing method of this embodiment may be executed by a machine learning model pre-set in the electronic device of this embodiment.

[0154] When the number of multiples N is set to 3, the structure of the machine learning model can be as follows: Figure 5 As shown in the dotted box, it can at least include: the first convolutional network, the first residual dense network, and the first residual efficient channel attention network corresponding to the first thumbnail image, the second convolutional network, the first feature combination module, the second residual dense network, and the second residual efficient channel attention network corresponding to the second thumbnail image, the third convolutional network, the second feature combination module, the third residual dense network, and the third residual efficient channel attention network corresponding to the third thumbnail image, and the fourth convolutional network, the third feature combination module, the fourth residual dense network, and the fourth residual efficient channel attention network corresponding to the original image.

[0155] The first convolutional network may perform a first feature extraction process on the first thumbnail image to obtain original features of the first thumbnail image.

[0156] The first residual dense network may perform local depth feature extraction processing on the original features of the first thumbnail image to obtain the local depth features of the first thumbnail image.

[0157] The first residual efficient channel attention network can perform global deep feature extraction processing on the local deep features of the first thumbnail image to obtain the second fusion feature corresponding to the first thumbnail image.

[0158] The second convolutional network may perform a first feature extraction process on the second thumbnail image to obtain original features of the second thumbnail image.

[0159] The first feature combination module may combine the original feature of the second thumbnail image and the second fused feature corresponding to the first thumbnail image to obtain the combined feature corresponding to the second thumbnail image.

[0160] The second residual dense network may perform local depth feature extraction processing on the combined features corresponding to the second thumbnail image to obtain local depth features of the second thumbnail image.

[0161] The second residual efficient channel attention network can perform global deep feature extraction processing on the local deep features of the second thumbnail image to obtain the second fusion feature corresponding to the second thumbnail image.

[0162] The third convolutional network may perform a first feature extraction process on the third thumbnail image to obtain original features of the third thumbnail image.

[0163] The second feature combination module may be configured to combine the original feature of the third thumbnail image and the second fused feature corresponding to the second thumbnail image to obtain the combined feature corresponding to the third thumbnail image.

[0164] The third residual dense network may perform local depth feature extraction processing on the combined features corresponding to the third thumbnail image to obtain local depth features of the third thumbnail image.

[0165] The third residual efficient channel attention network can perform global deep feature extraction processing on the local deep features of the third thumbnail image to obtain the second fusion feature corresponding to the third thumbnail image.

[0166] The fourth convolutional network can perform a first feature extraction process on the original image to obtain the original features of the original image.

[0167] The third feature combination module may combine the original features of the original image and the second fused features corresponding to the third thumbnail image to obtain the combined features corresponding to the original image.

[0168] The fourth residual dense network can perform local deep feature extraction processing on the combined features corresponding to the original image to obtain the local deep features of the original image.

[0169] The fourth residual efficient channel attention network can perform global depth feature extraction processing on the local depth features of the original image to obtain the third fusion feature corresponding to the original image.

[0170] According to the third fusion feature corresponding to the original image, a denoised image can be generated.

[0171] In the embodiments of the present disclosure, the structure of any residual dense network can be as Figure 6 shown. The processing process of this residual dense network can be to transfer the features input to this residual dense network (the corresponding combined features or the original features of the first thumbnail image) to the input of each convolutional layer. The outputs of each convolutional layer are also transferred to the input of the subsequent convolutional layers in a densely connected manner. Finally, the output of the last convolutional layer is connected to the outputs of the previous layers, and then feature fusion is performed through a 1*1 convolutional operation. Finally, it is added to the features input to this residual dense network to obtain the corresponding local depth features.

[0172] The structure of any residual efficient channel attention network can be as Figure 7 shown. Specifically, the processing process of this residual efficient channel attention network can be to perform global average pooling (GAP) processing on the input H*W*C local depth features to generate feature maps of size 1*1*C; calculate the adaptive convolutional kernel size (kernel_size) according to the feature maps; apply the kernel_size to the one-dimensional convolution to obtain the weights of each channel; according to the weights of each channel and the local depth features, the second fusion feature or the third fusion feature can be obtained.

[0173] <Example>

[0174] Figure 8 For the case where the number N of the set multiples is 3, it is a schematic diagram of the image processing method of the embodiments of the present disclosure.

[0175] According to Figure 8 shown, the original image is downsampled according to N set multiples respectively to obtain the first thumbnail image, the second thumbnail image, and the third thumbnail image.

[0176] Perform first feature extraction processing on the first thumbnail image to obtain the original features of the first thumbnail image. Input the original features of the first thumbnail image into the first residual dense network and the first residual efficient channel attention network connected in series, and the second fusion feature corresponding to the first thumbnail image can be obtained.

[0177] The first feature extraction process is performed on the second thumbnail image to obtain the original feature of the second thumbnail image. The original feature of the second thumbnail image is combined with the second fused feature corresponding to the first thumbnail image to obtain the combined feature corresponding to the second thumbnail image. The combined feature corresponding to the second thumbnail image is input into the second residual dense network and the second residual efficient channel attention network connected in series to obtain the second fused feature corresponding to the second thumbnail image.

[0178] Perform the first feature extraction process on the third thumbnail image to obtain the original feature of the third thumbnail image. Combine the original feature of the third thumbnail image with the second fused feature corresponding to the second thumbnail image to obtain the combined feature corresponding to the third thumbnail image. Input the combined feature corresponding to the third thumbnail image into the third residual dense network and the third residual efficient channel attention network connected in series to obtain the second fused feature corresponding to the third thumbnail image. Perform the first feature extraction process on the original image to obtain the original feature of the original image. Combine the original feature of the original image with the second fused feature corresponding to the third thumbnail image to obtain the combined feature corresponding to the original image. Input the combined feature corresponding to the original image into the fourth residual dense network and the fourth residual efficient channel attention network connected in series to obtain the third fused feature corresponding to the original image. According to the third fused feature, a denoised image can be obtained.

[0179] <Device Example>

[0180] In this embodiment, an image processing device 6000 is provided. Fig. 9 As shown, it includes an image acquisition module 6100, a downsampling module 6200, a feature analysis module 6300, a feature extraction module 6400 and a fusion processing module 6500. The image acquisition module 6100 is used to acquire the original image to be processed; the downsampling module 6200 is used to perform downsampling processing on the original image according to N set multiples to obtain N thumbnail images corresponding to the set multiples; wherein N is a positive integer; the feature analysis module 6300 is used to perform feature analysis on the N thumbnail images to obtain the first fusion feature characterizing the N thumbnail images; the feature extraction module 6400 is used to perform the first feature extraction processing on the original image to obtain the original feature of the original image; the fusion processing module 6500 is used to perform fusion processing according to the first fusion feature and the original feature of the original image to obtain a denoised image.

[0181] In one embodiment of the present disclosure, the N thumbnail images are arranged in order of a set multiple; the feature analysis module 6300 may further include:

[0182] The first extraction unit is used to perform a second feature extraction process on the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image:

[0183] A second extraction unit is used to perform a first feature extraction process on the nth thumbnail image to obtain an original feature of the nth thumbnail image; wherein n is a positive integer greater than 1 and less than or equal to N;

[0184] A fusion unit, used for fusing the original feature of the nth thumbnail image and the second fused feature corresponding to the n-1th thumbnail image to obtain the second fused feature corresponding to the nth thumbnail image;

[0185] The determining unit is used to use the second fusion feature corresponding to the Nth thumbnail image as the first fusion feature representing the N thumbnail images.

[0186] In one embodiment of the present disclosure, the first extraction unit may also be used to:

[0187] Performing a first feature extraction process on the first thumbnail image to obtain an original feature of the first thumbnail image;

[0188] Based on the first residual dense network, local deep feature extraction processing is performed on the original features of the first thumbnail image to obtain the local deep features of the first thumbnail image;

[0189] Based on the first residual efficient channel attention network, a global deep feature extraction process is performed on the local deep features of the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image.

[0190] In one embodiment of the present disclosure, the first feature extraction process may be performed in the following manner:

[0191] Based on the preset convolutional network, a first feature extraction process is performed on the input thumbnail image to obtain the original features of the corresponding thumbnail image.

[0192] In one embodiment of the present disclosure, the fusion unit may also be used to:

[0193] Combining the original feature of the nth thumbnail image and the second fused feature corresponding to the n-1th thumbnail image to obtain a combined feature corresponding to the nth thumbnail image;

[0194] Based on the nth residual dense network, local deep feature extraction processing is performed on the combined features corresponding to the nth thumbnail image to obtain the local deep features of the nth thumbnail image;

[0195] Based on the nth residual efficient channel attention network, a global deep feature extraction process is performed on the local deep features of the nth thumbnail image to obtain a second fusion feature corresponding to the nth thumbnail image.

[0196] In one embodiment of the present disclosure, the image processing device 6000 may further include:

[0197] The compression module is used to compress any second fusion feature.

[0198] In one embodiment of the present disclosure, the fusion processing module 6500 may also be used to:

[0199] Combining the first fused feature with the original feature of the original image to obtain a combined feature corresponding to the original image;

[0200] Based on the N+1th residual dense network, local deep feature extraction is performed on the combined features corresponding to the original image to obtain the local deep features of the original image;

[0201] Based on the N+1th residual efficient channel attention network, the local deep features of the original image are globally extracted to obtain the third fusion feature corresponding to the original image;

[0202] A denoised image is generated according to the third fusion feature.

[0203] Those skilled in the art will appreciate that the image processing device 6000 can be implemented in various ways. For example, the image processing device 6000 can be implemented by configuring a processor with instructions. For example, instructions can be stored in a ROM, and when the device is started, the instructions are read from the ROM into a programmable device to implement the image processing device 6000. For example, the image processing device 6000 can be solidified into a dedicated device (such as an ASIC). The image processing device 6000 can be divided into independent units, or they can be combined together for implementation. The image processing device 6000 can be implemented by one of the various implementations described above, or can be implemented by a combination of two or more of the various implementations described above.

[0204] In this embodiment, the image processing device 6000 can have multiple implementation forms. For example, the image processing device 6000 can be a functional module running in any software product or application that provides image processing services, or it can be an external embedded component, plug-in, patch, etc. of these software products or applications, or it can also be these software products or applications themselves.

[0205] <Electronic equipment>

[0206] In this embodiment, an electronic device 7000 is also provided. Fig.10 As shown, including:

[0207] The image processing device 6000 described above; or,

[0208] The memory 7100 and the processor 7200, the memory 7100 is used to store executable instructions; the instructions are used to control the processor 7200 to execute the aforementioned image processing method.

[0209] In this embodiment, the electronic device 7000 can be any electronic product having a memory 7100 and a processor 7200, such as a mobile phone, a tablet computer, a PDA, a desktop computer, a laptop computer, a workstation, a game console, or the like.

[0210] The embodiments of the present disclosure have been described above in conjunction with the accompanying drawings. Through the embodiments of the present disclosure, the original image is denoised according to the thumbnail image corresponding to each set magnification, and the denoised image obtained can effectively restore the details in the original image and remove the noise in the original image to improve the visual effect of the denoised image. Based on the image processing method of this embodiment, the original image with a higher compression rate can be transmitted between devices to save the bandwidth and bit rate required for transmitting the original image.

[0211] <Readable Storage Medium>

[0212] In this embodiment, a readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method of any embodiment of the present disclosure is implemented.

[0213] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0214] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0215] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0216] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0217] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0218] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0219] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0220] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and the module, a program segment or a part of an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0221] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.

Claims

1. An image processing method, characterized in that: include: Obtain the original image to be processed; According to N set multiples, downsampling the original image is performed to obtain N thumbnail images corresponding to the set multiples one by one; wherein N is a positive integer; Performing feature analysis on the N thumbnail images to obtain first fusion features representing the N thumbnail images; Performing a first feature extraction process on the original image to obtain original features of the original image; Performing fusion processing according to the first fusion feature and the original feature of the original image to obtain a denoised image, Wherein, the N thumbnail images are arranged in order of set multiples; The performing feature analysis on the N thumbnail images to obtain a first fusion feature characterizing the N thumbnail images includes: Performing a first feature extraction process on the first thumbnail image to obtain original features of the first thumbnail image; Based on the first residual dense network, performing local depth feature extraction processing on the original features of the first thumbnail image to obtain the local depth features of the first thumbnail image; Based on the first residual efficient channel attention network, performing global deep feature extraction processing on the local deep features of the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image; Performing a first feature extraction process on the nth thumbnail image to obtain an original feature of the nth thumbnail image; wherein n is a positive integer greater than 1 and less than or equal to N; Combining the original feature of the nth thumbnail image and the second fused feature corresponding to the n-1th thumbnail image to obtain a combined feature corresponding to the nth thumbnail image; Based on the nth residual dense network, performing local depth feature extraction processing on the combined features corresponding to the nth thumbnail image to obtain the local depth features of the nth thumbnail image; Based on the nth residual efficient channel attention network, perform global deep feature extraction processing on the local deep features of the nth thumbnail image to obtain a second fusion feature corresponding to the nth thumbnail image; The second fusion feature corresponding to the Nth thumbnail image is used as the first fusion feature representing the N thumbnail images.

2. The method according to claim 1, characterized in that The first feature extraction process is performed in the following manner: Based on the preset convolutional network, a first feature extraction process is performed on the input thumbnail image to obtain the original features of the corresponding thumbnail image.

3. The method according to claim 1, characterized in that: After obtaining the second fusion feature, the method further includes: The second fused features are compressed.

4. The method according to claim 1, characterized in that: The step of performing fusion processing according to the first fusion feature and the original feature of the original image to obtain a denoised image comprises: Combining the first fused feature with the original feature of the original image to obtain a combined feature corresponding to the original image; Based on the N+1th residual dense network, local depth feature extraction processing is performed on the combined features corresponding to the original image to obtain the local depth features of the original image; Based on the N+1th residual efficient channel attention network, perform global deep feature extraction processing on the local deep features of the original image to obtain a third fusion feature corresponding to the original image; The denoised image is generated according to the third fusion feature.

5. An image processing device, characterized in that: include: An image acquisition module, used to acquire the original image to be processed; A downsampling module, used to downsample the original image according to N set multiples to obtain N thumbnail images corresponding to the set multiples; wherein N is a positive integer; A feature analysis module, used for performing feature analysis on the N thumbnail images to obtain a first fusion feature representing the N thumbnail images; A feature extraction module, used for performing a first feature extraction process on the original image to obtain original features of the original image; a fusion processing module, configured to perform fusion processing according to the first fusion feature and the original feature of the original image to obtain a denoised image, Wherein, the N thumbnail images are arranged in order of set multiples; The feature analysis module also includes: A first extraction unit is used to perform a first feature extraction process on the first thumbnail image to obtain the original features of the first thumbnail image; based on the first residual dense network, perform a local depth feature extraction process on the original features of the first thumbnail image to obtain the local depth features of the first thumbnail image; based on the first residual efficient channel attention network, perform a global depth feature extraction process on the local depth features of the first thumbnail image to obtain a second fusion feature corresponding to the first thumbnail image; A second extraction unit is used to perform a first feature extraction process on the nth thumbnail image to obtain an original feature of the nth thumbnail image; wherein n is a positive integer greater than 1 and less than or equal to N; A fusion unit is used to combine the original features of the nth thumbnail image and the second fused features corresponding to the n-1th thumbnail image to obtain the combined features corresponding to the nth thumbnail image; based on the nth residual dense network, perform local depth feature extraction processing on the combined features corresponding to the nth thumbnail image to obtain the local depth features of the nth thumbnail image; based on the nth residual efficient channel attention network, perform global depth feature extraction processing on the local depth features of the nth thumbnail image to obtain the second fused features corresponding to the nth thumbnail image; The determining unit is configured to use the second fusion feature corresponding to the Nth thumbnail image as the first fusion feature representing the N thumbnail images.

6. An electronic device, characterized in that: include: The device as claimed in claim 5; or, A processor and a memory, wherein the memory is used to store instructions, and the instructions are used to control the processor to execute the method according to any one of claims 1 to 4.

7. A readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 4 when executed by a processor.

Images