Image interference removing method and device, electronic equipment and computer readable storage medium

Abstract

The application provides an image interference removing method and device, electronic equipment and computer readable storage medium. The method comprises: performing image decomposition on a to-be-processed image carrying interference information to obtain at least two target images; performing feature extraction on the to-be-processed image through a first feature extraction layer of an image interference removing model to obtain image features of the to-be-processed image; performing interference feature extraction on each target image through a second feature extraction layer of the image interference removing model to obtain corresponding interference features; performing fusion processing on the interference features of each target image through a feature fusion layer of the image interference removing model to obtain fused interference features; and performing interference removing processing on the to-be-processed image based on the fused interference features and the image features through an output layer of the image interference removing model to obtain a corresponding interference removed image. Through the application, interference information in the image can be accurately removed.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image de-interference method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] During digital transmission, images are easily affected by interference from imaging equipment or the external environment, resulting in interference such as image noise and jagged edges. To obtain images with better quality, it is usually necessary to perform interference removal processing.

[0003] The image denoising process used in related technologies involves directly inputting the image to be denoised into a pre-trained machine learning model, which then performs the denoising. However, this approach relies on a very limited amount of image information available to the machine learning model, resulting in inaccurate removal of denoising information based on this limited information. Summary of the Invention

[0004] This application provides an image de-interference method, apparatus, electronic device, and computer-readable storage medium that can accurately remove interference information from images.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image de-interference method, including:

[0007] The image to be processed carrying interference information is decomposed to obtain at least two target images;

[0008] The first feature extraction layer of the image de-interference model is used to extract features from the image to be processed, thereby obtaining the image features of the image to be processed.

[0009] The second feature extraction layer of the image de-interference model extracts interference features from each target image to obtain the corresponding interference features.

[0010] The feature fusion layer of the image de-interference model is used to fuse the interference features of each target image to obtain fused interference features.

[0011] The image to be processed is processed by the output layer of the image de-interference model based on the fused interference features and the image features to obtain the corresponding de-interference image.

[0012] This application provides an image interference removal device, including:

[0013] The image decomposition module is used to decompose the image to be processed carrying interference information to obtain at least two target images;

[0014] The image feature extraction module is used to extract features from the image to be processed through the first feature extraction layer of the image de-interference model to obtain the image features of the image to be processed.

[0015] The interference feature extraction module is used to extract interference features from each of the target images through the second feature extraction layer of the image de-interference model to obtain the corresponding interference features.

[0016] The feature fusion module is used to fuse the interference features of each target image through the feature fusion layer of the image de-interference model to obtain fused interference features;

[0017] The interference removal module is used to perform interference removal processing on the image to be processed based on the fused interference features and the image features through the output layer of the image interference removal model, so as to obtain the corresponding interference removal image.

[0018] In the above scheme, the image decomposition module is further used to perform multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed; and to use the image of each layer in the Gaussian pyramid as the target image.

[0019] In the above scheme, the image decomposition module is further configured to perform multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed; based on the Gaussian pyramid corresponding to the image to be processed, construct a Laplacian pyramid corresponding to the image to be processed; combine the images of each layer in the Gaussian pyramid with the images of the corresponding levels in the Laplacian pyramid to obtain at least two image combinations; and use the at least two image combinations as the target image.

[0020] In the above scheme, the image decomposition module is further configured to perform multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed; based on the Gaussian pyramid corresponding to the image to be processed, construct a Laplacian pyramid corresponding to the image to be processed; perform downsampling processing on the images of each layer in the Gaussian pyramid to obtain corresponding downsampled images; combine the images of each layer in the Gaussian pyramid, the images of the corresponding level in the Laplacian pyramid, and the downsampled images of the same size as the images of the corresponding level in the Laplacian pyramid to obtain at least two image combinations; and use the at least two image combinations as the target image.

[0021] In the above scheme, the second feature extraction layer includes N sub-feature extraction layers, the number of target images is N, and N is a positive integer not less than 2; the interference feature extraction module is further used to input each target image into one of the sub-feature extraction layers, and to extract interference features from the target image through the sub-feature extraction layers to obtain the corresponding interference features.

[0022] In the above scheme, the interference features include size features and channel features. The feature fusion module is further used to perform size transformation on the interference features of each target image to obtain target interference features corresponding to each interference feature; wherein, the size features of each target interference feature are the same; the channel features of each target interference feature are fused to obtain fused channel features; and the size features of the target interference features are concatenated with the fused channel features to obtain the fused interference features.

[0023] In the above scheme, when the number of target images is m, the corresponding number of interference features is m. The feature fusion module is further used to fuse the j-th interference feature with the (j+1)-th interference feature among the m interference features to obtain the j-th fused interference feature; where m is a positive integer not less than 3, j is a positive integer, and j∈[1, m-1]; the j-th fused interference feature is fused with the (j+2)-th interference feature to obtain the (j+1)-th fused interference feature; starting with j as 1, j is traversed, and when the value of j+2 is the same as the value of m, the (j+1)-th fused interference feature is taken as the fused interference feature.

[0024] In the above scheme, the interference removal module is further configured to determine the difference feature between the image feature and the fused interference feature through the output layer of the image interference removal model; and to decode the difference feature through the output layer of the image interference removal model to obtain the interference-removed image.

[0025] In the above scheme, the image de-interference device further includes: a model training module, used to decompose the sample interference image carrying interference information to obtain at least two sample target images; wherein, the sample interference image is obtained by adding interference information to the original sample image; through the first feature extraction layer of the image de-interference model, the sample interference image is used to extract features to obtain sample image features of the sample interference image; through the second feature extraction layer of the image de-interference model, interference features are extracted from each of the sample target images to obtain corresponding sample interference features; through the feature fusion layer of the image de-interference model, the sample interference features corresponding to each of the sample target images are fused to obtain sample fused interference features; through the output layer of the image de-interference model, based on the sample fused interference features and the sample image features, the sample interference image is used to de-interference the sample interference image to obtain a corresponding predicted de-interference image; based on the difference between the predicted de-interference image and the original sample image, the model parameters of the image de-interference model are updated.

[0026] In the above scheme, the original sample image corresponds to at least two sample interference images. Each sample interference image is obtained by adding interference information to the original sample image. Accordingly, the image decomposition module is also used to perform the operations of image decomposition of the sample interference image carrying interference information and the de-interference processing of the sample interference image for each sample interference image, to obtain the predicted de-interference image corresponding to each sample interference image. Accordingly, the model training module is also used to update the model parameters of the image de-interference model based on the differences between each predicted de-interference image and the original sample image.

[0027] In the above scheme, the interference information includes at least one of image jagged edges and image noise; correspondingly, the image de-interference device further includes: an interference information adding module, used to add image jagged edges to the original sample image to obtain a sample jagged image carrying image jagged edges, and use the sample jagged image as the sample interference image; or add image noise to the original sample image to obtain a sample noise image carrying image noise, and use the sample noise image as the sample interference image; or add image jagged edges and image noise to the original sample image to obtain a sample jagged noise image carrying both image jagged edges and image noise, and use the sample jagged noise image as the sample interference image.

[0028] This application provides a method for training an image de-interference model, including:

[0029] The sample interference image carrying interference information is decomposed to obtain at least two sample target images; wherein, the sample interference image is obtained by adding interference information to the original sample image;

[0030] The first feature extraction layer of the image de-interference model is used to extract features from the sample interference image to obtain the sample image features of the sample interference image.

[0031] The second feature extraction layer of the image de-interference model is used to extract interference features from each of the sample target images to obtain the corresponding sample interference features.

[0032] The feature fusion layer of the image de-interference model is used to fuse the sample interference features corresponding to each sample target image to obtain sample fused interference features.

[0033] The image de-interference model outputs a de-interference image based on the sample fusion interference features and the sample image features to obtain a corresponding predicted de-interference image.

[0034] Based on the difference between the predicted de-interference image and the original sample image, the parameters of the first feature extraction layer, the second feature extraction layer, the feature fusion layer, and the output layer are updated respectively.

[0035] This application provides an electronic device, including:

[0036] Memory, used to store executable instructions;

[0037] The processor, when executing executable instructions stored in the memory, implements the image de-interference method or the image de-interference model training method provided in the embodiments of this application.

[0038] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the image de-interference method or the image de-interference model training method provided in this application.

[0039] The embodiments of this application have the following beneficial effects:

[0040] This application embodiment decomposes the image to be processed into multiple target images. It can be understood that each target image contains image information of the image to be processed. As the image to be processed is decomposed, the image information of the image to be processed is also decomposed into multiple representations. Thus, the image information of the image to be processed can be represented more clearly and comprehensively through multiple target images. Then, the interference features of each target image are extracted using an image interference removal model, and the interference features are fused. The resulting fused interference features cover more and more accurate interference information in the image to be processed, thereby enabling the accurate removal of interference information in the image to be processed based on the fused interference features. Attached Figure Description

[0041] Figure 1 This is an optional structural schematic diagram of the image interference removal system provided in the embodiments of this application;

[0042] Figure 2 This is an optional structural schematic diagram of the electronic device provided in an embodiment of this application;

[0043] Figure 3 This is an optional flowchart illustrating the image interference removal method provided in this application embodiment;

[0044] Figure 4 This is an optional flowchart illustrating the image interference removal method provided in this application embodiment;

[0045] Figure 5 This is an optional structural diagram of the Gaussian pyramid provided in an embodiment of this application;

[0046] Figure 6 This is an optional schematic diagram of the downsampling process provided in an embodiment of this application;

[0047] Figure 7 This is an optional flowchart illustrating the image interference removal method provided in this application embodiment;

[0048] Figure 8 This is an optional schematic diagram illustrating the construction process of the Gaussian pyramid and the Laplace pyramid provided in the embodiments of this application;

[0049] Figure 9 This is an optional structural diagram of the Gaussian pyramid and the corresponding Laplace pyramid provided in the embodiments of this application;

[0050] Figure 10 This is an optional schematic diagram of the upsampling process provided in an embodiment of this application;

[0051] Figure 11 This is an optional schematic diagram of the image combination pyramid provided in the embodiments of this application;

[0052] Figure 12 This is an optional flowchart illustrating the image interference removal method provided in this application embodiment;

[0053] Figure 13 This is an optional schematic diagram of the downsampled image pyramid provided in an embodiment of this application;

[0054] Figure 14 This is an optional schematic diagram of the image combination pyramid provided in the embodiments of this application;

[0055] Figure 15 This is an optional schematic diagram of the downsampling process provided in an embodiment of this application;

[0056] Figure 16 This is an optional schematic diagram of the image combination pyramid provided in the embodiments of this application;

[0057] Figure 17 This is an optional schematic diagram of the image combination pyramid provided in the embodiments of this application;

[0058] Figure 18 This is an optional structural diagram of the image de-interference model provided in an embodiment of this application;

[0059] Figure 19 This is an optional structural diagram of the image de-interference model provided in an embodiment of this application;

[0060] Figure 20 This is an optional flowchart illustrating the image interference removal method provided in this application embodiment;

[0061] Figure 21 This is an optional schematic diagram of the image features provided in an embodiment of this application;

[0062] Figure 22 This is an optional flowchart illustrating the image interference removal method provided in this application embodiment;

[0063] Figure 23 This is an optional schematic diagram of the image de-interference process provided in the embodiments of this application;

[0064] Figure 24 This is an optional flowchart illustrating the training method of the image de-interference model provided in the embodiments of this application;

[0065] Figure 25 This is an optional schematic diagram of the sample set provided in the embodiments of this application;

[0066] Figure 26 This is an optional flowchart illustrating the image interference removal method provided in this application embodiment;

[0067] Figure 27 This is an optional schematic diagram of the image interference removal device provided in the embodiments of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0070] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0072] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0073] 1) Interference information refers to unnecessary or redundant information in image data that affects image quality, such as image noise and jagged edges.

[0074] 2) Image decomposition: The original image is decomposed into sub-images with different image features. These image features can be the image's structure, texture, noise, jagged edges, scale, etc.

[0075] Image scale includes spatial scale, temporal scale, and semantic scale. Spatial scale includes cartographic scale, geographic scale, operational scale, and resolution measurement scale. In the field of computer vision, image scale generally refers to resolution measurement scale.

[0076] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0077] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0078] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0079] The solutions provided in this application involve technologies such as machine learning in artificial intelligence, and are specifically illustrated through the following embodiments.

[0080] The inventors discovered that when image de-interference is performed by inputting the image to be de-interferenced into a machine learning model, downsampling the image to extract image features, then deconvolving the image features to enlarge their size to obtain interference information of the same size as the input image, and finally subtracting the image from the interference information to obtain the predicted de-interferenced image, the image features are prone to jagged edges during the upsampling process, resulting in poor image de-aliasing performance.

[0081] Based on this, the inventors discovered that by using dilated convolution instead of downsampling for image feature extraction, the receptive field can be enlarged while avoiding upsampling. However, dilated convolution introduces holes into the generated image, which is detrimental to image de-interference and results in poor image de-interference performance.

[0082] Based on this, embodiments of this application provide an image de-interference method, apparatus, electronic device, and computer-readable storage medium, which can accurately remove interference information from images.

[0083] First, the image interference removal system provided in the embodiments of this application will be described. Figure 1 This is an optional schematic diagram of the image de-interference system 100 provided in this application embodiment. To support an image de-interference application, an image de-interference client 410 is provided on the terminal 400. The terminal 400 is connected to the server 200 through a network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using a wireless link for data transmission. In some embodiments, the terminal 400 can be a laptop, tablet, desktop computer, smartphone, dedicated messaging device, portable gaming device, smart speaker, smartwatch, etc., but is not limited to these. The server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. The terminal 400 and the server 200 can be directly or indirectly connected through wired or wireless communication methods, which is not limited in this application embodiment.

[0084] Terminal 400 is used to, in response to an upload operation for an image to be processed, acquire the image to be processed, and in response to a de-interference instruction for the image to be processed, send the image to be processed to the server.

[0085] Server 200 is used to decompose the image to be processed into at least two target images, input the image to be processed and the at least two target images into an image de-interference model, extract features from the image to be processed through the first feature extraction layer of the image de-interference model to obtain image features of the image to be processed, extract interference features from each target image through the second feature extraction layer of the image de-interference model to obtain corresponding interference features, fuse the interference features of each target image through the feature fusion layer of the image de-interference model to obtain fused interference features, and perform de-interference processing on the image to be processed based on the fused interference features and the image features through the output layer of the image de-interference model to obtain a corresponding de-interference image, and send the de-interference image to terminal 400.

[0086] Terminal 400 is also used to receive and output the de-interference image sent by server 200.

[0087] The electronic device for implementing the above-described image de-interference method, provided in the embodiments of this application, will now be described. See [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. In practical applications, the electronic device 500 can be implemented as follows: Figure 1 Terminal 400 or server 200 in the middle, with electronic devices as Figure 1 Taking the server 200 shown as an example, an electronic device implementing the image de-interference method of this application will be described. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0088] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0089] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0090] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0091] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0092] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0093] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0094] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0095] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0096] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0097] In some embodiments, the image interference removal device provided in this application can be implemented in software. Figure 2 An image de-interference device 555 stored in memory 550 is shown. This device can be software in the form of programs or plug-ins, and includes the following software modules: image decomposition module 5551, image feature extraction module 5552, interference feature extraction module 5553, feature fusion module 5554, and de-interference module 5555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0098] In other embodiments, the image de-interference device provided in this application can be implemented in hardware. As an example, the image de-interference device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image de-interference method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0099] Next, the image de-interference method provided in this application embodiment will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0100] See Figure 3 , Figure 3 This is an optional flowchart illustrating an image de-interference method provided in an embodiment of this application, which will be combined with... Figure 3 The steps shown are explained.

[0101] Step 101: The server performs image decomposition on the image to be processed carrying interference information to obtain at least two target images.

[0102] Here, each target image possesses different image features. The server decomposes the set of images to be processed to obtain multiple target images with different image features. Here, the target images may include the images to be processed. In practice, the server can process multiple pixels constituting the image to be processed to obtain at least two target images. Specifically, the server can select multiple target pixels from the image to be processed and construct a single target image from these multiple target pixels.

[0103] In some embodiments, see Figure 4 , Figure 4 This is an optional flowchart illustrating an image de-interference method provided in an embodiment of this application, based on... Figure 3 Step 101 can also be achieved in the following way:

[0104] Step 201: The server performs multi-scale image decomposition on the image to be processed carrying interference information in order to construct a Gaussian pyramid for the corresponding image to be processed.

[0105] Step 202: Use the image of each layer in the Gaussian pyramid as the target image.

[0106] In practice, the server can perform at least one image decomposition on the image to be processed to obtain a target image at a different scale. Then, a Gaussian pyramid is constructed based on the image to be processed and the target image. The resulting Gaussian pyramid contains images at two different scales, namely the image to be processed and the target image. (Refer to...) Figure 5 , Figure 5 This is an optional structural diagram of a Gaussian pyramid provided in this application embodiment. It includes i+2 layers of images from layer 0 to layer i+1, where i is an integer greater than or equal to 0. The specific process of constructing the Gaussian pyramid in this application embodiment is as follows: The server uses the image to be processed I0 as the bottom layer image of the Gaussian pyramid, i.e., the 0th layer image. A Gaussian kernel is used to convolve the image to be processed I0 to obtain the Gaussian image G(I0) after convolution. Then, the Gaussian image G(I0) is downsampled to obtain the downsampled image down_sample(G(I0)). The downsampled image is used as the next layer image I1 of the image to be processed I0 in the Gaussian pyramid, i.e., I1 = down_sample(G(I0)). The image I1 of the (i+1)th layer of the Gaussian pyramid is obtained in the same way. i+1 =down_sample(G(I i )).

[0107] Furthermore, the downsampling process involved in constructing the Gaussian pyramid described above specifically involves: extracting pixels from the original image whose number of rows and columns is odd; constructing a new image based on the extracted pixels; and using the constructed image as the downsampled image. For example, refer to... Figure 6 , Figure 6 This is an optional schematic diagram of the downsampling process provided in this application embodiment. In this process, pixels in the original image P with an odd number of rows and columns are marked as 1, and the remaining pixels are marked as 0. Through the downsampling process described above in this application embodiment, pixels with an odd number of rows and columns are extracted from the original image P. Then, an image is constructed based on these pixels, namely the downsampled image down_sample(P). It can be understood that the size of the downsampled image obtained through this downsampling method is 1 / 4 of the original image. That is, the size of the image at the top layer of the Gaussian pyramid is 1 / 4 of the size of the image at the bottom layer, i.e., I... i+1 The size is I i 1 / 4. For example, Figure 5The size of the 0th layer image I0 in the Gaussian pyramid shown is (W, H), where W is the image width and H is the image height. Therefore, the size of the 1st layer image I1 is (W / 2, H / 2), meaning the size of I1 is W / 2 * H / 2 = 1 / 4 W * H, which is 1 / 4 of I0. Correspondingly, the size of the i-th layer image I... i The dimensions are (W / 2) i H / 2 i ),

[0108] In practice, the server uses each layer of the constructed Gaussian pyramid as the target image, which is then input into the image denoising model to denoise the image to be processed. In this way, the image to be processed is decomposed into target images at multiple scales. The target images are low-frequency signals of the image to be processed at different scales, and contain image information of different intensities. For example, images at higher levels in the Gaussian pyramid have lower image resolution, weaker detail information, more interference information, and stronger contour information than images at lower levels. This allows the image denoising model to learn image information at multiple scales, thereby enabling more accurate denoising and resulting in higher quality denoised images.

[0109] In some embodiments, see Figure 7 , Figure 7 This is an optional flowchart illustrating an image de-interference method provided in an embodiment of this application, based on... Figure 3 Step 101 can also be achieved in the following way:

[0110] Step 301: The server performs multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid for the corresponding image to be processed.

[0111] Step 302: Based on the Gaussian pyramid of the corresponding image to be processed, construct the Laplacian pyramid of the corresponding image to be processed.

[0112] Reference Figure 8 and Figure 9 , Figure 8 This is an optional schematic diagram illustrating the construction process of the Gaussian pyramid and the Laplace pyramid provided in the embodiments of this application. Figure 9This is an optional structural diagram of the Gaussian pyramid and the corresponding Laplace pyramid provided in the embodiments of this application. In actual implementation, the process of constructing the Laplace pyramid is as follows: The server upsamples the image I0 of the 0th layer of the Gaussian pyramid to obtain the upsampled image up_sample(I0), and then performs a convolution operation on the upsampled image using a Gaussian kernel to obtain the image G'(up_sample(I0)). The structure of the Gaussian kernel used here is the same as that used when constructing the Gaussian pyramid, but the value is 4 times that of the Gaussian kernel used when constructing the Gaussian pyramid. Then, the server obtains the image L0 of the 0th layer of the Laplace pyramid based on the image I0 of the 0th layer of the Gaussian pyramid and the image G'(up_sample(I0)). Here, the image L0 is obtained by subtracting the image G'(up_sample(I0)) from the image I0 of the 0th layer of the Gaussian pyramid, that is, L0 = I0 - G'(up_sample(I0)). In the same way, the image L of the i-th layer of the Laplace pyramid is obtained. i =I i -G'(up_sample(I i+1 ), where i is an integer greater than or equal to 0. It can be understood that each layer of the Laplacian pyramid is a residual image, which is a high-frequency signal of the image to be processed at different scales. The number of layers in the Laplacian pyramid is one less than the number of layers in the corresponding Gaussian pyramid, and the Laplacian image and the Gaussian image of the same level have the same size.

[0113] Furthermore, the upsampling process involved in constructing the Laplacian pyramid described above can specifically be: Upsampling the image I in the Gaussian pyramid... i Add new pixels, double the number of pixel rows and columns, and insert the newly added pixels into image I at intervals. i The pixels in each row are spaced apart, and the newly added columns of pixels are inserted into image I at intervals. i Between each column of pixels, the newly added pixels are then filled with channel values, for example, all of them can be filled with the channel value corresponding to white. For example, see [link to example]. Figure 10 , Figure 10 This is an optional schematic diagram of the upsampling process provided in the embodiments of this application. The server is in image I i A new row of pixels is added below each row of pixels, and a new column of pixels is added to the right of each column of pixels. Here, we take a channel value of 0 as an example for filling in the new pixels, resulting in an upsampled image up_sample(I i )like Figure 10 As shown, its size is enlarged to image I iFour times that of image I. In some embodiments, the server can also be in image I i Add a new row of pixels below each row of pixels and a new column of pixels to the left of each column of pixels, or in image I i Add a new row of pixels above each row of pixels and a new column of pixels to the left of each column of pixels, or in image I i Upsampling is performed by adding a row of pixels above each row of pixels and adding a column of pixels to the right of each column of pixels, or by adding pixels in the form of multiple rows or columns. This application does not specifically limit the upsampling method.

[0114] Step 303: Combine the images of each layer in the Gaussian pyramid with the corresponding images of the Laplace pyramid to obtain at least two image combinations.

[0115] Step 304: Combine at least two images as the target image.

[0116] See Figure 11 , Figure 11 This is an optional schematic diagram of the image combination pyramid provided in the embodiments of this application. Here, the image combination can be represented by a set. For example, the server combines the i-th layer image I in the Gaussian pyramid. i Image L of the i-th layer in the Laplace pyramid i By combining the images, we obtain image combination B. i ={I i ,L i In actual implementation, when the server performs image processing, it combines the images into B. i The images are processed in the form of image blocks. An image block is formed by superimposing two or more images. The size of an image block includes three dimensions: width, height, and the number of images. In this embodiment, the number of images is represented by the form a × b, where a is the number of images constituting the image block, and b is the number of channels of a pixel in a single image. For example, if the image is an RGB (Red, Green, Blue) image, the number of channels of the pixels in the image is 3, namely red, green, and blue channels. If the image is an RGGB (Red, Green, Green, Blue) image, the number of channels of the pixels in the image is 4, namely red, green, green, and blue channels, and so on. In this embodiment, the number of channels is described as 3. Figure 11 In the image combination shown, image combination B i The size characteristics are (W / 2) i H / 2 i ,2×3).

[0117] After the server combines the images of each level in the Gaussian pyramid with the corresponding images of the Laplacian pyramid, the resulting image combinations are all of different sizes, forming an image pyramid, for example... Figure 11 The image combination pyramid is shown. Since the Laplacian pyramid has one less layer than the Gaussian pyramid, the server, when performing image combination, uses the last layer image of the Gaussian pyramid, i.e., the topmost image I. i+1 Each image is treated as a separate image combination. It can be understood that in this embodiment, the server combines the Gaussian pyramid (containing layers 0 to i+1, a total of i+2 layers) with its corresponding Laplacian pyramid to obtain an image combination pyramid containing layers 0 to i+1, a total of i+2 layers, where the i-th layer image combination B... i ={I i ,L i}, the image combination B of the (i+1)th layer i+1 ={I i+1}

[0118] In practice, the server combines each image as a target image and inputs it into the image de-interference model to perform de-interference processing on the image to be processed. In this way, the image to be processed is decomposed into a Gaussian pyramid containing images of multiple scales and a Laplacian pyramid containing images of multiple scales. Then, the Gaussian pyramid and the Laplacian pyramid are combined, and the combined image is used as the target image. This allows the image processing based on the target image to take into account both low-frequency and high-frequency signals in the image to be processed, and remove the interference information in the low-frequency and high-frequency signals at the same time, thereby obtaining a de-interference image with higher image quality.

[0119] In some embodiments, see Figure 12 , Figure 12 This is an optional flowchart illustrating an image de-interference method provided in an embodiment of this application, based on... Figure 3 Step 101 can also be achieved in the following way:

[0120] Step 401: The server performs multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid for the corresponding image to be processed.

[0121] Step 402: Based on the Gaussian pyramid of the corresponding image to be processed, construct the Laplacian pyramid of the corresponding image to be processed.

[0122] Step 403: Downsample the images of each layer in the Gaussian pyramid to obtain the corresponding downsampled images.

[0123] Reference Figure 13 , Figure 13 This is an optional schematic diagram of the downsampled image pyramid provided in this application embodiment. Here, the server directly downsamples the images of each layer in the Gaussian pyramid to obtain downsampled images corresponding to each layer. In actual implementation, the server downsamples the images of each layer in the Gaussian pyramid except the top layer to obtain the corresponding downsampled images. For example, the server downsamples the images of layers 0 to 1 (i+1 layers in total) of the Gaussian pyramid with i+2 layers. i The downsampling process is performed to obtain the corresponding downsampling images ds(I0), ds(I1), ..., ds(I2). i It should be understood that the size of the downsampled image ds(I0) is smaller than that of the image I0 before downsampling; specifically, the size of the downsampled image ds(I0) is 1 / 4 of that of I0.

[0124] Step 404: Combine the images of each layer in the Gaussian pyramid, the corresponding layer in the Laplacian pyramid, and the downsampled image of the same size as the corresponding layer in the Laplacian pyramid to obtain at least two image combinations.

[0125] Step 405: Combine at least two images as the target image.

[0126] See Figure 14 , Figure 14 This is an optional schematic diagram of the image composite pyramid provided in the embodiments of this application. In actual implementation, the server will combine the i-th layer image I from the Gaussian pyramid. i Image L of the i-th layer in the Laplace pyramid i and with image I i The downsampled images of the same size are combined, here, with image I. i Image I is a downsampled image of the same size. i The downsampled image corresponding to the next layer in the Gaussian pyramid, i.e., the (i-1)th layer image I. i-1 The corresponding downsampled image ds(I) i-1 In other words, the server will send image I i Image L i and downsampled image ds(I i-1 Combine them to obtain the corresponding image combination B. i ={I i ,L i ,ds(I i-1 )}.

[0127] In this embodiment, based on the Gaussian pyramid and the Laplacian pyramid, the Gaussian pyramid is further downsampled to perform image decomposition, resulting in a downsampled image pyramid. Thus, the combined image pyramid obtained by combining the three has more dimensional image information, which makes the model training based on it perform better, thereby obtaining a higher quality de-interference image.

[0128] In some embodiments, based on Figure 12 Step 403 can also be achieved in the following way:

[0129] The server performs the following processing on each layer of the Gaussian pyramid image: determining the row and column codes of pixels in the Gaussian image; selecting multiple first pixels with odd row and column codes from the multiple pixels constituting the Gaussian image, and constructing a corresponding first downsampled image based on the multiple first pixels; selecting multiple second pixels with odd row and even column codes from the multiple pixels constituting the Gaussian image, and constructing a corresponding second downsampled image based on the multiple second pixels; selecting multiple third pixels with even row and odd column codes from the multiple pixels constituting the Gaussian image, and constructing a corresponding third downsampled image based on the multiple third pixels; selecting multiple fourth pixels with even row and even column codes from the multiple pixels constituting the Gaussian image, and constructing a corresponding fourth downsampled image based on the multiple fourth pixels; and using the first, second, third, and fourth downsampled images as the downsampled images.

[0130] For example, refer to Figure 15 and Figure 16 , Figure 15 This is an optional schematic diagram of the downsampling process provided in an embodiment of this application. Figure 16 This is an optional schematic diagram of a downsampled image pyramid provided in an embodiment of this application. In this embodiment, the first pixel is recorded as 0, the second pixel as 1, the third pixel as 2, and the fourth pixel as 3, respectively. Then, the i-th layer image I of the Gaussian pyramid... i The pixels can be represented as Figure 15 As shown, the server obtains the first to fourth downsampled images ds0(I) using the four downsampling methods described above. i ), ds1(I i ), ds2(I i ) and ds3(I i The pixels in the image are 0, 1, 2, and 3 respectively, and the size of the four downsampled images is the same as that of image I. i 1 / 4 of it.

[0131] In practice, the server performs downsampling processing on the image (excluding the top layer) of the Gaussian pyramid using the four methods described above, resulting in four corresponding downsampled image pyramids: the first downsampled image pyramid, the second downsampled image pyramid, the third downsampled image pyramid, and the fourth downsampled image pyramid. See also... Figure 17 , Figure 17 This is an optional schematic diagram of an image composite pyramid provided in an embodiment of this application. The image composite pyramid includes images from a Gaussian pyramid, a Laplace pyramid, and four downsampled image pyramids.

[0132] In this embodiment, by performing the four downsampling processes described above on the Gaussian pyramid, one layer of the image in the Gaussian pyramid is decomposed into four downsampled images. The pixels of these four downsampled images together constitute the pixels of the corresponding image in the Gaussian pyramid, so that all pixel information in the Gaussian pyramid is retained in the four downsampled images. This results in the image combination pyramid based on the Gaussian pyramid, the Laplacian pyramid, and the pyramids of the four downsampled images having more and more complete original image information of the image to be processed.

[0133] Step 102: The first feature extraction layer of the image de-interference model is used to extract features from the image to be processed, thereby obtaining the image features of the image to be processed.

[0134] See Figure 18 , Figure 18 This is a schematic diagram of an optional structure of the image de-interference model provided in an embodiment of this application. Here, the image de-interference model is a residual network model, wherein both the first and second feature extraction layers consist of a series of convolutional layers. Each convolutional layer can adopt a convolutional structure of a Residual Network (ResNet), Densely Connected Networks (DENZ), SERESNet, or Inception Network. In this embodiment, the convolutional operation performed by the first feature extraction layer is a long-term convolutional process (LTCP), and the convolutional operation performed by the second feature extraction layer is a short-term convolutional process (STCP). The feature fusion layer also consists of a series of convolutional layers, which fuse two or more features through convolutional operations.

[0135] In practice, after the server decomposes the image to be processed into at least two target images, it inputs both the image to be processed and the two target images into the image denoising model. Specifically, the server inputs the image to be processed into the first feature extraction layer of the image denoising model and the target images into the second feature extraction layer. Then, the first feature extraction layer of the image denoising model extracts features from the image to be processed, obtaining the image features of the image to be processed. Here, the image features include both the image information of the image to be processed and the interference information.

[0136] Step 103: Through the second feature extraction layer of the image de-interference model, interference features are extracted from each target image to obtain the corresponding interference features.

[0137] In practice, the server inputs each target image into the image de-interference model to extract the interference features of the target image. Here, the interference features are the encoded representation of the interference information present in the target image. In this embodiment, the image de-interference model extracts the interference information in the target image by learning the residual of the target image, thus obtaining the corresponding interference features.

[0138] In some embodiments, the second feature extraction layer includes N sub-feature extraction layers, where the number of target images is N, and N is a positive integer not less than 2. Based on Figure 3 Step 103 can also be implemented as follows: The server inputs each target image into a sub-feature extraction layer, and extracts interference features from the target image through the sub-feature extraction layer to obtain the corresponding interference features.

[0139] See Figure 19 , Figure 19 This is a schematic diagram of an optional structure of the image de-interference model provided in this application embodiment. It should be noted that the structures of each sub-feature extraction layer may not be completely identical, and even for two sub-feature extraction layers with identical structures, their parameters may differ. In actual implementation, the server inputs each target image into a separate sub-feature extraction layer, and extracts different target images through different sub-feature extraction layers. Since each target image obtained by image decomposition of the image to be processed in this application embodiment possesses different scale information, extracting target images of different scales through sub-feature extraction layers with different structures or parameters can specifically extract interference features of different scales, thereby improving the accuracy of feature extraction.

[0140] Step 104: The interference features of each target image are fused through the feature fusion layer of the image de-interference model to obtain fused interference features.

[0141] In practice, the server performs feature fusion processing on the interference features of all target images, fusing the interference features corresponding to each target image into a single fused interference feature. This fused interference feature can be understood as including interference information from all target images. Specifically, the server performs convolution operations on each interference feature through the feature fusion layer of the image de-interference model, fusing all interference features into a single fused interference feature.

[0142] In some embodiments, the interference features include size features and channel features. See also Figure 20 , Figure 20 This is an optional flowchart illustrating an image de-interference method provided in an embodiment of this application, based on... Figure 3 Step 104 can also be achieved in the following way:

[0143] Step 501: The server performs size transformation on the interference features of each target image through the feature fusion layer of the image de-interference model to obtain the target interference features corresponding to each interference feature; wherein, the size features of each target interference feature are the same.

[0144] It should be noted that, in the embodiments of this application, the image includes information in two dimensions: size and channels. That is, the image features of the image include size features and channel features. Here, the size features include width features and height features, for example, Figure 11 Image combination B shown i The size characteristics are (W / 2) i H / 2 i W / 2 i For width features, H / 2 i For height features. The channel features of an image include the sub-channel features corresponding to each pixel in the image. These sub-channel features include the channel value of each monochrome channel of the pixel in the color mode. Here, the color mode can be RGB (Red, Green, Blue) color mode, so the sub-channel features of a pixel include the channel values ​​of three channels: red, green, and blue. For example, refer to... Figure 21 , Figure 21 This is an optional schematic diagram of the image features provided in an embodiment of this application. Figure 21 The image shown has a width feature of W, a height feature of H, a sub-channel feature of (0,0,0) for the first row and first column of pixels, a sub-channel feature of (56,56,56) for the first row and second column of pixels, a sub-channel feature of (56,56,56) for the second row and first column of pixels, a sub-channel feature of (255,255,255) for the second row and first column of pixels, and so on.

[0145] In this embodiment, the feature dimension of the interference feature image is consistent with the corresponding target image, and it also has size and channel features. Furthermore, the size feature of the interference feature is consistent with the size feature of the corresponding target image. For example, if the size feature of the target image is (W / 2... i H / 2 i If the size of the interference feature obtained after extracting the interference feature from the target image is (W / 2×3), then the size of the interference feature is also (W / 2). i H / 2 i (2×3). The channel features of the interference features correspond to the channel features corresponding to the interference information in the target image.

[0146] In practice, the target images differ in scale and size characteristics. During feature fusion, the server first uses the feature fusion layer of the image de-interference model to perform size transformation on the interference features of each target image, ensuring that the size characteristics of the interference features corresponding to each target image are consistent. In this embodiment, size transformation is achieved through upsampling.

[0147] Step 502: The channel features of each target interference feature are fused to obtain the fused channel features.

[0148] In practice, the server can concatenate the channel features of all target interference features to obtain multi-dimensional concatenated channel features, and then fuse these concatenated channel features into a one-dimensional fused channel feature through convolution. Alternatively, the server can directly perform convolution operations on all target interference features to directly fuse them into a one-dimensional fused channel feature.

[0149] Step 503: The size feature of the target interference feature is combined with the feature of the fusion channel to obtain the fused interference feature.

[0150] Here, by concatenating the size features with the fusion channel features, the resulting fused interference features have features in both the size dimension and the channel dimension, thereby completing the feature fusion of all target interference features. The resulting fused interference features then contain interference information from target images at various scales.

[0151] In some embodiments, when the number of target images is m, the corresponding number of interfering features is m. See also Figure 22 , Figure 22 This is an optional flowchart illustrating an image de-interference method provided in an embodiment of this application, based on... Figure 3 Step 104 can also be achieved in the following way:

[0152] Step 601: The server uses the feature fusion layer of the image de-interference model to fuse the j-th interference feature with the (j+1)-th interference feature among the m interference features to obtain the j-th fused interference feature; where m is a positive integer not less than 3, j is a positive integer, and j∈[1,m-1].

[0153] Step 602: Fuse the j-th fused interference feature with the (j+2)-th interference feature to obtain the (j+1)-th fused interference feature.

[0154] Step 603: Starting with j as 1, traverse j, and when the value of j+2 is the same as the value of m, take the (j+1)th fusion interference feature as the fusion interference feature.

[0155] In this embodiment, the server sorts the interference features according to their size from smallest to largest. It can be understood that the size of the j-th interference feature is smaller than the size of the (j+1)-th interference feature. In actual implementation, the server uses the feature fusion layer of the image de-interference model to progressively fuse each interference feature, starting with the smallest interference feature. Specifically, the server iterates through j starting from a value of 1, fusing the j-th interference feature with the (j+1)-th interference feature to obtain the j-th fused interference feature. This process continues until the (j+2)-th interference feature is obtained, and then the j-th fused interference feature is fused with the (j+2)-th interference feature to obtain the (j+1)-th fused interference feature. This process is repeated until the traversal is complete, resulting in the final fused interference feature. The fusion process between the j-th and (j+1)-th interference features can be found in [reference needed]. Figure 19 The embodiments shown in this application will not be described again here.

[0156] For example, refer to Figure 23 , Figure 23 This is an optional schematic diagram of the image de-interference process provided in this application embodiment. In actual implementation, the feature fusion layer of the image de-interference model starts by fusing the interference features corresponding to the smallest target image. Here, the target images are B0, B1, ..., B... i B i+1 See Figure 17 The image pyramid shown depicts a target image whose size decreases progressively with each level. In practice, the feature fusion layer of the image de-interference model starts from the target image B. i+1 The corresponding interference feature P(B) i+1 Starting with the image de-interference model, feature fusion is performed layer by layer on each interfering feature. Specifically, the feature fusion layer of the image de-interference model fuses the interfering features P(B) one by one. i+1 Upsampling is performed to obtain the corresponding upsampled feature up(P(B)). i+1The upsampling operation here involves deconvolution of the interfering features to better predict features with larger scales. The upsampled feature is up(P(B)). i+1 The size characteristics of the target image B are similar to those of the target image B. i The corresponding interference feature P(B) i The size features are consistent, and then the upsampled features up(P(B)) with consistent size features are obtained. i+1 )) and interference features P(B i In the channel dimension, feature concatenation of channel features is performed to obtain the concatenated interference feature M(up(P(B)). i+1 Next, the splicing interference feature M(up(P(B)) will be added. i+1 Upsampling is performed so that its size features are similar to those of the target image B. i-1 The corresponding interference feature P(B) i-1 The two are then matched, and their features are concatenated until all target images are processed to obtain the final target concatenation interference feature. Then, the target concatenation interference feature is convolved into a one-dimensional channel feature through a convolution operation to obtain a fusion interference feature P(I0) of size (W,H,3).

[0157] Step 105: Through the output layer of the image de-interference model, based on the fused interference features and image features, the image to be processed is de-interference processed to obtain the corresponding de-interference image.

[0158] In some embodiments, based on Figure 3 Step 105 can also be implemented as follows: the server determines the difference features between the image features and the fused interference features through the output layer of the image de-interference model; the difference features are decoded through the output layer of the image de-interference model to obtain the de-interference image.

[0159] In practice, the server subtracts the image features F(I0) of the image to be processed I0 from the fused interference features P(I0) through the output layer of the image de-interference model, obtaining the difference feature pred(I0). This difference feature is the image feature corresponding to the de-interference image after removing interference information from the image to be processed. Then, the output layer of the image de-interference model decodes the difference feature, converting the encoded difference feature into an image, which is the de-interference image corresponding to the image to be processed. This completes the de-interference processing of the image to be processed.

[0160] In this embodiment, the server decomposes the image to be processed into multiple target images, each containing image information of the image to be processed. As the image to be processed is decomposed, its image information is also decomposed into multiple representations. Thus, the image information of the image to be processed can be represented more clearly and comprehensively through multiple target images. Then, the image interference removal model is used to extract the interference features of each target image, and the interference features are fused. The resulting fused interference features cover more and more accurate interference information in the image to be processed, thereby enabling the accurate removal of interference information in the image to be processed based on the fused interference features.

[0161] In some embodiments, based on Figure 3 Before step 102, the image de-interference model is trained. For details on the training process of the image de-interference model, please refer to [link to training instructions]. Figure 24 , Figure 24 This is an optional flowchart illustrating a training method for an image de-interference model provided in this application embodiment. This application embodiment provides a training method for an image de-interference model, including:

[0162] Step 701: The server performs image decomposition on the sample interference image carrying interference information to obtain at least two sample target images; wherein, the sample interference image is obtained by adding interference information to the original sample image.

[0163] In practical implementation, the server acquires high-quality original sample images, adds interference information to the original sample images, and obtains corresponding interference images. In some embodiments, the interference information includes at least one of image jagged edges and image noise, based on... Figure 24 Before step 701, the following can also be performed: the server adds image jagged edges to the original sample image to obtain a jagged sample image with image jagged edges, and uses the jagged sample image as a sample interference image; or the server adds image noise to the original sample image to obtain a noisy sample image with image noise, and uses the noisy sample image as a sample interference image; or the server adds image jagged edges to the original sample image to obtain a jagged sample image with image jagged edges, and adds image noise to the jagged sample image to obtain a jagged sample noise image with both image jagged edges and image noise, and uses the jagged sample noise image as a sample interference image.

[0164] In practice, the server can obtain a noisy sample image by adding noise to the original sample image, a noisy sample image by adding jagged edges, or a jagged-edged sample image by adding both noise and jagged edges simultaneously. The server can also add other interference information to the original sample image, such as adding occlusion blocks, to obtain other interference sample images. Each original sample image can correspond to one or more interference sample images.

[0165] For example, see Figure 25 , Figure 25 This is an optional schematic diagram of the sample set provided in this application embodiment. Here, the sample set includes a set of original sample images and a set of sample interference images. Each original sample image corresponds to a set of sample interference images composed of multiple sample interference images with different interference information. The set of sample interference images includes at least one of a set of sample noise images, a set of sample jagged images, a set of sample jagged noise images, and other sets of sample interference images. In actual implementation, after determining the original sample image, the server randomly selects a sample interference image from the set of sample interference images corresponding to the original sample image to form an image pair with the original sample image. This image pair is then input into the image de-interference model of this application embodiment to train the image de-interference model.

[0166] In practical implementation, the server can convert the original sample image's image storage format to obtain a jagged sample image with image jagged edges. For example, it can convert a Portable Network Graphics (PNG) image to a Joint Photographic Experts Group (JPEG) standard image, thereby adding image jagged edges to the original sample image during the image storage format conversion process. The server can also perform channel-dimensional format conversion on the original sample image to obtain a jagged sample image with image jagged edges, such as converting an RGB image to an RGGB image, to add image jagged edges to the original sample image during the image conversion process. In some embodiments, image jagged edges can also be added to the original sample image in other ways, which are not limited in this application. In practical implementation, the server can add image noise to the original sample image by adding Gaussian white noise with a mean of 0 and a variance of theta, obtaining a corresponding noisy sample image. In some embodiments, the server can also add image noise to the original sample image by adjusting the image format or scaling the image size. In practice, the server can add image jagged edges to the original sample image and then add image noise to the jagged image to obtain a jagged noise image, or add image jagged edges to the jagged image after adding image noise to the original sample image to obtain a jagged noise image, and so on. After adding interference information to multiple original sample images, the server constructs a sample set consisting of multiple original sample images and corresponding interference images.

[0167] In practice, the server can select one or more original sample images from the sample set, and based on the original sample image, select a corresponding sample interference image from the sample interference image set corresponding to the original sample image, and perform image decomposition on the sample interference image to obtain at least two sample target images.

[0168] Step 702: Through the first feature extraction layer of the image de-interference model, feature extraction is performed on the sample interference image to obtain the sample image features of the sample interference image.

[0169] Step 703: Through the second feature extraction layer of the image de-interference model, interference features are extracted from each of the sample target images to obtain the corresponding sample interference features.

[0170] Step 704: The feature fusion layer of the image de-interference model is used to fuse the sample interference features corresponding to each sample target image to obtain sample fused interference features.

[0171] Step 705: Through the output layer of the image de-interference model, based on the sample fusion interference features and the sample image features, the sample interference image is processed to obtain the corresponding predicted de-interference image.

[0172] Step 706: Update the model parameters of the image de-interference model based on the difference between the predicted de-interference image and the original sample image.

[0173] In practice, the server inputs a sample interference image and at least two sample target images into the image de-interference model to obtain the sample image features of the sample interference image and the interference features corresponding to the at least two sample target images. These interference features are then fused through a feature fusion layer to obtain the sample fused interference features. The output layer of the image de-interference model then calculates the sample difference feature between the sample image features and the sample fused interference features. The server also inputs the original sample image into the image de-interference model. The first feature extraction layer of the image de-interference model extracts the original sample image features. Next, based on the difference between the original sample image features and the sample difference feature, the server updates the model parameters of the image de-interference model. Specifically, the server updates the parameters of the first feature extraction layer, the second feature extraction layer, the feature fusion layer, and the output layer. Here, the difference between the original sample image features and the sample difference feature is obtained by the server calculating the value of the loss function corresponding to the image de-interference model. The server iterates through the above training process until the loss function converges, completing the training of the image de-interference model. Here, the loss function can be based on the Euclidean distance formula, specifically calculating the sum of the Euclidean distances between all pixels in the predicted de-interference image and the original sample image. For example, the loss function can be expressed as follows:

[0174] Loss(I0,J0)=(1 / (2*W*H))*∑_{i\in pred(I0),j\in J0}||ij||2;

[0175] Where I0 is the sample interference image, J0 is the original sample image, and pred(I0) is the predicted interference-free image.

[0176] In some embodiments, the original sample image corresponds to at least two sample interference images, each of which is obtained by adding interference information to the original sample image. Accordingly, the image de-interference method further includes: the server performing image decomposition on the sample interference image carrying interference information, followed by de-interference processing on the sample interference image, for each sample interference image, to obtain a predicted de-interference image corresponding to each sample interference image. Correspondingly, updating the model parameters of the image de-interference model based on the difference between the predicted de-interference image and the original sample image includes: the server updating the model parameters of the image de-interference model based on the difference between each predicted de-interference image and the original sample image.

[0177] In practical implementation, the interference information carried by each sample interference image is different. For example, it may be image noise of different intensities, or different types of interference information, such as image noise and image jagged edges. In this embodiment, the server can add various interference information to the original sample image to obtain multiple corresponding sample interference images. In some embodiments, the server can also be based on... Figure 23 The server obtains the original sample image and the corresponding set of sample interference images from the given sample set. Next, the server performs the image de-interference process described above on each sample interference image to obtain the sample difference features corresponding to each sample interference image. Here, the image corresponding to the sample difference features is the predicted de-interference image. The server also inputs the original sample image into the image de-interference model for feature extraction to obtain the original sample image features. Then, the server updates the model parameters of the image de-interference model based on the differences between the sample difference features corresponding to each sample interference image and the features of the original sample image. In some embodiments, the server can calculate the central feature of each sample difference feature and update the model parameters of the image de-interference model based on the difference between the central feature and the features of the original sample image. Here, the central feature can be the mean of each sample difference feature.

[0178] In the above embodiments, by inputting the sample interference images containing multiple interference information corresponding to the original sample image into the image de-interference model for training, the image de-interference model can learn more intensity or type of interference information at the same time, thereby further improving the de-interference accuracy of the image de-interference model.

[0179] In some embodiments, the second feature extraction layer includes N sub-feature extraction layers, and the number of target images is N, where N is a positive integer not less than 2. The step of extracting interference features from each sample target image to obtain corresponding sample interference features includes: inputting each target image into a sub-feature extraction layer, and extracting interference features from the target image through the sub-feature extraction layer to obtain corresponding sample interference features. Accordingly, based on... Figure 24 Step 706 includes: updating the parameters of the first feature extraction layer, the parameters of each of the second feature extraction layers, the parameters of the feature fusion layer, and the parameters of the output layer based on the difference between the predicted de-interference image and the original sample image.

[0180] In this embodiment, the server decomposes a sample interference image into multiple sample target images, each containing image information of the sample interference image. As the sample interference image is decomposed, its image information is also decomposed into multiple representations. Thus, the image information of the sample interference image can be represented more clearly and comprehensively through multiple sample target images. Then, the image de-interference model extracts the interference features of each sample target image and fuses these interference features. The resulting fused interference features cover more and more accurate interference information in the sample interference image, enabling accurate removal of interference information from the sample interference image based on these fused interference features, resulting in a higher-quality predicted interference image. The image de-interference model is then trained based on the difference between the predicted interference image and the original sample image, resulting in a highly accurate de-interference effect.

[0181] The image interference removal method provided in the embodiments of this application will be described next. Figure 26 This is an optional flowchart illustrating the image interference removal method provided in this application embodiment. See [link / reference]. Figure 26 The image de-interference method provided in this application embodiment is implemented collaboratively by the client and the server.

[0182] Step 801: The client responds to the upload operation for the original sample image and obtains the original sample image.

[0183] Here, the client can be an image set on the terminal to interfere with the client. The original sample image can be obtained by the user based on the human-computer interaction interface of the client, triggering the upload function in the human-computer interaction interface, so that the client presents an image selection interface in the human-computer interaction interface. The user then uploads the original sample image from the terminal based on the image selection interface, thereby enabling the client to obtain the uploaded original sample image.

[0184] In some embodiments, the original sample image may also be captured by a camera connected in communication with the terminal. After capturing the original sample image, the camera transmits the original sample image to the terminal, and the terminal automatically uploads it to the client.

[0185] Step 802: The client sends the original sample image to the server.

[0186] Step 803: The server adds interference information to the original sample image to obtain a sample interference image carrying the interference information.

[0187] Here, the server can add at least one of image noise and image jaggedness to the original sample image. The server can also add other interference information to the original sample image, such as adding occlusion blocks, etc.

[0188] Step 804: The server performs image decomposition on the sample interference image to obtain at least two sample target images.

[0189] Here, the server can construct a Gaussian pyramid, a Laplacian pyramid, and a downsampled image pyramid corresponding to the sample interference image. It can then combine images of the same size from the Gaussian pyramid, the Laplacian pyramid, and the downsampled image pyramid to obtain the corresponding image combination pyramid. Each layer of the image combination pyramid is then combined as the target image.

[0190] Step 805: The server inputs the sample interference image and at least two sample target images into the image de-interference model so that the image de-interference model can perform image de-interference processing on the sample interference image to obtain the corresponding predicted de-interference image.

[0191] Here, the first feature extraction layer of the image de-interference model extracts features from the sample interference image to obtain sample image features. The second feature extraction layer extracts interference features from each target image to obtain sample interference features corresponding to each target image. Then, the feature fusion layer fuses the sample interference features to obtain sample fused interference features. Finally, based on the sample image features and the fused interference features, the sample interference image is de-interference processed to obtain the predicted interference image.

[0192] Step 806: The server updates the model parameters of the image de-interference model based on the differences between each pixel in the predicted de-interference image and the corresponding pixel in the original sample image.

[0193] Here, the server uses the Euclidean distance loss function to calculate the sum of pixel differences between each pixel in the predicted de-interference image and the corresponding pixel in the original sample image, obtaining the value of the Euclidean distance loss function, and updating the model parameters of the image de-interference model based on this value. The server iterates through the above training process until the Euclidean distance loss function converges, completing the training of the image de-interference model.

[0194] Step 807: The server generates a notification message indicating that the image de-interference model training is complete.

[0195] Step 808: The server sends a notification message to the client.

[0196] Step 809: The client responds to the upload operation for the image to be processed and obtains the image to be processed.

[0197] Here, the image to be processed can also be sent to the client by other devices that are connected to the terminal.

[0198] Step 810: In response to the de-interference command for the image to be processed, the client sends the image to be processed to the server.

[0199] Here, the de-interference command can be automatically generated by the client after receiving a message that the image de-interference model has been trained, or it can be automatically generated by the client under certain triggering conditions. For example, the client can automatically generate a de-interference command for the image to be processed after acquiring an image to be processed. Alternatively, it can be sent to the client by other devices that are connected to the terminal, or it can be generated by the user based on the human-computer interaction interface of the client after triggering the corresponding de-interference function.

[0200] Step 811: The server performs image decomposition on the image to be processed to obtain at least two target images.

[0201] Step 812: The server inputs the image to be processed and at least two target images into the image denoising model, so that the image denoising model performs denoising processing on the image to be processed based on at least two target images to obtain the corresponding denoised image.

[0202] Step 813: The server sends the de-interference image to the client.

[0203] Step 814: The client outputs the de-interference image.

[0204] Here, the client can display the de-interference image in its human-computer interaction interface, save the de-interference image to the local terminal, and send the de-interference image to other devices that are connected to the terminal.

[0205] In this embodiment, the client sends a sample interference image to the server, which then decomposes the sample interference image into multiple sample target images. Each sample target image contains image information of the sample interference image. As the sample interference image is decomposed, its image information is also decomposed into multiple representations. Therefore, multiple sample target images can more clearly and comprehensively represent the image information of the sample interference image. Then, an image de-interference model is used to extract interference features from each sample target image, and these features are fused. The resulting fused interference features cover more and more accurate interference information from the sample interference image, enabling accurate removal of interference information from the sample interference image based on these fused features, resulting in a higher-quality predicted interference image. The image de-interference model is then trained based on the difference between the predicted interference image and the original sample image, resulting in a highly accurate de-interference model. Finally, the image to be processed is de-interferenced using this model, yielding a high-quality de-interference image.

[0206] The following will describe an exemplary application of the embodiments of this application in a practical application scenario. The image de-interference method provided in the embodiments will be described using a server implementation.

[0207] The server acquires N high-quality RGB images stored in PNG format, which are the original sample images. Here, the set of N high-quality original sample images is denoted as I = {i_1, i_2, ..., i_N}, where N is a positive integer. Next, based on the N images to be processed, the server generates low-quality jagged and noisy images. The server can generate jagged images as follows: the server converts i_v (v∈[1,N]) into the corresponding RGGB image i_v', generating the set I' = {i_1', i_2', ..., i_N'}; the server also converts the storage format of i_v from PNG to JPG, obtaining the corresponding JPG image i_v”, generating the set I” = {i_1”, i_2”, ..., i_N”}. It can be understood that the generated RGGB image i_v' and JPG image i_v” carry jagged edges and are jagged sample images corresponding to the original sample image i_v. The server can generate noisy images as follows: The server adds Gaussian noise with a mean of 0 and a variance of theta to image i_v, generating image i_v*. The generated images form a set I* = {i_1*, i_2*, ..., i_N*}. Here, the server can set theta = 0.25. It can be understood that the generated image i_v* carries image noise and is a sample noise image corresponding to the original sample image i_v.

[0208] In practice, the server trains the image de-interference model based on sets I, I', I" and I*. The server can randomly select one set from I', I" and I* as the sample interference image set, and then train the image de-interference model based on set I and the selected sample interference image set. The server also inputs sets I, I', I" and I* together into the image de-interference model for further training.

[0209] In actual implementation, see Figure 17 For each sample interference image, the server constructs a corresponding [structure / method] such as [example]. Figure 17 The diagram shows the Gaussian pyramid, the Laplacian pyramid, and four types of downsampled image pyramids. The process of constructing the Gaussian pyramid, the Laplacian pyramid, and the four types of downsampled image pyramids is described in the aforementioned embodiment of this application and will not be repeated here. Next, the server combines images of the same size from the Gaussian pyramid, the Laplacian pyramid, and the four types of downsampled image pyramids to obtain the corresponding image combination pyramid. For example, taking a Gaussian pyramid with 4 layers as an illustration, in this embodiment, for a sample interference image, the server decomposes it into four image blocks of different sizes, namely B0={I0,L0}, B1={I1,L1,ds0(I0),ds1(I0),ds2(I0),ds3(I0)}, B2={I2,L2,ds0(I1),ds1(I1),ds2(I1),ds3(I1)}, and B3={I3,ds0(I2),ds1(I2),ds2(I2),ds3(I2)}. Each image has 3 channels. Here, the sizes of the four image blocks are (W, H, 2x3), (W / 2, H / 2, 6x3), (W / 4, H / 4, 6x3), and (W, H, 5x3), respectively.

[0210] See Figure 19 The structure of the image de-interference module shown is as follows: Figure 23The image de-interference process is illustrated. Here, the image de-interference model is a residual network model, including a first feature extraction layer, a second feature extraction layer, a feature fusion layer, and an output layer. The second feature extraction layer comprises multiple sub-feature extraction layers, the same number as the number of image patches. The structures or parameters of each sub-feature extraction layer are not entirely consistent. Both the first feature extraction layer and each sub-feature extraction layer consist of multiple convolutional layers, which can employ convolutional structures from ResNet, DenseNet, SERESNet, or Inception networks. Next, the server inputs the sample interference image and the corresponding four image patches into the image de-interference model, enabling the model to extract the image features F(I0) of the image to be processed through the first feature extraction layer. Here, the first feature extraction layer uses long-term convolution operations to extract features from the image to be processed, obtaining image features containing both image information and interference information. Then, each sub-feature extraction layer of the image de-interference model extracts interference features from each image patch, using short-term convolution operations. Next, the feature fusion layer of the image de-interference model performs feature fusion operations step by step, starting from image block B3. Specifically, image block B3 is upsampled to make its size consistent with that of image block B2. Then, the upsampled image block B3 and image block B2 are concatenated using channel-dimensional features. The size of the concatenated image block is (W / 2, H / 2, 11x3). Through the above operations, the concatenated interference feature with the size of (W / 2, H / 2, 19x3) is obtained after feature concatenation of all image blocks. Then, the image de-interference model uses a convolutional layer in the feature fusion layer to perform channel-dimensional feature fusion on the concatenated interference feature with the size of (W / 2, H / 2, 19x3) to obtain a fused interference feature P(I0) with the size of (W / 2, H / 2, 3). Finally, the output layer subtracts the image feature F(I0) of the image to be processed from the fused interference feature P(I0) to obtain the sample difference feature pred(I0). In actual implementation, the server also inputs the original sample image into the image de-interference model. Through the first feature extraction layer of the image de-interference model, the original sample image is used to extract features to obtain the original sample image features F(I). Based on the original sample image features F(I) and the sample difference features pred(I0), the server determines the value of the loss function corresponding to the image de-interference model. Based on the value of the loss function, the model parameters of the image de-interference model are updated, and the training of the model is iterated until the loss function converges, thus completing the training of the image de-interference model.

[0211] After training the image denoising model, it can be used to denoise any image. In practice, the server takes an image to be processed as input, first calculating its Gaussian pyramid, Laplacian pyramid, and the corresponding four downsampled images ds0, ds1, ds2, and ds3, then constructing data blocks B0, B1, B2, and B3. The image to be processed, along with the constructed data blocks B0, B1, B2, and B3, is then input into the trained image denoising model. The output image of the model is a high-quality denoised image of the same size as the input image to be processed.

[0212] It is understood that in the embodiments of this application, data such as images are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0213] The following continues to describe the exemplary structure of the image de-interference device 555 provided in the embodiments of this application as a software module. In some embodiments, see [link to relevant documentation]. Figure 27 , Figure 27 This is an optional schematic diagram of the image interference removal device provided in the embodiments of this application. The software modules stored in the image interference removal device 555 in the memory 550 may include:

[0214] The image decomposition module 5551 is used to decompose the image to be processed carrying interference information to obtain at least two target images;

[0215] Image feature extraction module 5552 is used to extract features from the image to be processed through the first feature extraction layer of the image de-interference model to obtain the image features of the image to be processed;

[0216] The interference feature extraction module 5553 is used to extract interference features from each of the target images through the second feature extraction layer of the image de-interference model to obtain the corresponding interference features.

[0217] The feature fusion module 5554 is used to fuse the interference features of each target image through the feature fusion layer of the image de-interference model to obtain fused interference features;

[0218] The interference removal module 5555 is used to perform interference removal processing on the image to be processed based on the fused interference features and the image features through the output layer of the image interference removal model, so as to obtain the corresponding interference removal image.

[0219] In some embodiments, the image decomposition module 5551 is further configured to perform multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed; and to use the image of each layer in the Gaussian pyramid as the target image.

[0220] In some embodiments, the image decomposition module 5551 is further configured to perform multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed; construct a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid corresponding to the image to be processed; combine the images of each layer in the Gaussian pyramid with the images of the corresponding levels in the Laplacian pyramid to obtain at least two image combinations; and use the at least two image combinations as the target image.

[0221] In some embodiments, the image decomposition module 5551 is further configured to perform multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed; construct a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid corresponding to the image to be processed; perform downsampling processing on the images of each layer in the Gaussian pyramid to obtain corresponding downsampled images; combine the images of each layer in the Gaussian pyramid, the images of the corresponding level in the Laplacian pyramid, and the downsampled images of the same size as the images of the corresponding level in the Laplacian pyramid to obtain at least two image combinations; and use the at least two image combinations as the target image.

[0222] In some embodiments, the second feature extraction layer includes N sub-feature extraction layers, the number of target images is N, and N is a positive integer not less than 2; the interference feature extraction module 5553 is further configured to input each target image into one of the sub-feature extraction layers, and extract interference features from the target image through the sub-feature extraction layers to obtain the corresponding interference features.

[0223] In some embodiments, the interference features include size features and channel features. The feature fusion module 5554 is further configured to perform size transformation on the interference features of each target image to obtain target interference features corresponding to each interference feature; wherein, the size features of each target interference feature are the same; perform fusion processing on the channel features of each target interference feature to obtain fused channel features; and perform feature concatenation between the size features of the target interference features and the fused channel features to obtain the fused interference features.

[0224] In some embodiments, when the number of target images is m, the corresponding number of interference features is m. The feature fusion module 5554 is further configured to fuse the j-th interference feature with the (j+1)-th interference feature among the m interference features to obtain the j-th fused interference feature; where m is a positive integer not less than 3, j is a positive integer, and j∈[1, m-1]; fuse the j-th fused interference feature with the (j+2)-th interference feature to obtain the (j+1)-th fused interference feature; starting with j as 1, traverse j, and when the value of j+2 is the same as the value of m, use the (j+1)-th fused interference feature as the fused interference feature.

[0225] In some embodiments, the de-interference module 5555 is further configured to determine the difference features between the image features and the fused interference features through the output layer of the image de-interference model; and decode the difference features through the output layer of the image de-interference model to obtain the de-interference image.

[0226] In some embodiments, the software module stored in the image de-interference device 555 in the memory 550 may include: a model training module, used to perform image decomposition on a sample interference image carrying interference information to obtain at least two sample target images; wherein the sample interference image is obtained by adding interference information to the original sample image; through the first feature extraction layer of the image de-interference model, feature extraction is performed on the sample interference image to obtain sample image features of the sample interference image; through the second feature extraction layer of the image de-interference model, interference features are extracted from each of the sample target images to obtain corresponding sample interference features; through the feature fusion layer of the image de-interference model, the sample interference features corresponding to each of the sample target images are fused to obtain sample fused interference features; through the output layer of the image de-interference model, based on the sample fused interference features and the sample image features, the sample interference image is de-interference processed to obtain a corresponding predicted de-interference image; based on the difference between the predicted de-interference image and the original sample image, the model parameters of the image de-interference model are updated.

[0227] In some embodiments, the original sample image corresponds to at least two sample interference images, each of which is obtained by adding interference information to the original sample image. Accordingly, the image decomposition module 5551 is further configured to perform the operations of image decomposition of the sample interference image carrying interference information and the de-interference processing of the sample interference image for each sample interference image, to obtain the predicted de-interference image corresponding to each sample interference image. Accordingly, the model training module is further configured to update the model parameters of the image de-interference model based on the differences between each predicted de-interference image and the original sample image.

[0228] In some embodiments, the interference information includes at least one of image jagged edges and image noise; correspondingly, the software module stored in the image de-interference device 555 in the memory 550 may include: an interference information adding module, configured to add image jagged edges to the original sample image to obtain a sample jagged image carrying image jagged edges, and use the sample jagged image as the sample interference image; or add image noise to the original sample image to obtain a sample noise image carrying image noise, and use the sample noise image as the sample interference image; or add image jagged edges and image noise to the original sample image to obtain a sample jagged noise image carrying both image jagged edges and image noise, and use the sample jagged noise image as the sample interference image.

[0229] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated.

[0230] This application provides an electronic device, including:

[0231] Memory, used to store executable instructions;

[0232] The processor, when executing executable instructions stored in the memory, implements the image de-interference method or the image de-interference model training method provided in the embodiments of this application.

[0233] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the image de-interference method or the image de-interference model training method provided in this application.

[0234] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image de-interference method or image de-interference model training method described in this application.

[0235] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 3 The image de-interference method shown, or as... Figure 24 The training method for the image de-interference model is shown.

[0236] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0237] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0238] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0239] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0240] In summary, the embodiments of this application can accurately remove interference information from images.

[0241] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image interference removal method, characterized in that, The method includes: A multi-scale image decomposition is performed on the image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed. The downsampling process involved in constructing the Gaussian pyramid includes: extracting pixels from the image to be processed after Gaussian convolution where both the number of rows and columns are odd; constructing a new image based on the extracted pixels; using the constructed new image as the downsampled image after the downsampling process; and using the downsampled image as one layer of the image to be processed in the Gaussian pyramid. Based on the Gaussian pyramid corresponding to the image to be processed, a Laplacian pyramid corresponding to the image to be processed is constructed; wherein, the upsampling processing involved in constructing the Laplacian pyramid includes: adding new pixels to the image in the Gaussian pyramid, doubling both the number of pixel rows and the number of pixel columns; and interleaving the newly added rows of pixels between the rows of pixels in the image, and interleaving the newly added columns of pixels between the columns of pixels in the image; filling the channel values ​​of the newly added pixels to obtain the upsampled image after the upsampling processing; The Gaussian images in the Gaussian pyramid, excluding the top layer, are downsampled in four different ways to obtain four downsampled image pyramids. The pixels of the downsampled images in the four downsampled image pyramids together constitute the pixels of the corresponding images in the Gaussian pyramid, and all pixel information in the Gaussian pyramid is retained in the downsampled images in the four downsampled image pyramids. The images of each layer in the Gaussian pyramid, the corresponding layer in the Laplacian pyramid, and the downsampled images of the same size at the corresponding layers in the four types of downsampled image pyramids are combined to obtain at least two image combinations. The at least two images are combined to form the target image; The first feature extraction layer of the image de-interference model is used to extract features from the image to be processed, thereby obtaining the image features of the image to be processed. The second feature extraction layer of the image de-interference model extracts interference features from each target image to obtain the corresponding interference features. The feature fusion layer of the image de-interference model is used to fuse the interference features of each target image to obtain fused interference features. The image to be processed is processed by the output layer of the image de-interference model based on the fused interference features and the image features to obtain the corresponding de-interference image.

2. The method according to claim 1, characterized in that, The Gaussian image in the Gaussian pyramid, excluding the top layer, is downsampled in four different ways to obtain four types of downsampled image pyramids, including: Determine the row and column codes of pixels in a Gaussian image; From the multiple pixels that constitute the Gaussian image, select multiple first pixels with odd row and column codes, and construct the corresponding first downsampled image based on the multiple first pixels; From the multiple pixels that constitute the Gaussian image, select multiple second pixels with odd row encoding and even column encoding, and construct the corresponding second downsampled image based on the multiple second pixels; From the multiple pixels that make up the Gaussian image, select multiple third pixels with even row codes and odd column codes, and construct the corresponding third downsampled image based on the multiple third pixels; From the multiple pixels that constitute the Gaussian image, select multiple fourth pixels whose row and column codes are both even numbers, and construct the corresponding fourth downsampled image based on the multiple fourth pixels; The first downsampled image, the second downsampled image, the third downsampled image, and the fourth downsampled image are used as the downsampled image.

3. The method according to claim 1, characterized in that, The second feature extraction layer includes N sub-feature extraction layers, and the number of target images is N, where N is a positive integer not less than 2; The second feature extraction layer of the image de-interference model extracts interference features from each of the target images carrying interference information to obtain corresponding interference features, including: Each target image is input into a sub-feature extraction layer, and interference features are extracted from the target image through the sub-feature extraction layer to obtain the corresponding interference features.

4. The method according to claim 1, characterized in that, The interference features include size features and channel features. The process of fusing the interference features of each target image to obtain fused interference features includes: The size of the interference features of each target image is transformed to obtain the target interference features corresponding to each interference feature; wherein, the size features of each target interference feature are the same; The channel features of each of the target interference features are fused to obtain fused channel features; The size feature of the target interference feature is concatenated with the feature of the fusion channel to obtain the fused interference feature.

5. The method according to claim 1, characterized in that, When the number of target images is m, the corresponding number of interference features is m. The process of fusing the interference features of each target image to obtain fused interference features includes: The j-th interference feature among the m interference features is fused with the (j+1)-th interference feature to obtain the j-th fused interference feature; where m is a positive integer not less than 3, j is a positive integer, and j∈[1,m-1]; The j-th fused interference feature is fused with the (j+2)-th interference feature to obtain the (j+1)-th fused interference feature; Starting with j as 1, iterate through j, and when the value of j+2 is the same as the value of m, take the (j+1)th fusion interference feature as the fusion interference feature.

6. The method according to claim 1, characterized in that, The output layer of the image de-interference model performs de-interference processing on the image to be processed based on the fused interference features and the image features to obtain a corresponding de-interference image, including: The difference features between the image features and the fused interference features are determined through the output layer of the image de-interference model. The difference features are decoded through the output layer of the image de-interference model to obtain the de-interference image.

7. The method according to claim 1, characterized in that, Before performing feature extraction on the image to be processed by the first feature extraction layer of the image de-interference model, the method further includes: Image decomposition is performed on the sample interference image carrying interference information to obtain at least two sample target images; The sample interference image is obtained by adding interference information to the original sample image; The first feature extraction layer of the image de-interference model is used to extract features from the sample interference image to obtain the sample image features of the sample interference image. The second feature extraction layer of the image de-interference model is used to extract interference features from each of the sample target images to obtain the corresponding sample interference features. The feature fusion layer of the image de-interference model is used to fuse the sample interference features corresponding to each sample target image to obtain sample fused interference features. The image de-interference model outputs a de-interference image based on the sample fusion interference features and the sample image features to obtain a corresponding predicted de-interference image. Based on the difference between the predicted de-interference image and the original sample image, the model parameters of the image de-interference model are updated.

8. The method according to claim 7, characterized in that, The original sample image corresponds to at least two sample interference images, each of which is obtained by adding interference information to the original sample image. Accordingly, the method further includes: For each of the sample interference images, the operations of image decomposition of the sample interference image carrying interference information and the de-interference processing of the sample interference image are performed respectively to obtain the predicted de-interference image corresponding to each of the sample interference images. Accordingly, updating the model parameters of the image de-interference model based on the difference between the predicted de-interference image and the original sample image includes: The model parameters of the image de-interference model are updated based on the differences between the predicted de-interference images and the original sample images.

9. The method according to claim 7, characterized in that, The interference information includes at least one of image jagged edges and image noise; Before performing image decomposition on the sample interference image carrying interference information, the method further includes: Add image jagged edges to the original sample image to obtain a jagged sample image with jagged edges, and use the jagged sample image as the sample interference image; or Add image noise to the original sample image to obtain a sample noise image carrying image noise, and use the sample noise image as the sample interference image; or Image jagged edges and image noise are added to the original sample image to obtain a sample jagged noise image that carries both image jagged edges and image noise. The sample jagged noise image is used as the sample interference image.

10. A training method for an image de-interference model, characterized in that, The method includes: Multi-scale image decomposition is performed on the sample interference image carrying interference information to construct a Gaussian pyramid corresponding to the sample interference image; wherein, the downsampling processing involved in constructing the Gaussian pyramid includes: extracting pixels with an odd number of rows and columns from the sample interference image after Gaussian convolution, constructing a new image based on the extracted pixels, and using the constructed new image as the downsampled image after the downsampling processing; and using the downsampled image as a layer image of the sample interference image in the Gaussian pyramid; Based on the Gaussian pyramid corresponding to the sample interference image, a Laplacian pyramid corresponding to the sample interference image is constructed; wherein, the upsampling processing involved in constructing the Laplacian pyramid includes: adding new pixels to the image in the Gaussian pyramid, doubling both the number of pixel rows and the number of pixel columns; and interleaving the newly added rows of pixels between the rows of pixels in the image, and interleaving the newly added columns of pixels between the columns of pixels in the image; filling the channel values ​​of the newly added pixels to obtain the upsampled image after the upsampling processing; The Gaussian images in the Gaussian pyramid, excluding the top layer, are downsampled in four different ways to obtain four downsampled image pyramids. The pixels of the downsampled images in the four downsampled image pyramids together constitute the pixels of the corresponding images in the Gaussian pyramid, and all pixel information in the Gaussian pyramid is retained in the downsampled images in the four downsampled image pyramids. The images of each layer in the Gaussian pyramid, the corresponding layer in the Laplacian pyramid, and the downsampled images of the same size at the corresponding layers in the four types of downsampled image pyramids are combined to obtain at least two image combinations. The at least two images are combined to form the target image of the sample; wherein the sample interference image is obtained by adding interference information to the original sample image; The first feature extraction layer of the image de-interference model is used to extract features from the sample interference image to obtain the sample image features of the sample interference image. The second feature extraction layer of the image de-interference model is used to extract interference features from each of the sample target images to obtain the corresponding sample interference features. The feature fusion layer of the image de-interference model is used to fuse the sample interference features corresponding to each sample target image to obtain sample fused interference features. The image de-interference model outputs a de-interference image based on the sample fusion interference features and the sample image features to obtain a corresponding predicted de-interference image. Based on the difference between the predicted de-interference image and the original sample image, the parameters of the first feature extraction layer, the second feature extraction layer, the feature fusion layer, and the output layer are updated respectively.

11. An image interference removal device, characterized in that, include: An image decomposition module is used to perform multi-scale image decomposition on the image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed. The downsampling process involved in constructing the Gaussian pyramid includes: extracting pixels with an odd number of rows and columns from the image to be processed after Gaussian convolution; constructing a new image based on the extracted pixels; using the new image as the downsampled image after the downsampling process; using the downsampled image as a layer of the image to be processed in the Gaussian pyramid; and constructing a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid. The upsampling process involved in constructing the Laplacian pyramid includes: adding new pixels to the image in the Gaussian pyramid, doubling both the number of rows and columns of pixels; and interleaving the newly added pixels into the existing image at intervals. The newly added columns of pixels are interleaved between the rows of pixels in the image; the newly added pixels are padded with channel values ​​to obtain an upsampled image after the upsampling process; four downsampling processes are applied to the Gaussian images in the Gaussian pyramid (excluding the top layer) to obtain four downsampled image pyramids; the pixels of the downsampled images in the four downsampled image pyramids together constitute the pixels of the corresponding image in the Gaussian pyramid, and all pixel information in the Gaussian pyramid is retained in the downsampled images in the four downsampled image pyramids; the images of each layer in the Gaussian pyramid, the images of the corresponding level in the Laplacian pyramid, and the downsampled images of the same size at the corresponding level in the four downsampled image pyramids are combined to obtain at least two image combinations; the at least two image combinations are used as the target image. The image feature extraction module is used to extract features from the image to be processed through the first feature extraction layer of the image de-interference model to obtain the image features of the image to be processed. The interference feature extraction module is used to extract interference features from each of the target images through the second feature extraction layer of the image de-interference model to obtain the corresponding interference features. The feature fusion module is used to fuse the interference features of each target image through the feature fusion layer of the image de-interference model to obtain fused interference features; The interference removal module is used to perform interference removal processing on the image to be processed based on the fused interference features and the image features through the output layer of the image interference removal model, so as to obtain the corresponding interference removal image.

12. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the method of any one of claims 1 to 10 when executed by a processor.

14. A computer program product comprising executable instructions or a computer program, characterized in that, When the executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 10.

Images