Image processing method, image processing system training method and electronic device

Abstract

The embodiment of the present application is applied to the field of artificial intelligence, and provides an image processing method, an image processing system training method and an electronic device. The image processing method includes: obtaining an original image file to be processed and a visible image to be processed, wherein the visible image to be processed is an image obtained by arranging the data elements in the original image file to be processed according to pixels; using an image processing system to process the original image file to be processed and the visible image to be processed to obtain a deblurred original image file, wherein the image processing system is a trained neural network model. Based on the technical method of the present application, the image represented by the deblurred original image file can be made clearer.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an image processing method, an image processing system training method, and an electronic device. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] Machine learning is an important branch of artificial intelligence, and deep learning is an important branch of machine learning. Deep learning refers to the use of multi-layer neural network structures to learn from big data the representation of various things in the real world that can be directly used for computer calculations (for example, things in images, sounds in audio, etc.). In the field of image processing, deep learning has achieved excellent results in object detection, image generation, image segmentation and other problems.

[0004] When users use electronic devices to take photos, they will inevitably encounter hand shaking, capture moving scenes, and other needs. The image may be blurred due to shaking during shooting, movement of the subject, or camera defocus. Image deblurring is a classic problem in computer underlying vision, and its purpose is to restore a clear image from a blurred image. Restoring a blurred image to a clear image can improve the imaging quality of electronic devices and enhance the user's photography experience.

[0005] However, different regions of a single image captured in practice often contain different degrees of blur, and the cause of blur is unpredictable. Therefore, image deblurring is a highly ill-posed optimization problem. How to obtain a clear image is an urgent problem to be solved. Summary of the invention

[0006] The present application provides an image processing method, an image processing system training method and an electronic device, which can obtain clear images.

[0007] In a first aspect, an image processing method is provided, comprising: obtaining an original image file to be processed and a visible image to be processed, wherein the visible image to be processed is an image obtained by arranging data elements in the original image file to be processed according to pixels; using an image processing system to process the original image file to be processed and the visible image to be processed to obtain a deblurred original image file, wherein the image processing system is a trained neural network model.

[0008] The original image file can reflect information related to the position of the sensor element, that is, information related to space. And the information related to space has an important impact on the image quality obtained by image deblurring. The image processing method provided in the embodiment of the present application uses an image processing system to process the original image file to be processed and the visible image to be processed obtained by arranging the data elements in the original image file to be processed according to pixels to obtain a deblurred original image file. In the image processing process, the information related to space is considered to make the image represented by the deblurred original image file clearer.

[0009] It should be understood that the image represented by the deblurred original image file can be understood as an image obtained by arranging the data elements in the deblurred original image file according to pixels.

[0010] In one possible implementation, the image processing system includes a spatial feature extraction model, a color feature extraction model and an image generation model; the spatial feature extraction model is used to perform feature extraction on the original image file to be processed to obtain a first spatial feature; the color feature extraction model is used to perform feature extraction on the visible image to be processed to obtain a first color feature; the image generation model is used to process the first spatial feature and the first color feature to obtain the deblurred original image file.

[0011] By performing feature extraction on the original image file to be processed and the visible image to be processed respectively, the extracted features are made more consistent with the characteristics of the two different data, the original image file and the visible image. Thus, the deblurred original image file is determined according to the extracted features, which can make the image represented by the deblurred original image file clearer and have better image quality.

[0012] In one possible implementation, the image processing system is used to process the original image file to be processed and the visible image to be processed to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0013] The deblurred original image file is determined based on the difference image data obtained by the original image file to be processed and the image processing system, so that the processing process of the image processing system is more consistent with the purpose of image processing such as deblurring, and more attention is paid to the difference between the original image file to be processed and the deblurred original image file representing a clear image, thereby improving the clarity of the image represented by the deblurred original image file and improving the image quality.

[0014] In one possible implementation, the image processing system also includes a first spatial attention mechanism model and a first color feature adjustment model; the first spatial attention mechanism model is used to process the first spatial feature to obtain a first spatial weight, and the first spatial weight represents the first position importance of each first color position in multiple first color positions in the first color feature; the first color feature adjustment model is used to adjust the first color feature according to the first spatial weight; the image generation model is specifically used to process the first spatial feature and the adjusted first color feature to obtain difference image data, and the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0015] According to the first color feature obtained by processing the visible image to be processed, a first color weight is determined, and the first color weight can represent the importance of different colors for the deblurring process. According to the first color weight, the first spatial feature is adjusted so that the importance of the color information in the adjusted first spatial feature is adapted, thereby improving the clarity of the image represented by the deblurred original image file determined according to the first color feature and the adjusted first spatial feature.

[0016] In one possible implementation, the first spatial attention mechanism model is used to determine the first spatial weight by using the maximum value of multiple first spatial elements located at each first spatial position in multiple first spatial channels of the first spatial feature.

[0017] Spatial features can be understood as texture information in an image, and can be used to represent edge information in an image, that is, locations where there are differences between pixel values. The importance of the first spatial location can be determined by calculating the maximum value of multiple first spatial elements located at each first spatial location in multiple first spatial channels of the first spatial feature. Thus, the first spatial weight more accurately reflects the importance of the location, and the image represented by the deblurred original image file determined by the first spatial feature after adjusting the first spatial weight is clearer.

[0018] In a possible implementation, the image processing system also includes a first color attention mechanism model and a first spatial feature adjustment model, and the multiple image channels of the visible image to be processed correspond to multiple coordinate axes of the color space; the first color attention mechanism model is used to process the first color feature to obtain a first color weight, and the first color weight represents the importance of the first color of each first spatial channel in the multiple first spatial channels of the first spatial feature; the first spatial feature adjustment model is used to adjust the first spatial feature according to the first color weight; the image generation model is specifically used to process the first color feature and the adjusted first spatial feature to obtain difference image data, and the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0019] Multiple image channels in the visible image to be processed correspond to multiple coordinate axes of the color space, and the color channels in the first color feature obtained by processing the visible image to be processed are related to the color information. According to the first color feature, a first color weight is determined, and the first color weight can represent the importance of different colors for deblurring processing. According to the first color weight, the first spatial feature is adjusted so that the importance of the color information in the adjusted first spatial feature is adapted, and the deblurred original image file is determined according to the first color feature and the adjusted first spatial feature, so as to improve the clarity of the image represented by the deblurred original image file.

[0020] In a possible implementation, the first color attention mechanism model is specifically used to determine the first color weight according to an average value of multiple elements in each color channel in the first color feature.

[0021] In each color channel of the first color feature, there are a large number of elements. By calculating the average value of the multiple elements, the influence of abnormal values ​​can be eliminated.

[0022] In a possible implementation, the color feature extraction model includes a first color feature extraction sub-model, a second color feature extraction sub-model, a second spatial attention mechanism model, and a second color feature adjustment model; the spatial feature extraction model includes a first spatial feature extraction sub-model; the first spatial feature extraction sub-model is used to perform feature extraction on the original image file to be processed to obtain a second spatial feature, and the first spatial feature is determined based on the second spatial feature; the first color feature extraction sub-model is used to perform feature extraction on the visible image to be processed to obtain a second color feature; the second spatial attention mechanism model is used to process the second spatial feature to obtain a second spatial weight, and the second The spatial weight represents the second position importance of each second color position in the multiple second color positions in the second color feature; the second color feature adjustment model is used to adjust the second color feature according to the second spatial weight; the second color feature extraction sub-model is used to extract features of the adjusted second color feature to obtain the first color feature; the image generation model is specifically used to process the first spatial feature and the first color feature to obtain difference image data, the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0023] In the process of determining the first color feature according to the visible image to be processed, the second spatial weight is determined according to the original image file to be processed, and the first color feature is determined according to the second spatial weight. In the process of determining the first color feature, the importance of the position in the feature determined according to the original image file to be processed is considered, so that the deblurred original image file is more accurate, that is, the image represented by the deblurred original image file is clearer and has higher image quality.

[0024] In a possible implementation, the spatial feature extraction model includes a first spatial feature extraction sub-model, a second spatial feature extraction sub-model, a second color attention mechanism model, and a second spatial feature adjustment model; the color feature extraction model includes a first color feature extraction sub-model; the first color feature extraction sub-model is used to perform feature extraction on the visible image to be processed to obtain a second color feature, and the first color feature is determined based on the second color feature; the first spatial feature extraction sub-model is used to perform feature extraction on the original image file to be processed to obtain a second spatial feature; the second color attention mechanism model is used to process the second color feature to obtain a second color weight, and the second The color weight represents the importance of the second color of each second spatial channel in the multiple second spatial channels of the second spatial feature; the second spatial feature adjustment model is used to adjust the second spatial feature according to the second color weight; the second spatial feature extraction sub-model is used to extract features of the adjusted second spatial feature to obtain the first spatial feature; the image generation model is specifically used to process the first spatial feature and the first color feature to obtain difference image data, the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0025] In the process of determining the first spatial feature according to the original image file to be processed, the second color weight is determined according to the visible image to be processed, and the first spatial feature is determined according to the second color weight. In the process of determining the first spatial feature, the importance of the color-related channels in the features determined according to the visible image to be processed is considered, so that the deblurred original image file is more accurate, that is, the image represented by the deblurred original image file is clearer and has higher image quality.

[0026] In a possible implementation, the image generation model includes a first fusion model, a downsampling model, a feature map generation model and a second fusion model; the first fusion model is used to fuse the first spatial feature and the first color feature to obtain a fused feature; the downsampling model is used to downsample the fused feature to obtain a downsampled feature; the feature map generation model is used to generate feature images for the fused feature and the downsampled feature, respectively, to obtain a first generated feature map corresponding to the fused feature, and at least one second generated feature map corresponding to the downsampled feature; the second fusion model is used to fuse the first generated feature map with the at least one second generated feature map to obtain the deblurred original image file.

[0027] The fused features are downsampled by the downsampling model to obtain downsampled features, and the feature map generation model used to process the fused features to obtain the first generated feature map is used to process the downsampled features to obtain a second generated feature map, and then the second generated feature map is fused with the first generated feature map to obtain a deblurred original image file. Thus, different degrees of blur in the original image file to be processed are removed, so that the image corresponding to the obtained deblurred original image file is clearer.

[0028] In a second aspect, a method for training an image processing system is provided, comprising: obtaining training samples and label deblurred original image files, the training samples comprising training original image files and training visible images, the training visible images being images obtained by arranging training data elements in the training original image files according to pixels; processing the samples using an initial image processing system to obtain a training deblurred original image file; adjusting parameters of the initial image processing system according to training differences between the training deblurred original image file and the label deblurred original image file, the adjusted initial image processing system being the trained image processing system.

[0029] In a possible implementation, the initial image processing system includes an initial spatial feature extraction model, an initial color feature extraction model, and an initial image generation model. The initial spatial feature extraction model is used to perform feature extraction on the training original image file to obtain a first training spatial feature. The initial color feature extraction model is used to perform feature extraction on the training visible image to obtain a first training color feature. The initial image generation model is used to process the first training spatial feature and the first training color feature to obtain a training deblurred original image file. The adjusting the parameters of the initial image processing system according to the training difference between the training deblurred original image file and the label deblurred original image file includes: adjusting the parameters of the initial spatial feature extraction model, the initial color feature extraction model, and the initial image generation model according to the training difference, the adjusted initial spatial feature extraction model is a spatial feature extraction model, the adjusted initial color feature extraction model is a color feature extraction model, the adjusted initial image generation model is an image generation model, and the image processing system includes the spatial feature extraction model, the color feature extraction model, and the image generation model.

[0030] In a possible implementation, the initial image generation model is specifically used to process the first training spatial feature and the first training color feature to obtain training difference image data, wherein the training difference image data represents the difference between the training deblurred original image file and the training original image file. The training difference represents the training difference between the training difference image data and the label difference image data, and the label difference image data represents the difference between the label deblurred original image file and the training deblurred original image file.

[0031] In a possible implementation, the initial image processing system further includes an initial first spatial attention mechanism model and a first color feature adjustment model. The initial first spatial attention mechanism model is used to process the first training spatial feature to obtain a first training spatial weight, and the first training spatial weight represents the first training position importance of each first training color position in the first training color feature. The first color feature adjustment model is used to adjust the first training color feature according to the first training spatial weight. The initial image generation model is specifically used to process the first training spatial feature and the adjusted first training color feature to obtain training difference image data. The adjusting the parameters of the initial image processing system according to the training difference between the training deblurred original image file and the label deblurred original image file includes: adjusting the parameters of the initial first spatial attention mechanism model according to the training difference, the adjusted initial first spatial attention mechanism model is the first spatial attention mechanism model, and the image processing system includes the first spatial attention mechanism model and the first color feature adjustment model.

[0032] Optionally, the initial first spatial attention mechanism model is used to determine the first training spatial weight by taking the maximum value of multiple first training spatial elements located at each first training spatial position in multiple first training spatial channels of the first training spatial feature.

[0033] Optionally, the initial image processing system also includes an initial first color attention mechanism model and a first spatial feature adjustment model, and the multiple training image channels of the training visual image correspond to multiple coordinate axes of the color space. The initial first color attention mechanism model is used to process the first training color feature to obtain a first training color weight, and the first training color weight represents the importance of the first training color of each first training spatial channel in the multiple first training spatial channels of the first training spatial feature. The first spatial feature adjustment model is used to adjust the first training spatial feature according to the first training color weight. The initial image generation model is specifically used to process the first training color feature and the adjusted first training spatial feature to obtain the difference image data.

[0034] The adjusting the parameters of the initial image processing system according to the training difference between the training deblurred original image file and the label deblurred original image file includes: adjusting the parameters of the initial first color attention mechanism model according to the training difference. The adjusted initial first color attention mechanism model is the first color attention mechanism model, and the image processing system includes the first color attention mechanism model and the first spatial feature adjustment model.

[0035] Optionally, the initial first color attention mechanism model is specifically used to determine the first training color weight according to an average value of multiple elements in each training color channel in the first training color feature.

[0036] Optionally, the initial color feature extraction model includes an initial first color feature extraction sub-model, an initial second color feature extraction sub-model, an initial second space attention mechanism model and a second color feature adjustment model. The initial space feature extraction model includes an initial first space feature extraction sub-model. The initial first space feature extraction sub-model is used to perform feature extraction on the training original image file to obtain the second training space feature, and the first training space feature is determined according to the second training space feature. The initial first color feature extraction sub-model is used to perform feature extraction on the training visual image to obtain the second training color feature. The initial second space attention mechanism model is used to process the second training space feature to obtain the second training space weight, and the second training space weight represents the second position importance of each second color position in the multiple second color positions in the second color feature. The second color feature adjustment model is used to adjust the second training color feature according to the second training space weight. The initial second color feature extraction sub-model is used to perform feature extraction on the adjusted second color feature to obtain the first color feature. The adjusting of the parameters of the initial image processing system includes: adjusting the parameters of the initial first color feature extraction submodel, the parameters of the initial second color feature extraction submodel, the parameters of the initial second spatial attention mechanism model, and the parameters of the initial first spatial feature extraction submodel. The adjusted initial first color feature extraction submodel is the first color feature extraction submodel, the adjusted initial second color feature extraction submodel is the second color feature extraction submodel, and the adjusted initial second spatial attention mechanism model is the second spatial attention mechanism model. The adjusted initial first spatial feature extraction submodel is the first spatial feature extraction submodel. The color feature extraction model includes the first color feature extraction submodel, the second color feature extraction submodel, the second spatial attention mechanism model, and the second color feature adjustment model. The spatial feature extraction model includes the first spatial feature extraction submodel.

[0037] Optionally, the initial spatial feature extraction model includes an initial first spatial feature extraction sub-model, an initial second spatial feature extraction sub-model, an initial second color attention mechanism model, and a second spatial feature adjustment model. The initial color feature extraction model includes an initial first color feature extraction sub-model. The initial first color feature extraction sub-model is used to perform feature extraction on the training visible image to obtain the second training color feature, and the first training color feature is determined according to the second training color feature. The initial first spatial feature extraction sub-model is used to perform feature extraction on the training original image file to obtain the second training spatial feature. The initial second color attention mechanism model is used to process the second training color feature to obtain the second training color weight, and the second training color weight represents the importance of the second training color of each second training spatial channel in the multiple second training spatial channels of the second training spatial feature. The second spatial feature adjustment model is used to adjust the second training spatial feature according to the second training color weight. The initial second spatial feature extraction sub-model is used to perform feature extraction on the adjusted second training spatial feature to obtain the first training spatial feature. The adjusting of the parameters of the initial image processing system includes: adjusting the parameters of the initial first spatial feature extraction submodel, the parameters of the initial second spatial feature extraction submodel, the parameters of the initial second color attention mechanism model, and the parameters of the initial first color feature extraction submodel. The adjusted parameters of the initial first spatial feature extraction submodel are the first spatial feature extraction submodel, the adjusted initial second spatial feature extraction submodel is the second spatial feature extraction submodel, and the adjusted initial second color attention mechanism model is the second color attention mechanism model. The adjusted initial first color feature extraction submodel is the first color feature extraction submodel. The spatial feature extraction model includes the first spatial feature extraction submodel, the second spatial feature extraction submodel, the second color attention mechanism model, and the second spatial feature adjustment model, and the color feature extraction model includes the first color feature extraction submodel.

[0038] Optionally, the initial image generation model includes an initial first fusion model, a downsampling model, an initial feature map generation model, an upsampling model, and an initial second fusion model. The initial first fusion model is used to fuse the first training spatial feature and the first training color feature to obtain a training fusion feature. The downsampling model is used to downsample the training fusion feature to obtain a training downsampling feature. The initial feature map generation model is used to generate feature images for the training fusion feature and the training downsampling feature, respectively, to obtain a first training generated feature map corresponding to the training fusion feature, and a second training generated feature map corresponding to the training downsampling feature. The initial second fusion model is used to fuse the first generated feature map with the second generated feature map to obtain the deblurred original image file. The adjusting the parameters of the initial image processing system includes: adjusting the parameters of the initial feature map generation model. The adjusted initial feature map generation model is the feature map generation model.

[0039] In a third aspect, an image processing apparatus is provided, comprising a unit for executing the method of the first aspect or the second aspect. The apparatus may be a terminal device or a chip in the terminal device.

[0040] In a fourth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method of the first aspect or the second aspect.

[0041] In a fifth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to implement the method of the first aspect or the second aspect.

[0042] According to a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code, and the computer program code is used to implement the method of the first aspect.

[0043] According to a seventh aspect, a computer program product is provided, the computer program product comprising: a computer program code, wherein the computer program code is used to implement the method according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

[0045] Figure 2 is a schematic diagram of a software system for an electronic device applicable to the present application;

[0046] Figures 3 to 5 is a schematic diagram of a graphical user interface of an electronic device provided in an embodiment of the present application;

[0047] Figure 6 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

[0048] Figure 7 is a schematic structural diagram of an image processing system provided in an embodiment of the application;

[0049] Figure 8 is a schematic structural diagram of another image processing system provided in an embodiment of the application;

[0050] Fig. 9 is a schematic flow chart of an image processing system training method provided in an embodiment of the application;

[0051] Fig.10 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0052] Fig.11 It is a schematic diagram of a system architecture provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0054] As the functions of electronic devices become increasingly rich, electronic devices have become indispensable communication and entertainment tools in people's work and life. People can use the camera, video and other functions of electronic devices such as mobile phones to collect media data and view the collected image data.

[0055] When users use electronic devices to take photos, they will inevitably encounter the need to shake their hands or capture moving scenes. The captured image may be blurred due to camera shaking, target movement, or camera defocus. Image deblurring is a classic problem in computer underlying vision, and its purpose is to restore a clear image from a blurred image. Restoring a blurred image to a clear image can improve the imaging quality of electronic devices and enhance the user's photography experience.

[0056] However, different regions of a single image captured in practice often contain different degrees of blur, and the cause of blur is unpredictable. Therefore, image deblurring is a highly ill-posed optimization problem. How to restore an accurate and clear image is an urgent problem to be solved.

[0057] In order to solve the above problems, the present application provides an image processing method, an image processing system training method and an electronic device.

[0058] Figure 1 A hardware system of an electronic device suitable for the present application is shown.

[0059] The method provided in the embodiments of the present application can be applied to various electronic devices capable of image acquisition, such as mobile phones, tablet computers, wearable devices, laptop computers, netbooks, personal digital assistants (PDAs), and vehicle-mounted devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0060] Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0061] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0063] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0064] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0065] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0066] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0067] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0068] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0069] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0070] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0071] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0072] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0073] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0074] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0075] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0076] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0077] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0078] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the system library of the Android runtime (Android runtime), and the kernel layer. The application layer can include a series of application packages.

[0079] like Figure 2As shown, the application layer may include applications such as camera, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, wallpaper, gallery, multimedia editor, etc.

[0080] Applications such as cameras can be used to execute the image processing method provided in the embodiments of the present application.

[0081] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0082] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a wallpaper framework, and the like.

[0083] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for the electronic device 100. The resource manager provides various resources for applications. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction.

[0084] The wallpaper framework is used to display, schedule, store, etc. images based on user operations in the user interface provided by applications such as wallpapers, galleries, and multimedia editors.

[0085] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0086] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0087] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0088] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc. For example, the system library may include subject detection module, face detection module, key person detection module, aesthetic scoring module, image quality enhancement module, image cropping module, etc.

[0089] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0090] The kernel layer is the layer between hardware and software. The kernel layer may include driver modules such as display driver, camera driver, audio driver, and sensor driver.

[0091] Combine the following Figures 3 to 5 , the application scenarios of the image processing method provided in the embodiments of the present application are described.

[0092] Figure 3 (a) of the figure shows a graphical user interface (GUI) of an electronic device, and the GUI is a desktop 310 of the electronic device. When a camera start operation is detected in which a user clicks a camera icon 311 of a camera application (APP) on the desktop 310, the electronic device can start the camera application and display a Figure 3 Another GUI shown in (b) in the figure may be referred to as a shooting interface 320 .

[0093] The shooting interface 320 may include a viewfinder 321 and a photo icon 322. A preview image may be displayed in real time in the viewfinder 321. The preview image may be a visible image determined based on an original image file output in real time by a camera in the electronic device.

[0094] When detecting that the user clicks the photo icon 322, the electronic device can deblur the original image file output by the camera, determine the deblurred visible image based on the deblurred original image file image obtained after the deblurring process, and store the deblurred visible image.

[0095] The visible image determined according to the original image file output by the camera when the user clicks the photo icon 322 is as follows: Figure 3 As shown in (c) in .

[0096] The shooting interface 320 may also include a photo icon 323. When detecting that the user clicks the photo icon 323, the electronic device may display the following Figure 3 The album interface 330 shown in (d) in FIG. The album interface 330 includes a stored deblurred visible image 332 .

[0097] The deblurred image 412 may be obtained by deblurring the original image file output by the camera when the user clicks the photo icon 322 .

[0098] In some cases, the user may want to save the blurred image. In order to meet the user's needs, when it is detected that the user clicks the photo icon 322 in the shooting interface 320, the electronic device can display the following Figure 4 The album interface 410 shown in (a) of FIG. The album interface may include a visible image 411 represented by the original image file and a deblur icon 412. When detecting that the user clicks the deblur icon 412, the electronic device may display the following Figure 4 The album interface 420 shown in (b) of FIG. 40 may include a deblurred visible image 421 .

[0099] Or, if Figure 5 As shown in (a) of FIG. 3 , the shooting interface 320 may further include a shooting option icon 324. When detecting that the user clicks the shooting option icon 324, the electronic device may display the following Figure 5 Another GUI shown in (b) in FIG. 5 , which may be referred to as a shooting setting interface 510 .

[0100] The shooting setting interface 510 includes a switch icon 511. When the switch icon indicates that the deblurring process is in the on state, if it is detected that the user clicks the switch icon 511, the electronic device can set the deblurring process to the off state; conversely, when the switch icon indicates that the deblurring process is in the off state, if it is detected that the user clicks the switch icon 511, the electronic device can set the deblurring process to the on state. Thus, when it is detected that the user clicks the photo icon 322 in the shooting interface 320, the electronic device can display the following information: Figure 4 The album interface 420 shown in (b) in FIG.

[0101] If the deblurring process is on, when it is detected that the user clicks the photo icon 322 in the shooting interface 320, the electronic device can deblur the original image file output by the camera, determine the deblurred visible image based on the deblurred original image file image obtained after the deblurring process, and store the deblurred visible image.

[0102] On the contrary, if the deblurring process is turned on, when it is detected that the user clicks the photo icon 322 in the shooting interface 320, the electronic device can determine the blurred visible image based on the original image file output by the camera and store the blurred visible image.

[0103] Electronic devices can be Figure 6 The image processing method shown here performs deblurring on the original image file output by the camera.

[0104] Combine the following Figure 6 The image processing method provided in the embodiment of the present application is described in detail. The execution subject of the method provided in the present application can be an electronic device, or a software / hardware module capable of image processing in the electronic device. For the sake of convenience, the electronic device is used as an example in the following embodiments.

[0105] Figure 6 is a schematic flow chart of an image processing method provided by an embodiment of the present application. The method may include steps S610 to S620, and these steps are described in detail below.

[0106] Step S610, obtaining an original image file to be processed and a visible image to be processed, wherein the visible image to be processed is an image obtained by arranging data elements in the original image file to be processed according to pixels.

[0107] A raw image file may also be called a raw domain image, an original image, or a digital negative. A raw image file contains processed data from an image sensor such as a digital camera, scanner, or film scanner. In other words, a raw image file is image data captured from an image sensor and has not been processed. In other words, a raw image file contains physical information about the light intensity and color of the scene, and contains more information related to the image sensor.

[0108] The original image file may include multiple data elements, each data element representing data collected by a sensor element in the image sensor. The multiple data elements may be arranged in the order of the multiple sensor elements in the image sensor that collect the multiple data elements.

[0109] A visible image may also be called a color image or a color space image. A visible image may be obtained by arranging the data elements in the original image file according to pixels. In order to obtain a visible image, the original image file may be preprocessed. The preprocessing may include arranging the data elements according to pixels, and may also include processing such as color space conversion.

[0110] Visual images can be represented by color spaces such as red, green, blue (RGB) color space, hue, saturation, value (HSV) color space, Commission Internationale de L'Eclairage (CIE) XYZ color space, CIE Lab LCH color space, etc. Color space can also be called color space, color model, color model, color system, etc.

[0111] The RGB color space can also be described by a three-dimensional cube. Red (red, R), green (green, G), and blue (blue, B) are the three coordinate axes of the Cartesian coordinate system, and one vertex of the cube is located at the origin of the coordinates. The three edges adjacent to the vertex are located on the coordinate axis. Any color light can be added and mixed by the three primary colors of R, G, and B in different proportions. When the components of the three primary colors are all 0 (weakest), the mixed color light is black; when the components of the three primary colors are all the strongest, the mixed color light is white.

[0112] The visible image may include multiple channels, which correspond to multiple coordinate axes in the color space. The original image file can be understood as one channel. By arranging the data elements in the original image file according to pixels, the data elements corresponding to the same pixel in the original image file are located at the same position in multiple channels in the visible image. That is, the pixels corresponding to the same position in each channel in the visible image are the same.

[0113] The method of obtaining the original image file to be processed may be to receive the original image file to be processed sent by other electronic devices, or to read the original image file to be processed from a memory. Figure 6 The method shown can be applied in an electronic device, and the electronic device may include a camera. By receiving the original image file to be processed sent by the camera, the original image file to be processed can be obtained.

[0114] The method of obtaining the visible image to be processed may be to receive or read the visible image to be processed, or to determine the visible image to be processed according to the original image file to be processed.

[0115] Step S620, using an image processing system to process the original image file to be processed and the visible image to be processed to obtain a deblurred original image file, wherein the image processing system is a trained neural network model.

[0116] In order to facilitate the processing of the visual image to be processed by the neural network model, the visual image to be processed can represent the coordinate axes in the color space through 4 channels, and two of the 4 channels can represent the same coordinate axis. For example, the coordinate axes represented by the 4 channels of the visual image to be processed can be R, G, B, G respectively.

[0117] The original image file can be represented in the form of an array, and different sensing elements of the image sensor can correspond to different elements in the array. Therefore, the original image file has a linear mapping relationship with the scene irradiance. Each pixel in the visible image can be determined according to the elements corresponding to multiple sensing elements. Therefore, the original image file can reflect the information related to the position of the sensing element, that is, the information related to the space. The information related to the space has an important influence on the image quality obtained by image deblurring. Moreover, compared with the visible image represented by the original image file, the original image file generally has a higher bit width (about 10 to 20 bits (bit)) and contains richer information. Therefore, through steps S610 to S620, the obtained deblurred original image file can have a higher image quality.

[0118] It should be understood that the visible image to be processed can be obtained by rearranging the elements of the original image file to be processed according to pixels. The visible image to be processed can have 1 channel.

[0119] The image processing system can splice the original image file to be processed and the visible image to be processed, and process the spliced ​​image to obtain a deblurred original image file.

[0120] Alternatively, the image processing system may include a spatial feature extraction model, a color feature extraction model, and an image generation model. The spatial feature extraction model may be used to perform feature extraction on the original image file to be processed to obtain a first spatial feature. The color feature extraction model may be used to perform feature extraction on the visible image to be processed to obtain a first color feature. The image generation model may be used to process the first spatial feature and the first color feature to obtain a deblurred original image file.

[0121] By performing feature extraction on the original image file to be processed and the visible image to be processed respectively, the extracted features are made more consistent with the characteristics of the two different data, the original image file and the visible image, and the deblurred original image file is determined based on the extracted features, thereby making the image represented by the deblurred original image file clearer and having better image quality.

[0122] The image quality of an image represented by an original image file may be represented according to a peak signal-to-noise ratio (PSNR) of the original image file.

[0123] PSNR is an indicator used to measure video or image quality. The larger the PSNR value, the better the image quality.

[0124] Alternatively, the image quality of the image represented by the original image file can also be determined by converting the original image file into a visible image and then determining the image quality of the visible image. For example, the image quality can be subjectively evaluated through human perception of the visible image.

[0125] The image processing system processes the original image file to be processed and the visible image to be processed. The image processing system may output a deblurred original image file, or the image processing system may output difference image data. The difference image data may represent the difference between the deblurred original image file and the original image file to be processed. The deblurred original image file may be determined based on the difference image data and the original image file to be processed.

[0126] When the image processing system includes a spatial feature extraction model, a color feature extraction model and an image generation model, the image generation model can be used to process the first spatial feature and the first color feature, and can output a deblurred original image file or difference image data.

[0127] The image processing system processes the original image file to be processed and the visible image to be processed to obtain difference image data, which can make the processing process of the image processing system more compatible with the purpose of image processing such as deblurring, improve the flexibility of image deblurring processing, and improve the clarity of the image determined based on the deblurred original image file, thereby improving the image quality.

[0128] The visible image to be processed may include multiple image channels, which may correspond to multiple coordinate axes of a color space.

[0129] When the image generation model is used to obtain difference image data, the image processing system may further include a first spatial attention mechanism model and a first color feature adjustment model, and / or the image processing system also includes a first color attention mechanism model and a first spatial feature adjustment model.

[0130] In the case where the image processing system includes a first spatial attention mechanism model and a first color feature adjustment model, the first spatial attention mechanism model can be used to process the first color feature to obtain a first color weight. The first color weight represents the importance of the first color of each first spatial channel in the plurality of first spatial channels of the first spatial feature. The first spatial feature adjustment model can be used to adjust the first spatial feature according to the first color weight.

[0131] Thus, the image generation model can be used to process the first color feature and the adjusted first spatial feature to obtain difference image data.

[0132] The channel of the first spatial characteristic can be called the first spatial channel.

[0133] The first color weight may include a weight corresponding to each first spatial channel.

[0134] Multiple image channels in the visible image to be processed correspond to multiple coordinate axes of the color space, and the color channels in the first color feature obtained by processing the visible image to be processed are related to the color information. According to the first color feature, a first color weight is determined, and the first color weight can represent the importance of different colors for deblurring processing. According to the first color weight, the first spatial feature is adjusted so that the importance of the color information in the adjusted first spatial feature is adapted, and the deblurred original image file is determined according to the first color feature and the adjusted first spatial feature, so as to improve the clarity of the image represented by the deblurred original image file.

[0135] The first color attention mechanism model can determine the first color weight according to the average value or maximum value of multiple elements in each color channel in the first color feature.

[0136] In each color channel of the first color feature, there are a large number of elements. By calculating the average value of the multiple elements, the influence of abnormal values ​​can be eliminated.

[0137] Since the number of channels of the first color feature and the first spatial feature may not be equal, the first color attention mechanism model can perform feature extraction on a vector representing the average or maximum value of multiple elements in each color channel of the first color feature, so that the number of channels of the adjusted vector is equal to that of the first spatial feature.

[0138] The adjustment of the first spatial feature by the first spatial feature adjustment model according to the first color weight can be understood as multiplying the weight corresponding to each first spatial channel represented by the first color weight by the element of the first spatial channel in the first spatial feature to obtain the adjusted first spatial feature. In other words, each element in the adjusted first spatial feature is the product of the weight corresponding to the element in the first spatial feature before adjustment and the first spatial channel where the element is located.

[0139] The channels in the first color feature may also be referred to as color channels or first color channels.

[0140] In the case where the image processing system includes a first color attention mechanism model and a first spatial feature adjustment model, the first spatial attention mechanism model can be used to process the first spatial feature to obtain a first spatial weight. The first spatial weight represents the first position importance of each of the multiple first color positions in the first color feature. Each of the multiple first color channels of the first color feature includes a first color element located at the multiple first color positions.

[0141] Thus, the image generation model can be used to process the first spatial feature and the adjusted first color feature to obtain difference image data.

[0142] The first spatial weight may include a weight corresponding to each first color position.

[0143] According to the first spatial feature obtained by processing the original image file to be processed, a first spatial weight is determined, and the first spatial weight can represent the importance of different positions for deblurring processing. The first spatial feature is adjusted according to the first color weight so that the importance of the position in the adjusted first color feature is adapted, thereby improving the clarity of the image represented by the deblurred original image file determined according to the first spatial feature and the adjusted first color feature.

[0144] When the image processing system includes a first spatial attention mechanism model and a first color feature adjustment model, the first spatial attention mechanism model can determine the first spatial weight based on the average value or maximum value of multiple first spatial elements located at each first spatial position in multiple first spatial channels in the first spatial feature.

[0145] The size of the first color feature in each first color channel and the size of the first spatial feature in each first spatial channel may be equal or equal.

[0146] In some embodiments, the first spatial attention mechanism model can also perform feature extraction on the first spatial feature to adjust the number of channels of the first spatial feature so that the number of first spatial channels in the adjusted first spatial feature is equal to the number of first color channels in the first color feature.

[0147] The size of the first spatial feature after the number of channels is adjusted in each spatial channel may be equal to the size of the first spatial feature before the adjustment in each first color channel. The first spatial attention mechanism model may also determine the first spatial weight according to the average value or maximum value of multiple first spatial elements located at each first spatial position in multiple first spatial channels in the first spatial feature after the number of channels is adjusted.

[0148] The first spatial attention mechanism model can also use a feature map representing the average or maximum value of multiple first spatial elements located at each first spatial position in multiple first spatial channels in the first spatial feature as the first spatial weight, or the first spatial attention mechanism can perform feature extraction on the feature map to obtain the first spatial weight.

[0149] Spatial features can be understood as texture information in an image, and can be used to represent edge information in an image, that is, locations where there are differences between pixel values. The importance of the first spatial location can be determined by calculating the maximum value of multiple first spatial elements located at each first spatial location in multiple first spatial channels of the first spatial feature. Thus, the first spatial weight more accurately reflects the importance of the location, and the image represented by the deblurred original image file determined by the first spatial feature after adjusting the first spatial weight is clearer.

[0150] The adjustment of the first color feature by the first color feature adjustment model according to the first spatial weight can be understood as multiplying the weight corresponding to each first color position represented by the first spatial weight by the element of the first color position in the first color feature to obtain the adjusted first color feature. In other words, each element in the adjusted first color feature is the product of the weight corresponding to the element in the first color feature before adjustment and the first color position where the element is located.

[0151] When the image processing system includes a first spatial attention mechanism model, a first color feature adjustment model, a first color attention mechanism model and a first spatial feature adjustment model, the image generation model can determine the deblurred original image file based on the adjusted first spatial feature and the adjusted first color feature.

[0152] According to the color features obtained from the processed visual image, the adjustment of the spatial features obtained from the original image file to be processed can also be performed during the spatial feature extraction process. Similarly, according to the spatial features obtained from the original image file to be processed, the adjustment of the color features obtained from the visual image to be processed can also be performed during the color feature extraction process.

[0153] Exemplarily, the color feature extraction model may include a first color feature extraction sub-model, a second color feature extraction sub-model, a second spatial attention mechanism model and a second color feature adjustment model; the spatial feature extraction model includes a first spatial feature extraction sub-model.

[0154] The first spatial feature extraction sub-model is used to perform feature extraction on the original image file to be processed to obtain the second spatial feature, and the first spatial feature is determined according to the second spatial feature.

[0155] The first color feature extraction sub-model is used to extract features from the visible image to be processed to obtain a second color feature.

[0156] The second spatial attention mechanism model is used to process the second spatial feature to obtain a second spatial weight, where the second spatial weight represents the second position importance of each second color position in a plurality of second color positions in the second color feature, and each second color channel in a plurality of second color channels of the second color feature includes second color elements located at a plurality of second color positions.

[0157] The second color feature adjustment model is used to adjust the second color feature according to the second spatial weight.

[0158] The second color feature extraction sub-model is used to perform feature extraction on the adjusted second color feature to obtain the first color feature.

[0159] The adjustment of the second color feature according to the second spatial weight can be understood as multiplying the weight corresponding to each second color position represented by the second spatial weight by the element of the second color position in the second color feature to obtain the adjusted second color feature. In other words, each element in the adjusted second color feature is the product of the weight corresponding to the element in the second color feature before adjustment and the second color position where the element is located.

[0160] In the case where a second color feature is obtained by extracting a feature of the visible image to be processed, the second color feature obtained by extracting a feature of the visible image to be processed is adjusted according to a second spatial weight, the second spatial weight is determined according to the second spatial feature obtained by extracting a feature of the original image file to be processed, and the second spatial weight represents the second position importance of each second color position in a plurality of second color positions in the second color feature. The first color feature can be obtained by extracting a feature of the adjusted second color feature, and a deblurred original image file can be obtained according to the first color feature.

[0161] Thus, in the process of determining the first color feature based on the visible image to be processed, the second spatial weight is determined based on the original image file to be processed, and the first color feature is determined based on the second spatial weight. In the process of determining the first color feature, the importance of the position in the feature determined based on the original image file to be processed is considered, so that the deblurred original image file is more accurate, that is, the image determined based on the deblurred original image file is clearer and has a higher image quality.

[0162] The second spatial attention mechanism model can determine the second color weight according to the average value or maximum value of multiple second spatial elements located at each second spatial position in multiple second spatial channels in the second spatial feature.

[0163] The second spatial attention mechanism may use a feature map representing the average or maximum value of multiple second spatial elements located at each second spatial position in multiple second spatial channels in the second spatial feature as the second spatial weight, or the second spatial attention mechanism may perform feature extraction on the feature map to obtain the second spatial weight.

[0164] The second spatial attention mechanism can also perform feature extraction on the second spatial feature to adjust the number of channels of the second spatial feature so that the number of first spatial channels in the adjusted first spatial feature is equal to the number of first color channels in the first color feature. The first spatial attention mechanism model can also determine the second spatial weight according to the average value or maximum value of multiple second spatial elements located at each second spatial position in multiple second spatial channels in the second spatial feature after the number of channels is adjusted.

[0165] Exemplarily, the spatial feature extraction model may include a first spatial feature extraction sub-model, a second spatial feature extraction sub-model, a second color attention mechanism model and a second spatial feature adjustment model; the color feature extraction model may include a first color feature extraction sub-model.

[0166] The first color feature extraction sub-model is used to perform feature extraction on the visible image to be processed to obtain a second color feature, and the first color feature is determined based on the second color feature.

[0167] The first spatial feature extraction sub-model is used to perform feature extraction on the original image file to be processed to obtain the second spatial feature.

[0168] The second color attention mechanism model is used to process the second color feature to obtain a second color weight, where the second color weight represents the second color importance of each second spatial channel in multiple second spatial channels of the second spatial feature.

[0169] The second spatial feature adjustment model is used to adjust the second spatial feature according to the second color weight.

[0170] The second spatial feature extraction sub-model is used to extract features from the adjusted second spatial features to obtain the first spatial features.

[0171] The adjustment of the second spatial feature according to the second color weight can be understood as multiplying the weight corresponding to each second spatial channel represented by the second color weight by the element of the second spatial channel in the second spatial feature to obtain the adjusted second spatial feature. In other words, each element in the adjusted second spatial feature is the product of the weight corresponding to the element in the second spatial feature before adjustment and the second spatial channel where the element is located.

[0172] The second color attention mechanism model can determine the second spatial weight according to the average value or maximum value of multiple elements in each second color channel in the second color feature.

[0173] For example, the second color attention mechanism model can perform feature extraction on a vector representing the average value or maximum value of multiple elements in each second color channel in the second color feature to obtain a second spatial weight.

[0174] In the case where the second spatial feature is obtained by feature extraction of the original image file to be processed, the second spatial feature is adjusted according to the second color weight, the second spatial weight is determined according to the second color feature obtained by feature extraction of the visible image to be processed, and the second color weight represents the importance of the second color of each second spatial channel in the plurality of second spatial channels of the second spatial feature. The first spatial feature can be obtained by feature extraction of the adjusted second spatial feature, and the deblurred original image file can be obtained according to the first spatial feature.

[0175] Thus, in the process of determining the first spatial feature based on the original image file to be processed, the second color weight is determined based on the visible image to be processed, and the first spatial feature is determined based on the second color weight. In the process of determining the first spatial feature, the importance of the color-related channels in the feature determined based on the visible image to be processed is considered, so that the deblurred original image file is more accurate, that is, the image determined based on the deblurred original image file is clearer and has a higher image quality.

[0176] For spatial features such as the first spatial feature and the second spatial feature, and color features such as the first color feature and the second color feature, when the image generation model is used to obtain difference image data, the value of each element in the spatial feature and the color feature can reflect the degree of difference between the deblurred original image file and the original image file to be processed. Therefore, when the image generation model is used to obtain difference image data, the spatial weight obtained by processing the spatial feature using the attention mechanism model can reflect the importance of the spatial position; the color weight obtained by processing the color feature using the color attention mechanism model can reflect the importance of the channel.

[0177] After determining the first spatial feature and the second color feature, the image generation model may process the first spatial feature and the second color feature.

[0178] The image generation model may include a fusion module and a feature map generation model. The fusion module may be used to fuse the first spatial feature and the second color feature. The feature map generation model may process the fused feature to obtain a deblurred original image file.

[0179] Exemplarily, the image generation model may further include an image feature extraction module, which may be used to perform feature extraction on the output of the feature map generation model to obtain a deblurred original image file.

[0180] Alternatively, the image generation model may include a first fusion model, a downsampling model, a feature map generation model, and a second fusion model.

[0181] The first fusion model is used to fuse the first spatial feature and the first color feature to obtain a fusion feature.

[0182] The downsampling model is used to downsample the fused features to obtain downsampled features.

[0183] It should be understood that the number of downsampled features obtained by downsampling the fused features through the downsampling model may be one or more. In the case where the number of downsampled features is multiple, different downsampled features correspond to different downsampling multiples.

[0184] In the case where there are multiple downsampled features, the downsampling model can downsample the fused features using different downsampling multiples to obtain multiple downsampled features. Alternatively, the downsampling model can downsample the fused features according to a preset downsampling multiple, and then downsample the obtained downsampled features again according to a preset downsampling multiple to obtain another downsampled feature. Thus, by downsampling the features obtained by the last downsampling again through the downsampling model, multiple downsampled features with different downsampling multiples relative to the fused features can be obtained.

[0185] The feature map generation model is used to generate feature images for the fused features and the downsampled features respectively, so as to obtain a first generated feature map corresponding to the fused features and a second generated feature map corresponding to the downsampled features.

[0186] The second fusion model is used to fuse the first generated feature map with at least one second generated feature map to obtain a deblurred original image file.

[0187] The fused features are downsampled by the downsampling model to obtain downsampled features, and the feature map generation model used to process the fused features to obtain the first generated feature map is used to process the downsampled features to obtain a second generated feature map, and then the second generated feature map is fused with the first generated feature map to obtain a deblurred original image file. Thus, different degrees of blur in the original image file to be processed are removed, and a coarse to fine process is achieved, so that the image corresponding to the obtained deblurred original image file is clearer.

[0188] The sizes of the first generated feature map and the second generated feature map input into the second fusion model may be the same or different.

[0189] In order to make the second generated feature map have the same size as the first generated feature map, the image generation model may further include an upsampling model.

[0190] The upsampling model is used to upsample the feature map output by the feature map generation model after processing the downsampled features to obtain a second generated feature map, so that the second generated feature map has the same size as the first generated feature map. The feature map output by the feature map generation model after processing the downsampled features can be called a third generated feature map.

[0191] The feature map generation model may include a first generation sub-model and at least one second generation sub-model.

[0192] The first generation sub-model can be used to generate a feature image for the fused feature to obtain a first generated feature map corresponding to the fused feature.

[0193] Different downsampling models may correspond to different downsampling multiples. Each second generation sub-model may be used to generate a feature image for a downsampling feature having a downsampling multiple corresponding to the second generation sub-model, so as to obtain a second generated feature map corresponding to the downsampling feature.

[0194] When there are multiple down-sampled features, there may also be multiple second generation sub-models.

[0195] Regarding the parameters of the first generation sub-model and the at least one second generation sub-model, the parameters of these generation sub-models may be equal or unequal.

[0196] When the parameters of the first generation sub-model and the second generation sub-model are the same, since the parameters are equal, when storing the first generation sub-model and the second generation sub-model, only one of the first generation sub-model and the second generation sub-model may be stored, thereby reducing the storage space occupied by the image processing system.

[0197] It should be understood that Figure 6 The method shown can be processed by a CPU, or by a CPU and a GPU together, or by using other processors suitable for neural network calculations without using a GPU, and this application does not impose any restrictions.

[0198] Figure 6 The method can be specifically carried out as follows Fig.11 The execution device 1110 shown executes, Figure 6 The original image file to be processed in the method described above can be as follows Fig.11The pre-processing module 1113 in the execution device 1110 can be used to process the raw image file to be processed to obtain the visible image to be processed, given the input data provided by the client device 1140. The calculation module 1111 in the execution device 1110 can be used to execute steps S610 to S620.

[0199] The method provided in the embodiment of the present application utilizes the trained image processing system to process the original image file to be processed and the visible image to be processed determined based on the original image file to be processed, and comprehensively considers the color information in the visible image to be processed and the space-related information recorded in the original image file to be processed, so that the deblurred original image file has a higher image quality, that is, the image processing method of the present application has a higher deblurring effect.

[0200] Combine the following Figure 7 ,right Figure 6 The image processing method shown is used in an image processing system as described below.

[0201] Figure 7 It is a schematic structural diagram of an image processing system provided in an embodiment of the present application.

[0202] The image processing system 700 may be a deep neural network (DNN).

[0203] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0204]

[0205] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0206] DNN can also be called a multi-layer neural network, which can be understood as a neural network with multiple hidden layers. According to the position of different layers, DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0207] Although DNN looks complicated, the work of each layer is not complicated. The work of each layer in the deep neural network can be expressed mathematically as To describe: From a physical level, the work of each layer in a deep neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a deep neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0208] Therefore, DNN is simply the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0209] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0210] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0211] The image processing system 700 may include a spatial feature extraction model 710, a color feature extraction model 720, a first spatial attention mechanism model 731, a first color feature adjustment model 741, a first color attention mechanism model 751, a first spatial feature adjustment model 761 and an image generation model 770.

[0212] The spatial feature extraction model 710 includes a first spatial feature extraction sub-model 711, a second spatial feature extraction sub-model 712, a second color attention mechanism model 752 and a second spatial feature adjustment model 762.

[0213] The color feature extraction model 720 includes a first color feature extraction sub-model 721 , a second color feature extraction sub-model 722 , a second spatial attention mechanism model 732 and a second color feature adjustment model 742 .

[0214] The image generation model 770 includes a first fusion model 771 , a downsampling model 772 , a feature map generation model 773 , an upsampling model 774 , and a second fusion model 775 .

[0215] The first spatial feature extraction sub-model 711 is used to extract features from the original image data to be processed to obtain second spatial features.

[0216] Figure 8 700 is a structural diagram of an image processing system 700 provided in an embodiment of the present application. The model in the image processing system 700 may have the following Figure 8 The structure shown.

[0217] The first spatial feature extraction sub-model 711 may include convolutional layers 811 and 812 .

[0218] A convolutional layer refers to a neuron layer in a convolutional neural network that performs convolution processing on the input signal. In a convolutional layer of a convolutional neural network, a neuron can only be connected to some neurons in adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as a way of extracting audio information that is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0219] The convolution layer can include many convolution operators, also called kernels. Its role in natural language processing is equivalent to a filter that extracts specific information from the input speech or semantic information. The convolution operator can essentially be a weight matrix, which is usually predefined.

[0220] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input data, thereby helping the image processing system 700 to make correct predictions.

[0221] When input data of an image processing system is processed by multiple convolutional layers in sequence, the initial convolutional layer (for example, convolutional layer 811) tends to extract more general features, which can also be called low-level features. As the depth of the deep neural network increases, the features extracted by the subsequent convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0222] The first color feature extraction sub-model 721 is used to extract features from the visible image to be processed to obtain a second color feature.

[0223] The first color feature extraction submodel 721 may include a convolution layer 813 and a convolution layer 814 , and the first color feature extraction submodel 721 may further include a downsampling layer 821 .

[0224] The raw image data may be represented in an array form, and each data element in the array of the raw image data is used to represent data collected by a sensor element corresponding to the data element in the image sensor array. The relative position relationship of the multiple data elements in the array of the raw image data is the same as the relative position relationship of the sensor elements corresponding to the multiple data elements in the image sensor.

[0225] The visible image to be processed may be obtained by rearranging elements in the original image data. The original image data includes a plurality of data elements corresponding to each pixel. The plurality of data elements corresponding to the same pixel may correspond to a plurality of coordinate axes in the color space. Among the plurality of data elements corresponding to different pixels, the positions of the data elements corresponding to the same coordinate axis in the pixels where they are located may be the same or different in different pixels.

[0226] The data elements corresponding to the same coordinate axis in the color space in the visible image to be processed are extracted, and the elements corresponding to different coordinate axes in the color space can be used as different channels in the visible image to be processed. For example, the four channels in the visible image to be processed can correspond to the coordinate axes R, G, B, and G of the color space respectively. Exemplarily, in the original image data to be processed with a size of 512×512, the multiple data elements corresponding to each pixel can correspond to R, G, B, and G respectively. The size of the visible image to be processed determined according to the original image data to be processed with a size of 512×512 can be expressed as 4×256×256, where 4 represents the four channels R, G, B, and G of the visible image to be processed.

[0227] The lengths and widths of the two features obtained by processing the visible image to be processed and the original image data to be processed respectively using the multiple convolution layers in the first color feature extraction submodel 721 and the multiple convolution layers in the first spatial feature extraction submodel 711 may be unequal. Adding a downsampling layer after the convolution layer with a larger feature length, for example, adding a downsampling layer 821 after the convolution layer 814, can make the length and width of the second spatial feature equal to the second color feature.

[0228] For example, the number of input channels of the convolution layer 811 may be 4, the number of output channels may be 4, and the convolution kernel size may be 3; the number of input channels of the convolution layer 812 may be 4, the number of output channels may be 8, and the convolution kernel size may be 3; the number of input channels of the convolution layer 813 may be 1, the number of output channels may be 4, and the convolution kernel size may be 3; the number of input channels of the convolution layer 814 may be 4, the number of output channels may be 8, and the convolution kernel size may be 3. For the original image data of 1×512×512 to be processed, the size of the feature output by the convolution layer 811 may be 4×256×256, and the size of the second color feature may be 8×256×256; the size of the feature output by the convolution layer 813 may be 4×512×512, and the size of the feature output by the convolution layer 814 may be 8×512×512. The feature output by the convolution layer 814 is processed by the downsampling layer 821, and the size of the second color feature may be 16×256×256.

[0229] The second color attention mechanism model 752 is used to process the second color feature to obtain a second color weight. The second color weight represents the importance of the second color of each second spatial channel in the plurality of second spatial channels of the second spatial feature.

[0230] The attention mechanism is a method that mimics the human visual and cognitive system. The attention mechanism mimics the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sensations to increase the fineness of observation of some areas, and can use limited attention resources to quickly filter out high-value information from a large amount of information. The attention mechanism allows the neural network to focus on relevant parts when processing input data. By introducing the attention mechanism, the neural network can automatically learn and selectively focus on important information in the input, improving the performance and generalization ability of the model.

[0231] The second color attention mechanism model 752 may include an average pooling layer 831 , a convolution layer 832 , a convolution layer 833 , and an activation function 834 .

[0232] Pooling layers are used to reduce the spatial size of data.

[0233] Average pooling refers to dividing the input data into several rectangular or long sub-regions and outputting the average value of all elements in each sub-region.

[0234] The average pooling layer 831 may also be referred to as a global average pooling layer, and is used to calculate the average value of each channel in the second color feature. The channel of the second color feature may be referred to as a second color channel.

[0235] The convolution layer 832 and the convolution layer 833 may be used for feature extraction. Furthermore, the convolution layer 832 and / or the convolution layer 833 may adjust the number of channels so that the number of channels of the output of the convolution layer 833 is equal to the number of channels of the second spatial feature.

[0236] For the 1×512×512 original image data to be processed, the size of the output of the average pooling layer 831 can be 8×1×1, the size of the output of the convolution layer 832 can be 8×1×1, and the size of the output of the convolution layer 833 can be 16×1×1.

[0237] The activation function 834 may be a sigmoid. The activation function sigmoid is a nonlinear function that can map an input real value to a value between 0 and 1 and has saturation. Its input can be any real number, and the output result is always between 0 and 1.

[0238] The second color feature adjustment model 742 is used to adjust the second color feature using the second spatial weight, that is, using the second position importance of each second position as the weight of the second position in the second color feature. Each of the multiple second color channels of the second color feature includes a second initial color element located at the multiple second positions.

[0239] The second color feature extraction sub-model 722 is used to extract features from the adjusted second color features obtained by the second color feature adjustment model 742 to obtain the first color features.

[0240] The second color feature extraction sub-model 722 may include a convolution layer 815 , a convolution layer 816 , and a down-sampling layer 822 .

[0241] Convolutional layer 815 and convolutional layer 816 are used to extract features from input data. The number of input channels of convolutional layer 815 may be 8, the number of output channels may be 16, and the convolution kernel size may be 3; the number of input channels of convolutional layer 816 may be 8, the number of output channels may be 16, and the convolution kernel size may be 3.

[0242] For the 1×512×512 original image data to be processed, the size of the output of the convolution layer 815 can be 8×256×256, the size of the output of the convolution layer 816 can be 16×256×256, and the size of the output of the downsampling layer 822 can be 32×128×128.

[0243] The second spatial attention mechanism model 732 is used to process the second spatial feature to obtain a second spatial weight. The second spatial weight represents the second position importance of each second position in the plurality of second positions of the second spatial feature. The second spatial feature includes a second spatial element located at a plurality of second positions in each second spatial channel in the plurality of second spatial channels.

[0244] The second spatial attention mechanism model 732 includes a convolution layer 841, a maximum pooling layer 842, a convolution layer 843, and an activation function 844.

[0245] The convolution layer 841 is used to extract the second spatial feature. The output of the convolution layer 841 is equal to the length and width of the second spatial feature.

[0246] The maximum pooling layer 842 is used to maximize the different positions in the output of the convolution layer 841. That is, the output of the maximum pooling layer 842 is 1 channel, and in this channel, the element at each position is the maximum value of the elements at this position in each channel in the output of the convolution layer 841.

[0247] The convolution layer 843 is used to extract features from the output of the maximum pooling layer 842. The size of the output of the convolution layer 843 may be equal to the size of the output of the maximum pooling layer 842.

[0248] The activation function 844 may be a sigmoid.

[0249] For the 1×512×512 original image data to be processed, the size of the output of the convolution layer 841 can be 8×256×256, the size of the output of the maximum pooling layer 842 can be 1×256×256, the size of the output of the convolution layer 843 can be 1×256×256, and the size of the output of the activation function 844 can be 1×256×256.

[0250] The second spatial feature adjustment model 762 is used to adjust the second spatial feature by using the second color importance of each second spatial channel as the weight of the second spatial channel in the second spatial feature, that is, by using the second color weight.

[0251] The second spatial feature extraction sub-model 712 is used to extract features from the adjusted second spatial features obtained by the second spatial feature adjustment model 762 to obtain the first spatial features.

[0252] Convolutional layer 815 and convolutional layer 816 are used to extract features from input data. The number of input channels of convolutional layer 817 may be 16, the number of output channels may be 16, and the convolution kernel size may be 3; the number of input channels of convolutional layer 816 may be 16, the number of output channels may be 32, and the convolution kernel size may be 3.

[0253] For the 1×512×512 original image data to be processed, the size of the output of the convolution layer 817 can be 16×256×256, the size of the output of the convolution layer 816 can be 32×256×256, and the size of the output of the downsampling layer 822 can be 64×128×128.

[0254] The first spatial attention mechanism model 731 can be used to process the first spatial features to obtain the first spatial weights.

[0255] The first spatial weight indicates the first position importance of each first position in the plurality of first positions of the second spatial feature. The first spatial feature includes first spatial elements located at the plurality of first positions in each first spatial channel in the plurality of first spatial channels.

[0256] The first color attention mechanism model 751 can be used to process the first color feature to obtain the first color weight.

[0257] The first color weight indicates the importance of the first color of each first spatial channel in the first spatial feature.

[0258] The first color weight and the first spatial weight may have the same length and width.

[0259] Similar to the second spatial attention mechanism model 732 , the first spatial attention mechanism model 731 may include a convolution layer 861 , a maximum pooling layer 862 , a convolution layer 863 , and an activation function 864 .

[0260] The convolution layer 861 is used to extract the first spatial feature. The output of the convolution layer 831 is equal to the length and width of the first spatial feature.

[0261] The maximum pooling layer 862 is used to maximize the different positions in the output of the convolution layer 861. That is, the output of the maximum pooling layer 862 is 1 channel, and in this channel, the element at each position is the maximum value of the elements at this position in each channel in the output of the convolution layer 861.

[0262] The convolution layer 863 is used to extract features from the output of the maximum pooling layer 862. The size of the output of the convolution layer 863 may be equal to the size of the output of the maximum pooling layer 862.

[0263] The activation function 864 may be a sigmoid.

[0264] Similar to the second color attention mechanism model 752 , the first color attention mechanism model 751 may include an average pooling layer 851 , a convolution layer 852 , a convolution layer 853 , and an activation function 854 .

[0265] The average pooling layer 851 may also be referred to as a global average pooling layer, and is used to calculate the average value of each channel in the first color feature. The channel of the first color feature may be referred to as a first color channel.

[0266] The convolution layer 852 and the convolution layer 853 may be used for feature extraction. Furthermore, the convolution layer 852 and / or the convolution layer 853 may adjust the number of channels so that the number of channels of the output of the convolution layer 853 is equal to the number of channels of the first spatial feature.

[0267] The activation function 854 may be a sigmoid.

[0268] The first spatial feature adjustment model 761 is used to adjust the first spatial feature by using the first color importance of each first spatial channel as the weight of the first spatial channel in the first spatial feature, that is, by using the first color weight.

[0269] The channels in the first spatial feature may be referred to as first spatial channels.

[0270] The first color feature adjustment model 741 is used to adjust the first color feature by taking the first position importance of each first position as the weight of the second position in the first color feature, that is, by taking the first spatial weight. Each second color channel in the multiple first color channels of the first color feature includes a second initial color element located at the multiple second positions.

[0271] The image generation model 770 is used to process the adjusted first color feature obtained by the first color feature adjustment model 741 and the adjusted first spatial feature obtained by the first spatial feature adjustment model 761 to generate deblurred original image data.

[0272] The first fusion model 771 is used to fuse the adjusted first color feature and the adjusted first space feature to obtain a fusion feature. For example, the first fusion model 771 can splice the adjusted first color feature and the adjusted first space feature to obtain a fusion feature.

[0273] The downsampling model 772 may downsample the fused feature to obtain at least one downsampled feature. In the case where there are multiple downsampled features, the multiple downsampled features may be obtained by the downsampling model 772 downsampling the fused feature by different multiples.

[0274] The downsampling model 772 may include a downsampling layer 881 and a downsampling layer 882 .

[0275] The downsampling layer 881 is used to downsample the fused feature to obtain a first downsampled feature. The downsampling layer 882 is used to downsample the first downsampled feature to obtain a second downsampled feature.

[0276] Therefore, the downsampling multiples of the first downsampled feature and the second downsampled feature relative to the fused feature are different.

[0277] The feature map generation model 773 is used to generate feature images for the fusion feature and at least one downsampled feature output by the downsampled model 772, respectively, to obtain a first generated feature map corresponding to the fusion feature and at least one second generated feature map corresponding to the at least one downsampled feature.

[0278] The feature map generation model 773 may include a generation sub-model 871, a generation sub-model 872, and a generation sub-model 873. The generation sub-model 871, the generation sub-model 872, and the generation sub-model 873 may all be U-type deep neural networks (U-net). The generation sub-model may be lightweight and improved according to the support level of the electronic device in which it is located to reduce the memory occupied by the model and the reasoning time.

[0279] The generating submodel 871 is used to process the fused feature to obtain a first generating feature map corresponding to the fused feature. The generating submodel 872 is used to process the first down-sampled feature to obtain a third generating feature map corresponding to the first down-sampled feature. The generating submodel 873 is used to process the second down-sampled feature to obtain a third generating feature map corresponding to the second down-sampled feature. The generating submodel 871, the generating submodel 872, and the generating submodel 873 may have the same structure, and the parameters in the generating submodel 871, the generating submodel 872, and the generating submodel 873 may be the same or different.

[0280] The upsampling model 774 is used to upsample the at least one third generated feature map to obtain at least one second generated feature map. The at least one second generated feature map has the same size as the first generated feature map.

[0281] The upsampling model 774 may include an upsampling layer 883 and an upsampling layer 884 .

[0282] The upsampling layer 884 is used to upsample the third generated feature map corresponding to the second downsampled feature. The upsampling multiple of the upsampling layer 882 is the same as the downsampling multiple of the downsampling layer 884.

[0283] The upsampling layer 883 is used to upsample the third generated feature map corresponding to the first downsampled feature to obtain the first downsampled feature to obtain the second generated feature map. The upsampling layer 883 is also used to upsample the feature map output by the upsampling layer 884 to obtain the second downsampled feature to obtain the second generated feature map. The upsampling multiple of the upsampling layer 883 is the same as the downsampling multiple of the downsampling layer 881.

[0284] The fusion model 775 is used to fuse the first generated feature map and at least one second generated feature map to obtain a deblurred original image file.

[0285] The output of the fusion model 775 may be referred to as a fused image feature.

[0286] The image generation model may further include an image extraction model 776. The image extraction model 776 is used to extract features of the fused image to obtain a deblurred original image file. The image extraction model 776 may include a convolution layer 874 and a convolution layer 875.

[0287] Convolutional layer 874 and convolutional layer 875 can be used to extract features from the input feature map.

[0288] Combine the following Fig. 9 ,right Figure 6 The image processing method shown is used to explain the training method of the image processing system.

[0289] Fig. 9 It is a schematic flowchart of an image processing system training method provided in an embodiment of the present application. Fig. 9 The method shown includes steps S910 to S930.

[0290] Step S910, obtaining training samples and label deblurred original image files, wherein the training samples include training original image files and training visible images, and the training visible images are images obtained by arranging training data elements in the training original image files according to pixels.

[0291] The training original image file can be obtained by adding noise to the label deblurred original image file.

[0292] Alternatively, the label deblurred original image file and the training original image file may be images captured by two image sensors for the same field of view. For example, a beam splitter may be used to allow light passing through the same lens to be transmitted to the two image sensors. The exposure time of the image sensor used to capture the label deblurred original image file may be shorter than the exposure time of the image sensor used to capture the training original image file.

[0293] For moving objects, the original image file captured by the image sensor with a shorter exposure time indicates a clearer image.

[0294] Step S920: Process the sample using the initial image processing system to obtain a training deblurred original image file.

[0295] Step S930, adjusting the parameters of the initial image processing system according to the training difference between the training deblurred original image file and the label deblurred original image file, and the adjusted initial image processing system is the trained image processing system.

[0296] The training difference between the deblurred original image file and the label deblurred original image file can be represented by a loss value.

[0297] In the process of training the neural network model, because the output of the neural network model is as close as possible to the value that we really want to predict, we can compare the predicted value of the current network with the target value that we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network model). For example, if the model's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network model can predict the target value that we really want or a value that is very close to the target value that we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value of the loss function, that is, the loss value (loss), the greater the difference, so the training of the neural network model becomes a process of minimizing this loss as much as possible.

[0298] The back propagation (BP) algorithm can be used to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0299] The training difference may be determined based on a mean square error and / or structural similarity between the training deblurred original image file and the label deblurred original image file.

[0300] Structural similarity (SSIM) is a measure of the similarity between two images. From the perspective of image composition, the structural similarity index defines structural information as an attribute that is independent of brightness and contrast and reflects the structure of objects in the scene, and models distortion as a combination of three different factors: brightness, contrast, and structure. Structural similarity uses the mean as an estimate of brightness, the standard deviation as an estimate of contrast, and the covariance as a measure of structural similarity.

[0301] For example, the training difference L can be expressed as L = L2 + λ·L SSIM , where L2 represents the mean square error, L SSIM represents the structural similarity, and λ represents the preset coefficient.

[0302] According to the training difference between the training deblurred original image file and the label deblurred original image file, the parameters of the initial image processing system are adjusted to minimize the training difference. When the training difference converges or the training difference meets the requirements, the training is terminated to obtain the adjusted neural network model.

[0303] Optionally, the initial image processing system includes an initial spatial feature extraction model, an initial color feature extraction model and an initial image generation model.

[0304] The initial spatial feature extraction model is used to extract features from the original training image file to obtain the first training spatial feature.

[0305] The initial color feature extraction model is used to extract features from the training visible image to obtain the first training color feature.

[0306] The initial image generation model is used to process the first training space feature and the first training color feature to obtain a training deblurred original image file.

[0307] In step S930, adjusting the parameters of the initial image processing system may include: adjusting the parameters of the initial spatial feature extraction model, the initial color feature extraction model and the initial image generation model, and the image processing system includes the adjusted initial spatial feature extraction model, the adjusted initial color feature extraction model and the adjusted initial image generation model.

[0308] The adjusted initial spatial feature extraction model may be referred to as a spatial feature extraction model, the adjusted initial color feature extraction model may be referred to as a color feature extraction model, and the adjusted initial image generation model may be referred to as an image generation model.

[0309] Optionally, the initial image generation model is specifically used to process the first training space feature and the first training color feature to obtain training difference image data, where the training difference image data represents the difference between the training deblurred original image file and the training original image file.

[0310] In step S930, the parameters of the initial image processing system are adjusted according to the training difference between the training deblurred original image file and the label deblurred original image file. It can be understood that the parameters of the initial image processing system are adjusted according to the difference between the training difference image data and the label difference image data. The label difference image data represents the difference between the label deblurred original image file and the training deblurred original image file.

[0311] That is, the training difference can be expressed as the difference between the training difference image data and the label difference image data,

[0312] Optionally, the initial image processing system also includes an initial first spatial attention mechanism model and a first color feature adjustment model.

[0313] The initial first spatial attention mechanism model is used to process the first training spatial feature to obtain a first training spatial weight, where the first training spatial weight represents the first training position importance of each first training color position in the plurality of first training color positions in the first training color feature.

[0314] The first color feature adjustment model is used to adjust the first training color feature according to the first training space weight.

[0315] The initial image generation model is specifically used to process the first training space feature and the adjusted first training color feature to obtain training difference image data.

[0316] The adjustment of the parameters of the initial image processing system in step S930 may include the adjustment of the parameters of the initial first spatial attention mechanism model. The adjusted initial first spatial attention mechanism model is the first spatial attention mechanism model. The image processing system may include the first spatial attention mechanism model and the first color feature adjustment model.

[0317] Optionally, the initial first spatial attention mechanism model is used to determine the first training spatial weight by taking the maximum value of multiple first training spatial elements located at each first training spatial position in multiple first training spatial channels of the first training spatial feature.

[0318] Optionally, the initial image processing system also includes an initial first color attention mechanism model and a first spatial feature adjustment model, and multiple training image channels of the training visible image correspond to multiple coordinate axes of the color space.

[0319] The initial first color attention mechanism model is used to process the first training color feature to obtain a first training color weight, where the first training color weight represents the importance of the first training color of each first training space channel in multiple first training space channels of the first training space feature.

[0320] The first spatial feature adjustment model is used to adjust the first training spatial feature according to the first training color weight.

[0321] The initial image generation model is specifically used to process the first training color feature and the adjusted first training space feature to obtain the difference image data.

[0322] The parameter adjustment of the image processing system includes adjusting the parameters of the initial first color attention mechanism model. The adjusted initial first color attention mechanism model is the first color attention mechanism model, and the image processing system includes the first color attention mechanism model and the first spatial feature adjustment model.

[0323] Optionally, the initial first color attention mechanism model is specifically used to determine the first training color weight according to an average value of multiple elements in each training color channel in the first training color feature.

[0324] Optionally, the initial color feature extraction model includes an initial first color feature extraction sub-model, an initial second color feature extraction sub-model, an initial second spatial attention mechanism model and a second color feature adjustment model. The initial spatial feature extraction model includes an initial first spatial feature extraction sub-model.

[0325] The initial first spatial feature extraction sub-model is used to extract features from the original training image file to obtain the second training spatial features, and the first training spatial features are determined based on the second training spatial features.

[0326] The initial first color feature extraction sub-model is used to extract features from the training visible image to obtain the second training color features.

[0327] The initial second spatial attention mechanism model is used to process the second training spatial features to obtain second training spatial weights, where the second training spatial weights represent the second position importance of each second color position in the plurality of second color positions in the second color features.

[0328] The second color feature adjustment model is used to adjust the second training color feature according to the second training space weight.

[0329] The initial second color feature extraction sub-model is used to perform feature extraction on the adjusted second color feature to obtain the first color feature.

[0330] The adjustment of the parameters of the initial color feature extraction model includes the adjustment of the parameters of the initial first color feature extraction sub-model, the parameters of the initial second color feature extraction sub-model, and the parameters of the initial second spatial attention mechanism model. The adjustment of the parameters of the initial spatial feature extraction model includes the adjustment of the parameters of the initial first spatial feature extraction sub-model.

[0331] The adjusted initial first color feature extraction submodel can be referred to as the first color feature extraction submodel, the adjusted initial second color feature extraction submodel can be referred to as the second color feature extraction submodel, the adjusted initial second spatial attention mechanism model can be referred to as the second spatial attention mechanism model. The adjusted initial first spatial feature extraction submodel can be referred to as the first spatial feature extraction submodel.

[0332] The color feature extraction model may include a first color feature extraction sub-model, a second color feature extraction sub-model, a second spatial attention mechanism model, and a second color feature adjustment model. The spatial feature extraction model may include a first spatial feature extraction sub-model.

[0333] Optionally, the initial spatial feature extraction model includes an initial first spatial feature extraction sub-model, an initial second spatial feature extraction sub-model, an initial second color attention mechanism model and a second spatial feature adjustment model. The initial color feature extraction model includes an initial first color feature extraction sub-model.

[0334] The initial first color feature extraction sub-model is used to extract features from the training visible image to obtain the second training color feature, and the first training color feature is determined according to the second training color feature.

[0335] The initial first spatial feature extraction sub-model is used to extract features from the original training image file to obtain the second training spatial features.

[0336] The initial second color attention mechanism model is used to process the second training color feature to obtain a second training color weight, where the second training color weight represents the second training color importance of each second training space channel in multiple second training space channels of the second training space feature.

[0337] The second spatial feature adjustment model is used to adjust the second training spatial feature according to the second training color weight.

[0338] The initial second space feature extraction sub-model is used to perform feature extraction on the adjusted second training space features to obtain the first training space features.

[0339] The adjustment of the parameters of the initial spatial feature extraction model may include the adjustment of the parameters of the initial first spatial feature extraction sub-model, the parameters of the initial second spatial feature extraction sub-model, and the parameters of the initial second color attention mechanism model. The adjustment of the parameters of the initial color feature extraction model may include the adjustment of the parameters of the initial first color feature extraction sub-model.

[0340] The adjusted parameters of the initial first spatial feature extraction submodel can be referred to as the first spatial feature extraction submodel, the adjusted initial second spatial feature extraction submodel can be referred to as the second spatial feature extraction submodel, the adjusted initial second color attention mechanism model can be referred to as the second color attention mechanism model. The adjusted initial first color feature extraction submodel can be referred to as the first color feature extraction submodel.

[0341] The spatial feature extraction model includes a first spatial feature extraction sub-model, a second spatial feature extraction sub-model, a second color attention mechanism model and a second spatial feature adjustment model, and the color feature extraction model includes a first color feature extraction sub-model.

[0342] Optionally, the initial image generation model includes an initial first fusion model, a downsampling model, an initial feature map generation model, an upsampling model and an initial second fusion model.

[0343] The initial first fusion model is used to fuse the first training space feature and the first training color feature to obtain a training fusion feature.

[0344] The downsampling model is used to downsample the training fusion feature to obtain at least one training downsampled feature, and different training downsampled features correspond to different downsampling multiples.

[0345] The initial feature map generation model is used to generate feature images for the training fusion feature and at least one training down-sampled feature respectively to obtain a first training generated feature map corresponding to the training fusion feature and at least one second training generated feature map corresponding to the at least one training down-sampled feature.

[0346] It should be understood that different training down-sampled features may correspond to different training second generated feature maps.

[0347] The initial second fusion model is used to fuse the first generated feature map with the at least one second generated feature map to obtain the deblurred original image file.

[0348] Adjusting the parameters of the initial image generation model may include adjusting the parameters of the initial feature map generation model.

[0349] In the case where the initial first fusion model is used to splice the first training space feature and the first training color feature, the initial first fusion model can be used as the first fusion model. The splicing of features can also be understood as achieving feature fusion by feature concatenation.

[0350] When the initial first fusion model is used to fuse the first training spatial features and the first training color features by weighted fusion, feature superposition, etc., the adjustment of the parameters of the initial image generation model may also include adjustment of the parameters of the initial first fusion model, and the adjusted initial first fusion model is the first fusion model.

[0351] In the case where the initial first fusion model is fused in a weighted fusion manner, the parameters of the initial first fusion model adjusted during the training process may be weights. In the case where the initial first fusion model is fused in a feature superposition manner, the initial first fusion model may perform convolution operations and / or pooling operations on the first training spatial features and the first training color features, and the parameters of the initial first fusion model adjusted during the training process may be parameters of a convolution layer used for convolution operations and / or parameters of a pooling layer used for pooling operations in the initial first fusion model.

[0352] Similar to the initial first fusion model, when the second fusion model is fused by splicing, the initial second fusion model can be used as the second fusion model. When the second fusion model is fused by weighted fusion, feature superposition, etc., the adjustment of the parameters of the initial image generation model can also include the adjustment of the parameters of the initial second fusion model, and the adjusted initial second fusion model is the second fusion model.

[0353] The image generation model may include a first fusion model, a downsampling model, a feature map generation model, and a second fusion model.

[0354] In some embodiments, the initial image generation model may further include an upsampling model. The upsampling model is used to upsample the feature maps corresponding to at least one training downsampled feature output by the initial feature map generation model to obtain at least one second training generated feature map. The at least one second training generated feature map output by the upsampling model has the same size as the first training generated feature map.

[0355] Image generation models can also include upsampling models.

[0356] The trained image processing system can be verified using verification data, which includes verification samples and labeled deblurred original image files, the verification samples include verification original image files and verification visible images, and the verification visible images are images determined based on the verification original image files.

[0357] The verification sample is processed by an image processing system to obtain a verification deblurred original image file. According to the verification deblurred original image file and the marked deblurred original image file, the learned perceptual image patch similarity (LPIPS) can be calculated.

[0358] LPIPS can also be called perceptual loss, which is used to measure the difference between two images. LPIPS is more consistent with human perception than traditional methods (such as PSNR). The lower the LPIPS value, the more similar the two images are, and vice versa, the greater the difference.

[0359] pass Fig. 9 The image processing system trained by the method shown can be applied in Figure 6 In the image processing method shown. Figure 6 The image processing method shown is similar to Fig. 9 The image processing system training method shown can be executed by the same or different electronic devices. For example, the terminal device can be used to execute Figure 6 The image processing method shown in the figure can be used by the server to Fig. 9 The terminal device may be Figure 1 The electronic device 100 shown in FIG. performs Fig. 9 The apparatus of the method shown may also be Fig.11 The training device 1120 shown in FIG. 1120. The training samples and labels used in step S910 to deblur the original image file may be as follows: Fig.11 The training data maintained in the database 1130 is shown. Alternatively, the label deblurred original image file and the training original image file in the training sample can be the training data maintained in the database 1130.

[0360] Optionally, Fig. 9 S910 to S930 of the method shown can be executed in the training device 1120, or can be pre-executed by other functional modules before the training device 1120, that is, the training data received or acquired from the database 1130 is pre-processed, such as processing the training original image files in the training samples to obtain the training visible images in the training samples as the input of the training device 1120, and the training device 1120 executes S910 to S930.

[0361] Optionally, Fig. 9 The method shown can be processed by a CPU, or by a CPU and a GPU together, or by using other processors suitable for neural network calculations without using a GPU, and this application does not impose any restrictions.

[0362] The server may send the trained image processing system to the electronic device. When the electronic device obtains the original image file to be processed and the visible image to be processed, the electronic device may use the image processing system to process the acquired data to obtain the deblurred original image file. Thus, the electronic device may obtain a clear image according to the deblurred original image file.

[0363] It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, rather than to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0364] Combination of the above Figures 6 to 9 , describes in detail the image processing method and image processing system training method of the embodiment of the present application, and will now be combined with Fig.10 and Fig.11 , describes the device embodiment of the present application in detail. It should be understood that the image processing device in the embodiment of the present application can execute the various image processing methods and image processing system training methods in the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

[0365] Fig.10 It is a schematic diagram of an image processing device provided in an embodiment of the present application.

[0366] The image processing apparatus 1000 includes an acquisition unit 1010 and a processing unit 1020 .

[0367] In some embodiments, the image processing apparatus 1000 may execute Figure 6 The image processing method shown.

[0368] The acquisition unit 1010 is used to acquire an original image file to be processed and a visible image to be processed, wherein the visible image to be processed is an image obtained by arranging data elements in the original image file to be processed according to pixels;

[0369] The processing unit 1020 is used to process the original image file to be processed and the visible image to be processed by using an image processing system to obtain a deblurred original image file, and the image processing system is a trained neural network model.

[0370] Optionally, the image processing system includes a spatial feature extraction model, a color feature extraction model and an image generation model.

[0371] The spatial feature extraction model is used to extract features from the original image file to be processed to obtain a first spatial feature.

[0372] The color feature extraction model is used to extract features from the visible image to be processed to obtain a first color feature.

[0373] The image generation model is used to process the first spatial feature and the first color feature to obtain the deblurred original image file.

[0374] Optionally, the image processing system also includes a first spatial attention mechanism model and a first color feature adjustment model.

[0375] The first spatial attention mechanism model is used to process the first spatial feature to obtain a first spatial weight, where the first spatial weight represents the first position importance of each first color position in multiple first color positions in the first color feature.

[0376] The first color feature adjustment model is used to adjust the first color feature according to the first spatial weight.

[0377] The image generation model is specifically used to process the first spatial feature and the adjusted first color feature to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0378] Optionally, the first spatial attention mechanism model is used to determine the first spatial weight by using the maximum value of multiple first spatial elements located at each first spatial position in multiple first spatial channels of the first spatial feature.

[0379] Optionally, the image processing system further includes a first color attention mechanism model and a first spatial feature adjustment model, and the multiple image channels of the visible image to be processed correspond to multiple coordinate axes of the color space.

[0380] The first color attention mechanism model is used to process the first color feature to obtain a first color weight, where the first color weight represents the importance of the first color of each first spatial channel in multiple first spatial channels of the first spatial feature.

[0381] The first spatial feature adjustment model is used to adjust the first spatial feature according to the first color weight.

[0382] The image generation model is specifically used to process the first color feature and the adjusted first spatial feature to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0383] Optionally, the first color attention mechanism model is specifically used to determine the first color weight according to an average value of multiple elements in each color channel in the first color feature.

[0384] Optionally, the color feature extraction model includes a first color feature extraction sub-model, a second color feature extraction sub-model, a second spatial attention mechanism model and a second color feature adjustment model; the spatial feature extraction model includes a first spatial feature extraction sub-model.

[0385] The first spatial feature extraction sub-model is used to extract features from the original image file to be processed to obtain a second spatial feature, and the first spatial feature is determined based on the second spatial feature.

[0386] The first color feature extraction sub-model is used to extract features from the visible image to be processed to obtain a second color feature.

[0387] The second spatial attention mechanism model is used to process the second spatial feature to obtain a second spatial weight, where the second spatial weight represents the second position importance of each second color position in multiple second color positions in the second color feature.

[0388] The second color feature adjustment model is used to adjust the second color feature according to the second spatial weight.

[0389] The second color feature extraction sub-model is used to perform feature extraction on the adjusted second color feature to obtain the first color feature.

[0390] The image generation model is specifically used to process the first spatial feature and the first color feature to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0391] Optionally, the spatial feature extraction model includes a first spatial feature extraction sub-model, a second spatial feature extraction sub-model, a second color attention mechanism model and a second spatial feature adjustment model; the color feature extraction model includes a first color feature extraction sub-model; and the multiple image channels of the visible image to be processed correspond to multiple coordinate axes of the color space.

[0392] The first color feature extraction sub-model is used to perform feature extraction on the visible image to be processed to obtain a second color feature, and the first color feature is determined based on the second color feature.

[0393] The first spatial feature extraction sub-model is used to extract features from the original image file to be processed to obtain a second spatial feature.

[0394] The second color attention mechanism model is used to process the second color feature to obtain a second color weight, where the second color weight represents the second color importance of each second spatial channel in multiple second spatial channels of the second spatial feature.

[0395] The second spatial feature adjustment model is used to adjust the second spatial feature according to the second color weight.

[0396] The second spatial feature extraction sub-model is used to perform feature extraction on the adjusted second spatial feature to obtain the first spatial feature.

[0397] The image generation model is specifically used to process the first spatial feature and the first color feature to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

[0398] Optionally, the image generation model includes a first fusion model, a downsampling model, a feature map generation model, an upsampling model and a second fusion model.

[0399] The first fusion model is used to fuse the first spatial feature and the first color feature to obtain a fusion feature.

[0400] The downsampling model is used to downsample the fused features to obtain downsampled features.

[0401] The feature map generation model is used to generate feature images for the fused features and the down-sampled features respectively, so as to obtain a first generated feature map corresponding to the fused features and at least one second generated feature map corresponding to the down-sampled features.

[0402] The second fusion model is used to fuse the first generated feature map with the at least one second generated feature map to obtain the deblurred original image file.

[0403] In some other embodiments, the image processing apparatus 1000 may execute Fig. 9 The image processing system training method shown.

[0404] The acquisition unit 1010 is used to acquire training samples and label deblurred original image files. The training samples include training original image files and training visible images. The training visible images are images obtained by arranging training data elements in the training original image files according to pixels.

[0405] The processing unit 1020 is used to process the sample using the initial image processing system to obtain a training deblurred original image file.

[0406] The processing unit 1020 is further configured to adjust the parameters of the initial image processing system according to the training difference between the training deblurred original image file and the label deblurred original image file, and the adjusted initial image processing system is the trained image processing system.

[0407] Optionally, the initial image processing system includes an initial spatial feature extraction model, an initial color feature extraction model and an initial image generation model.

[0408] The initial spatial feature extraction model is used to extract features from the original training image file to obtain the first training spatial feature.

[0409] The initial color feature extraction model is used to extract features from the training visible image to obtain the first training color feature.

[0410] The initial image generation model is used to process the first training space feature and the first training color feature to obtain a training deblurred original image file.

[0411] The processing unit 1020 is specifically used to adjust the parameters of the initial spatial feature extraction model, the initial color feature extraction model and the initial image generation model according to the training differences. The adjusted initial spatial feature extraction model is the spatial feature extraction model, the adjusted initial color feature extraction model is the color feature extraction model, and the adjusted initial image generation model is the image generation model. The image processing system includes a spatial feature extraction model, a color feature extraction model and an image generation model.

[0412] Optionally, the initial image generation model is specifically used to process the first training space feature and the first training color feature to obtain training difference image data, where the training difference image data represents the difference between the training deblurred original image file and the training original image file.

[0413] The training difference represents the difference between the training difference image data and the label difference image data, and the label difference image data represents the difference between the label deblurred original image file and the training deblurred original image file.

[0414] Optionally, the initial image processing system also includes an initial first spatial attention mechanism model and a first color feature adjustment model.

[0415] The initial first spatial attention mechanism model is used to process the first training spatial feature to obtain a first training spatial weight, where the first training spatial weight represents the first training position importance of each first training color position in the plurality of first training color positions in the first training color feature.

[0416] The first color feature adjustment model is used to adjust the first training color feature according to the first training space weight.

[0417] The initial image generation model is specifically used to process the first training space feature and the adjusted first training color feature to obtain training difference image data.

[0418] The processing unit 1020 is specifically used to adjust the parameters of the initial first spatial attention mechanism model according to the training differences. The adjusted initial first spatial attention mechanism model is the first spatial attention mechanism model. The image processing system includes the first spatial attention mechanism model and the first color feature adjustment model.

[0419] Optionally, the initial first spatial attention mechanism model is used to determine the first training spatial weight by taking the maximum value of multiple first training spatial elements located at each first training spatial position in multiple first training spatial channels of the first training spatial feature.

[0420] Optionally, the initial image processing system also includes an initial first color attention mechanism model and a first spatial feature adjustment model, and multiple training image channels of the training visible image correspond to multiple coordinate axes of the color space.

[0421] The initial first color attention mechanism model is used to process the first training color feature to obtain a first training color weight, where the first training color weight represents the importance of the first training color of each first training space channel in multiple first training space channels of the first training space feature.

[0422] The first spatial feature adjustment model is used to adjust the first training spatial feature according to the first training color weight.

[0423] The initial image generation model is specifically used to process the first training color feature and the adjusted first training space feature to obtain the difference image data.

[0424] The processing unit 1020 is specifically used to adjust the parameters of the initial first color attention mechanism model. The adjusted initial first color attention mechanism model is the first color attention mechanism model, and the image processing system includes the first color attention mechanism model and the first spatial feature adjustment model.

[0425] Optionally, the initial first color attention mechanism model is specifically used to determine the first training color weight according to an average value of multiple elements in each training color channel in the first training color feature.

[0426] Optionally, the initial color feature extraction model includes an initial first color feature extraction sub-model, an initial second color feature extraction sub-model, an initial second spatial attention mechanism model and a second color feature adjustment model. The initial spatial feature extraction model includes an initial first spatial feature extraction sub-model.

[0427] The initial first spatial feature extraction sub-model is used to extract features from the original training image file to obtain the second training spatial features, and the first training spatial features are determined based on the second training spatial features.

[0428] The initial first color feature extraction sub-model is used to extract features from the training visible image to obtain the second training color features.

[0429] The initial second spatial attention mechanism model is used to process the second training spatial features to obtain second training spatial weights, where the second training spatial weights represent the second position importance of each second color position in the plurality of second color positions in the second color features.

[0430] The second color feature adjustment model is used to adjust the second training color feature according to the second training space weight.

[0431] The initial second color feature extraction sub-model is used to perform feature extraction on the adjusted second color feature to obtain the first color feature.

[0432] The processing unit 1020 is specifically used to adjust the parameters of the initial first color feature extraction sub-model, the parameters of the initial second color feature extraction sub-model, the parameters of the initial second spatial attention mechanism model, and the parameters of the initial first spatial feature extraction sub-model.

[0433] The adjusted initial first color feature extraction submodel is the first color feature extraction submodel, the adjusted initial second color feature extraction submodel is the second color feature extraction submodel, the adjusted initial second spatial attention mechanism model is the second spatial attention mechanism model. The adjusted initial first spatial feature extraction submodel is the first spatial feature extraction submodel.

[0434] The color feature extraction model includes a first color feature extraction sub-model, a second color feature extraction sub-model, a second spatial attention mechanism model and a second color feature adjustment model. The spatial feature extraction model includes a first spatial feature extraction sub-model.

[0435] Optionally, the initial spatial feature extraction model includes an initial first spatial feature extraction sub-model, an initial second spatial feature extraction sub-model, an initial second color attention mechanism model and a second spatial feature adjustment model. The initial color feature extraction model includes an initial first color feature extraction sub-model.

[0436] The initial first color feature extraction sub-model is used to extract features from the training visible image to obtain the second training color feature, and the first training color feature is determined according to the second training color feature.

[0437] The initial first spatial feature extraction sub-model is used to extract features from the original training image file to obtain the second training spatial features.

[0438] The initial second color attention mechanism model is used to process the second training color feature to obtain a second training color weight, where the second training color weight represents the second training color importance of each second training space channel in multiple second training space channels of the second training space feature.

[0439] The second spatial feature adjustment model is used to adjust the second training spatial feature according to the second training color weight.

[0440] The initial second space feature extraction sub-model is used to perform feature extraction on the adjusted second training space features to obtain the first training space features.

[0441] The processing unit 1020 is specifically used to adjust the parameters of the initial first spatial feature extraction sub-model, the parameters of the initial second spatial feature extraction sub-model, the parameters of the initial second color attention mechanism model, and the parameters of the initial first color feature extraction sub-model.

[0442] The adjusted parameters of the initial first spatial feature extraction submodel are the first spatial feature extraction submodel, the adjusted initial second spatial feature extraction submodel is the second spatial feature extraction submodel, the adjusted initial second color attention mechanism model is the second color attention mechanism model. The adjusted initial first color feature extraction submodel is the first color feature extraction submodel.

[0443] The spatial feature extraction model includes a first spatial feature extraction sub-model, a second spatial feature extraction sub-model, a second color attention mechanism model and a second spatial feature adjustment model, and the color feature extraction model includes a first color feature extraction sub-model.

[0444] Optionally, the initial image generation model includes an initial first fusion model, a downsampling model, an initial feature map generation model, an upsampling model and an initial second fusion model.

[0445] The initial first fusion model is used to fuse the first training space feature and the first training color feature to obtain a training fusion feature.

[0446] The downsampling model is used to downsample the training fusion features to obtain training downsampled features.

[0447] The initial feature map generation model is used to generate feature images for the training fusion features and the training down-sampling features respectively, so as to obtain a first training generated feature map corresponding to the training fusion features and a second training generated feature map corresponding to the training down-sampling features.

[0448] The initial second fusion model is used to fuse the first generated feature map with the second generated feature map to obtain the deblurred original image file.

[0449] The processing unit 1020 is specifically used to adjust the parameters of the initial feature map generation model. The adjusted initial feature map generation model is the feature map generation model.

[0450] It should be noted that the above-mentioned image processing device 1000 is embodied in the form of a functional unit. The term "unit" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0451] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0452] Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0453] Fig.11 This is a system architecture provided by an embodiment of the present invention.

[0454] As shown in the system architecture 1100, the data acquisition device 1160 is used to collect training data. In the embodiment of the present application, the training data includes: training samples and label deblurred original image files. The training samples include training original image files and training visible images, and the training visible images are images determined based on the training original image files. The data acquisition device 1160 is also used to store the training data in the database 1130.

[0455] The training device 1120 obtains the target model / rule 1101 based on the training data maintained in the database 1130. The target model / rule 1101 can be a trained image processing system. The following will describe in more detail how the training device 1120 obtains the target model / rule 1101 based on the training data in Example 1. The target model / rule 1101 can be used to implement the image processing method provided in the embodiment of the present application, that is, the original image file to be processed and the visible image to be processed are input into the target model / rule 1101 after relevant preprocessing, and the deblurred original image file can be obtained. The training device 1120 can be used to perform Fig. 9 The image processing system method shown.

[0456] It should be noted that, in actual applications, the training data maintained in the database 1130 may not all come from the data acquisition device 1160, but may also be received from other devices. It should also be noted that the training device 1120 may not train the target model / rule 1101 entirely based on the training data maintained by the database 1130, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0457] The target model / rule 1101 obtained by training the training device 1120 can be applied to different systems or devices, such as Fig.11 The execution device 1110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle terminal, etc., or a server or a cloud. Fig.11In the embodiment, the execution device 1110 is configured with an I / O interface 1112 for data interaction with an external device. The user can input data to the I / O interface 1112 through the client device 1140. The input data in the embodiment of the present application may include: using a camera to collect to obtain a raw image file to be processed. The input data in the embodiment of the present application may also include: controlling the processor to process the raw image file to be processed to obtain a visible image to be processed.

[0458] The preprocessing module 1113 and the preprocessing module 1114 are used to preprocess the input data (such as the original image file to be processed) received by the I / O interface 1112. In the embodiment of the present application, the preprocessing module 1113 can be used to process the original image file to be processed to obtain a visible image to be processed.

[0459] In the embodiment of the present application, the preprocessing module 1113 and the preprocessing module 1114 may be omitted (or only one of the preprocessing modules may be provided), and the calculation module 1111 may be directly used to process the input data.

[0460] When the execution device 1110 preprocesses the input data, or when the computing module 1111 of the execution device 1110 performs calculations and other related processing, the execution device 1110 can call the data, code, etc. in the data storage system 1150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 1150.

[0461] Finally, the I / O interface 1112 returns the processing result, such as the deblurred original image file obtained above, to the client device 1140 for providing to the user.

[0462] It is worth noting that the training device 1120 can generate corresponding target models / rules 1101 based on different training data for different goals or different tasks. The corresponding target models / rules 1101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0463] In the attached Fig.11In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 1112. In another case, the client device 1140 can automatically send input data to the I / O interface 1112. If the client device 1140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 1140. The user can view the results output by the execution device 1110 on the client device 1140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 1140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 1112 and the output results of the output I / O interface 1112 as new sample data, and store them in the database 1130. Of course, it is also possible not to collect through the client device 1140, but the I / O interface 1112 directly stores the input data of the input I / O interface 1112 and the output results of the output I / O interface 1112 as new sample data in the database 1130.

[0464] It is worth noting that Fig.11 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Fig.11 In the embodiment, the data storage system 1150 is an external memory relative to the execution device 1110. In other cases, the data storage system 1150 can also be placed in the execution device 1110.

[0465] like Fig.11 As shown, the target model / rule 1101 is obtained by training according to the training device 1120, and the target model / rule 1101 may be an image processing system in the embodiment of the present application. Specifically, the image processing system provided in the embodiment of the present application may include a spatial feature extraction model, a color feature extraction model and an image generation model. The image processing system may also include a combination of a first spatial attention mechanism model and a first color feature adjustment model, and / or a combination of a first color attention mechanism model and a first spatial feature adjustment model. The spatial feature extraction model may include a first spatial feature extraction sub-model, a second spatial feature extraction sub-model, a second color attention mechanism model and a second spatial feature adjustment model. The color feature extraction model may include a first color feature extraction sub-model, a second color feature extraction sub-model, a second spatial attention mechanism model and a second color feature adjustment model. The image generation model may include a first fusion model, a downsampling model, a feature map generation model, and a second fusion model.

[0466] The first spatial attention mechanism model, the first color attention mechanism model, the first spatial feature extraction sub-model, the second spatial feature extraction sub-model, the second color attention mechanism model, the first color feature extraction sub-model, the second color feature extraction sub-model, the second spatial attention mechanism model, the feature map generation model, and the second fusion model can all be convolutional neural networks.

[0467] As mentioned in the previous basic concept introduction, convolutional neural network is a deep neural network with convolution structure, which is a deep learning architecture. Deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the data input into it.

[0468] The present application also provides a chip, the chip comprising a data interface and one or more processors. When the one or more processors execute instructions, the one or more processors read instructions stored in a memory through the data interface to implement the image processing method and / or image processing system training method described in the above method embodiment.

[0469] The one or more processors may be general-purpose processors or special-purpose processors. For example, the one or more processors may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices such as discrete gates, transistor logic devices, or discrete hardware components.

[0470] The chip may be used as a component of a terminal device or other electronic devices. For example, the chip may be located in the electronic device 100 .

[0471] The processor and the memory may be provided separately or integrated together. For example, the processor and the memory may be integrated on a system on chip (SOC) of a terminal device. That is, the chip may also include a memory.

[0472] A program may be stored in the memory, and the program may be executed by the processor to generate instructions, so that the processor executes the image processing method and / or the image processing system training method described in the above method embodiment according to the instructions.

[0473] Optionally, data may be stored in the memory. Optionally, the processor may read data stored in the memory, and the data may be stored at the same storage address as the program, or may be stored at a different storage address than the program.

[0474] Exemplarily, the memory can be used to store relevant programs of the image processing method provided in the embodiments of the present application, and the processor can be used to call relevant programs of the image processing method stored in the memory to implement the image processing method in the embodiments of the present application.

[0475] For example, the processor can be used to obtain an original image file to be processed and a visible image to be processed, wherein the visible image to be processed is an image determined based on the original image file to be processed; and use an image processing system to process the original image file to be processed and the visible image to be processed to obtain a deblurred original image file, wherein the image processing system is a trained neural network model.

[0476] Exemplarily, the memory can be used to store relevant programs of the image processing system training method provided in the embodiments of the present application, and the processor can be used to call relevant programs of the image processing system training method stored in the memory to implement the image processing system training method in the embodiments of the present application.

[0477] For example, the processor can be used to obtain training samples and label deblurred original image files, wherein the training samples include training original image files and training visible images, and the training visible images are images determined based on the training original image files; the samples are processed using an initial image processing system to obtain a training deblurred original image file; and the parameters of the initial image processing system are adjusted based on the training differences between the training deblurred original image file and the label deblurred original image file, and the adjusted initial image processing system is the trained image processing system.

[0478] The chip can be arranged in an electronic device.

[0479] The present application also provides a computer program product, which, when executed by a processor, implements the touch recognition method described in any method embodiment of the present application.

[0480] The computer program product may be stored in a memory, for example, a program, which is finally converted into an executable target file that can be executed by a processor after preprocessing, compiling, assembling and linking.

[0481] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the touch recognition method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

[0482] The computer-readable storage medium is, for example, a memory. The memory may be a volatile memory or a nonvolatile memory, or the memory may include both a volatile memory and a nonvolatile memory. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0483] The embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0484] In the description of this application, the terms "first", "second", etc. are used only for descriptive purposes and should not be understood as indicating or implying relative importance, or a specific order or sequence. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0485] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0486] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0487] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0488] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0489] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0490] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0491] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0492] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire an original image file to be processed and a visible image to be processed, wherein the visible image to be processed is an image obtained by arranging data elements in the original image file to be processed according to pixels; Processing the original image file to be processed and the visible image to be processed by an image processing system to obtain a deblurred original image file, wherein the image processing system is a trained neural network model; The image processing system includes a spatial feature extraction model, a color feature extraction model and an image generation model; The spatial feature extraction model is used to extract features from the original image file to be processed to obtain a first spatial feature; The color feature extraction model is used to extract features from the visible image to be processed to obtain a first color feature; The image generation model is used to process the first spatial feature and the first color feature to obtain the deblurred original image file; The color feature extraction model includes a first color feature extraction sub-model, a second color feature extraction sub-model, a second spatial attention mechanism model and a second color feature adjustment model; the spatial feature extraction model includes a first spatial feature extraction sub-model; The first spatial feature extraction sub-model is used to extract features from the original image file to be processed to obtain a second spatial feature, and the first spatial feature is determined according to the second spatial feature; The first color feature extraction sub-model is used to extract features from the visible image to be processed to obtain a second color feature; The second spatial attention mechanism model is used to process the second spatial feature to obtain a second spatial weight, where the second spatial weight represents a second position importance of each second color position in a plurality of second color positions in the second color feature; The second color feature adjustment model is used to adjust the second color feature according to the second spatial weight; The second color feature extraction sub-model is used to perform feature extraction on the adjusted second color feature to obtain the first color feature; The image generation model is specifically used to process the first spatial feature and the first color feature to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

2. The method according to claim 1, characterized in that The image processing system also includes a first spatial attention mechanism model and a first color feature adjustment model, The first spatial attention mechanism model is used to process the first spatial feature to obtain a first spatial weight, where the first spatial weight represents a first position importance of each first color position in a plurality of first color positions in the first color feature; The first color feature adjustment model is used to adjust the first color feature according to the first spatial weight; The image generation model is specifically used to process the first spatial feature and the adjusted first color feature to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

3. The method according to claim 2, characterized in that The first spatial attention mechanism model is used to determine the first spatial weight by taking the maximum value of multiple first spatial elements located at each first spatial position in multiple first spatial channels of the first spatial feature.

4. The method according to any one of claims 1 to 3, characterized in that The image processing system further includes a first color attention mechanism model and a first spatial feature adjustment model, wherein the plurality of image channels of the visual image to be processed correspond to a plurality of coordinate axes of the color space; The first color attention mechanism model is used to process the first color feature to obtain a first color weight, where the first color weight represents the importance of the first color of each first spatial channel in the multiple first spatial channels of the first spatial feature; The first spatial feature adjustment model is used to adjust the first spatial feature according to the first color weight; The image generation model is specifically used to process the first color feature and the adjusted first spatial feature to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

5. The method according to claim 4, characterized in that The first color attention mechanism model is specifically used to determine the first color weight according to the average value of multiple elements in each color channel in the first color feature.

6. The method according to any one of claims 1 to 5, characterized in that The spatial feature extraction model includes a first spatial feature extraction sub-model, a second spatial feature extraction sub-model, a second color attention mechanism model and a second spatial feature adjustment model; the color feature extraction model includes a first color feature extraction sub-model; The multiple image channels of the visible image to be processed correspond to multiple coordinate axes of the color space; The first color feature extraction sub-model is used to extract features from the visible image to be processed to obtain a second color feature, and the first color feature is determined according to the second color feature; The first spatial feature extraction sub-model is used to extract features from the original image file to be processed to obtain a second spatial feature; The second color attention mechanism model is used to process the second color feature to obtain a second color weight, where the second color weight represents the importance of the second color of each second spatial channel in the plurality of second spatial channels of the second spatial feature; The second spatial feature adjustment model is used to adjust the second spatial feature according to the second color weight; The second spatial feature extraction sub-model is used to extract features from the adjusted second spatial features to obtain the first spatial features; The image generation model is specifically used to process the first spatial feature and the first color feature to obtain difference image data, wherein the difference image data represents the difference between the deblurred original image file and the original image file to be processed, and the deblurred original image file is determined based on the difference image data and the original image file to be processed.

7. The method according to any one of claims 1 to 6, characterized in that The image generation model includes a first fusion model, a downsampling model, a feature map generation model, an upsampling model and a second fusion model; The first fusion model is used to fuse the first spatial feature and the first color feature to obtain a fusion feature; The downsampling model is used to downsample the fused features to obtain downsampled features; The feature map generation model is used to generate feature images for the fused feature and the down-sampled feature, respectively, to obtain a first generated feature map corresponding to the fused feature and at least one second generated feature map corresponding to the down-sampled feature; The second fusion model is used to fuse the first generated feature map with the at least one second generated feature map to obtain the deblurred original image file.

8. A method for training an image processing system, characterized in that: The method comprises: Acquire a training sample and a label deblurred original image file, wherein the training sample includes a training original image file and a training visible image, wherein the training visible image is an image obtained by arranging the training data elements in the training original image file according to pixels; Processing the sample using an initial image processing system to obtain a training deblurred original image file; According to the training difference between the training deblurred original image file and the label deblurred original image file, adjusting the parameters of the initial image processing system, the adjusted initial image processing system being the trained image processing system; The initial image processing system includes an initial space feature extraction model, an initial color feature extraction model and an initial image generation model; The initial spatial feature extraction model is used to extract features from the training original image file to obtain a first training spatial feature; The initial color feature extraction model is used to extract features from the training visible image to obtain a first training color feature; The initial image generation model is used to process the first training space feature and the first training color feature to obtain the training deblurred original image file The initial color feature extraction model includes an initial first color feature extraction sub-model, an initial second color feature extraction sub-model, an initial second spatial attention mechanism model and a second color feature adjustment model, and the initial spatial feature extraction model includes an initial first spatial feature extraction sub-model; The initial first spatial feature extraction sub-model is used to perform feature extraction on the training original image file to obtain a second training spatial feature, wherein the first training spatial feature is determined based on the second training spatial feature; The initial first color feature extraction sub-model is used to extract features from the training visible image to obtain second training color features; The initial second spatial attention mechanism model is used to process the second training spatial feature to obtain a second training spatial weight, wherein the second training spatial weight represents a second position importance of each second color position in a plurality of second color positions in the second color feature; The second color feature adjustment model is used to adjust the second training color feature according to the second training space weight; The initial second color feature extraction sub-model is used to extract the adjusted second color feature to obtain the first training color feature The initial image generation model is specifically used to process the first training spatial feature and the first training color feature to obtain training difference image data, wherein the training difference image data represents the difference between the training deblurred original image file and the training original image file, the training difference represents the difference between the training difference image data and the label difference image data, and the label difference image data represents the difference between the label deblurred original image file and the training deblurred original image file.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

10. A chip, characterized in that: The method comprises a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for implementing the method according to any one of claims 1 to 8.

Images