Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: determining a noise adjustment parameter according to the imaging information of a to-be-processed image, and calling a target neural network model to carry out the noise reduction of the to-be-processed image based on the noise adjustment parameter, and obtaining a processed image; and then, based on the imaging information, an image signal processor is called to carry out noise reduction processing on the processed image, and a target image is obtained. In the scheme, based on the imaging information of the to-be-processed image, linkage of the target neural network model and the image signal processor is realized, the imaging information is used as an influence factor to guide the target neural network model and the image signal processor to optimize the image in the same direction, and joint noise reduction of the model and hardware equipment is realized. Therefore, a better noise reduction effect is achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] Image noise refers to unnecessary or redundant interference information in image data. In the process of image formation and transmission, images will inevitably be interfered by noise. The noise and signal in the image are often intertwined and difficult to distinguish. The existence of noise seriously affects the quality of remote sensing images, so it must be corrected before image enhancement and classification processing. Image denoising is an important part of image processing. In recent years, the use of intelligent image denoising models to denoise images has gradually become widespread in various image applications.

[0003] As the computing power of various device platforms gradually increases, image denoising models have begun to be applied in various electronic devices. However, the image denoising effect cannot be maximized by relying solely on the denoising results provided by the image denoising model and the generalization of the model itself. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can improve the image noise reduction effect.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device is integrated with an image signal processor, and the method comprises:

[0006] Acquire an image to be processed and imaging information corresponding to the image to be processed;

[0007] determining a noise adjustment parameter according to the imaging information;

[0008] Based on the noise adjustment parameters, calling the target neural network model to perform noise reduction processing on the image to be processed to obtain a processed image;

[0009] Based on the imaging information, the image signal processor is called to perform noise reduction processing on the processed image to obtain a target image.

[0010] In a second aspect, the present application provides another image processing method, which is applied to an electronic device, wherein the electronic device is integrated with an image signal processor, and the method comprises:

[0011] Acquire an image to be processed and imaging information corresponding to the image to be processed;

[0012] Based on the imaging information, calling the image signal processor to perform noise reduction processing on the image to be processed to obtain a processed image;

[0013] determining a noise adjustment parameter according to the imaging information;

[0014] Based on the noise adjustment parameters, a target neural network model is called to perform noise reduction processing on the processed image to obtain a target image.

[0015] In a third aspect, the present application provides an image processing device, which is applied to an electronic device, wherein the electronic device is integrated with an image signal processor, and the device comprises:

[0016] An acquisition unit, used for acquiring an image to be processed and imaging information corresponding to the image to be processed;

[0017] A determination unit, configured to determine a noise adjustment parameter according to the imaging information;

[0018] A first noise reduction unit, configured to call a target neural network model to perform noise reduction processing on the image to be processed based on the noise adjustment parameter to obtain a processed image;

[0019] The second noise reduction unit is used to call the image signal processor to perform noise reduction processing on the processed image based on the imaging information to obtain a target image.

[0020] In a fourth aspect, the present application provides another image processing device, which is applied to an electronic device, wherein the electronic device is integrated with an image signal processor, and the device comprises:

[0021] An acquisition unit, used for acquiring an image to be processed and imaging information corresponding to the image to be processed;

[0022] A second noise reduction unit, configured to call the image signal processor to perform noise reduction processing on the image to be processed based on the imaging information to obtain a processed image;

[0023] a determining unit, configured to determine a noise adjustment parameter according to the imaging information;

[0024] The first denoising unit is used to call a target neural network model to perform denoising on the processed image based on the noise adjustment parameter to obtain a target image.

[0025] In a fifth aspect, the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the image processing method provided in the present application.

[0026] In a sixth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for loading by a processor to implement the steps in the image processing method provided in the present application.

[0027] In the embodiment of the present application, the noise adjustment parameters are determined according to the imaging information of the image to be processed, and based on the noise adjustment parameters, the target neural network model is called to perform noise reduction processing on the image to be processed to obtain a processed image; then, based on the imaging information, the image signal processor is called to perform noise reduction processing on the processed image to obtain a target image. In this solution, based on the imaging information of the image to be processed, the linkage between the target neural network model and the image signal processor is realized, and the imaging information is used as an influencing factor to guide the target neural network model and the image signal processor to optimize the image in the same direction, so as to realize the joint noise reduction of the model and the hardware device, thereby achieving a better noise reduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solution in this embodiment, the drawings required for use in the description of the embodiment will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is a schematic diagram of a scene of an image processing system provided in an embodiment of the present application;

[0030] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application;

[0031] Figure 3 This is a schematic diagram of the first architecture of the image processing system provided in the embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of the internal flow of the parameter control module provided in the embodiment of the present application;

[0033] Figure 5 is a schematic diagram of a second architecture of an image processing system provided in an embodiment of the present application;

[0034] Figure 6 This is a first structural diagram of the target neural network model provided in the embodiment of the present application;

[0035] Figure 7 It is a second structural schematic diagram of the target neural network model provided in the embodiment of the present application;

[0036] Figure 8 It is a third structural schematic diagram of the target neural network model provided in the embodiment of the present application;

[0037] Fig. 9 It is a fourth structural schematic diagram of the target neural network model provided in the embodiment of the present application;

[0038] Fig.10This is a fifth structural diagram of the target neural network model provided in the embodiment of the present application;

[0039] Fig.11 This is a sixth structural diagram of the target neural network model provided in the embodiment of the present application;

[0040] Fig.12 is another flowchart of the image processing method provided by an embodiment of the present application;

[0041] Fig.13 is a schematic diagram of a third architecture of the image processing system provided in an embodiment of the present application;

[0042] Fig.14 is another flowchart of the image processing method provided in the embodiment of the present application;

[0043] Fig.15 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0044] Fig.16 is a structural schematic diagram of another image processing device provided in an embodiment of the present application;

[0045] Fig.17 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0047] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] As the computing power of various device platforms gradually increases, image denoising models have begun to be applied in various electronic devices. In practical applications, the final presentation of image effects is not only related to the image denoising model, but also closely related to the hardware modules in the electronic devices. Therefore, in order to improve the image denoising effect, the image denoising model and the hardware modules in the electronic devices can be used simultaneously for image denoising processing.

[0051] However, for such usage scenarios, image denoising models only provide denoising results and lack the flexibility to be debugged to combine with hardware modules in electronic devices to achieve collaborative denoising. It is difficult to maximize the denoising effect by relying solely on the generalization of the model itself or the debugging capabilities of the device platform.

[0052] Based on this, the embodiments of the present application provide an image processing method, an image processing device, an electronic device, and a computer-readable storage medium. The image processing method can be executed by the image processing device, or by an electronic device integrated with the image processing device.

[0053] The following will be combined with the drawings in this embodiment to clearly and completely describe the technical solution in this embodiment. Obviously, the described embodiment is only a part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0054] Please refer to Figure 1 , the present application also provides an image processing system, such as Figure 1 As shown, the image processing system includes an electronic device 100, and the image processing device provided by the present application is integrated in the electronic device 100. For example, the electronic device 100 can obtain an image to be processed and imaging information corresponding to the image to be processed; determine a noise adjustment parameter according to the imaging information; based on the noise adjustment parameter, call the target neural network model to perform noise reduction processing on the image to be processed to obtain a processed image; based on the imaging information, call the image signal processor integrated in the electronic device to perform noise reduction processing on the processed image to obtain a target image.

[0055] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as a smart phone, a tablet computer, a PDA, a laptop computer, a virtual reality device, an augmented reality device or a mixed reality device, etc., a mobile electronic device with a processor, or a desktop computer, a television, a server, an industrial equipment, etc., a fixed electronic device with a processor.

[0056] In addition, if Figure 1As shown, the image processing system may also include a data warehouse 200 for storing original data, intermediate data and result data in the image processing process. For example, the electronic device 100 acquires the image to be processed (original data), performs denoising processing on the target neural network model to obtain a processed image (intermediate data), and finally stores the target image (result data) obtained by the image signal processor performing secondary denoising processing on the processed image in the data warehouse 200.

[0057] It should be noted that Figure 1 The scene diagram of the image processing system shown is only an example. The image processing system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the image processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0058] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0059] Please refer to Figure 2 and Figure 3 , Figure 2 is a flowchart of an image processing method provided in an embodiment of the present application; Figure 3 Schematic diagram of the first architecture of the image processing system provided in the embodiment of the present application. Figure 2 As shown, the process of the image processing method provided by this application is as follows:

[0060] 101. Obtain an image to be processed and imaging information corresponding to the image to be processed.

[0061] In the embodiment of the present application, firstly, an image to be processed that needs to be subjected to image noise reduction processing is obtained, and the image to be processed is an image containing image noise. In practical applications, image noise comes from many aspects, which can be noise caused by components, such as thermal noise caused by electronic components, shot noise and flicker noise caused by vacuum devices, particle noise generated by transistors, photon noise of photoelectric tubes, and noise formed by different transformations of these noises caused by components forming various electronic circuits and devices; it can also be image optical noise caused by optical phenomena.

[0062] In this embodiment, the image to be processed may come from image data collected by the electronic device itself, or may come from third-party image data. When the electronic device itself is integrated with an image acquisition component, image acquisition can be performed through the built-in image acquisition component, and the image to be processed is obtained based on the acquired image. That is, in one embodiment, the electronic device of this scheme is integrated with an image sensor. When acquiring the image to be processed, the original image of the current shooting scene can be collected by the image sensor, and then the image to be processed is generated based on the original image. It should be noted that the original image can be in RAW image format, and the RAW image format image can capture the shooting characteristics of the scene as much as possible, that is, it contains physical information such as the light intensity and color of the scene.

[0063] In actual applications, the images collected in different shooting scenes will be different, and the corresponding image noise will also be different. For example, under the same other conditions, the original image directly collected by the image sensor in a bright environment scene will contain relatively less image noise, while the original image directly collected by the image sensor in a dark environment scene will contain relatively more image noise.

[0064] Therefore, in some implementations, the original image may be directly used as the image to be processed, and the image noise is not enhanced, so as to simplify the subsequent noise reduction process.

[0065] In addition, the electronic device can also obtain the image to be processed from other electronic devices equipped with image acquisition components, and can also obtain pre-captured images from the server, which are uploaded to the server after being captured by other electronic devices equipped with image acquisition components.

[0066] In the embodiment of the present application, the imaging information can reflect the current real shooting scene and shooting characteristics, and refers to the physical parameters (referred to as Meta information) obtained after analysis based on the information collected by the image sensor in the current shooting scene when the image to be processed is imaged, which are used to constrain the imaging results, such as exposure parameters (such as aperture, shutter, sensitivity, focal length, exposure compensation, etc.), dynamic range, brightness, etc. During specific implementation, the relevant environmental information of the on-site shooting scene can be reversely analyzed based on the imaging information and the final image.

[0067] 102. Determine a noise adjustment parameter according to the imaging information.

[0068] In the embodiment of the present application, the imaging information can be introduced into the image denoising process as an adjustment factor to debug and optimize the denoising effect for different scenes. Specifically, corresponding noise adjustment parameters can be generated based on the imaging information to adjust the denoising effect of the subsequent image denoising model.

[0069] In some embodiments, the imaging information may include: at least one of the sensitivity, brightness and dynamic range corresponding to the original image. Then, when determining the noise adjustment parameter according to the imaging information, the following process may be specifically included:

[0070] Determine the ambient noise of the shooting scene where the original image is located according to the sensitivity, determine the ambient brightness of the shooting scene where the original image is located according to the brightness, and / or determine the light intensity range of the shooting scene where the original image is located according to the dynamic range;

[0071] Determining a scene type of a captured scene based on ambient noise, ambient brightness, and / or a light intensity range;

[0072] The noise adjustment parameters are determined according to the scene type.

[0073] Specifically, the information of the shooting scene, such as scene brightness and noise conditions, can be reversely analyzed based on the imaging information, and the shooting scene can be located based on the analyzed information to analyze its scene type. For example, daytime or nighttime, high noise or low noise, dark environment or bright environment, etc. Finally, the corresponding required noise adjustment parameters are determined based on the scene type. In specific implementation, the noise adjustment parameter can be a noise adjustment ratio, that is, a weighting coefficient.

[0074] In actual applications, in order to improve the accuracy of subsequent noise reduction, the noise adjustment parameters can also be determined based on more fine-grained scene classification. For example, each scene type can be classified twice, and different levels will match different degrees of noise adjustment parameters.

[0075] It should be noted that the noise adjustment parameter in the embodiment of the present application is not limited to one dimensional evaluation index, and it can be obtained based on evaluation indicators of multiple dimensions.

[0076] refer to Figure 3 In a specific implementation, the imaging information (ie, Meta information) may be input into a parameter control module, and the parameter module is configured with a relevant parameter generation strategy for outputting the noise adjustment parameters required by the image denoising model based on the input imaging information.

[0077] refer to Figure 4 In some embodiments, the parameter control module can be configured to implement scene differentiation and parameter control. In the scene differentiation stage, scene differentiation can be implemented through the incoming imaging information (i.e., Meta information), and the output indicator is the noise level. In the parameter control stage, the noise adjustment parameters required by the image denoising model can be output according to the incoming noise level.

[0078] 103. Based on the noise adjustment parameters, the target neural network model is called to perform noise reduction processing on the image to be processed to obtain a processed image.

[0079] In the embodiment of the present application, the target neural network model is a trained artificial intelligence (AI) network model for intelligent noise reduction of images. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0080] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in fields such as machine vision and natural language processing.

[0081] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding training sets, network models that implement different functions can be trained. For example, a gender classification model can be trained based on one training set, and an image optimization model can be trained based on another training set.

[0082] In the embodiment of the present application, when the model is trained, a training set can be constructed based on the unprocessed image obtained by real shooting of the electronic device and the clean image after noise removal, and the basic image denoising model (such as the convolutional neural network model) can be trained based on the constructed training set to obtain a target neural network model with better image denoising ability. In practical applications, the target neural network model can be deployed locally on the electronic device or in a cloud server, and can be provided by the cloud service when needed.

[0083] refer to Figure 3In the embodiment of the present application, the target neural network model is used as an image denoising model, and a debugging entry is provided, which is a debuggable image denoising model. When the target network model is used for denoising, a noise adjustment parameter can be input based on the network node where the debugging entry is located to cooperate with the output of the target neural network model at the network node, and the output of the network node is adjusted by the noise adjustment parameter, and the processed image is output based on the adjusted result to obtain the denoising effect of the model on the image to be processed.

[0084] It should be noted that in the embodiment of the present application, the input noise adjustment parameters need to be adjusted in advance based on the imaging information and the model noise reduction effect, and the model noise reduction effect is optimized after multiple tests and debugging. Subsequently, according to the debugging results of multiple tests, a parameter generation strategy for generating the optimal noise adjustment parameters based on the imaging information is fitted, so that when performing noise reduction based on the image noise reduction model, matching noise adjustment parameters can be generated for different scenes. Figure 3 , the parameter generation strategy can be configured in the parameter control module to generate matching noise adjustment parameters based on the input Meta information.

[0085] 104. Based on the imaging information, call the image signal processor to perform noise reduction on the processed image to obtain a target image.

[0086] Continue to refer Figure 3 Specifically, after the target neural network model is used to perform noise reduction on the image to be processed, the hardware module in the electronic device, namely the image signal processor (ISP), is called to perform secondary noise reduction on the image to obtain the final target image.

[0087] In actual applications, image signal processors have complex functions and may contain multiple image algorithm processing modules. Image signal processors of different chips may have different processing flows and modules, but their principles and implementation functions are basically the same. For example, image signal processors can be used to remove image noise, remove bad pixel data, linearize data, convert data formats, convert color spaces, perform 3A processing (automatic white balance, auto focus, auto exposure), adjust brightness, color correction, sharpen images, and zoom images.

[0088] In the embodiment of the present application, the image signal processor is a hardware image processing engine, which includes many noise reduction (NR) modules, such as a noise reduction module in the Raw domain, a noise reduction module in the YUV domain, etc. Specifically, the noise reduction effect of the image signal processor is related to the incoming imaging information. It will distinguish scenes through the incoming imaging information, adaptively control the noise reduction strength in the noise reduction module based on the imaging information, and combine with the above-mentioned target neural network model to achieve joint noise reduction, and obtain the final noise reduction effect, so that the image expressiveness is optimal.

[0089] As can be seen from the above, the image processing method provided by the embodiment of the present application determines the noise adjustment parameters according to the imaging information of the image to be processed, and based on the noise adjustment parameters, calls the target neural network model to perform noise reduction processing on the image to be processed to obtain a processed image; then, based on the imaging information, calls the image signal processor to perform noise reduction processing on the processed image to obtain a target image. In this solution, the linkage between the target neural network model and the image signal processor can be realized based on the imaging information of the image to be processed, and the imaging information is used as an influencing factor to guide the target neural network model and the image signal processor to optimize the image in the same direction, so as to realize the joint noise reduction of the model and the hardware device, thereby achieving a better noise reduction effect.

[0090] In some embodiments, during specific implementation, since the original image has too much information, in order to reduce the complexity of the subsequent image denoising model, a portion of denoising operation may be performed on the original image before the denoising operation is performed using the image denoising model.

[0091] In an embodiment of the present application, the image signal processor may be composed of a plurality of hardware modules with different functions, and each hardware module undertakes different image processing tasks. For example, some hardware modules may directly receive the output of the image sensor (i.e., the original image), and the hardware module may perform image preprocessing operations on the output of the image sensor. For example, the output of the image sensor may be directly subjected to color correction, downsampling, demosaicing, statistical 3A data and other processing operations. For another example, some hardware modules may be used for bad pixel removal, phase focusing, demosaicing, downsampling, HDR processing and Bayer hybrid noise reduction processing of photographic image data; some hardware modules may be used for hardware noise reduction, image cropping, color processing, detail enhancement and other image processing tasks; some hardware modules may be used to encode photographic data to obtain images of different formats.

[0092] refer to Figure 5In some embodiments, when the image to be processed is obtained, the image signal processor integrated in the electronic device can be called to pre-process the original image based on the imaging information (i.e., Meta information) to obtain the image to be processed (the image to be processed at this time can be a clean image, or an image with some noise), and then passed to the image denoising model for in and out processing. By pre-processing the original image, the amount of information contained in the image is reduced, thereby reducing the performance requirements and power consumption of the system in the subsequent denoising process. Next, the image denoising model outputs the processed image, and calls other functional modules of the image signal processor (i.e., functional modules other than pre-processing) to perform denoising on the processed image, and finally obtains the final target image.

[0093] That is, in this embodiment, the image denoising model processes data processed by a part of the device hardware (i.e., some functional modules of the image signal processor), and the data may be an image in RAW, YUV or RGB format. The data processed by the image denoising model (i.e., the processed image) is then processed by another part of the device hardware (i.e., other functional modules of the image signal processor) to obtain the final denoised image.

[0094] It can be seen that the system architecture provided by this embodiment can be flexibly applied to application scenarios where algorithm nodes cannot be added between the image sensor and the device hardware (i.e., the image signal processor). In addition, since there is a noise reduction module in the back-end device hardware, the requirements for the image noise reduction model are lower. For example, the image noise reduction model can perform noise reduction on a certain channel in a targeted manner, and the other channels are handled by the back-end device hardware. For example, the image noise reduction model can only perform noise reduction on the Y channel in the YUV image, and the noise reduction of the UV channel is handled by the back-end device hardware. The amount of data processed by the image noise reduction model is further reduced, which can further reduce the performance requirements and power consumption of the model.

[0095] In an embodiment of the present application, the target neural network model serves as an adjustable image denoising model, which can be a globally adjustable image denoising model or a locally adjustable image denoising model, and can be configured according to actual needs.

[0096] In specific implementation, the residual network structure can be used to build the target neural network model. Figure 6 For a globally adjustable image denoising model, it is only necessary to introduce the noise adjustment parameters into the residual network structure and adjust the entire residual output of the model to achieve global debugging of the image denoising result. That is, in some embodiments, based on the noise adjustment parameters, when calling the target neural network model to perform denoising on the image to be processed, the following process may be specifically included:

[0097] Obtaining a residual image corresponding to the image to be processed based on the target neural network model;

[0098] The parameter value of each pixel in the residual image is adjusted according to the noise adjustment parameter;

[0099] The image to be processed is subjected to denoising based on the adjusted residual image.

[0100] In the embodiment of the present application, the residual image is the difference between the noise image (i.e., the image to be processed) predicted by the model and the clear image (i.e., the processed image); the parameter value of the pixel point can be a gray value or a color component. Figure 6 In the network structure shown, the noise adjustment parameter is a global debugging parameter. The noise adjustment parameter is passed in to control the overall noise reduction strength of the image, so that the image noise reduction model can be debugged as a whole. The model output satisfies the following formula:

[0101] Out data=In data+res*para

[0102] Among them, Out data represents output image data, In data represents input image data, res represents residual image, and para represents noise adjustment parameter.

[0103] refer to Figure 7 For a locally adjustable image denoising model, a local mask generation module may be configured in the network structure to adaptively generate a local mask image based on the noise adjustment parameters and the image content of the image to be processed, so as to subsequently control the denoising intensity of each pixel in the corresponding area of ​​the image to be processed through the local mask image. That is, in some embodiments, when the target neural network model is called to perform denoising on the image to be processed based on the noise adjustment parameters, the following process may be specifically included:

[0104] Obtain a residual image corresponding to the image to be processed through the target neural network model;

[0105] Perform noise reduction on the image to be processed based on the residual image to obtain an intermediate image;

[0106] Generate a local mask image according to the noise adjustment parameter and the intermediate image;

[0107] Adjusting the parameter value of each pixel in the residual image according to the local mask image;

[0108] The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

[0109] As above, the residual image is the difference between the noise image (i.e., the image to be processed) predicted by the model and the clear image (i.e., the processed image). Then, the intermediate image after the noise reduction processing of the image to be processed based on the residual image can be considered to be a clear image to a certain extent. In the embodiment of the present application, the size of the generated local mask image is consistent with the size of the output image (i.e., the processed image), and each pixel of the output image corresponds to a noise level. The significance of the local mask is that the noise reduction strength in the image can be adjusted according to the image content, and each pixel in the local mask has its corresponding noise reduction strength. For parts that do not need to retain too many image details (such as the image background), a relatively strong noise reduction strength can be used. For parts that need to retain image details (such as the image foreground), a relatively weak noise reduction strength can be used. Compared with global unified debugging, more image details can be retained in areas where details need to be retained, and the noise reduction in areas where details do not need to be retained is cleaner. The model output satisfies the following formula:

[0110] Out data=In data+res*local mask

[0111] Among them, Out data represents output image data, In data represents input image data, res represents residual image, and local mask represents local mask image.

[0112] It should be noted that the local mask image in the embodiment of the present application is not a binary mask containing only two values ​​0 and 1, but can contain many values. For example, each mask value in the local mask image can be any value in [0, 1], and the mask value can be used to indicate the noise reduction strength for each pixel.

[0113] In the embodiment of the present application, there are many ways to generate the local mask. For example, refer to Figure 8 , a mask generation module for generating a local mask can be constructed by using multiple convolution layers, and a local mask image for controlling the noise reduction strength of each pixel is generated under the action of the noise adjustment parameter. In some embodiments, the local mask image is generated according to the noise adjustment parameter and the intermediate image, which can specifically include the following process:

[0114] dividing the intermediate image into a plurality of image regions;

[0115] Determining an image classification corresponding to the image region according to image content of the image region;

[0116] Determine the noise reduction level corresponding to the image area according to the image classification;

[0117] Determine the noise reduction intensity corresponding to each pixel in the intermediate image according to the noise adjustment parameter and the noise reduction level;

[0118] Generate a local mask image based on the denoising strength.

[0119] The image classification may include a texture region type and a flat region type. Specifically, different noise reduction levels may be set for different types of image classification. For image classifications that need to retain more image details (such as texture region types), a lower noise reduction level may be set; for image classifications that do not need to retain too many image details (such as flat region types), a higher noise reduction level may be set to improve the cleanliness of the image.

[0120] In some implementations, when dividing the intermediate image into a plurality of image regions, the following process may be specifically included:

[0121] Extracting image features of the intermediate image;

[0122] The intermediate image is segmented according to the image features to obtain multiple image regions.

[0123] Specifically, first, the image features of the intermediate image can be extracted through multiple convolution layers to obtain a reduced feature map; then, the reduced feature map is restored to a feature map of the same size as the intermediate image by using deconvolution; then, the restored feature map is compared with the intermediate image; finally, the classification prediction of each pixel in the intermediate image is realized based on the comparison result. After the classification prediction of each pixel is obtained, the intermediate image can be segmented to obtain multiple image regions.

[0124] In some implementations, the corresponding noise reduction level may be determined based on the classification prediction result of the pixel points, and the noise adjustment parameter may be directly applied to each pixel point to determine the noise reduction intensity corresponding to each pixel point.

[0125] It should be noted that in this embodiment, the local mask image (i.e., local mask) and the output image (i.e., Out data) are both used as the output of the target neural network model. When training the model, a loss function needs to be constructed based on the loss of the output image and the loss of the mask to train the model. In this embodiment, an absolute value loss function can be used, and the constructed loss function can be expressed as follows:

[0126] Loss=L1(GT image-output image)+L1(GT mask-local mask)

[0127] Among them, Loss represents the total loss of the model, output image represents the output image of the model, GT image represents the annotated true value image, local mask represents the mask image output by the model, and GT mask represents the annotated mask image.

[0128] In practical applications, when constructing a loss function, other applicable types of loss functions may also be used, without limitation.

[0129] refer to Fig. 9 In the embodiment of the present application, a convolution layer and a normalization layer with preset weights may also be used to construct a mask generation module for generating a local mask, and under the action of the noise adjustment parameter, a local mask image for controlling the noise reduction strength of each pixel is generated. That is, in some embodiments, when a local mask image is generated according to the noise adjustment parameter and the intermediate image, the following process may be specifically included:

[0130] Performing image filtering on the intermediate image to obtain a filtered intermediate image;

[0131] According to the noise adjustment parameter, the noise reduction strength corresponding to each pixel in the intermediate image after filtering is adjusted;

[0132] The intermediate image after adjusting the noise reduction intensity is normalized to generate a local mask image.

[0133] Specifically, a convolution layer with preset weights can be used to perform image filtering on the intermediate image. Among them, the convolution layer with preset weights can be obtained by setting the weights of the initialized convolution layer based on the Laplace operator. In this embodiment, the convolution layer with preset weights can be used as a high-pass filter to filter the intermediate image, and based on the filtering result, it can be distinguished whether each pixel belongs to a flat area or a texture area, and combined with the classification area of ​​each pixel under the action of the noise adjustment parameter to generate a mask image for controlling the noise reduction strength of each pixel. After the convolution layer, a BN layer, an LN layer and other structures can be connected as a normalization layer to normalize the mask image output by the convolution layer to obtain a local mask image.

[0134] refer to Fig.10 In the embodiment of the present application, a mask generation module for generating a local mask can also be constructed by using a collection layer and a split layer, and a local mask image for controlling the noise reduction strength of each pixel point is generated under the action of the noise adjustment parameter. In some embodiments, based on the noise adjustment parameter, the target neural network model is called to perform noise reduction on the image to be processed, which can specifically include the following process:

[0135] Obtain a feature map of the image to be processed through a target neural network model;

[0136] The feature map is split according to the dimension of its channel number to obtain the residual image and local mask image corresponding to the image to be processed;

[0137] According to the noise adjustment parameter, the noise reduction intensity corresponding to each pixel in the local mask image is adjusted;

[0138] According to the local mask image after adjusting the noise reduction strength, the parameter value of each pixel in the residual image is adjusted;

[0139] The image to be processed is subjected to denoising based on the adjusted residual image.

[0140] In specific implementation, the target neural network model can be a trained U-Net neural network model, and the image features of the image to be processed are extracted through it, and the corresponding feature map is output and passed into the collection layer. Among them, the collection layer is an 8-channel convolution layer. Subsequently, the split layer is used to split the feature map according to the dimension of its channel number, and the residual image and local mask image corresponding to the image to be processed are separated from the feature map.

[0141] It should be noted that in this embodiment, both the local mask image and the output image are used as the output of the target neural network model. When training the model, a loss function needs to be constructed based on the loss of the output image and the loss of the mask to train the model. In this embodiment, the absolute value loss function can also be used, and the constructed loss function can be expressed as follows:

[0142] Loss=L1(GT image-output image)+L1(GT mask-local mask)

[0143] In practical applications, when constructing a loss function, other applicable types of loss functions may also be used, without limitation.

[0144] refer to Fig.11 In one embodiment, the mask generation module can also be set at the front end of the image denoising model (i.e., after the original input), so that more original image information can be obtained, and then a suitable local mask image can be adaptively generated. That is, in one embodiment, based on the noise adjustment parameter, when the target neural network model is called to perform denoising on the image to be processed, the image content of the image to be processed can be first obtained; then, according to the noise adjustment parameter and the image content of the image to be processed, a local mask image is generated; then, the residual image corresponding to the image to be processed is obtained through the target neural network model, and the parameter value of each pixel in the residual image is adjusted according to the local mask image, and finally, the image to be processed is subjected to denoising based on the adjusted residual image.

[0145] In one embodiment, the mask generation module can also be set in the middle of the image denoising model, that is, after some network connection layers in the image denoising model and before the last network connection layer. At this time, after more layers of calculation, a suitable local mask image is adaptively generated.

[0146] From the above, it can be seen that the embodiment of the present application provides an adjustable image denoising model, which can be jointly debugged with the hardware modules in the electronic device, and under the action of Meta information, it is combined into an image processing system that can be jointly debugged for different scenarios. On the one hand, for each scenario, this solution has flexibility, so that in different usage scenarios, it can achieve better image denoising effects; on the other hand, for some special scenarios, specific strategies can be adopted to make the entire image denoising model and device hardware collection into a more generalized and reliable image processing system.

[0147] In addition, the image processing system proposed in the embodiment of the present application can be applied to image noise reduction, and can also be extended to other application fields, such as image enhancement, image segmentation, etc., without limitation.

[0148] Please refer to Fig.12 and Fig.13 , Fig.12 is a flowchart of another image processing method provided in an embodiment of the present application; Fig.13 The third architecture diagram of the image processing system provided in the embodiment of the present application is shown in FIG. Fig.12 As shown, the process of the image processing method provided by this application is as follows:

[0149] 201. Acquire an image to be processed and imaging information corresponding to the image to be processed;

[0150] 202. Based on the imaging information, call an image signal processor to perform noise reduction processing on the image to be processed to obtain a processed image;

[0151] 203. Determine a noise adjustment parameter according to the imaging information;

[0152] 204. Based on the noise adjustment parameter, call the target neural network model to perform noise reduction on the processed image to obtain a target image.

[0153] Specifically, refer to Fig.13 In the embodiment of the present application, the image to be processed is first processed by the image signal processor to obtain a clean or partially noisy image to be processed, and the image to be processed is then passed to the adjustable image denoising model for processing to obtain the final target image. During the operation of the system, the image denoising model is debugged by the noise adjustment parameters generated by the parameter control module, and the final denoising result is obtained based on the debugged result.

[0154] The image processing system architecture provided by the embodiment of the present application can be applied to application scenarios where it is impossible to add algorithm nodes between the image sensor and the device hardware (i.e., the image signal processor). In addition, the image size will be reduced after the image is processed by the image signal processor. Therefore, the image denoising model processes an image of a smaller size, which can reduce the performance requirements and power consumption of the model to a certain extent.

[0155] In one embodiment, the electronic device is further integrated with an image sensor; the step of acquiring the image to be processed includes:

[0156] The original image of the current shooting scene is collected by the image sensor;

[0157] The original image is used as the image to be processed.

[0158] In one embodiment, the imaging information includes: at least one of sensitivity, brightness and dynamic range corresponding to the original image; and determining the noise adjustment parameter according to the imaging information includes:

[0159] Determine the ambient noise of the shooting scene where the original image is located according to the sensitivity, determine the ambient brightness of the shooting scene where the original image is located according to the brightness, and / or determine the light intensity range of the shooting scene where the original image is located according to the dynamic range;

[0160] Determining a scene type of the shooting scene based on the ambient noise, the ambient brightness and / or the light intensity range;

[0161] A noise adjustment parameter is determined according to the scene type.

[0162] In one embodiment, based on the noise adjustment parameter, calling the target neural network model to perform noise reduction processing on the processed image includes:

[0163] Acquire a residual image corresponding to the processed image based on the target neural network model;

[0164] Adjusting the parameter value of each pixel in the residual image according to the noise adjustment parameter;

[0165] The processed image is subjected to noise reduction processing based on the adjusted residual image.

[0166] In one embodiment, based on the noise adjustment parameter, calling the target neural network model to perform noise reduction processing on the processed image includes:

[0167] Obtaining a residual image corresponding to the processed image through the target neural network model;

[0168] Performing noise reduction on the processed image based on the residual image to obtain an intermediate image;

[0169] generating a local mask image according to the noise adjustment parameter and the intermediate image;

[0170] Adjusting the parameter value of each pixel in the residual image according to the local mask image;

[0171] The processed image is subjected to noise reduction processing based on the adjusted residual image.

[0172] In one embodiment, generating a local mask image according to the noise adjustment parameter and the intermediate image includes:

[0173] dividing the intermediate image into a plurality of image regions;

[0174] Determine an image classification corresponding to the image region according to the image content of the image region;

[0175] Determine the noise reduction level corresponding to the image area according to the image classification;

[0176] Determining the noise reduction intensity corresponding to each pixel in the intermediate image according to the noise adjustment parameter and the noise reduction level;

[0177] The local mask image is generated based on the noise reduction strength.

[0178] In one embodiment, the intermediate image is divided into a plurality of image regions, including:

[0179] extracting image features of the intermediate image;

[0180] The intermediate image is segmented according to the image features to obtain a plurality of image regions.

[0181] In one embodiment, generating a local mask image according to the noise adjustment parameter and the intermediate image includes:

[0182] Performing image filtering on the intermediate image to obtain a filtered intermediate image;

[0183] According to the noise adjustment parameter, the noise reduction strength corresponding to each pixel in the intermediate image after filtering is adjusted;

[0184] The intermediate image after adjusting the noise reduction strength is normalized to generate the local mask image.

[0185] In one embodiment, based on the noise adjustment parameter, calling the target neural network model to perform noise reduction processing on the processed image includes:

[0186] Acquire a feature map of the processed image through the target neural network model;

[0187] The feature map is split according to the dimension of the number of channels to obtain a residual image and a local mask image corresponding to the processed image;

[0188] According to the noise adjustment parameter, adjusting the noise reduction strength corresponding to each pixel point in the local mask image;

[0189] According to the local mask image after adjusting the noise reduction strength, adjusting the parameter value of each pixel in the residual image;

[0190] The processed image is subjected to noise reduction processing based on the adjusted residual image.

[0191] In one embodiment, based on the noise adjustment parameter, calling the target neural network model to perform noise reduction processing on the processed image includes:

[0192] Obtaining the image content of the processed image;

[0193] generating a local mask image according to the noise adjustment parameter and the image content of the processed image;

[0194] Obtaining a residual image corresponding to the processed image through the target neural network model;

[0195] Adjusting the parameter value of each pixel in the residual image according to the local mask image;

[0196] The processed image is subjected to noise reduction processing based on the adjusted residual image.

[0197] The specific implementation of the above implementation mode can refer to the above embodiments, which will not be described in detail.

[0198] From the above, it can be seen that the image processing method provided in the embodiment of the present application realizes the linkage between the target neural network model and the image signal processor based on the imaging information of the image to be processed, and uses the imaging information as an influencing factor to guide the target neural network model and the image signal processor to optimize the image in the same direction, thereby realizing the joint noise reduction of the model and the hardware equipment, and achieving a better noise reduction effect.

[0199] According to the image processing method provided by the above embodiment, the image processing method provided by this embodiment is further described in detail below by taking the image to be processed as the original image, the imaging information as the sensitivity, and the image processing device integrated in the electronic device as an example.

[0200] Please refer to Fig.14 , the process of the image processing method can be specifically as follows:

[0201] In step 301, the electronic device collects an original image of the current shooting scene through an image sensor.

[0202] In step 302, the electronic device obtains the sensitivity of the image sensor when capturing the original image.

[0203] In step 303, the electronic device determines the noise condition of the current shooting scene according to the sensitivity, and determines the scene type of the current shooting scene according to the noise condition.

[0204] In step 304, the electronic device determines a noise adjustment parameter according to the scene type.

[0205] In step 305, the electronic device calls the target neural network model to perform noise reduction processing on the original image according to the noise adjustment parameters to obtain a processed image.

[0206] In step 306, the electronic device calls the image signal processor to perform secondary noise reduction processing on the processed image based on the scene type of the current shooting scene to obtain a target image.

[0207] It should be noted that for the technical details of the above steps, please refer to the relevant description in the above embodiments, which will not be repeated here.

[0208] The image processing method provided in the embodiment of the present application distinguishes scenes according to the sensitivity of the image sensor when the original image is taken, realizes the linkage between the target neural network model and the image signal processor based on the scene type, and uses the sensitivity as an influencing factor to guide the target neural network model and the image signal processor to optimize the image in the same direction, thereby realizing the joint noise reduction of the model and the hardware equipment, thereby achieving a better noise reduction effect.

[0209] In order to facilitate better implementation of the image processing method provided in the embodiment of the present application, the embodiment of the present application also provides an image processing device based on the above image processing method. The image processing device can be integrated in an electronic device provided with an image signal processor, such as a server or a terminal, and the terminal can include a tablet computer, a laptop computer and / or a personal computer. The meaning of the terms is the same as in the above image processing method. For specific implementation details, please refer to the description in the above method embodiment.

[0210] Please refer to Fig.15 , Fig.15 The structure diagram of the image processing device provided in the embodiment of the present application is as follows. The image processing device may include an acquisition unit 401, a determination unit 402, a first noise reduction unit 403 and a second noise reduction unit 404, wherein:

[0211] An acquisition unit 401 is used to acquire an image to be processed and imaging information corresponding to the image to be processed;

[0212] A determination unit 402, configured to determine a noise adjustment parameter according to the imaging information;

[0213] A first noise reduction unit 403 is used to call a target neural network model to perform noise reduction processing on the image to be processed based on the noise adjustment parameter to obtain a processed image;

[0214] The second noise reduction unit 404 is used to call the image signal processor to perform noise reduction on the processed image based on the imaging information to obtain a target image.

[0215] In some implementations, the electronic device is further integrated with an image sensor; the acquisition unit 401 may be specifically used for:

[0216] The original image of the current shooting scene is collected by the image sensor;

[0217] The image to be processed is generated based on the original image.

[0218] In some implementations, when generating the image to be processed based on the original image, the acquisition unit 401 may further be used to:

[0219] The original image is used as the image to be processed; or,

[0220] Based on the imaging information, calling the image signal processor to pre-process the original image to obtain the image to be processed;

[0221] In some implementations, the imaging information includes: at least one of the sensitivity, brightness, and dynamic range corresponding to the original image; the determination unit 402 may be specifically configured to:

[0222] Determine the ambient noise of the shooting scene where the original image is located according to the sensitivity, determine the ambient brightness of the shooting scene where the original image is located according to the brightness, and / or determine the light intensity range of the shooting scene where the original image is located according to the dynamic range;

[0223] Determining a scene type of the shooting scene based on the ambient noise, the ambient brightness and / or the light intensity range;

[0224] A noise adjustment parameter is determined according to the scene type.

[0225] In some implementations, the first noise reduction unit 403 may be configured to:

[0226] Acquire a residual image corresponding to the image to be processed based on the target neural network model;

[0227] Adjusting the parameter value of each pixel in the residual image according to the noise adjustment parameter;

[0228] The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

[0229] In some implementations, the first noise reduction unit 403 may be configured to:

[0230] Obtaining a residual image corresponding to the image to be processed through the target neural network model;

[0231] Performing noise reduction processing on the image to be processed based on the residual image to obtain an intermediate image;

[0232] generating a local mask image according to the noise adjustment parameter and the intermediate image;

[0233] Adjusting the parameter value of each pixel in the residual image according to the local mask image;

[0234] The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

[0235] In some implementations, when generating a local mask image according to the noise adjustment parameter and the intermediate image, the first noise reduction unit 403 may be specifically configured to:

[0236] dividing the intermediate image into a plurality of image regions;

[0237] Determine an image classification corresponding to the image region according to the image content of the image region;

[0238] Determine the noise reduction level corresponding to the image area according to the image classification;

[0239] Determining the noise reduction intensity corresponding to each pixel in the intermediate image according to the noise adjustment parameter and the noise reduction level;

[0240] The local mask image is generated based on the noise reduction strength.

[0241] In some implementations, when the intermediate image is divided into a plurality of image regions, the first noise reduction unit 403 may be further configured to:

[0242] extracting image features of the intermediate image;

[0243] The intermediate image is segmented according to the image features to obtain a plurality of image regions.

[0244] In some implementations, when generating a local mask image according to the noise adjustment parameter and the intermediate image, the first noise reduction unit 403 may be specifically configured to:

[0245] Performing image filtering on the intermediate image to obtain a filtered intermediate image;

[0246] According to the noise adjustment parameter, the noise reduction strength corresponding to each pixel in the intermediate image after filtering is adjusted;

[0247] The intermediate image after adjusting the noise reduction strength is normalized to generate the local mask image.

[0248] In some implementations, the first noise reduction unit 403 may be configured to:

[0249] Acquire a feature map of the image to be processed through the target neural network model;

[0250] The feature map is split according to the dimension of the number of channels to obtain a residual image and a local mask image corresponding to the image to be processed;

[0251] According to the noise adjustment parameter, adjusting the noise reduction strength corresponding to each pixel point in the local mask image;

[0252] According to the local mask image after adjusting the noise reduction strength, adjusting the parameter value of each pixel in the residual image;

[0253] The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

[0254] In some implementations, the first noise reduction unit 403 may be configured to:

[0255] Obtaining the image content of the image to be processed;

[0256] Generate a local mask image according to the noise adjustment parameter and the image content of the image to be processed;

[0257] Obtaining a residual image corresponding to the image to be processed through the target neural network model;

[0258] Adjusting the parameter value of each pixel in the residual image according to the local mask image;

[0259] The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

[0260] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0261] The image processing device provided in the embodiment of the present application realizes the linkage between the target neural network model and the image signal processor based on the imaging information of the image to be processed, and uses the imaging information as an influencing factor to guide the target neural network model and the image signal processor to optimize the image in the same direction, thereby realizing the joint noise reduction of the model and the hardware equipment, and achieving a better noise reduction effect.

[0262] Based on the above image processing method, the embodiment of the present application also provides another image processing device. The image processing device can be integrated in an electronic device provided with an image signal processor, such as a server or a terminal, and the terminal can include a tablet computer, a laptop computer and / or a personal computer. The meaning of the terms is the same as in the above image processing method. For specific implementation details, please refer to the description in the above method embodiment.

[0263] Please refer to Fig.16 , Fig.16 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may also include an acquisition unit 401, a determination unit 402, a first noise reduction unit 403, and a second noise reduction unit 404, wherein:

[0264] An acquisition unit 401 is used to acquire an image to be processed and imaging information corresponding to the image to be processed;

[0265] A second noise reduction unit 404 is used to call the image signal processor to perform noise reduction processing on the image to be processed based on the imaging information to obtain a processed image;

[0266] A determination unit 402, configured to determine a noise adjustment parameter according to the imaging information;

[0267] The first denoising unit 403 is used to call the target neural network model to perform denoising on the processed image based on the noise adjustment parameter to obtain a target image.

[0268] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0269] The image processing device provided in the embodiment of the present application realizes the linkage between the target neural network model and the image signal processor based on the imaging information of the image to be processed, and uses the imaging information as an influencing factor to guide the target neural network model and the image signal processor to optimize the image in the same direction, thereby realizing the joint noise reduction of the model and the hardware equipment and achieving a better noise reduction effect.

[0270] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps of the image processing method provided in this embodiment by calling a computer program stored in the memory.

[0271] Please refer to Fig.17 , Fig.17 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0272] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, an input unit 504, and an image signal processor 505. Those skilled in the art will appreciate that Fig.17 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0273] The processor 501 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, the processor 501 performs various functions of the electronic device and processes data. Optionally, the processor 501 may include one or more processing cores; optionally, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0274] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0275] The electronic device also includes a power supply 503 for supplying power to each component. Optionally, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 503 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0276] The electronic device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0277] The electronic device may also include an image signal processor 505, which is logically connected to the processor 501 and can be used to match image sensors with different performances, process image signals at high speed, and realize a high-performance image processing engine for high-definition images. In the embodiment of the present application, the image signal processor 505 can be used to perform noise reduction processing on the image.

[0278] Although not shown, the electronic device may also include a display unit, an image acquisition component (such as an image sensor), etc., which will not be described in detail here. Specifically in this embodiment, the processor 501 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 502 according to the following instructions, and the processor 501 will execute the steps in the image processing method provided in this application, such as:

[0279] Acquire an image to be processed and imaging information corresponding to the image to be processed;

[0280] determining a noise adjustment parameter based on the imaging information;

[0281] Based on the noise adjustment parameter, calling the target neural network model to perform noise reduction processing on the image to be processed to obtain a processed image;

[0282] Based on the imaging information, the image signal processor is called to perform noise reduction processing on the processed image to obtain a target image.

[0283] or,

[0284] Acquire an image to be processed and imaging information corresponding to the image to be processed;

[0285] Based on the imaging information, calling the image signal processor to perform noise reduction processing on the image to be processed to obtain a processed image;

[0286] determining a noise adjustment parameter based on the imaging information;

[0287] Based on the noise adjustment parameters, the target neural network model is called to perform noise reduction on the processed image to obtain a target image.

[0288] The electronic device provided in the embodiment of the present application realizes the linkage between the target neural network model and the image signal processor based on the imaging information of the image to be processed, and uses the imaging information as an influencing factor to guide the target neural network model and the image signal processor to optimize the image in the same direction, thereby realizing the joint noise reduction of the model and the hardware device, thereby achieving a better noise reduction effect.

[0289] It should be noted that the electronic device provided in the embodiment of the present application and the image processing method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0290] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device executes the steps in the image processing method provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0291] The above is a detailed introduction to an image processing method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0292] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. An image processing method, applied to an electronic device, characterized in that: The electronic device is integrated with an image signal processor, and the method comprises: Acquire an image to be processed and imaging information corresponding to the image to be processed; determining a noise adjustment parameter according to the imaging information; Based on the noise adjustment parameters, calling the target neural network model to perform noise reduction processing on the image to be processed to obtain a processed image; Based on the imaging information, the image signal processor is called to perform noise reduction processing on the processed image to obtain a target image.

2. The image processing method according to claim 1, characterized in that: The electronic device is also integrated with an image sensor; the step of obtaining the image to be processed includes: Collecting the original image of the current shooting scene by the image sensor; The image to be processed is generated based on the original image.

3. The image processing method according to claim 2, characterized in that: The step of generating the image to be processed based on the original image comprises: Using the original image as the image to be processed; or, Based on the imaging information, the image signal processor is called to pre-process the original image to obtain the image to be processed.

4. The image processing method according to claim 2, characterized in that: The imaging information includes: at least one of the sensitivity, brightness and dynamic range corresponding to the original image; and determining the noise adjustment parameter according to the imaging information includes: Determine the environmental noise of the shooting scene where the original image is located according to the sensitivity, determine the environmental brightness of the shooting scene where the original image is located according to the brightness, and / or determine the light intensity range of the shooting scene where the original image is located according to the dynamic range; Determining a scene type of the shooting scene based on the ambient noise, the ambient brightness and / or the light intensity range; A noise adjustment parameter is determined according to the scene type.

5. The image processing method according to claim 1, characterized in that: The step of calling a target neural network model to perform noise reduction processing on the image to be processed based on the noise adjustment parameter includes: Acquire a residual image corresponding to the image to be processed based on the target neural network model; Adjusting the parameter value of each pixel in the residual image according to the noise adjustment parameter; The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

6. The image processing method according to claim 1, characterized in that: The step of calling a target neural network model to perform noise reduction processing on the image to be processed based on the noise adjustment parameter includes: Acquire a residual image corresponding to the image to be processed through the target neural network model; Performing noise reduction processing on the image to be processed based on the residual image to obtain an intermediate image; generating a local mask image according to the noise adjustment parameter and the intermediate image; Adjusting the parameter value of each pixel in the residual image according to the local mask image; The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

7. The image processing method according to claim 6, characterized in that: The step of generating a local mask image according to the noise adjustment parameter and the intermediate image comprises: dividing the intermediate image into a plurality of image regions; Determining an image classification corresponding to the image area according to image content of the image area; Determining a noise reduction level corresponding to the image area according to the image classification; Determining the noise reduction intensity corresponding to each pixel in the intermediate image according to the noise adjustment parameter and the noise reduction level; The local mask image is generated based on the noise reduction strength.

8. The image processing method according to claim 7, characterized in that: The step of dividing the intermediate image into a plurality of image regions comprises: extracting image features of the intermediate image; The intermediate image is subjected to image segmentation processing according to the image features to obtain a plurality of image regions.

9. The image processing method according to claim 6, characterized in that: The step of generating a local mask image according to the noise adjustment parameter and the intermediate image comprises: Performing image filtering on the intermediate image to obtain a filtered intermediate image; According to the noise adjustment parameter, adjusting the noise reduction strength corresponding to each pixel in the intermediate image after filtering; The intermediate image after adjusting the noise reduction strength is normalized to generate the local mask image.

10. The image processing method according to claim 1, characterized in that: The step of calling a target neural network model to perform noise reduction processing on the image to be processed based on the noise adjustment parameter includes: Acquire a feature map of the image to be processed through the target neural network model; Splitting the feature map according to the dimension of the number of channels to obtain a residual image and a local mask image corresponding to the image to be processed; According to the noise adjustment parameter, adjusting the noise reduction strength corresponding to each pixel in the local mask image; According to the local mask image after adjusting the noise reduction strength, adjusting the parameter value of each pixel in the residual image; The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

11. The image processing method according to claim 1, characterized in that: The step of calling a target neural network model to perform noise reduction processing on the image to be processed based on the noise adjustment parameter includes: Acquire the image content of the image to be processed; generating a local mask image according to the noise adjustment parameter and the image content of the image to be processed; Acquire a residual image corresponding to the image to be processed through the target neural network model; Adjusting the parameter value of each pixel in the residual image according to the local mask image; The image to be processed is subjected to noise reduction processing based on the adjusted residual image.

12. An image processing method, applied to an electronic device, characterized in that: The electronic device is integrated with an image signal processor, and the method comprises: Acquire an image to be processed and imaging information corresponding to the image to be processed; Based on the imaging information, calling the image signal processor to perform noise reduction processing on the image to be processed to obtain a processed image; determining a noise adjustment parameter according to the imaging information; Based on the noise adjustment parameters, a target neural network model is called to perform noise reduction processing on the processed image to obtain a target image.

13. An image processing device, applied to an electronic device, characterized in that: The electronic device is integrated with an image signal processor, and the device comprises: An acquisition unit, used for acquiring an image to be processed and imaging information corresponding to the image to be processed; A determination unit, configured to determine a noise adjustment parameter according to the imaging information; A first noise reduction unit, configured to call a target neural network model to perform noise reduction processing on the image to be processed based on the noise adjustment parameter to obtain a processed image; The second noise reduction unit is used to call the image signal processor to perform noise reduction processing on the processed image based on the imaging information to obtain a target image.

14. An image processing device, applied to an electronic device, characterized in that: The electronic device is integrated with an image signal processor, and the device comprises: An acquisition unit, used for acquiring an image to be processed and imaging information corresponding to the image to be processed; A second noise reduction unit, configured to call the image signal processor to perform noise reduction processing on the image to be processed based on the imaging information to obtain a processed image; A determination unit, configured to determine a noise adjustment parameter according to the imaging information; The first denoising unit is used to call a target neural network model to perform denoising on the processed image based on the noise adjustment parameter to obtain a target image.

15. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the image processing method according to any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the image processing method according to any one of claims 1 to 12.

Images