Image processing method and electronic device

Abstract

The application is suitable for the technical field of computers, and provides an image processing method and electronic equipment, wherein the method comprises the following steps: firstly, inputting continuous multiple frames of to-be-processed images into an encoder of a preset image processing model to obtain a first feature map; then inputting one frame of the multiple frames of to-be-processed images into a preset visual large model to obtain a second feature map containing high-level semantic information; then splicing the first feature map and the second feature map to obtain a third feature map; finally, inputting the third feature map into a decoder of the image processing model to obtain a target image. Thus, the high-level semantic information extracted by the visual large model is used to guide the image processing model to better complete an image processing task, and the effect of image processing is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method and an electronic device. BACKGROUND

[0002] Image processing refers to a technology of performing various operations and changes on a digital image, including noise reduction, demosaicing, super resolution (SR for short), defogging, and deraining.

[0003] In the related art, a neural network model can be used to process an image. However, a general neural network model cannot understand entities in the image, and thus cannot guarantee consistency of processing on the same entity, and further leads to poor image processing effect. For example, when performing noise reduction on an image containing a sky region, because part of the noise in the sky is similar to texture, the neural network model cannot well distinguish the two, and finally leads to poor image processing effect. SUMMARY

[0004] The present application provides an image processing method and an electronic device, which can optimize the effect of image processing.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which comprises: inputting continuous multiple frames of to-be-processed images into an encoder of a preset image processing model to obtain a first feature map; inputting one frame of the multiple frames of to-be-processed images into a preset visual large model to obtain a second feature map containing high-level semantic information; splicing the first feature map and the second feature map to obtain a third feature map; and inputting the third feature map into a decoder of the image processing model to obtain a target image.

[0006] In the technical solution of the present application, the first feature map extracted by the image processing model and the second feature map extracted by the visual large model are spliced, wherein the visual large model is used to extract high-level semantic information in the image to identify an entity region present in the image, and further the third feature map obtained by splicing is input into the decoder of the image processing model to finally obtain the target image. Thus, the high-level semantic information extracted by the visual large model is used to guide the image processing model to better complete the image processing task, and further optimize the effect of image processing.

[0007] Optionally, in a possible implementation manner of the first aspect, the tensor shape of the first feature map and the second feature map both contains at least four dimensions: image quantity, feature channel number, image height, and image width, and the splicing of the first feature map and the second feature map to obtain the third feature map comprises: splicing the first feature map and the second feature map in the feature channel number dimension to obtain the third feature map.

[0008] In the process of splicing the first feature map and the second feature map, the image height and the image width dimensions of the first feature map and the second feature map remain unchanged, and the splicing is performed according to the dimension of the feature channel number, which can effectively integrate the feature information of different channels together. The third feature map obtained by splicing is rich in high-level semantic information and can provide more abundant feature representation. In addition, the splicing in the dimension of the feature channel number ensures the consistency of the spatial structure and the channel information of the first feature map and the second feature map, so that the subsequent image processing model can correctly process the third feature map.

[0009] Optionally, in another possible implementation manner of the first aspect, the method further includes: in the training process of the image processing model, updating the encoder weight and the decoder weight of the image processing model according to the target image and the real image processing result.

[0010] The parameters of the image processing model can be continuously optimized by the target image and the real image processing result, the performance and the generalization ability of the image processing model are improved, and the effect of image processing is optimized.

[0011] Optionally, in another possible implementation manner of the first aspect, the inputting of the one frame of the plurality of frames of to-be-processed images into the preset visual large model to obtain the second feature map containing high-level semantic information includes: determining visual feature data contained in each frame of to-be-processed image; determining a to-be-processed image containing the most visual feature data as a reference frame; and inputting the reference frame into the visual large model to obtain the second feature map.

[0012] The to-be-processed image containing the most visual feature data is selected as the input of the visual large model, which can help the visual large model better understand the semantic information in the to-be-processed image.

[0013] Optionally, in another possible implementation manner of the first aspect, each frame of to-be-processed image contains visual feature data of a plurality of regions, and the second feature map contains high-level semantic information corresponding to each region.

[0014] When the to-be-processed image contains visual feature data of a plurality of regions, the visual large model can extract high-level semantic information of each region in the to-be-processed image, and the high-level semantic information can be used for identifying, segmenting, positioning and understanding different regions in the image, so as to provide more abundant and accurate information for the image processing model.

[0015] Optionally, in another possible implementation manner of the first aspect, the to-be-processed image is a noisy image, and the image processing model is a denoising model; the inputting the continuous multiple frames of to-be-processed images into the encoder of the preset image processing model to obtain the first feature map includes: obtaining noise intensity information corresponding to the multiple frames of noisy images; inputting the multiple frames of noisy images and the noise intensity information into the encoder of the denoising model to obtain the first feature map; and the inputting the third feature map into the decoder of the image processing model to obtain the target image includes: inputting the third feature map into the decoder of the denoising model to obtain a first processed image; and dynamically adjusting noise intensity information corresponding to each region in the first processed image based on the visual feature data of each region to determine the target image.

[0016] When the image processing task is a denoising task, the image processing model is a denoising model. Generally, in addition to the multiple frames of noisy images, the input of the denoising model also includes noise intensity information corresponding to the noisy images. The noise intensity information is usually obtained according to the exposure time and the ISO (International Standards Organization) of the camera on the electronic device, and therefore, the noise intensity information is consistent, which leads to the same denoising intensity of the denoising model for each region of the noisy image. Therefore, the application improves the denoising effect by dynamically adjusting the intensity of the noise intensity information.

[0017] Optionally, in another possible implementation manner of the first aspect, the image processing model is an HDR (High Dynamic Range) model, and the multiple frames of to-be-processed images include at least one long-exposure-time image and at least one short-exposure-time image.

[0018] When the image processing task is HDR, the long-exposure-time image can provide dark details, and the short-exposure-time image can provide bright details. The HDR model can combine the details of the two to finally generate a target image with a wider dynamic range and higher quality.

[0019] Optionally, in another possible implementation manner of the first aspect, the to-be-processed image is a to-be-super-resolution image, and the image processing model is a super-resolution model; the inputting the third feature map into the decoder of the image processing model to obtain the target image includes: inputting the third feature map into the decoder of the super-resolution model to obtain a second processed image; and performing up-sampling processing on the second processed image to determine the target image.

[0020] Since the target of the super-resolution task is to convert a low-resolution image into a high-resolution image, the resolution and details of the to-be-processed image are increased through the up-sampling processing, so that the target image obtained finally has higher resolution while maintaining the clarity of details.

[0021] In a second aspect, an embodiment of the present application provides an image processing apparatus, the apparatus comprising a unit composed of software and / or hardware for executing any of the methods of the first aspect.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors, and a memory; the memory is coupled to the one or more processors, and the memory is configured to store computer program codes, the computer program codes comprising computer instructions, and the one or more processors are configured to invoke the computer instructions to enable the electronic device to implement any of the methods of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a chip system applied to an electronic device, comprising one or more processors, and the one or more processors are configured to invoke computer instructions to enable the electronic device to implement any of the methods of the first aspect.

[0024] Optionally, the chip system further comprises a memory, and the memory is electrically connected to the processor.

[0025] Optionally, the chip system can further comprise a communication interface.

[0026] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium comprising instructions, and when the instructions are executed on an electronic device, the electronic device is enabled to implement any of the methods of the first aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by an electronic device, the electronic device is enabled to implement any of the methods of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0029] Figure 1 is an execution process schematic diagram of a conventional deep learning processing method;

[0030] Figure 2 is a flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0031] Figure 3 is a flow schematic diagram of a noise reduction method provided by an embodiment of the present application;

[0032] Figure 4 This is a schematic diagram illustrating the execution process of an existing image processing method;

[0033] Figure 5 This is a schematic diagram illustrating the execution process of an image processing method provided in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0036] The embodiments of this application will now be described in conjunction with the accompanying drawings.

[0037] Figure 1 This is a schematic diagram illustrating the execution process of a typical deep learning method. For example... Figure 1 As shown, the image to be processed is first acquired, and then input into the encoder of the image processing model. Feature extraction is performed on the image to learn its features, texture, shape, and other information, converting it into a data format that the computer can understand and process. After encoding, the image processing model decodes the image, using its decoder to perform image processing tasks on the information extracted by the decoder, ultimately generating the target image.

[0038] In the embodiments of this application, image processing may include low-level visual tasks such as noise reduction, demosaic, super resolution (SR), dehazing, deraining, and multi-frame fusion.

[0039] It should be noted that the encoder refers to the part that converts the input data (such as images) into feature representations, and the encoder is usually composed of a series of convolutional layers and pooling layers, which are used to gradually extract abstract features of the input data. The main function of the encoder is to reduce the spatial dimension of the input data through layer-by-layer convolution and pooling operations, and extract features with semantic information that can capture important patterns and structures in the input data and context information related to the task. Specifically, the encoder filters the input data through convolution operations to extract features of different scales and directions. Each convolutional layer is usually composed of multiple filters (or convolution kernels), each of which performs convolution operations on the input data to generate a set of feature maps. Each feature map represents the response of the input data in a certain feature direction. As the layers of the encoder deepen, the number of channels of the feature map usually increases, while the spatial dimension decreases. This is achieved through pooling operations, which aggregate each small region of the feature map to reduce the size of the feature map. Common pooling operations include max pooling and average pooling, which select the maximum or average value in a small region as the pooled value. The feature map output by the last layer of the encoder usually has a lower spatial resolution but contains higher-level semantic information. These feature maps can be passed to the decoder or other task-related modules for further processing and generation of the final output result.

[0040] Corresponding to the encoder, the decoder is responsible for converting the feature representation converted by the encoder into the final output result. The main task of the decoder is to perform inverse operations on the feature maps extracted by the encoder, gradually recover the spatial dimension, and generate an output corresponding to the original input data. The decoder is usually composed of a series of deconvolutional layers (also known as transpose convolutional layers) and up-sampling layers. Among them, the deconvolutional layer is the inverse operation of the convolutional layer, which can expand the spatial dimension of the feature map while keeping the number of feature channels unchanged. In the deconvolution operation, the value of each feature map pixel is copied to a larger output region, thereby realizing the recovery of the spatial dimension. In addition to the deconvolutional layer, the decoder can also use the up-sampling layer to recover the spatial dimension. The up-sampling layer increases the size of the feature map through interpolation or other techniques, thereby realizing the recovery of the spatial dimension. Common up-sampling methods include bilinear interpolation, deconvolution operation, etc. The decoder is usually symmetrical to the layer structure of the encoder, with gradually decreasing layers and gradually recovering the spatial dimension. The output of the last layer of the decoder usually has the same spatial dimension as the original input data and can be used as the final output result. It should be noted that the design of the decoder can be adjusted according to the needs of specific tasks.

[0041] However, general image processing models cannot understand the entities in the image, which cannot guarantee the consistency of processing the same entity, and thus leads to poor image processing effect. For example, when denoising an image containing a sky area, due to the similarity between part of the noise and the texture in the sky, the neural network model cannot distinguish between the two, ultimately resulting in poor image processing effect.

[0042] Therefore, the embodiment of the present application discloses an image processing method, mainly by splicing the first feature map extracted by the image processing model and the second feature map extracted by the visual large model, wherein the visual large model is used to extract high-level semantic information in the image to identify the entity region existing in the image, and further by inputting the third feature map obtained by splicing into the decoder of the image processing model, finally obtaining the target image. Thus, the high-level semantic information extracted by the visual large model is used to guide the image processing model to better complete the image processing task, thereby optimizing the effect of image processing.

[0043] Reference Figure 2 , shows a flowchart of an image processing method provided by the embodiment of the present application. The steps shown in Figure 2 will be introduced below.

[0044] Step 201, inputting continuous multiple frames of to-be-processed images into the encoder of a preset image processing model to obtain a first feature map.

[0045] It should be noted that each frame of to-be-processed image contains the same scene or object. First, continuous multiple frames of to-be-processed images can provide better image detail recovery effect, by fusing or aligning multiple frames of to-be-processed images, the noise, artifacts or blur in the to-be-processed image can be reduced. For example, in the super-resolution task, by using multiple low-resolution images to reconstruct a high-resolution image, the clarity and detail of the image can be improved. Second, for HDR tasks (such as HDR image synthesis, HDR photo enhancement, etc.), continuous multiple frames of to-be-processed images can provide a wider dynamic range. In some cases, a single frame of image may not be able to capture all the brightness details in the scene, especially in high-contrast or uneven lighting conditions. By using multiple frames of to-be-processed images, images under different exposure conditions can be obtained, thereby capturing a wider range of brightness, which is particularly important for processing HDR tasks. It should be understood that the processing of multiple frames of to-be-processed images may increase the complexity of calculation and storage, therefore, when designing the image processing model, the number of frames used needs to be determined according to the specific task requirements and resource limitations, for example, 3 continuous frames.

[0046] The to-be-processed image can be a RAW (Raw) image or an RGB (Red Green Blue) image. Specifically, the RAW image and the RGB image are two different image representation formats. The RAW image is raw data obtained directly from an image sensor of a digital camera without any processing. The RAW image records the photoelectric signal value of each pixel and is usually saved in a specific raw format, such as CR2 (Canon RAW) or NEF (Nikon RAW). The RAW image is not processed by the built-in image processing algorithm, and more image information is retained, including a higher dynamic range and color depth. Since the RAW image is not processed, it usually needs to be decoded, denoised, white balanced, color corrected, and color space converted by post-processing software to generate a final visual image. The RGB image is one of the most common color image representation forms. The RGB image is composed of three color channels: red (R), green (G), and blue (B). The color of each pixel is represented by the values of the three channels, usually in the range of 0 to 255. By adjusting the intensity of each channel, various colors and color combinations can be generated.

[0047] In some image processing processes, the RAW image can be converted into an RGB image. In this conversion process, operations such as decoding RAW data, applying white balance, color correction, and color space conversion are included, and finally a visual RGB image is generated. Since the RAW image retains more image information, it is usually selected to use the RAW format to shoot and process images when more post-processing and higher quality images are required.

[0048] In an implementation manner, the image processing model can adopt a Unet structure, which is a convolutional neural network structure commonly used in image segmentation tasks. In Unet, parameter feature fusion is only performed at the last layer of the encoder. Specifically, each layer of the encoder will splice the feature map with the feature map of the next layer to retain more spatial information. This splicing operation can be regarded as a feature fusion method, which combines low-level features and high-level features, so that the decoder can utilize information from different levels at the same time. After feature fusion, a residual connection can be used between the encoder and the decoder. The residual connection refers to adding the feature map of a certain layer of the encoder to the feature map of the corresponding layer of the decoder. This connection method can help information propagate faster in the network, alleviate the problem of gradient disappearance, and retain more detailed information, which helps to improve the image processing effect.

[0049] It should be understood that the image processing model can be obtained in advance by training according to actual needs and deployed at the end side. For example, when the image processing task is an HDR task, the image processing model can be an HDR model.

[0050] In an implementation, when the image processing task is HDR, the image processing model can be an HDR model, and the multi-frame to-be-processed images need to include at least one long-exposure image and at least one short-exposure image. It should be noted that HDR is an image processing technology aiming to improve the dynamic range of an image, so as to better show the bright and dark details in the scene. Traditional images can usually only capture a limited dynamic range, resulting in loss of details or overexposure or underexposure in high-light and shadow areas. The HDR technology combines multiple images with different exposure levels to show a wider brightness range.

[0051] Specifically, when processing an HDR task, the input of the HDR model needs to include at least one long-exposure image and at least one short-exposure image, because images under different exposure conditions contain different information. A long-exposure image is an image taken under a relatively long exposure time, and the long-exposure image is very useful for capturing low-brightness details, because of the longer exposure time, the bright part details in the long-exposure image are better preserved, but the dark part details can be overexposed or lost. A short-exposure image is an image taken under a relatively short exposure time, and the short-exposure image is very useful for capturing high-brightness details, because of the shorter exposure time, the dark part details in the short-exposure image are better preserved, but the bright part details can be truncated or lost. By using both long-exposure images and short-exposure images, the HDR model can utilize the differences between the two to recover the lost details. The long-exposure image provides dark part details, and the short-exposure image provides bright part details, and the HDR model can combine the information of the two to generate an HDR image with a wider dynamic range.

[0052] Further, the multi-frame to-be-processed images can also include at least one medium-exposure image, and the HDR model can use the additional medium-exposure image to provide more reference information. The medium-exposure image is between the long-exposure image and the short-exposure image in exposure time, and the medium-exposure image can provide more balanced brightness and details, which helps to further improve the quality of the HDR image.

[0053] Step 202, inputting one frame of the multi-frame to-be-processed images into a preset visual large model to obtain a second feature map containing high-level semantic information.

[0054] In an implementation manner, the visual large model can be a Transformer-based visual large model, such as VIT (Vision Transformer), SAM (Spatially Adaptive Multi-scale Attention), etc. The Transformer architecture can be used to extract high-level semantic information in images. Specifically, both VIT and SAM use the attention mechanism of the Transformer to model images, thereby realizing attention and feature extraction of different regions in the image. Among them, VIT can divide the input image into a series of image blocks (patches), and then map each image block to a vector through an embedding layer. These vectors are input sequences processed by the Transformer encoder, thereby learning the global feature representation of the image. SAM can model images by constructing spatially adaptive multi-scale attention at different levels. The SAM model can better capture the context information and spatial relationship of the target by learning the importance weight of different positions and scales in the image. The Transformer-based visual large model can model the global information and context in the image, thereby capturing more rich high-level semantic information.

[0055] It should be noted that the high-level semantic information refers to the abstract, semantic content or concept in the image, and the high-level semantic information represents a deeper understanding of objects, scenes, relationships and semantic concepts in the image. High-level semantic information can help the electronic device better understand the image and perform higher-level analysis and reasoning. High-level semantic information can include recognition and classification of different objects appearing in the image, such as recognizing the type of object, such as people, cars, cats, etc., and accurately classifying different objects. High-level semantic information can involve understanding and classification of scenes in the image, such as recognizing different types of scenes, such as indoor, outdoor, city, natural scenery, sky, etc., and more detailed descriptions of the scene, such as beach, park, office, etc. High-level semantic information can involve modeling and understanding of relationships between objects in the image, including recognizing spatial relationships (such as relative position, distance) between objects, semantic relationships (such as "in", "holding") and contextual relationships (such as the probability of the appearance of an object in a specific scene), etc. High-level semantic information can involve segmentation and labeling of different regions in the image, i.e., decomposing the image into semantically meaningful parts, such as objects, backgrounds, boundaries, etc.

[0056] In the embodiment of the present application, each frame of the to-be-processed image contains visual feature data of multiple regions, and the second feature map contains high-level semantic information corresponding to each region. When the to-be-processed image contains visual feature data of multiple regions, the visual large model can extract high-level semantic information of each region in the to-be-processed image, and be used for identifying, segmenting, locating and understanding different regions in the image, thereby providing more rich and accurate information for the image processing model.

[0057] The visual large model only needs one frame of to-be-processed image (for example, the first frame of the continuous multiple frames of to-be-processed image) to extract the second feature map containing high-level semantic information. In addition, the visual large model usually has large computing and storage overhead, and selecting one frame of to-be-processed image as the reference frame, that is, only one frame of to-be-processed image needs to be processed instead of all to-be-processed images, can reduce the amount of calculation, improve the processing speed, and reduce the computing resource demand of the visual large model.

[0058] In one embodiment, the step 202 can include determining visual feature data contained in each frame of to-be-processed image, determining the to-be-processed image with the most visual feature data as the reference frame, and inputting the reference frame into the visual large model to obtain the second feature map. Selecting the to-be-processed image with the most visual feature data as the input of the visual large model can help the visual large model better understand the semantic information in the to-be-processed image.

[0059] In one embodiment, the visual large model can be obtained by training. In order to reduce the training cost, the encoder of the visual large model can also use the currently open-source lightweight model and the corresponding weight, such as MobileSAM (Mobile Spatial Attention Module), FastSAM (Fast Spatial Attention Module), EfficientSAM (Efficient Spatial Attention Module) and the like. The above models can effectively extract high-level semantic information of the image while maintaining lightweight. MobileSAM, FastSAM and EfficientSAM are all models based on attention mechanism (Attention Mechanism), and have the characteristics of lightweight and high efficiency.

[0060] Specifically, MobileSAM is a lightweight attention module that learns spatial attention weights for images, allowing visual large models to focus more on important image regions. MobileSAM has fewer parameters and computational complexity, making it suitable for use on resource-constrained devices. FastSAM is a fast spatial attention module designed to improve the speed and efficiency of visual large models. FastSAM reduces the complexity and memory consumption of attention calculations to speed up the inference process of visual large models. EfficientSAM is a high-efficiency spatial attention module designed to improve the performance and efficiency of visual large models in image processing tasks. EfficientSAM uses a lightweight attention calculation method to effectively capture important information in images while maintaining low computational and memory overhead. These models can reduce the parameter quantity, computational complexity, and memory consumption of the model while maintaining high performance to adapt to the resource constraints of electronic devices.

[0061] In step 203, the first feature map and the second feature map are spliced to obtain a third feature map.

[0062] In the embodiments of the present application, the tensor shape of the first feature map and the second feature map each contains at least four dimensions: image quantity, feature channel number, image height, and image width. The tensor shape of the first feature map can be represented as [batch_size, cA, h, w], and the tensor shape of the second feature map can be represented as [batch_size, cB, h, w]. Wherein, batch_size represents the image quantity, specifically the number of images in each batch; cA and cB represent the feature channel number, also known as the channel dimension or depth, each channel contains the information of the feature map in a certain abstract feature dimension, for example, the channel can represent different color channels (such as red, green, and blue); h represents the image height, that is, the height size of the feature map in the spatial dimension, in image processing tasks, the image height usually corresponds to the number of vertical direction pixels of the image; w represents the image width, that is, the width size of the feature map in the spatial dimension, in image processing tasks, the image width usually corresponds to the number of horizontal direction pixels of the image.

[0063] In an implementation manner, the step 203 can include: splicing the first feature map and the second feature map in a feature channel number dimension to obtain a third feature map. When the first feature map and the second feature map are spliced, the image height and the image width dimensions of the first feature map and the second feature map remain unchanged, and the splicing is performed according to the feature channel number dimension. The feature information of different channels can be effectively integrated together. The third feature map obtained by splicing is rich in high-level semantic information and can provide more abundant feature representation. In addition, the splicing in the feature channel number dimension ensures the consistency of the spatial structure and the channel information of the first feature map and the second feature map, so that the subsequent image processing model can correctly process the third feature map.

[0064] For example, if the tensor shape of the first feature map is [batch_size, cA, h, w] and the tensor shape of the second feature map is [batch_size, cB, h, w], after the first feature map and the second feature map are spliced in the feature channel number dimension, the tensor shape of the third feature map obtained can be represented as [batch_size, cA+cB, h, w].

[0065] The step 204 includes: inputting the third feature map into a decoder of the image processing model to obtain a target image.

[0066] The second feature map extracted by the visual large model can contain the representation of any category object, and the fusion is performed at the deep feature level through the step 203, so that the image processing capability of the image processing model can be effectively improved.

[0067] In an embodiment, when the image processing task is super resolution, the image to be processed can be a super-resolution image to be processed, and the image processing model can be a super-resolution model. The step 204 can include: inputting the third feature map into a decoder of the super-resolution model to obtain a second processed image; and performing up-sampling processing on the second processed image to determine the target image. Since the target of the super-resolution task is to convert a low-resolution image into a high-resolution image, the resolution and the details of the image to be processed are increased through the up-sampling processing, so that the target image obtained finally has higher resolution while maintaining the clarity of the details.

[0068] As another possible implementation manner of the present application, after the target image is obtained, the image processing method further includes: updating the encoder weight and the decoder weight of the image processing model according to the target image in a training process of the image processing model. The parameters of the image processing model can be continuously optimized through the target image and the real image processing result, the performance and the generalization capability of the image processing model are improved, and the effect of the image processing is optimized.

[0069] As an example, the target image can be compared with the real image processing result to provide a clear supervision signal for the image processing model, so that the image processing model can learn how to better process the image to make its output closer to the real result. Supervised learning with labeled data can guide the learning process of the image processing model and improve the performance of the model. Specifically, the loss can be calculated and the weights of the image processing model can be updated using the backpropagation algorithm to optimize the parameters of the image processing so that it can better fit the training data and continuously optimize the processing effect of the image processing task. After multiple optimization processes, the generalization ability of the image processing model can be greatly improved. The image processing model can learn the commonalities and rules between different samples, so as to perform better when processing new unseen samples.

[0070] The image processing method provided by the embodiment of the present application first inputs the continuous multiple frames of to-be-processed images into the encoder of the preset image processing model to obtain a first feature map; then inputs one frame of the multiple frames of to-be-processed images into the preset visual large model to obtain a second feature map containing high-level semantic information; then splices the first feature map and the second feature map to obtain a third feature map; and finally inputs the third feature map into the decoder of the image processing model to obtain a target image. The method mainly splices the first feature map extracted by the image processing model and the second feature map extracted by the visual large model, wherein the visual large model is used to extract high-level semantic information in the image to identify the entity region existing in the image, and further inputs the third feature map obtained by splicing into the decoder of the image processing model to finally obtain the target image. Thus, the high-level semantic information extracted by the visual large model is used to guide the image processing model to better complete the image processing task, and the effect of image processing is optimized.

[0071] Referring to Figure 3 , a flowchart of a noise reduction method provided by an embodiment of the present application is shown. The steps shown in Figure 3 will be introduced below.

[0072] In step 301, multiple frames of noisy images and noise intensity information are input into the encoder of the noise reduction model to obtain a first feature map.

[0073] Step 301 can be regarded as an example of step 201. Generally, the input of the denoising model usually contains a NoiseMap channel in addition to the multi-frame noisy images, which is used to indicate the noise intensity information of the current input image. The noise intensity information is usually estimated according to the exposure time and ISO setting of the camera on the electronic device, so the value of NoiseMap is the same in the same group of input images. However, this may cause the denoising model to have the same denoising intensity for different regions of the image, because the denoising model cannot distinguish the noise level of different regions in the image according to the different values of NoiseMap, for example, over-smoothing details in low-noise regions or not sufficiently denoising in high-noise regions.

[0074] Step 302, inputting a frame of the multi-frame noisy images into a preset visual large model to obtain a second feature map containing high-level semantic information.

[0075] Step 303, splicing the first feature map and the second feature map to obtain a third feature map.

[0076] Step 304, inputting the third feature map into the decoder of the denoising model to obtain a first processed image.

[0077] The specific implementation process and principles of steps 302-304 described above can refer to the detailed description of the above embodiments, which will not be repeated here.

[0078] Step 305, based on the visual feature data of each region, dynamically adjusting the noise intensity information corresponding to each region in the first processed image to determine a target image.

[0079] Steps 304-305 can be regarded as an example of step 204.

[0080] In one embodiment, for the relatively smooth regions in the noisy image, the noise intensity information of such regions can be increased, thereby effectively reducing the noise of such regions and improving the overall quality of the image. For regions with more texture details, the noise intensity information of such regions can be reduced, so that the denoising model can adopt a lighter denoising strategy in these regions to preserve more detail information.

[0081] For example, assume that the noisy image contains a sky region and a building region. Since the sky is a smooth region, the noise level of this region can be informed to the denoising model by increasing the noise intensity information of the sky region, so that the denoising model can adopt a stronger denoising strategy to reduce the noise residue of the sky region; as for the building region, this kind of region has rich texture details, and the smearing problem can be reduced by reducing the noise intensity information of the building region, because the texture details of these regions are crucial to the image quality, and excessive denoising may cause loss of details and blurring effect.

[0082] In Figure 3 In the scheme shown, first, the multi-frame noisy image and the noise intensity information are input into the encoder of the denoising model to obtain a first feature map, then one frame of the multi-frame noisy image is input into a preset visual large model to obtain a second feature map containing high-level semantic information, then the first feature map and the second feature map are spliced to obtain a third feature map, and further the third feature map is input into the decoder of the denoising model to obtain a first processed image, and finally the noise intensity information corresponding to each region in the first processed image is dynamically adjusted based on the visual feature data of each region to determine a target image. Thus, by dynamically adjusting the intensity of the noise intensity information, the denoising effect is improved.

[0083] Figure 4 is an execution process diagram of an existing image processing method. As Figure 4 shown, the image to be processed and the pre-segmented Mask result can be input into the model for image processing, where the Mask result refers to a binary mask generated in the image segmentation task, used to identify the target region of interest in the image. The Mask result is usually used to separate the target from the background for further analysis, processing or application of other algorithms. It should be noted that the pre-segmented Mask result is single-channel and usually a binary image, in which the target region is marked as foreground (usually 1) and the background region is marked as background (usually 0). By fusing the Mask result with the image to be processed, the information of the Mask result can be combined with the pixel values of the image to form new input data. This fusion can be performed at each pixel position of the input image, for example, multiplying or adding the value of the Mask result with the pixel value at the corresponding position. In this way, the encoder module of the image processing model can use the information of the Mask result to guide the encoding and processing of the target region to extract more distinctive and important features.

[0084] However, the Mask result is single-channel, contains a closed set of categories (a limited number of categories, consistent with the number of categories in the training samples), and performs fusion at the input image level, which places a heavy burden on the encoding capabilities of the image processing model's encoder module. In contrast, the large-scale visual model in this application deals with an open-set world. Taking the SAM model as an example, the SAM model can segment any object; therefore, the second feature map obtained through the SAM model can be used to represent any object, and fusion at the depth feature level can effectively improve the image processing capabilities of the image processing model.

[0085] Figure 5 This diagram illustrates the execution process of an image processing method provided in an embodiment of this application. Figure 5 As shown, firstly, multiple frames of images to be processed ( Figure 5 The first feature map is obtained by inputting only 3 frames (shown in the image) into the decoder of the image processing model. Then, a reference frame is determined from multiple frames of the image to be processed and input into the encoder of the large visual model to obtain a second feature map containing high-level semantic information. Next, the first and second feature maps are concatenated to obtain a third feature map. Finally, the third feature map is input into the decoder of the image processing model to obtain the processed target image. Compared to... Figure 4 The conventional image processing method shown in this application involves stitching together a first feature map extracted by an image processing model and a second feature map extracted by a large visual model. The large visual model is used to extract high-level semantic information from the image to identify entity regions. The stitched third feature map is then input into the decoder of the image processing model to obtain the target image. Thus, the high-level semantic information extracted by the large visual model guides the image processing model to better complete the image processing task, thereby optimizing the image processing effect.

[0086] The methods of the embodiments of this application have been described above with reference to the accompanying drawings. It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially, these steps are not necessarily executed in the order shown in the figures. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps. The apparatus of the embodiments of this application will now be described with reference to the accompanying drawings.

[0087] Figure 6is a structural schematic diagram of an image processing apparatus according to an embodiment of the present application. For ease of illustration, only parts related to the embodiments of the present application are shown. As shown in Figure 6 The image processing apparatus 600 includes an input unit 601 and a processing unit 602. The image processing apparatus 600 can be integrated in an electronic device such as a mobile phone, a computer, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like.

[0088] The image processing apparatus 600 can be used to perform any of the image processing methods described above. For example, the input unit 601 can be used to perform steps 201-202 and step 204, and the processing unit 602 can be used to perform step 203. For another example, the input unit 601 can be used to perform steps 301-302 and step 304, and the processing unit 602 can be used to perform steps 303 and step 305.

[0089] In an implementation manner, the image processing apparatus 600 can further include a storage unit for storing related data. The storage unit can be integrated in any of the above-mentioned units, or can be a unit independent of all the above-mentioned units.

[0090] The image processing apparatus provided by the above-mentioned embodiments of the present application mainly performs splicing on the first feature map extracted by the image processing model and the second feature map extracted by the visual large model, wherein the visual large model is used to extract high-level semantic information in the image to identify the entity region existing in the image, and further inputs the third feature map obtained by splicing into the decoder of the image processing model to finally obtain the target image. Thus, the high-level semantic information extracted by the visual large model is used to guide the image processing model to better complete the image processing task, thereby optimizing the effect of image processing.

[0091] Figure 7 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. As shown in Figure 7As shown, the electronic device 700 can include a processor 710, an external memory interface 720, an internal memory 721, a universal serial bus (USB) interface 730, a charging management module 740, a power management module 741, a battery 742, an antenna 1, an antenna 2, a mobile communication module 750, a wireless communication module 760, an audio module 770, a speaker 770A, a receiver 770B, a microphone 770C, a headset jack 770D, a sensor module 780, a key 790, a motor 791, an indicator 792, a camera 793, a display screen 794, and a subscriber identification module (SIM) card interface 795, and the like. Among them, the sensor module 780 can include a pressure sensor 780A, a gyroscope sensor 780B, a barometric sensor 780C, a magnetic sensor 780D, an acceleration sensor 780E, a distance sensor 780F, a proximity light sensor 780G, a fingerprint sensor 780H, a temperature sensor 780J, a touch sensor 780K, an ambient light sensor 780L, a bone conduction sensor 780M, and the like.

[0092] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 700. In some other embodiments of the present application, the electronic device 700 can include more or fewer components than the illustration, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0093] Exemplarily, Figure 7 The processor 710 shown can include one or more processing units, for example: the processor 710 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), and the like. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0094] Among them, the controller can be the nerve center and command center of the electronic device 700. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0095] The processor 710 can also include a memory for storing instructions and data. In some embodiments, the memory in the processor 710 is a cache memory. The cache memory can hold instructions or data that the processor 710 has recently used or is likely to use again. If the processor 710 needs to use the instructions or data again, it can be retrieved directly from the cache memory. This avoids repeated accesses to the main memory, reducing the latency of the processor 710 and thus improving the efficiency of the system.

[0096] In some embodiments, the MIPI interface can be used to connect the processor 710 and the display screen 794, the camera 793, and other peripheral devices. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. The processor 710 and the display screen 794 communicate through the DSI interface to realize the display function of the electronic device 700.

[0097] In some embodiments, the GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. The GPIO interface can be used to connect the processor 710 and the camera 793, the display screen 794, the wireless communication module 760, the audio module 770, the sensor module 780, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0098] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 700. In some other embodiments of the present application, the electronic device 700 can also use different interface connection methods or a combination of multiple interface connection methods in the above embodiments.

[0099] The electronic device 700 realizes the display function through the GPU, the display screen 794, and the application processor. The GPU is a microprocessor for image processing, connected to the display screen 794 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 710 can include one or more GPUs that execute program instructions to generate or change display information.

[0100] The display screen 794 is used to display images, videos, and the like. The display screen 794 includes a display panel. In some embodiments, the electronic device 700 can include one or N display screens 794, where N is a positive integer greater than 1.

[0101] The internal memory 721 can be used to store computer-executable program codes including instructions. The processor 710 performs various functional applications and data processing of the electronic device 700 by executing the instructions stored in the internal memory 721. The internal memory 721 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program (e.g., a sound play function, an image play function, etc.) required for at least one function, etc. The data storage area can store data (e.g., audio data, a phone book, etc.) created during the use of the electronic device 700, etc. In addition, the internal memory 721 can include a high-speed random access memory and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0102] The pressure sensor 780A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 780A can be disposed on the display screen 794. There are many types of pressure sensors 780A, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates having a conductive material. When a force is applied to the pressure sensor 780A, the capacitance between the electrodes changes. The electronic device 700 determines the intensity of the pressure according to the change in capacitance. When a touch operation is applied to the display screen 794, the electronic device 700 detects the intensity of the touch operation according to the pressure sensor 780A. The electronic device 700 can also calculate the position of the touch according to the detection signal of the pressure sensor 780A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view a short message is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0103] The touch sensor 780K, also referred to as a "touch panel". The touch sensor 780K can be disposed on the display screen 794, and the touch sensor 780K and the display screen 794 together form a touch screen, also referred to as a "touch panel". The touch sensor 780K is used to detect a touch operation applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through the display screen 794. In other embodiments, the touch sensor 780K can also be disposed on the surface of the electronic device 700, which is different from the position where the display screen 794 is located.

[0104] It should be noted that the information interaction, execution process and the like between the above apparatuses / units are based on the same concept as the method embodiments of the present application, and specific functions and brought technical effects can be referred to the method embodiments part, which will not be repeated here.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0106] The embodiments of the present application also provide an electronic device, which comprises one or more processors, a memory, and a computer program stored in the memory and executable on the one or more processors, wherein the one or more processors execute the computer program to enable the electronic device to implement the steps in any of the above methods. The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, wherein the computer program is executed by an electronic device to implement the steps in each of the above methods.

[0107] The computer-readable medium can at least include any entity or device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, which can carry the computer program code to the photographing apparatus / electronic device. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0108] The embodiment of the present application provides a computer program product, which comprises a computer program. When the computer program is executed by an electronic device, the steps in each of the above-mentioned method embodiments can be implemented. The computer program comprises computer program code. The computer program code can be in the form of source code, object code, an executable file, or some intermediate form.

[0109] In the above embodiments, the description of each embodiment has its own focus. The parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0110] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0111] In the embodiments provided in the present application, it should be understood that the disclosed apparatuses / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0112] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. can be located in one place or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0113] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or sets thereof.

[0114] It should also be understood that, in the description of the application and the appended claims, the term "and / or" is used to mean one or more of the associated listed items, as well as the sum of all possible combinations of the associated listed items.

[0115] In addition, in the description of the application and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0116] In the description of the application, the reference "one embodiment" or "some embodiments" and the like means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0117] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining noise intensity information corresponding to a plurality of continuous frames of to-be-processed images, the plurality of frames of to-be-processed images being noisy images, each frame of the to-be-processed images comprising visual feature data of a plurality of regions; inputting the plurality of continuous frames of to-be-processed images and the noise intensity information into an encoder of a preset image processing model to obtain a first feature map, the image processing model being a denoising model; inputting one frame of the plurality of frames of to-be-processed images into a preset visual large model to obtain a second feature map comprising high-level semantic information, the second feature map comprising high-level semantic information corresponding to each of the regions respectively; splicing the first feature map and the second feature map to obtain a third feature map; inputting the third feature map into a decoder of the image processing model to obtain a first processed image; dynamically adjusting noise intensity information corresponding to each region in the first processed image based on visual feature data of each region to determine a target image.

2. The image processing method of claim 1, wherein, The tensor shape of the first feature map and the second feature map each comprises at least four dimensions: image quantity, feature channel number, image height, and image width, and the splicing of the first feature map and the second feature map to obtain the third feature map comprises: splicing the first feature map and the second feature map in the feature channel number dimension to obtain the third feature map.

3. The image processing method of claim 1, wherein, The method further comprises: updating the encoder weight and the decoder weight of the image processing model according to the target image during a training process of the image processing model.

4. The image processing method of claim 1, wherein, The inputting of one frame of the plurality of frames of to-be-processed images into the preset visual large model to obtain the second feature map comprising high-level semantic information comprises: determining visual feature data contained in each frame of the to-be-processed images; determining a to-be-processed image with the most visual feature data as a reference frame; inputting the reference frame into the visual large model to obtain the second feature map.

5. An electronic device, comprising: The electronic device comprises one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is configured to store computer program code comprising computer instructions, and the one or more processors are configured to invoke the computer instructions to enable the electronic device to perform the method according to any one of claims 1 to 4.

6. A chip system, characterized by The chip system is applied to an electronic device, and the chip system comprises one or more processors configured to invoke computer instructions to enable the electronic device to perform the method according to any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer readable storage medium comprises instructions configured to enable an electronic device to perform the method according to any one of claims 1 to 4 when the instructions are executed on the electronic device.

Images