Image processing method and electronic equipment

Abstract

The invention is suitable for the technical field of computers, and provides an image processing method and electronic device.The method comprises the steps that firstly, continuous multiple frames of images to be processed are input into an encoder of a preset image processing model, and a first feature map is obtained; inputting one frame in the multiple frames of images to be processed into a preset visual large model to obtain a second feature map containing advanced semantic information; splicing the first feature map and the second feature map to obtain a third feature map; and finally, inputting the third feature map into a decoder of an image processing model to obtain a target image. Therefore, the image processing model is guided to better complete the image processing task through the advanced semantic information extracted by the visual large model, and the image processing effect is further optimized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] Image processing refers to the technology of performing various operations and changes on digital images, including noise reduction, demosaicing, super resolution (SR, abbreviated as super resolution), defogging, deraining, etc.

[0003] In related technologies, neural network models can be used to process images. However, general neural network models cannot understand the entities in the image, and cannot guarantee the consistency of processing the same entity, which leads to poor image processing effects. For example, when denoising an image containing a sky area, since some noise in the sky is similar to texture, the neural network model cannot distinguish between the two well, which ultimately leads to poor image processing effects. Summary of the invention

[0004] The present application provides an image processing method and an electronic device, which can optimize the effect of image processing.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which includes: inputting a plurality of continuous frames of images to be processed into an encoder of a preset image processing model to obtain a first feature map; inputting one frame of the plurality of frames of images to be processed into a preset large visual model to obtain a second feature map containing high-level semantic information; splicing the first feature map and the second feature map to obtain a third feature map; and inputting the third feature map into a decoder of the image processing model to obtain a target image.

[0006] In the technical solution of the present application, the first feature map extracted by the image processing model and the second feature map extracted by the large visual model are spliced, wherein the large visual model is used to extract high-level semantic information in the image to identify the entity area in the image, and further the third feature map obtained by splicing is input into the decoder of the image processing model to finally obtain the target image. Thus, the high-level semantic information extracted by the large visual model is used to guide the image processing model to better complete the image processing task, thereby optimizing the effect of image processing.

[0007] Optionally, in a possible implementation of the first aspect, the tensor shapes of the first feature map and the second feature map each include at least four dimensions: the number of images, the number of feature channels, the image height, and the image width, and the first feature map and the second feature map are concatenated to obtain a third feature map, including: concatenating the first feature map and the second feature map in the dimension of the number of feature channels to obtain the third feature map.

[0008] Among them, when the first feature map and the second feature map are spliced, the image height and image width dimensions of the first feature map and the second feature map remain unchanged, and the splicing is performed according to the dimension of the number of feature channels, which can effectively integrate the feature information of different channels together, and the spliced ​​third feature map is rich in high-level semantic information and can provide richer feature representation. In addition, splicing is performed on the dimension of the number of feature channels to ensure the consistency of the spatial structure and channel information of the first feature map and the second feature map, so that the subsequent image processing model can correctly process the third feature map.

[0009] Optionally, in another possible implementation of the first aspect, the method further includes: during the training of the image processing model, updating the encoder weights and decoder weights of the image processing model according to the target image.

[0010] Among them, through the target image and the actual image processing results, the parameters of the image processing model can be continuously optimized, the performance and generalization ability of the image processing model can be improved, and then the image processing effect can be optimized.

[0011] Optionally, in another possible implementation of the first aspect, the above-mentioned inputting one frame of the multiple frames of images to be processed into a preset visual large model to obtain a second feature map containing high-level semantic information includes: determining the visual feature data contained in each frame of the image to be processed; determining the image to be processed with the most visual feature data as a reference frame; and inputting the reference frame into the visual large model to obtain a second feature map.

[0012] Among them, selecting the image to be processed with the most visual feature data as the input of the large visual model can help the large visual model better understand the semantic information in the image to be processed.

[0013] Optionally, in another possible implementation of the first aspect, each frame of the image to be processed contains visual feature data of multiple regions, and the second feature map contains high-level semantic information corresponding to each region.

[0014] Among them, when the image to be processed contains visual feature data of multiple regions, the high-level semantic information of each region in the image to be processed can be extracted through the visual big model, and used to identify, segment, locate and understand different regions in the image, thereby providing richer and more accurate information for the image processing model.

[0015] Optionally, in another possible implementation of the first aspect, the image to be processed is a noisy image, and the image processing model is a noise reduction model; the continuous multiple frames of images to be processed are input into an encoder of a preset image processing model to obtain a first feature map, including: obtaining noise intensity information corresponding to the multiple frames of noisy images; inputting the multiple frames of noisy images and the noise intensity information into the encoder of the noise reduction model to obtain the first feature map; the third feature map is input into the decoder of the image processing model to obtain the target image, including: inputting the third feature map into the decoder of the noise reduction model to obtain the first processed image; based on the visual feature data of each area, dynamically adjusting the noise intensity information corresponding to each area in the first processed image to determine the target image.

[0016] Among them, when the image processing task is a noise reduction task, the image processing model is a noise reduction model. Usually, the input of the noise reduction model includes noise intensity information corresponding to the noisy image in addition to multiple frames of noisy images. The noise intensity information is usually obtained based on the exposure time and sensitivity (International Standards Organization, ISO) of the camera on the electronic device. Therefore, the noise intensity information is consistent, which leads to the same noise reduction intensity for each area on the noisy image in the subsequent noise reduction model. Therefore, the present application improves the noise reduction effect by dynamically adjusting the intensity of the noise intensity information.

[0017] Optionally, in another possible implementation of the first aspect, the image processing model is a high dynamic range (HDR) model, and the multiple frames of images to be processed include at least one frame of long exposure image and at least one frame of short exposure image.

[0018] Among them, when the image processing task is HDR, the image can provide dark details when long exposure, and the image can provide bright details when short exposure. The HDR model can merge the details of the two and ultimately generate a target image with a wider dynamic range and higher quality.

[0019] Optionally, in another possible implementation of the first aspect, the image to be processed is an image to be super-resolved, and the image processing model is a super-resolution model; the third feature map is input into the decoder of the image processing model to obtain a target image, including: inputting the third feature map into the decoder of the super-resolution model to obtain a second processed image; and upsampling the second processed image to determine the target image.

[0020] Among them, since the goal of the super-resolution task is to convert a low-resolution image into a high-resolution image, the resolution and details of the image to be processed are increased through upsampling, so that the final target image has a higher resolution while maintaining detail clarity.

[0021] In a second aspect, an embodiment of the present application provides an image processing device, which includes a unit composed of software and / or hardware for executing any one of the methods of the first aspect.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to implement any one of the methods of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device and includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device can implement any one of the methods of the first aspect.

[0024] Optionally, the chip system also includes a memory, and the memory is electrically connected to the processor.

[0025] Optionally, the chip system may also include a communication interface.

[0026] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device can implement any one of the methods of the first aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by an electronic device, the electronic device can implement any one of the methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 It is a schematic diagram of the execution process of a conventional deep learning processing method;

[0030] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0031] Figure 3 It is a flowchart of a noise reduction method provided in an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of the execution process of an existing image processing method;

[0033] Figure 5 is a schematic diagram of an execution process of an image processing method provided in an embodiment of the present application;

[0034] Figure 6 is a structural schematic diagram of an image processing device according to an embodiment of the present application;

[0035] Figure 7 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] The scheme of the embodiment of the present application is introduced below with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic diagram of the execution process of a conventional deep learning method. Figure 1 As shown, first obtain the image to be processed, then input the image to be processed into the encoder of the image processing model, extract features from the image to be processed, learn the features, textures, shapes and other information in the image to be processed, and convert them into data forms that can be understood and processed by the computer. After the image processing model is encoded, the model is decoded, that is, the decoder of the image processing model is used to perform image processing tasks on the information extracted by the decoder, and finally generate the target image.

[0038] In the embodiment of the present application, image processing may include underlying visual tasks such as noise reduction, demosaicing, super resolution (SR), defogging, deraining, and multi-frame fusion.

[0039] It should be noted that the encoder refers to the part that converts input data (such as images) into feature representation. The encoder is usually composed of a series of convolutional layers and pooling layers to gradually extract abstract features of the input data. The main function of the encoder is to reduce the spatial dimension of the input data through layers of convolution and pooling operations, and extract features with semantic information. These features can capture important patterns and structures in the input data, as well as contextual information related to the task. Specifically, the encoder filters the input data through convolution operations to extract features of different scales and directions. Each convolutional layer is usually composed of multiple filters (or convolution kernels), each of which performs convolution operations on the input data to generate a set of feature maps. Each feature map represents the response of the input data in a specific feature direction. As the encoder goes deeper, the number of channels of the feature map usually increases, while the spatial dimension decreases. This is achieved through pooling operations, which aggregate each small area of ​​the feature map to reduce the size of the feature map. Commonly used pooling operations include maximum pooling and average pooling, that is, selecting the maximum value or average value in the small area as the pooled value. The feature maps output by the last layer of the encoder usually have lower spatial resolution but contain higher-level semantic information. These feature maps can be passed to the decoder or other task-related modules for further processing and generating the final output results.

[0040] Corresponding to the encoder, the decoder is responsible for converting the feature representation transformed by the encoder into the final output result. The main task of the decoder is to perform the inverse operation on the feature map extracted by the encoder, gradually restore the spatial dimension, and generate an output corresponding to the original input data. The decoder usually consists of a series of deconvolution layers (also called transposed convolution layers) and upsampling layers. Among them, the deconvolution layer is the inverse operation of the convolution layer, which can expand the spatial dimension of the feature map while keeping the number of feature channels unchanged. In the deconvolution operation, the value of each feature map pixel is copied to a larger output area, thereby restoring the spatial dimension. In addition to the deconvolution layer, the decoder can also use upsampling layers to restore the spatial dimension. The upsampling layer increases the size of the feature map through interpolation or other techniques to restore the spatial dimension. Common upsampling methods include bilinear interpolation, deconvolution operations, etc. The decoder is usually symmetrical with the encoder layer structure, with gradually decreasing layers and gradually restoring the spatial dimension. The output of the last layer of the decoder usually has the same spatial dimension as the original input data and can be used as the final output result. It should be noted that the design of the decoder can be adjusted according to the needs of specific tasks.

[0041] However, general image processing models cannot understand the entities in the image, and therefore cannot guarantee the consistency of processing the same entity, which leads to poor image processing results. For example, when denoising an image containing a sky area, since some noise in the sky is similar to texture, the neural network model cannot distinguish between the two well, which ultimately leads to poor image processing results.

[0042] In view of this, the embodiment of the present application discloses an image processing method, which mainly splices the first feature map extracted by the image processing model and the second feature map extracted by the visual large model, wherein the visual large model is used to extract high-level semantic information in the image to identify the entity area in the image, and further inputs the spliced ​​third feature map into the decoder of the image processing model to finally obtain the target image. Thus, the high-level semantic information extracted by the visual large model is used to guide the image processing model to better complete the image processing task, thereby optimizing the effect of image processing.

[0043] Reference Figure 2 , shows a schematic diagram of a flow chart of an image processing method provided by an embodiment of the present application. Figure 2 The steps shown are introduced.

[0044] Step 201: input multiple consecutive frames of images to be processed into an encoder of a preset image processing model to obtain a first feature map.

[0045] It should be noted that each frame of the image to be processed contains the same scene or object. First, continuous multiple frames of images to be processed can provide better image detail recovery effects. By fusing or aligning multiple frames of images to be processed, noise, artifacts or blur in the image to be processed can be reduced. For example, in super-resolution tasks, by using multiple low-resolution images to reconstruct a high-resolution image, the clarity and details of the image can be improved. Secondly, for HDR tasks (such as HDR image synthesis, HDR photo enhancement, etc.), continuous multiple frames of images to be processed can provide a wider dynamic range. In some cases, a single frame image may not be able to capture all the brightness details of the scene, especially in high contrast or uneven lighting conditions. By using multiple frames of images to be processed, images under different exposure conditions can be obtained, thereby capturing a wider range of brightness, which is particularly important for processing HDR tasks. It should be understood that the processing of multiple frames of images to be processed may increase the complexity of calculation and storage. Therefore, when designing an image processing model, it is necessary to determine the number of frames to be used based on the requirements of the specific task and resource constraints, such as 3 consecutive frames.

[0046] The image to be processed may be a RAW (raw) image or an RGB (red, green, and blue) image. Specifically, RAW images and RGB images are two different image representation formats. RAW images are raw data directly obtained from the image sensor of a digital camera without any processing. RAW images record the photoelectric signal value of each pixel and are usually saved in a specific raw format, such as CR2 (Canon RAW) or NEF (Nikon RAW). RAW images do not go through built-in image processing algorithms and retain more image information, including a higher dynamic range and color depth. Since they are not processed, RAW images usually require post-processing software to perform operations such as decoding, noise reduction, white balance, color correction, and color space conversion to generate the final visual image. RGB images are one of the most common color image representations. RGB images consist of three color channels: red (R), green (G), and blue (B). The color of each pixel is represented by the numerical values ​​of the three channels, usually in the range of 0 to 255. By adjusting the intensity of each channel, various colors and color combinations can be generated.

[0047] In some image processing processes, RAW images can be converted to RGB images. In this conversion process, operations such as decoding RAW data, applying white balance, color correction, and color space conversion are included to finally generate a visual RGB image. Since RAW images retain more image information, they are usually chosen to be shot and processed in RAW format when more post-processing and higher quality images are required.

[0048] In one implementation, the image processing model may adopt a Unet structure, which is a convolutional neural network structure commonly used for image segmentation tasks. In Unet, parameter feature fusion is only performed in the last layer of the encoder. Specifically, each layer of the encoder splices the feature map with the feature map of the next layer to retain more spatial information. This splicing operation can be regarded as a feature fusion method that combines low-level features with high-level features so that the decoder can use information from different levels at the same time. After feature fusion, a residual connection can be used between the encoder and the decoder. The residual connection refers to the addition operation of the feature map of a certain layer of the encoder with the feature map of the corresponding layer of the decoder. This connection method can help information propagate faster in the network, alleviate the problem of gradient disappearance, and retain more detailed information, which helps to improve the effect of image processing.

[0049] It should be understood that the image processing model can be obtained through training in advance in combination with actual needs and deployed on the terminal side. For example, when the image processing task is an HDR task, the image processing model can be an HDR model.

[0050] In one implementation, when the image processing task is HDR, the image processing model can be an HDR model, and the multiple frames of images to be processed need to include at least one frame of long exposure image and at least one frame of short exposure image. It should be noted that HDR is an image processing technology that aims to improve the dynamic range of the image so that it can better express the bright and dark details in the scene. Traditional images can usually only capture a limited dynamic range, resulting in loss of details or overexposure or underexposure in highlight and shadow areas. HDR technology synthesizes multiple images with different exposure levels to show a wider range of brightness.

[0051] Specifically, when processing HDR tasks, the input of the HDR model needs to include at least one long-exposure image and at least one short-exposure image, because images under different exposure conditions contain different information. Long-exposure images are images taken at relatively long exposure times. Long-exposure images are very useful for capturing low-brightness details. Due to the longer exposure time, the bright details in the long-exposure images are better preserved, but the dark details may be overexposed or lost. Short-exposure images are images taken at relatively short exposure times. Short-exposure images are very useful for capturing high-brightness details. Due to the shorter exposure time, the dark details in the short-exposure images are better preserved, but the bright details may be truncated or lost. By using long-exposure images and short-exposure images at the same time, the HDR model can use the difference between the two to restore the lost details. Long-exposure images provide dark details, and short-exposure images provide bright details. The HDR model can generate HDR images with a wider dynamic range by merging the information of the two.

[0052] Furthermore, the multiple frames of images to be processed may also include at least one frame of medium exposure image, and the HDR model may use the additional medium exposure image to provide more reference information. The medium exposure image is between the long exposure image and the short exposure image in terms of exposure time, and the medium exposure image can provide more balanced brightness and details, which helps to further improve the quality of the HDR image.

[0053] Step 202: input one of the multiple frames of images to be processed into a preset large visual model to obtain a second feature map containing high-level semantic information.

[0054] In one implementation, the visual big model can be a Transformer-based visual big model, such as VIT (Vision Transformer), SAM (Spatially Adaptive Multi-scale Attention), etc. The Transformer architecture can be used to extract high-level semantic information in the image. Specifically, both VIT and SAM use the attention mechanism of the Transformer to model the image, thereby achieving attention to different regions in the image and feature extraction. Among them, VIT can divide the input image into a series of image patches, and then map each image patch to a vector through the embedding layer. These vectors are processed by the Transformer encoder as input sequences to learn the global feature representation of the image. SAM can model the image by constructing spatially adaptive multi-scale attention at different levels. The SAM model can better capture the contextual information and spatial relationship of the target by learning the importance weights of different positions and scales in the image. The Transformer-based visual big model can model the global information and context in the image, thereby capturing richer high-level semantic information.

[0055] It should be noted that high-level semantic information refers to abstract, semantic content or concepts in an image, and high-level semantic information represents a deeper understanding of objects, scenes, relationships, and semantic concepts in an image. High-level semantic information can help electronic devices better understand images and perform higher-level analysis and reasoning. High-level semantic information can include the recognition and classification of different objects appearing in an image, such as identifying the types of objects, such as people, cars, cats, etc., and accurately classifying different objects. High-level semantic information can involve the understanding and classification of scenes in an image, such as identifying different types of scenes such as indoors, outdoors, cities, natural scenery, and the sky, as well as more detailed descriptions of scenes, such as beaches, parks, offices, etc. High-level semantic information can involve modeling and understanding the relationship between objects in an image, including identifying spatial relationships (such as relative positions, distances), semantic relationships (such as "in", "holding"), and contextual relationships (such as the probability of an object appearing in a specific scene). High-level semantic information can involve segmentation and labeling of different regions in an image, that is, decomposing an image into parts with semantic meanings, such as objects, backgrounds, boundaries, etc.

[0056] In the embodiment of the present application, each frame of the image to be processed contains visual feature data of multiple regions, and the second feature map contains high-level semantic information corresponding to each region. When the image to be processed contains visual feature data of multiple regions, the high-level semantic information of each region in the image to be processed can be extracted through the visual big model, and used to identify, segment, locate and understand different regions in the image, thereby providing richer and more accurate information for the image processing model.

[0057] Among them, the visual large model only needs one frame of the image to be processed (for example, the first frame of multiple consecutive frames of images to be processed) to extract the second feature map containing high-level semantic information. In addition, the visual large model usually has a large computational and storage overhead. Selecting one frame of the image to be processed as a reference frame, that is, only one frame of the image to be processed is needed to be processed instead of all the images to be processed, which can reduce the amount of calculation, improve the processing speed, and reduce the computing resource requirements of the visual large model.

[0058] In one embodiment, the above step 202 may include: determining the visual feature data contained in each frame of the image to be processed; determining the image to be processed with the most visual feature data as a reference frame; and inputting the reference frame into the visual macro model to obtain a second feature map. Among them, selecting the image to be processed with the most visual feature data as the input of the visual macro model can help the visual macro model better understand the semantic information in the image to be processed.

[0059] In one embodiment, the visual large model can be obtained through training. In order to reduce the training cost, the encoder of the visual large model can also adopt the currently open-source lightweight models and corresponding weights, such as MobileSAM (Mobile SpatialAttention Module), FastSAM (Fast Spatial Attention Module), EfficientSAM (Efficient Spatial Attention Module), etc. The above models can effectively extract high-level semantic information of the image while maintaining lightweight. MobileSAM, FastSAM and EfficientSAM are all models based on the Attention Mechanism, and are lightweight and efficient.

[0060] Specifically, MobileSAM is a lightweight attention module. By learning the spatial attention weights of the image, MobileSAM enables the visual large model to pay more attention to important image areas. MobileSAM has fewer parameters and computational complexity, and is suitable for use on resource-constrained devices; FastSAM is a fast spatial attention module that aims to improve the speed and efficiency of the visual large model. FastSAM accelerates the reasoning process of the visual large model by reducing the complexity of attention calculation and memory consumption. EfficientSAM is an efficient spatial attention module that aims to improve the performance and efficiency of the visual large model in image processing tasks. EfficientSAM uses a lightweight attention calculation method that can effectively capture important information in the image. EfficientSAM provides good attention representation capabilities while maintaining low computational and memory overhead. These models can reduce the model's parameter volume, computational complexity, and memory consumption while maintaining high performance to adapt to the resource constraints of electronic devices.

[0061] Step 203: concatenate the first feature map and the second feature map to obtain a third feature map.

[0062] In an embodiment of the present application, the tensor shapes of the first feature map and the second feature map each contain at least four dimensions: the number of images, the number of feature channels, the image height, and the image width. The tensor shape of the first feature map can be expressed as [batch_size, cA, h, w], and the tensor shape of the second feature map can be expressed as [batch_size, cB, h, w]. Wherein, batch_size represents the number of images, specifically the number of images in each batch; cA and cB represent the number of feature channels, also known as channel dimension or depth, each channel contains information about the feature map on a certain abstract feature dimension, for example, a channel can represent different color channels (such as red, green, and blue); h represents the image height, that is, the height of the feature map in the spatial dimension. In image processing tasks, the image height usually corresponds to the number of pixels in the vertical direction of the image; w represents the image width, that is, the width of the feature map in the spatial dimension. In image processing tasks, the image width usually corresponds to the number of pixels in the horizontal direction of the image.

[0063] In one implementation, the above step 203 may include: splicing the first feature map and the second feature map in the dimension of the number of feature channels to obtain a third feature map. When splicing the first feature map and the second feature map, the image height and image width dimensions of the first feature map and the second feature map remain unchanged, and splicing is performed according to the dimension of the number of feature channels, which can effectively integrate the feature information of different channels together, and the spliced ​​third feature map is rich in high-level semantic information and can provide richer feature representation. In addition, splicing in the dimension of the number of feature channels ensures the consistency of the spatial structure and channel information of the first feature map and the second feature map, so that the subsequent image processing model can correctly process the third feature map.

[0064] For example, if the tensor shape of the first feature map is [batch_size, cA, h, w] and the tensor shape of the second feature map is [batch_size, cB, h, w], then after concatenating the first feature map and the second feature map in the dimension of the number of feature channels, the tensor shape of the third feature map can be expressed as [batch_size, cA+cB, h, w].

[0065] Step 204, input the third feature map into the decoder of the image processing model to obtain the target image.

[0066] The second feature map extracted by the large visual model may include representations of objects of any category, and is fused at the deep feature level through step 203, thereby effectively improving the image processing capability of the image processing model.

[0067] In one embodiment, when the image processing task is super-resolution, the image to be processed may be an image to be super-resolved, and the image processing model may be a super-resolved model. The above step 204 may include: inputting the third feature map into the decoder of the super-resolved model to obtain a second processed image; and performing upsampling processing on the second processed image to determine a target image. Since the goal of the super-resolution task is to convert a low-resolution image into a high-resolution image, the resolution and details of the image to be processed are increased by upsampling processing, so that the target image finally obtained has a higher resolution while maintaining the clarity of the details.

[0068] As another possible implementation of the present application, after obtaining the target image, the above-mentioned image processing method further includes: in the training process of the image processing model, according to the target image, updating the encoder weight and decoder weight of the image processing model. Among them, through the target image and the real image processing results, the parameters of the image processing model can be continuously optimized, the performance and generalization ability of the image processing model can be improved, and then the effect of image processing can be optimized.

[0069] As an example, the target image can be compared with the actual image processing results to provide a clear supervision signal for the image processing model, so that the image processing model can learn how to better process the image so that its output is closer to the actual result. Supervised learning with labeled data can guide the learning process of the image processing model and improve the performance of the model. Specifically, the image processing parameters can be optimized by calculating the loss and updating the weights of the image processing model using the back propagation algorithm, so that it can better fit the training data and continuously optimize the processing effect of the image processing task. In particular, after multiple optimization processes, the generalization ability of the image processing model can be greatly improved. The image processing model can learn the commonalities and rules between different samples, so that it performs better when processing new unseen samples.

[0070] The image processing method provided by the embodiment of the present application first inputs a plurality of continuous frames of images to be processed into the encoder of a preset image processing model to obtain a first feature map; then inputs one frame of the plurality of frames of images to be processed into a preset large visual model to obtain a second feature map containing high-level semantic information; then splices the first feature map and the second feature map to obtain a third feature map; finally, inputs the third feature map into the decoder of the image processing model to obtain a target image. The method mainly splices the first feature map extracted by the image processing model and the second feature map extracted by the large visual model, wherein the large visual model is used to extract high-level semantic information in the image to identify the entity area existing in the image, and further inputs the spliced ​​third feature map into the decoder of the image processing model to finally obtain the target image. Thus, the high-level semantic information extracted by the large visual model is used to guide the image processing model to better complete the image processing task, thereby optimizing the effect of image processing.

[0071] Reference Figure 3 , shows a schematic diagram of a noise reduction method provided by an embodiment of the present application. Figure 3 The steps shown are introduced.

[0072] Step 301: input multiple frames of noisy images and noise intensity information into an encoder of a denoising model to obtain a first feature map.

[0073] Step 301 can be considered as an example of step 201. Usually, in addition to multiple frames of noisy images, the input of the denoising model usually also includes a NoiseMap channel, which is used to indicate the noise intensity information of the current input image. The noise intensity information is usually estimated based on the exposure time and ISO setting of the camera on the electronic device. Therefore, in the same set of input images, the value of NoiseMap is the same. However, this may cause the denoising model to have the same denoising intensity for each area of ​​the image, because the denoising model cannot distinguish the noise levels of different areas in the image according to the different values ​​of NoiseMap, such as over-smoothing details in low-noise areas or insufficient noise reduction in high-noise areas.

[0074] Step 302: input one of the multiple frames of noisy images into a preset large visual model to obtain a second feature map containing high-level semantic information.

[0075] Step 303: concatenate the first feature map and the second feature map to obtain a third feature map.

[0076] Step 304: input the third feature map into the decoder of the denoising model to obtain a first processed image.

[0077] The specific implementation process and principle of the above steps 302-304 can refer to the detailed description of the above embodiment, which will not be repeated here.

[0078] Step 305 : dynamically adjusting the noise intensity information corresponding to each region in the first processed image based on the visual feature data of each region to determine the target image.

[0079] Steps 304 - 305 may be considered as an example of step 204 .

[0080] In one embodiment, for relatively smooth areas in a noisy image, the noise intensity information of such areas can be increased, thereby effectively reducing the noise in such areas and improving the overall quality of the image. For areas with more texture details, the noise intensity information of such areas can be reduced, so that the denoising model can adopt a lighter denoising strategy in these areas to retain more detail information.

[0081] For example, suppose that a noisy image contains sky areas and building areas. Since the sky is a smooth area, the noise intensity information of the sky area can be increased to inform the denoising model that the noise level in this area is high. In this way, the denoising model can adopt a stronger denoising strategy to reduce the noise residue in the sky area. As for the building area, this type of area has rich texture details, and the smearing problem can be reduced by reducing the noise intensity information of the building area. This is because the texture details of these areas are crucial to image quality, and excessive denoising may result in loss of details and blurring effects.

[0082] exist Figure 3 In the scheme shown, multiple frames of noisy images and noise intensity information are first input into the encoder of the denoising model to obtain a first feature map, and then one frame of the multiple frames of noisy images is input into a preset large visual model to obtain a second feature map containing high-level semantic information, and then the first feature map and the second feature map are spliced ​​to obtain a third feature map, and the third feature map is further input into the decoder of the denoising model to obtain a first processed image, and finally based on the visual feature data of each area, the noise intensity information corresponding to each area in the first processed image is dynamically adjusted to determine the target image. Thus, by dynamically adjusting the intensity of the noise intensity information, the denoising effect is improved.

[0083] Figure 4 This is a schematic diagram of the execution process of an existing image processing method. Figure 4 As shown, the image to be processed and the pre-segmented Mask result can be input into the model for image processing, wherein the Mask result refers to a binary mask generated in the image segmentation task, which is used to identify the target area of ​​interest in the image. The Mask result is usually used to separate the target from the background for further analysis, processing or application of other algorithms. It should be noted that the pre-segmented Mask result is a single channel and is usually a binary image, in which the target area is marked as foreground (usually 1) and the background area is marked as background (usually 0). By fusing the Mask result with the image to be processed, the information of the Mask result can be combined with the pixel value of the image to form new input data. This fusion can be performed at each pixel position of the input image, for example, by multiplying or adding the value of the Mask result with the pixel value of the corresponding position. In this way, the encoder module of the image processing model can use the information of the Mask result to guide the encoding and processing of the target area to extract more distinguishing and important features.

[0084] However, the Mask result is a single channel, the categories included are closed sets (the categories are limited, and the number of categories is consistent with the categories of the training samples), and the fusion is performed at the input image level, which particularly tests the encoding ability of the encoder module of the image processing model. However, the visual large model in the embodiment of the present application faces an open set world. Taking the SAM model as an example, the SAM model can segment any object. Therefore, the second feature map obtained by the SAM model can be used to represent any object, and fusion at the deep feature level can effectively improve the image processing ability of the image processing model.

[0085] Figure 5 FIG. 1 is a schematic diagram showing the execution process of an image processing method provided by an embodiment of the present application. Figure 5 As shown, firstly multiple frames of images to be processed ( Figure 5 Only 3 frames are shown in the figure) are input into the decoder of the image processing model to obtain the first feature map. Then the reference frame is determined from the multiple frames of the image to be processed and input into the encoder of the visual model to obtain the second feature map containing high-level semantic information. Then the first feature map and the second feature map are spliced ​​to obtain the third feature map. Finally, the third feature map is input into the decoder of the image processing model to obtain the processed target image. Compared with Figure 4 The traditional image processing method shown in the figure, the embodiment of the present application splices the first feature map extracted by the image processing model and the second feature map extracted by the large visual model, wherein the large visual model is used to extract high-level semantic information in the image to identify the entity area in the image, and further inputs the spliced ​​third feature map into the decoder of the image processing model to finally obtain the target image. Thus, the high-level semantic information extracted by the large visual model is used to guide the image processing model to better complete the image processing task, thereby optimizing the effect of image processing.

[0086] The above mainly introduces the method of the embodiment of the present application in conjunction with the accompanying drawings. It should be understood that although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence, these steps are not necessarily performed in sequence in the order shown in the figure. Unless clearly stated in this article, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a part of the steps or stages in other steps. The device of the embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0087] Figure 6FIG. 1 is a schematic diagram of the structure of an image processing device according to an embodiment of the present application. For the sake of convenience, only the parts related to the embodiment of the present application are shown. Figure 6 As shown, the image processing device 600 includes an input unit 601 and a processing unit 602. The image processing device 600 can be integrated in electronic devices such as mobile phones, computers, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs).

[0088] The image processing device 600 can be used to perform any of the above image processing methods. For example, the input unit 601 can be used to perform steps 201-202 and step 204, and the processing unit 602 can be used to perform step 203. For another example, the input unit 601 can be used to perform steps 301-302 and step 304, and the processing unit 602 can be used to perform steps 303 and step 305.

[0089] In one implementation, the image processing device 600 may further include a storage unit for storing relevant data. The storage unit may be integrated in any one of the above units, or may be a unit independent of all the above units.

[0090] The image processing device provided in the above embodiment of the present application mainly splices the first feature map extracted by the image processing model and the second feature map extracted by the large visual model, wherein the large visual model is used to extract high-level semantic information in the image to identify the entity area in the image, and further inputs the spliced ​​third feature map into the decoder of the image processing model to finally obtain the target image. Thus, the high-level semantic information extracted by the large visual model is used to guide the image processing model to better complete the image processing task, thereby optimizing the effect of image processing.

[0091] Figure 7 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Figure 7As shown, the electronic device 700 may include a processor 710, an external memory interface 720, an internal memory 721, a universal serial bus (USB) interface 730, a charging management module 740, a power management module 741, a battery 742, an antenna 1, an antenna 2, a mobile communication module 750, a wireless communication module 760, an audio module 770, a speaker 770A, a receiver 770B, a microphone 770C, an earphone interface 770D, a sensor module 780, a button 790, a motor 791, an indicator 792, a camera 793, a display screen 794, and a subscriber identification module (SIM) card interface 795, etc. Among them, the sensor module 780 may include a pressure sensor 780A, a gyroscope sensor 780B, an air pressure sensor 780C, a magnetic sensor 780D, an acceleration sensor 780E, a distance sensor 780F, a proximity light sensor 780G, a fingerprint sensor 780H, a temperature sensor 780J, a touch sensor 780K, an ambient light sensor 780L, a bone conduction sensor 780M, etc.

[0092] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 700. In other embodiments of the present application, the electronic device 700 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0093] For example, Figure 7 The processor 710 shown may include one or more processing units, for example, the processor 710 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0094] The controller may be the nerve center and command center of the electronic device 700. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0095] The processor 710 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 710 is a cache memory. The memory may store instructions or data that the processor 710 has just used or cyclically used. If the processor 710 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 710, and thus improves the efficiency of the system.

[0096] In some embodiments, the MIPI interface can be used to connect the processor 710 with peripheral devices such as the display screen 794 and the camera 793. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. The processor 710 and the display screen 794 communicate through the DSI interface to realize the display function of the electronic device 700.

[0097] In some embodiments, the GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. The GPIO interface can be used to connect the processor 710 with the camera 793, the display screen 794, the wireless communication module 760, the audio module 770, the sensor module 780, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0098] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 700. In other embodiments of the present application, the electronic device 700 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0099] The electronic device 700 implements the display function through a GPU, a display screen 794, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 794 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 710 may include one or more GPUs, which execute program instructions to generate or change display information.

[0100] The display screen 794 is used to display images, videos, etc. The display screen 794 includes a display panel. In some embodiments, the electronic device 700 may include 1 or N display screens 794, where N is a positive integer greater than 1.

[0101] The internal memory 721 can be used to store computer executable program codes, which include instructions. The processor 710 executes various functional applications and data processing of the electronic device 700 by running the instructions stored in the internal memory 721. The internal memory 721 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 700 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 721 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0102] The pressure sensor 780A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 780A can be set on the display screen 794. There are many types of pressure sensors 780A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 780A, the capacitance between the electrodes changes. The electronic device 700 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 794, the electronic device 700 detects the touch operation intensity according to the pressure sensor 780A. The electronic device 700 can also calculate the touch position according to the detection signal of the pressure sensor 780A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0103] The touch sensor 780K is also called a "touch panel". The touch sensor 780K can be set on the display screen 794, and the touch sensor 780K and the display screen 794 form a touch screen, also called a "touch screen". The touch sensor 780K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 794. In other embodiments, the touch sensor 780K can also be set on the surface of the electronic device 700, which is different from the position of the display screen 794.

[0104] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0105] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0106] The embodiment of the present application also provides an electronic device, which includes: one or more processors, a memory, and a computer program stored in the memory and executable on the one or more processors, wherein the one or more processors execute the computer program so that the electronic device can implement the steps in any of the above methods. The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by the electronic device, the steps in each of the above method embodiments can be implemented.

[0107] The computer-readable medium may include at least: any entity or device capable of carrying computer program codes to a camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable medium cannot be an electric carrier signal or a telecommunication signal.

[0108] The present application embodiment provides a computer program product, which includes a computer program, and when the computer program is executed by an electronic device, the steps in the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form.

[0109] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0110] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0111] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0114] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0115] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0116] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0117] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Inputting a plurality of consecutive frames of images to be processed into an encoder of a preset image processing model to obtain a first feature map; Inputting one of the multiple frames of images to be processed into a preset visual macro model to obtain a second feature map containing high-level semantic information; Concatenating the first feature map and the second feature map to obtain a third feature map; The third feature map is input into the decoder of the image processing model to obtain a target image.

2. The image processing method according to claim 1, characterized in that: The tensor shapes of the first feature map and the second feature map each include at least four dimensions: the number of images, the number of feature channels, the image height, and the image width. The first feature map and the second feature map are concatenated to obtain a third feature map, including: The first feature map and the second feature map are concatenated in the dimension of the number of feature channels to obtain the third feature map.

3. The image processing method according to claim 1, characterized in that: The method further comprises: During the training process of the image processing model, the encoder weights and decoder weights of the image processing model are updated according to the target image.

4. The image processing method according to claim 1, characterized in that: The step of inputting one of the multiple frames of images to be processed into a preset large visual model to obtain a second feature map containing high-level semantic information includes: Determining visual feature data contained in each frame of the image to be processed; The image to be processed with the most visual feature data is determined as a reference frame; The reference frame is input into the visual macro model to obtain the second feature map.

5. The image processing method according to any one of claims 1 to 4, characterized in that: Each frame of the image to be processed contains visual feature data of multiple regions, and the second feature map contains high-level semantic information corresponding to each of the regions.

6. The image processing method according to claim 5, characterized in that: The image to be processed is a noisy image, and the image processing model is a denoising model; The step of inputting a plurality of consecutive frames of images to be processed into an encoder of a preset image processing model to obtain a first feature map includes: obtaining noise intensity information corresponding to the plurality of frames of noisy images; Input the multiple frames of noisy images and the noise intensity information into an encoder of the denoising model to obtain the first feature map; The step of inputting the third feature map into a decoder of the image processing model to obtain a target image comprises: inputting the third feature map into a decoder of the denoising model to obtain a first processed image; Based on the visual feature data of each area, the noise intensity information corresponding to each area in the first processed image is dynamically adjusted to determine the target image.

7. The image processing method according to claim 1, characterized in that: The image processing model is a high dynamic range rendering (HDR) model, and the multiple frames of images to be processed include at least one frame of a long exposure image and at least one frame of a short exposure image.

8. The image processing method according to claim 1, characterized in that: The image to be processed is an image to be super-resolved, and the image processing model is a super-resolved model; and the step of inputting the third feature map into a decoder of the image processing model to obtain a target image includes: Inputting the third feature map into the decoder of the super-resolution model to obtain a second processed image; The second processed image is up-sampled to determine the target image.

9. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 8.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when executed on an electronic device, enable the electronic device to perform the method as claimed in any one of claims 1 to 8.

Images